kicad-developers team mailing list archive

[Miguel Angel Ajo Pelayo <miguelangel@xxxxxxx>]

I'm thinking too, I really *love* the idea, and I'm sure that building on
the simple idea -later, with time- we can do even more.

But a simple github connector it's a great start.

I have a piece of code that I put together some time ago (over a single
weekend), to make a kicad-service website, that I will open at some time:

(please use nbee/nbeepass as credentials)

http://kicadhub.com/pcbs/2
http://kicadhub.com/assembly_guides/3
http://kicadhub.com/components/107
http://kicadhub.com/kicadnetlists/2

[it's ruby written, with shell scripts in python to use the kicad python
api for rendering and gathering information]

My plan /at that time/ was to make an aggregator of kicad projects
(crawling *github via api *+ octopart api) to make them documented
(assembly guides + priced boms, etc..). And also a* library part aggregator*,
where you could vote, comment, and auto-categorize (by pitch, number of
pins, pin distribution, function, etc..)

But at this time* I thought that my time was rather limited and that I was
better spending it at kicad itself*, finishing and polishing what I
started, so I stopped it.


Now if we mix my old idea, with this new approach:

Everybody has his own github library repositories, and then we could build
a community aggregator, where you can add your git url, it keeps your parts
synced to the aggregator, it also keeps track of voting, pulls your library
updates, and builds an index/categorization that kicad can read,
then it re-exports on a single github repo (agreggated) and later, exports
an API (extension of the git idea) to keep track of votes, comments, etc.

That aggregator might need a server (quite scalable as we relay fileserving
to github), but it must be fully open, same kicad license, so anybody could
build his own server if wanted (for example, within companies...)





2013/8/14 Chris Morgan <chmorgan@xxxxxxxxx>

> On Tue, Aug 13, 2013 at 12:54 AM, Dick Hollenbeck <dick@xxxxxxxxxxx>
> wrote:
> > On 08/12/2013 01:11 PM, Chris Morgan wrote:
> >> On Mon, Aug 12, 2013 at 12:05 PM, Dick Hollenbeck <dick@xxxxxxxxxxx>
> wrote:
> >>> On 08/12/2013 10:22 AM, Chris Morgan wrote:
> >>>> On Mon, Aug 12, 2013 at 11:13 AM, Dick Hollenbeck <dick@xxxxxxxxxxx>
> wrote:
> >>>>> FYI,
> >>>>>
> >>>>> In looking at an http plugin for pcbnew, here are my findings so far:
> >>>>>
> >>>>> It occurred to me that only three class PLUGIN functions seem to be
> necessary
> >>>>>
> >>>>>   *) IsWriteable()  this is one line, "return false".
> >>>>>   *) FootprintEnumerate()
> >>>>>   *) FootprintLoad()
> >>>>>
> >>>>>
> >>>>> So basically two functions.  I wrote FootprintLoad() in about 4
> hours using wxHTTP,
> >>>>> complete with "redirect" handling that I added.
>  FootprintEnumerate() is website dependent
> >>>>> since you have to parse the dir listing coming back.
> >>>>>
> >>>>>
> >>>>> This is a trivial amount of work, but you have something website
> specific that is read
> >>>>> only for ever.  And that is not a terrible thing, quite useful.
> >>>>>
> >>>>>
> >>>>> Then and found an API at GITHUB, and formulated more thoughts about
> that, so what follows
> >>>>> is the formation of a work package:
> >>>>> ==============================================================
> >>>>>
> >>>>>
> >>>>>
> >>>>>    http://developer.github.com/v3/
> >>>>>
> >>>>>
> >>>>> is a marriage made in heaven for pcbnew.  The only missing building
> block is https
> >>>>> support.  We have to parse a little JSON, but this is in
> boost::property tree already in
> >>>>> the source tree.
> >>>>>
> >>>>> It also gave me a better handle on the use cases of the fp lib table
> dialog window, which
> >>>>> now seems to require a little better support for the options column.
>  I can envision a
> >>>>> property table two column popup dialog that lets you show the
> options in name value form.
> >>>>>  (Also maybe an additional PLUGIN api function that returns the
> options that each plugin
> >>>>> knows about as fodder for this generic dialog window.)  Upon return
> from the popup dialog,
> >>>>> you get a string in form
> >>>>>
> >>>>>   "option1=value1,option2,option3=value3"
> >>>>>
> >>>>> That goes back into the fp lib table dialog "options" column.
> >>>>>
> >>>>> And of course this property dialog must parse this string upon
> invocation.  The parser can
> >>>>> be re-used if the results go into a PROPERTIES class instance so it
> can also be passed to
> >>>>> the PLUGIN api functions too.
> >>>>>
> >>>>> RE: https.  I have a couple of options in mind for that, after which
> I am thinking we
> >>>>> could start with a read only GITHUB_PLUGIN implementation, and grow
> into something with
> >>>>> write capabilities down the line.
> >>>>>
> >>>>>
> >>>>> I need to solve the https gap in the least painful way.
>  Possibilities I see are:
> >>>>>
> >>>>> a) cpp-netlib which has HTTPS client, looks like it needs g++ 4.7 to
> compile, although
> >>>>> I've yet to verify this is the case when sub-setting it.
> >>>>>
> >>>>> b) cherry pick libpoco, which I have used, and has HTTPS client.
> >>>>>
> >>>>> c) add ssl support on top of wxHTTP using boost and openssl in the
> same way boost does it
> >>>>> for non SSL boost::ASIO sockets.
> >>>>>
> >>>>>
> >>>>> These are currently about equal in my mind.  Note that merely having
> SSL sockets is not as
> >>>>> rich as having an HTTPS client, since much header and replay parsing
> support is need.
> >>>>>
> >>>>> If anyone is a networking jockey and wants to work with me on this
> let me know.  I think I
> >>>>> could and would code the read only form of the plugin in about 2
> days if we had the HTTPS
> >>>>> client support in place.  That is bogging me down right now, and
> brings the total work to
> >>>>> over 3 days, which I have to defer because of other work.
> >>>>>
> >>>>>
> >>>>> Dick
> >>>>>
> >>>>
> >>>>
> >>>> I had seen the github api when I was putting together some of the
> >>>> ideas about the online library sharing. It looks like a lot of the
> >>>> approaches they've used could be borrowed. I presume you are referring
> >>>> to using github directly?
> >>>
> >>> Correct.
> >>>
> >>>  I was more thinking the api was suitable to
> >>>> build upon for a custom server implementation with additional commands
> >>>> to handle our application specific usage.
> >>>
> >>>
> >>> That is also possible if you find the development resources to do it.
>   There are no fixed
> >>> limits on the number nor the types of pcbnew PLUGINs that get written.
> >>>
> >>>
> >>> If you decide to do that, *start by identifying the development
> resources*.  In my case of
> >>> GITHUB plugin, because of past discussions that we have had on the
> topic, and all the work
> >>> and planning that I put into the PLUGIN API so that it COULD HANDLE
> REMOTE ACCESS,
> >>>
> >>>    *I am prepared to provide the development resources*
> >>>
> >>> to do GITHUB_PLUGIN.  Although if I have to also do the HTTPS client
> layer work, it will
> >>> not be in 2 days from now.
> >>>
> >>>
> >>> Having a larger code base to borrow from makes any future plugin work
> easier of course also.
> >>>
> >>>
> >>> If that does not meet your needs, then do your means analysis up
> front.  Do not assume you
> >>> can rally the troops to donate valuable time to a large design.  You
> end up with 200 email
> >>> messages and a piece of paper if you are lucky.
> >>>
> >>>
> >>> The dirty secret about open source is that the code is most always
> written FOR the guy or
> >>> organization writing it.  It is only coincidental that it solves
> someone else's problem too.
> >>>
> >>> I think a GITHUB plugin will serve my needs, it only looks like I am
> doing it for someone
> >>> else.
> >>>
> >>>
> >>> Dick
> >>>
> >>>
> >>
> >>
> >> Your thoughts on enumerating parts? Would that be something done
> >> manually (user would have to plug in each of the parts they wanted to
> >> pull down via url or blob id or whatever), or would that be something
> >> you'd do via the github api (or another api)?
> >
> > Each row entry in the fp lib table is a library.  Once a row is in place
> with a libpath
> > pointing to the "github API form of a repo" [1], you then can read any
> footprint in the
> > repo the way you do now in pcbnew.  It is seemless, because we'll use
> > PLUGIN::FootprintsEnumerate() on whatever plugin is associated with that
> row.  We already
> > use FootprintsEnumerate, but is it hard coded to
> LEGACY_PLUGIN::FootprintsEnumerate(),
> > until Wayne's work is complete.
> >
> > Until we grow into the GITHUB_PLUGIN write support, the owner(s) of the
> repo can simply
> > git push to it to update parts.  The repo owner(s) can generate
> footprints in that library
> > any they wish, offline.  Scripting, coffeescript, whatever, as long as
> you have
> > *.kicad_mod files in the repo when its ready to use.
> >
> > Writing to the repo *from kicad* means adding support for
> PLUGIN::FootprintSave(), and
> > that won't be bad.  But creating a github library is probably harder
> using the PLUGIN API.
> >  Fortunately the github browser interface can do the creation easily.
>  The options for the
> > fp lib table row will require username and password for your github
> account, even to use
> > someone else's repo on a read only basis, but these two options will
> remain secret since
> > the github API only uses the SSL encrypted sockets to fulfil its API in
> the link I provided.
> >
> > For reading footprints, they can return the pretty format in "raw" form
> directly, if in
> > the API you ask for raw content. That means the reply to the https
> request done in
> > FootprintLoad() gets you a *.kicad_mod file back, and it is parsed in
> RAM before the
> > MODULE* is returned.
> >
> > For enumerating footpringts, what we ask of the github API is the
> contents of a directory,
> > and that is returned in JSON format.  It needs to be parsed before
> FootprinsEnumerate()
> > returns with a wxArrayString, and the '.kicad_mod' extension needs to be
> chopped off to
> > fulfil that PLUGIN API function.
> >
> >
> > [1] https://api.github.com/username/repo  I think
> >
> >
>
>
>
>
> I've had some time to think about this and I think that the github api
> may be entirely suitable for the purposes of kicad.
>
> I'd like to be able to compare it to what CircuitHub has but without
> any documentation that's a bit tough to do.
>
> The github api provides:
> - Something that works today
> - Ability to manipulate it with the filesystem and git tools
> - A way to enumerate (list files in a directory)
> - Versioning
> - A way to pull blobs
>
> I'm thinking that we could leverage github today, they already have a
> site and tools to manage parts. This isn't perfect, for instance, it
> would be handy to not have to authenticate. We could bolt on some api
> calls for that and other things that we want to add.
>
> In terms of a longer term community maintained system, there isn't any
> reason we can't mimic the git internals to attach things like comments
> and ratings, and to be able to generate a simulated directory listing
> for the github api.
>
> Dick, thoughts? I think this may address your goal of "simple" from
> the KiCad side, and my goal of building on something that has the
> potential to be a fancy web managed community site, without breaking
> the KiCad api to do so.
>
> Chris
>
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers
> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>



-- 

Miguel Angel Ajo Pelayo
http://www.nbee.es
+34 636 52 25 69
skype: ajoajoajo