kicad-developers team mailing list archive

[Chris Morgan <chmorgan@xxxxxxxxx>]

On Tue, Aug 13, 2013 at 12:54 AM, Dick Hollenbeck <dick@xxxxxxxxxxx> wrote:
> On 08/12/2013 01:11 PM, Chris Morgan wrote:
>> On Mon, Aug 12, 2013 at 12:05 PM, Dick Hollenbeck <dick@xxxxxxxxxxx> wrote:
>>> On 08/12/2013 10:22 AM, Chris Morgan wrote:
>>>> On Mon, Aug 12, 2013 at 11:13 AM, Dick Hollenbeck <dick@xxxxxxxxxxx> wrote:
>>>>> FYI,
>>>>>
>>>>> In looking at an http plugin for pcbnew, here are my findings so far:
>>>>>
>>>>> It occurred to me that only three class PLUGIN functions seem to be necessary
>>>>>
>>>>>   *) IsWriteable()  this is one line, "return false".
>>>>>   *) FootprintEnumerate()
>>>>>   *) FootprintLoad()
>>>>>
>>>>>
>>>>> So basically two functions.  I wrote FootprintLoad() in about 4 hours using wxHTTP,
>>>>> complete with "redirect" handling that I added.  FootprintEnumerate() is website dependent
>>>>> since you have to parse the dir listing coming back.
>>>>>
>>>>>
>>>>> This is a trivial amount of work, but you have something website specific that is read
>>>>> only for ever.  And that is not a terrible thing, quite useful.
>>>>>
>>>>>
>>>>> Then and found an API at GITHUB, and formulated more thoughts about that, so what follows
>>>>> is the formation of a work package:
>>>>> ==============================================================
>>>>>
>>>>>
>>>>>
>>>>>    http://developer.github.com/v3/
>>>>>
>>>>>
>>>>> is a marriage made in heaven for pcbnew.  The only missing building block is https
>>>>> support.  We have to parse a little JSON, but this is in boost::property tree already in
>>>>> the source tree.
>>>>>
>>>>> It also gave me a better handle on the use cases of the fp lib table dialog window, which
>>>>> now seems to require a little better support for the options column.  I can envision a
>>>>> property table two column popup dialog that lets you show the options in name value form.
>>>>>  (Also maybe an additional PLUGIN api function that returns the options that each plugin
>>>>> knows about as fodder for this generic dialog window.)  Upon return from the popup dialog,
>>>>> you get a string in form
>>>>>
>>>>>   "option1=value1,option2,option3=value3"
>>>>>
>>>>> That goes back into the fp lib table dialog "options" column.
>>>>>
>>>>> And of course this property dialog must parse this string upon invocation.  The parser can
>>>>> be re-used if the results go into a PROPERTIES class instance so it can also be passed to
>>>>> the PLUGIN api functions too.
>>>>>
>>>>> RE: https.  I have a couple of options in mind for that, after which I am thinking we
>>>>> could start with a read only GITHUB_PLUGIN implementation, and grow into something with
>>>>> write capabilities down the line.
>>>>>
>>>>>
>>>>> I need to solve the https gap in the least painful way.  Possibilities I see are:
>>>>>
>>>>> a) cpp-netlib which has HTTPS client, looks like it needs g++ 4.7 to compile, although
>>>>> I've yet to verify this is the case when sub-setting it.
>>>>>
>>>>> b) cherry pick libpoco, which I have used, and has HTTPS client.
>>>>>
>>>>> c) add ssl support on top of wxHTTP using boost and openssl in the same way boost does it
>>>>> for non SSL boost::ASIO sockets.
>>>>>
>>>>>
>>>>> These are currently about equal in my mind.  Note that merely having SSL sockets is not as
>>>>> rich as having an HTTPS client, since much header and replay parsing support is need.
>>>>>
>>>>> If anyone is a networking jockey and wants to work with me on this let me know.  I think I
>>>>> could and would code the read only form of the plugin in about 2 days if we had the HTTPS
>>>>> client support in place.  That is bogging me down right now, and brings the total work to
>>>>> over 3 days, which I have to defer because of other work.
>>>>>
>>>>>
>>>>> Dick
>>>>>
>>>>
>>>>
>>>> I had seen the github api when I was putting together some of the
>>>> ideas about the online library sharing. It looks like a lot of the
>>>> approaches they've used could be borrowed. I presume you are referring
>>>> to using github directly?
>>>
>>> Correct.
>>>
>>>  I was more thinking the api was suitable to
>>>> build upon for a custom server implementation with additional commands
>>>> to handle our application specific usage.
>>>
>>>
>>> That is also possible if you find the development resources to do it.   There are no fixed
>>> limits on the number nor the types of pcbnew PLUGINs that get written.
>>>
>>>
>>> If you decide to do that, *start by identifying the development resources*.  In my case of
>>> GITHUB plugin, because of past discussions that we have had on the topic, and all the work
>>> and planning that I put into the PLUGIN API so that it COULD HANDLE REMOTE ACCESS,
>>>
>>>    *I am prepared to provide the development resources*
>>>
>>> to do GITHUB_PLUGIN.  Although if I have to also do the HTTPS client layer work, it will
>>> not be in 2 days from now.
>>>
>>>
>>> Having a larger code base to borrow from makes any future plugin work easier of course also.
>>>
>>>
>>> If that does not meet your needs, then do your means analysis up front.  Do not assume you
>>> can rally the troops to donate valuable time to a large design.  You end up with 200 email
>>> messages and a piece of paper if you are lucky.
>>>
>>>
>>> The dirty secret about open source is that the code is most always written FOR the guy or
>>> organization writing it.  It is only coincidental that it solves someone else's problem too.
>>>
>>> I think a GITHUB plugin will serve my needs, it only looks like I am doing it for someone
>>> else.
>>>
>>>
>>> Dick
>>>
>>>
>>
>>
>> Your thoughts on enumerating parts? Would that be something done
>> manually (user would have to plug in each of the parts they wanted to
>> pull down via url or blob id or whatever), or would that be something
>> you'd do via the github api (or another api)?
>
> Each row entry in the fp lib table is a library.  Once a row is in place with a libpath
> pointing to the "github API form of a repo" [1], you then can read any footprint in the
> repo the way you do now in pcbnew.  It is seemless, because we'll use
> PLUGIN::FootprintsEnumerate() on whatever plugin is associated with that row.  We already
> use FootprintsEnumerate, but is it hard coded to LEGACY_PLUGIN::FootprintsEnumerate(),
> until Wayne's work is complete.
>
> Until we grow into the GITHUB_PLUGIN write support, the owner(s) of the repo can simply
> git push to it to update parts.  The repo owner(s) can generate footprints in that library
> any they wish, offline.  Scripting, coffeescript, whatever, as long as you have
> *.kicad_mod files in the repo when its ready to use.
>
> Writing to the repo *from kicad* means adding support for PLUGIN::FootprintSave(), and
> that won't be bad.  But creating a github library is probably harder using the PLUGIN API.
>  Fortunately the github browser interface can do the creation easily.  The options for the
> fp lib table row will require username and password for your github account, even to use
> someone else's repo on a read only basis, but these two options will remain secret since
> the github API only uses the SSL encrypted sockets to fulfil its API in the link I provided.
>
> For reading footprints, they can return the pretty format in "raw" form directly, if in
> the API you ask for raw content. That means the reply to the https request done in
> FootprintLoad() gets you a *.kicad_mod file back, and it is parsed in RAM before the
> MODULE* is returned.
>
> For enumerating footpringts, what we ask of the github API is the contents of a directory,
> and that is returned in JSON format.  It needs to be parsed before FootprinsEnumerate()
> returns with a wxArrayString, and the '.kicad_mod' extension needs to be chopped off to
> fulfil that PLUGIN API function.
>
>
> [1] https://api.github.com/username/repo  I think
>
>




I've had some time to think about this and I think that the github api
may be entirely suitable for the purposes of kicad.

I'd like to be able to compare it to what CircuitHub has but without
any documentation that's a bit tough to do.

The github api provides:
- Something that works today
- Ability to manipulate it with the filesystem and git tools
- A way to enumerate (list files in a directory)
- Versioning
- A way to pull blobs

I'm thinking that we could leverage github today, they already have a
site and tools to manage parts. This isn't perfect, for instance, it
would be handy to not have to authenticate. We could bolt on some api
calls for that and other things that we want to add.

In terms of a longer term community maintained system, there isn't any
reason we can't mimic the git internals to attach things like comments
and ratings, and to be able to generate a simulated directory listing
for the github api.

Dick, thoughts? I think this may address your goal of "simple" from
the KiCad side, and my goal of building on something that has the
potential to be a fancy web managed community site, without breaking
the KiCad api to do so.

Chris