kicad-developers team mailing list archive

Thread
Date

Re: [PATCH] Add remote lib retrieval to SCH_LEGACY_PLUGIN_CACHE

To: Nick Østergaard <oe.nick@xxxxxxxxx>, Badr Hack&Invent <badr@xxxxxxxxxxxxxx>
From: Wayne Stambaugh <stambaughw@xxxxxxxxx>
Date: Wed, 20 May 2020 08:23:22 -0400
Autocrypt: addr=stambaughw@xxxxxxxxx; prefer-encrypt=mutual; keydata= mQGiBEM0hxQRBAC2fNh3YOVLu1d5GZ0SbrTNldGiGnCJPLqzEnqFX9v6jmf33TMt6EmSLkl6 Wtfkoj0nVwKxcYmJkA8DX0QAokBkwNIzhSsBzQvthBLIk/5LnPVVKrEXOcL4mUyH1doKlkaE slgJozNa6Av+oavcvD02o1zJOloBbaHlNlyRt7fKswCgtIFlVjWggVH/15KfWk+Qo5JVPbME AIUBAQyL2OAx0n60AWec2WHnO9buHuG0ibtICgUMkE+2MRmYyKwYRdyVwGoIUemFuOyHp0AJ InX4T+vy2E7vkwODqjtMLfIoRkokW74Fi4nrvjlhOAw/vdq/twLbAmR9MOfPTpR4y7kQy1O2 /n+RkkRvh26vTzfbQmrH7cBJhk6aA/9Uwvu3E4zNJgHVZeS0HyWtmR1eOPPRbnkPgJTToX5O KMKzTJI/FX6kT7cFoCamitHrW3BJP4Dx+cMMsa47EGxqVTdbVJ4LjogsXTXxb+0Fn1u4zBdx x3Cer6O7+hqWy7zvpzeC6nSREjqDKa5CgHtv/GLm5uFPOmsjAsnHj2tlBrQmV2F5bmUgU3Rh bWJhdWdoIDxzdGFtYmF1Z2h3QGdtYWlsLmNvbT6IeAQTEQIAOBYhBOffs6CbblRzBkv33BtR cWlZ+CReBQJbFBS2AhsDBQsJCAcCBhUKCQgLAgQWAgMBAh4BAheAAAoJEBtRcWlZ+CReMI8A nRbrLkzp7+c2f0vX7sfg4ICX8LAKAJ9uClo4uJajmZa5zZrL2nKdZlUwIrkCDQRDNIcxEAgA gCru+3/aOC6RCjpvYC72wY+d5SmHphC6yeiV2/mOumyt5MLo/Ps2GznZr11JspqFk5K/Zpvp MMLqqjDZ39+50a2iKRQFJ6NlK+hJWMmj6eJygQrCwYo3Gjc6CqfrqUv+8VSnf/i5sIZmtOVA 4ZjML18MuBvMSsNdVLFJd5HNnYb1iOECpvqdPVh/21LLCEw7MUUGGnHBhCrmk2aJe5hFmcSN g4ldBcXrgMQBwf7aMVoobXBMFDb/IENByXn0llB7Gr2IFMRmNS9/p8s/II1Yl2bTqyX4FSz8 cfn7C9KEz7faZ7wzAcpwHFC/zs3JoAjJ0IEKdNUpIwAlKMzT3CzctwADBQf/cxpG28MKyrqk nNmq/8LQLy+x6FSYXBLjxQz9BiBNYeesDZQ6J5UbL1mjpJzMa5tLZypPYo4bbGyR22hrbyDF K7m6AcVaMIJKl98g4ukMutFfAJyRDaREH5Zl/X1P4u1Z/yaAIy9mKaNbaK1/5djNJ5wCTFen TUgAp9xdc30kGkFDdLJFp5uxDY4P0vaZiZdjUCvDM3Zjv5IzpNOfxVqTUBQNUP/BnnKhkk0p DTD6s3X8S+D0rOtEBQ8K0cwERI/E8EFa8nj0TNw4e2MYGR8wg+SxqJ7z5f0zPY0bO6G9DDFB wYCqzzPWGqdAh9vA5971TAbPERtdFybhkurozp2SfYhJBBgRAgAJBQJDNIcxAhsMAAoJEBtR cWlZ+CResHUAniULLCWiT26ieRTl7N2vS6vBo/DuAJ4m7Ss/gyiW6ybTn1ctDXAUgm2QVQ==
Cc: Kicad-developers <kicad-developers-bounces+badr=hackinvent.com@xxxxxxxxxxxxxxxxxxx>, kicad-developers@xxxxxxxxxxxxxxxxxxx
In-reply-to: <CAOuK9LipDHfRz5CWFB44yx64+0iZnCe+Tngg_dWaJKOGcaB0VQ@mail.gmail.com>
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.8.0

Creating a merge request on GitLab is now the only accepted way to
contribute to KiCad.  Please create a merge request on GitLab so we can
evaluate your change.

Cheers,

Wayne

On 5/20/20 6:51 AM, Nick Østergaard wrote:
> Maybe submit it as a MergeRequest on gitlab.
> 
> ons. 20. maj 2020 12.16 skrev Badr Hack&Invent <badr@xxxxxxxxxxxxxx
> <mailto:badr@xxxxxxxxxxxxxx>>:
> 
>     Hi Wayne,
> 
>     Is there any update regarding this patch?
> 
>     If there is any thing else to adjust please let me know.
> 
>     For information, the plugin is working without any bug since 2018 in
>     my for :)
> 
>     Regards,
> 
>     Badr
> 
>     Le 2018-10-30 11:07, Badr Hack&Invent a écrit :
> 
>>     Hi Wayne,
>>
>>     Did you have a chance to give a look at this patch?
>>     From our side we are using it almost a year now, it works fine.
>>
>>     Else, I don't know if an equivalent function is now under going in
>>     the version 6?
>>     If so, I will be glad to help in testing.
>>
>>     Regards,
>>
>>     Badr
>>
>>     Le 2017-12-24 15:31, Wayne Stambaugh a écrit :
>>
>>         Badr,
>>
>>         Thank you for your patch but we are currently in feature
>>         freeze for the
>>         stable 5 release.  This patch will have to wait until the stable 5
>>         version is release and the version 6 development is opened. 
>>         I'm not
>>         sure exactly when that will happen but I will review your
>>         patch then.
>>
>>         Cheers,
>>
>>         Wayne
>>
>>         On 12/24/2017 05:56 AM, Badr Hack&Invent wrote:
>>
>>             Hi,
>>             Here is a patch to add the remote library management as a
>>             plugin.
>>             I added the possibility to specify multiple urls  in a
>>             single file, it
>>             was useful for our case.
>>             We tested this approach several weeks, it seems fine in
>>             term of
>>             performance.
>>             It doesn't support remote zip files as for pcbnew. I can
>>             manage to
>>             implement it in a second time.
>>             If this patch is ok I will write a detailed documentation
>>             on how to use it.
>>             Regards,
>>             Badr
>>
>>
>>             Le 2017-08-14 15:21, Wayne Stambaugh a écrit :
>>
>>                 You could modify the SCH_LEGACY_PLUGIN_CACHE code to
>>                 handle urls instead
>>                 of file names.  That way you could hand remote and
>>                 local files in the
>>                 same plugin.  wxWidgets has classes to handle urls
>>                 (wxUrl) and input
>>                 streams (wxInputStream) which can be use as the
>>                 LINE_READER
>>                 (INPUTSTREAM_LINE_READER takes a pointer to a
>>                 wxInputStream object).
>>                 The primary drawback to this is performance.  Past
>>                 testing has shown
>>                 that wxInputStream has a lot more overhead and is
>>                 significantly slower
>>                 than either our FILE_LINE_READER and std::istream so
>>                 that is why we
>>                 haven't used it anywhere.  It is also why I
>>                 recommended writing a new
>>                 plugin.  I'm guessing with remote I/O, the
>>                 wxInputStream performance hit
>>                 will be far less noticeable than the remote I/O time.
>>
>>                 Wayne
>>
>>                 On 8/14/2017 3:29 AM, Badr Hack&Invent wrote:
>>
>>                     I see,
>>                     I thought it would be better to upgrade
>>                     SCH_LEGACY_PLUGIN_CACHE to
>>                     handle remote libraries seemlessly rather than
>>                     creating a new plugin.
>>                     I will check how to add a new plugin for remote
>>                     libs without adding
>>                     additional actions for the user.
>>
>>                     Badr
>>
>>                     Le 2017-08-14 02:44, Wayne Stambaugh a écrit :
>>
>>                         I think you misunderstood what I was
>>                         implying.  You should not be
>>                         looking for a different symbol library
>>                         header.  This is a change that I
>>                         would reject since you are effectively
>>                         changing the library file
>>                         format.
>>                          What I suggested was that you create new
>>                         plugin that reads from a
>>                         remote url similar to the github plugin used
>>                         for footprint libraries.
>>
>>                         I also do not like the idea of a separate
>>                         LoadRemote() method being
>>                         added to the SCH_PLUGIN object.  There is no
>>                         reason to add it.  All of
>>                         the symbol library methods take a wxString
>>                         which could be a url instead
>>                         of a file name.  It is not limited to files
>>                         and up to the plugin
>>                         type to
>>                         handle it correctly.  My preference is that
>>                         you create a new plugin for
>>                         remote files similar in concept to the github
>>                         plugin.  That is the
>>                         whole
>>                         point of the current plugin interface.  Take a
>>                         look at the board
>>                         plugins
>>                         to see the preferred method of creating plugins.
>>
>>                         I never really intended for the legacy symbol
>>                         file format to have
>>                         multiple plugins like the kicad board and
>>                         footprint library file
>>                         formats
>>                         so I didn't create separate parser and
>>                         formatter objects like I did
>>                         with
>>                         the board file formats.  Nothing is preventing
>>                         that from happening with
>>                         the current schematic and symbol library file
>>                         formats.  I would not
>>                         have
>>                         any objection to splitting the parser out of the
>>                         SCH_LEGACY_PLUGIN_CACHE
>>                         object.  I am planning on make the parsers and
>>                         formatters for the new
>>                         schematic and symbol library file format code
>>                         separate so it can be
>>                         used
>>                         by any plugin.
>>
>>                         Cheers,
>>
>>                         Wayne
>>
>>                         On 8/13/2017 3:26 PM, Badr Hack&Invent wrote:
>>
>>                             Hi,
>>
>>                             Here in the attachment the patch that add
>>                             the remote lib retrieval.
>>
>>                             Badr
>>                             Le 2017-08-13 17:29, Badr Hack&Invent a
>>                             écrit :
>>
>>                                 Hi,
>>
>>                                 Those couple of days I was checking
>>                                 how to update EESCHEMA to add
>>                                 remote libraries retrieval function.
>>
>>                                 Since am familiar with legacy format,
>>                                 I updated the plugin:
>>                                 SCH_LEGACY_PLUGIN_CACHE in charge of
>>                                 parsing the *.lib files.
>>
>>                                 The idea was to create a new type of
>>                                 library
>>                                 EESchema-REMOTELIBRARY (I
>>                                 put an example in the attachment)
>>                                 The content of this library is the
>>                                 following:
>>                                 EESchema-REMOTELIBRARY Version 1.0
>>                                 URL https://www.example.com/mylib1.lib
>>                                 URL https://www.example.com/mylib2.lib
>>                                 ...
>>
>>                                 This lib file is saved localy and
>>                                 specify the path of each remote
>>                                 library you want to retrieve.
>>
>>                                 The updated code seemlessly check the
>>                                 type of the library, if it is
>>                                 EESchema-LIBRARY it parse it like
>>                                 always, else if it is
>>                                 EESchema-REMOTELIBRARY it download
>>                                 each remote lib and parse it when
>>                                 it is EESchema-LIBRARY (no recusivity
>>                                 with EESchema-REMOTELIBRARY).
>>
>>                                 The impacted files are: 
>>                                 sch_legacy_plugin.cpp and
>>                                 sch_legacy_plugin.h
>>                                 -> I implemented the algo and made
>>                                 some tweeks to use LINE_READER
>>                                 instead of FILE_LINE_READER as
>>                                 argument to manage to use
>>                                 STRING_LINE_READER
>>
>>                                 I also modified
>>                                 KICAD_CURL_EASY::KICAD_CURL_EASY() to
>>                                 set the option
>>                                 CURLOPT_SSL_VERIFYHOST to 0 to disable
>>                                 ssl certificate checking in
>>                                 https requests. This modification is
>>                                 not required, but was useflul
>>                                 for
>>                                 our case where our server is behind
>>                                 ssl without certificate on the
>>                                 domaine, just ip addresses.
>>
>>                                 I made a prototype in the attachment,
>>                                 it is woring.
>>
>>                                 I don't know if this modification is
>>                                 inline with the arachitecture of
>>                                 kicad?
>>
>>                                 Badr
>>
>>
>>
>>                             _______________________________________________
>>                             Mailing list:
>>                             https://launchpad.net/~kicad-developers
>>                             Post to     :
>>                             kicad-developers@xxxxxxxxxxxxxxxxxxx
>>                             <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>>                             Unsubscribe :
>>                             https://launchpad.net/~kicad-developers
>>                             More help   :
>>                             https://help.launchpad.net/ListHelp
>>
>>
>>
>>                         _______________________________________________
>>                         Mailing list:
>>                         https://launchpad.net/~kicad-developers
>>                         Post to     :
>>                         kicad-developers@xxxxxxxxxxxxxxxxxxx
>>                         <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>>                         Unsubscribe :
>>                         https://launchpad.net/~kicad-developers
>>                         More help   : https://help.launchpad.net/ListHelp
>>
>>
> 
>     _______________________________________________
>     Mailing list: https://launchpad.net/~kicad-developers
>     Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>     <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>     Unsubscribe : https://launchpad.net/~kicad-developers
>     More help   : https://help.launchpad.net/ListHelp
>

Follow ups

Re: [PATCH] Add remote lib retrieval to SCH_LEGACY_PLUGIN_CACHE
From: Badr Hack&Invent, 2020-11-11

References

[RFC] Long pin numbers/pad names support
From: Maciej Suminski, 2017-08-12
Re: [RFC] Long pin numbers/pad names support
From: jp charras, 2017-08-13
[EESCHEMA] remote lib retrieval
From: Badr Hack&Invent, 2017-08-13
[PATCH] Add remote lib retrieval to SCH_LEGACY_PLUGIN_CACHE
From: Badr Hack&Invent, 2017-08-13
Re: [PATCH] Add remote lib retrieval to SCH_LEGACY_PLUGIN_CACHE
From: Wayne Stambaugh, 2017-08-14
Re: [PATCH] Add remote lib retrieval to SCH_LEGACY_PLUGIN_CACHE
From: Badr Hack&Invent, 2017-08-14
Re: [PATCH] Add remote lib retrieval to SCH_LEGACY_PLUGIN_CACHE
From: Wayne Stambaugh, 2017-08-14
Re: [PATCH] Add remote lib retrieval to SCH_LEGACY_PLUGIN_CACHE
From: Badr Hack&Invent, 2017-12-24
Re: [PATCH] Add remote lib retrieval to SCH_LEGACY_PLUGIN_CACHE
From: Wayne Stambaugh, 2017-12-24
Re: [PATCH] Add remote lib retrieval to SCH_LEGACY_PLUGIN_CACHE
From: Badr Hack&Invent, 2018-10-30
Re: [PATCH] Add remote lib retrieval to SCH_LEGACY_PLUGIN_CACHE
From: Badr Hack&Invent, 2020-05-20
Re: [PATCH] Add remote lib retrieval to SCH_LEGACY_PLUGIN_CACHE
From: Nick Østergaard, 2020-05-20