kicad-developers team mailing list archive

[Jeff Young <jeff@xxxxxxxxx>]

Hi Wayne,

Metrics don’t always tell the full story. ;)  As it turns out fixing the == operator just moves the performance hit downstream.  

The real culprit is loading the file descriptor into the OS’s cache.  If the == operator’s check-file-existence doesn’t load it then the following check-file-mod-date has to.

Essentially our current implementation of validate-cache (checking the file mod-date) negates the performance benefit of the cache except for bulk operations.  I considered something like using the system clock to check the file-mod-date no more often than every 2 seconds or something, but that seemed error-prone as well.

Cheers,
Jeff.


> On 25 Jan 2018, at 00:25, Wayne Stambaugh <stambaughw@xxxxxxxxx> wrote:
> 
> Jeff,
> 
> Your patch bypasses the issue but the underlying problem still exists
> anywhere the library cache file name is checked repetitively.  Looking
> at your analysis, the guilty call is the wxFileName == operator.  This
> comparison should be a simple string comparison since that is what
> matters here.  I will merge your patch since it is faster than doing the
> comparison when loading each symbol sequentially.  Please keep in mind,
> keeping pointers to library symbols lying around is a segfault waiting
> to happen because they can change out from under you which is why I did
> it this way.  I erred on the side of caution.  This patch is riskier but
> since it's only using the pointers temporarily to copy the LIB_ALIAS
> information and not stored anywhere, I think it's fairly safe.  However,
> if someone edited a symbol library at the same time the symbol chooser
> was being loaded it could segfault.
> 
> Thanks,
> 
> Wayne
> 
> On 01/24/2018 06:43 PM, Jeff Young wrote:
>> It’d probably be better if I actually attached the patch….
>> 
>> 
>> 
>> 
>>> On 24 Jan 2018, at 23:34, Jeff Young <jeff@xxxxxxxxx <mailto:jeff@xxxxxxxxx>
>>> <mailto:jeff@xxxxxxxxx <mailto:jeff@xxxxxxxxx>>> wrote:
>>> 
>>> The attached patch speeds loading the symbol libraries from the cache
>>> to near-instantaneous (down from about 1.5 seconds on my machine).
>>> 
>>> I don’t claim it’s the right patch.  Someone with more knowledge of
>>> this area should at least review it.
>>> 
>>> And I still get a hard-load (presumably a cache miss of some sort)
>>> each time I do a cmd-E.  But this at least fixes symbol placement.
>>> 
>>> Cheers,
>>> Jeff.
>>> 
>>> 
>>>> On 24 Jan 2018, at 16:11, Jeff Young <jeff@xxxxxxxxx <mailto:jeff@xxxxxxxxx>
>>>> <mailto:jeff@xxxxxxxxx <mailto:jeff@xxxxxxxxx>>> wrote:
>>>> 
>>>> I couldn’t figure out how to save a portion of the file.  Here’s a
>>>> screen capture of the stack-trace:
>>>> 
>>>> <PastedGraphic-4.png>
>>>> 
>>>>> On 24 Jan 2018, at 16:02, Maciej Sumiński <maciej.suminski@xxxxxxx <mailto:maciej.suminski@xxxxxxx>
>>>>> <mailto:maciej.suminski@xxxxxxx <mailto:maciej.suminski@xxxxxxx>>> wrote:
>>>>> 
>>>>> I believe there is only one code path leading to this function. Would
>>>>> you post the full stacktrace?
>>>>> 
>>>>> On 01/24/2018 04:59 PM, Jeff Young wrote:
>>>>>> 75% (!) of the time building the component tree is spent here:
>>>>>> 
>>>>>> bool SCH_LEGACY_PLUGIN_CACHE::IsFile( const wxString&
>>>>>> aFullPathAndFileName ) const
>>>>>> {
>>>>>>    return m_libFileName == aFullPathAndFileName;
>>>>>> }
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On 24 Jan 2018, at 15:31, Jeff Young <jeff@xxxxxxxxx <mailto:jeff@xxxxxxxxx>
>>>>>>> <mailto:jeff@xxxxxxxxx <mailto:jeff@xxxxxxxxx>>> wrote:
>>>>>>> 
>>>>>>> I’ve been trying to profile it to see exactly where the time is
>>>>>>> spent, but I’m having trouble getting valgrind to work.  I might
>>>>>>> put this on the back burner while I fix the issues you identified
>>>>>>> in the Update Footprints stuff.
>>>>>>> 
>>>>>>> 
>>>>>>>> On 24 Jan 2018, at 15:27, Wayne Stambaugh <stambaughw@xxxxxxxxx <mailto:stambaughw@xxxxxxxxx>
>>>>>>>> <mailto:stambaughw@xxxxxxxxx <mailto:stambaughw@xxxxxxxxx>>> wrote:
>>>>>>>> 
>>>>>>>> Jeff,
>>>>>>>> 
>>>>>>>> I still think there is something wrong with populating the component
>>>>>>>> tree model.  It was working fine (even with the footprint viewer
>>>>>>>> loading) until Orson made changes to it support the new symbol
>>>>>>>> library
>>>>>>>> editor.  He reverted those changes but maybe he missed something
>>>>>>>> subtle
>>>>>>>> that is causing the issue.  I know Chris said he was going to
>>>>>>>> look at it
>>>>>>>> when he got a chance since he wrote the original implementation.  He
>>>>>>>> seems to understand the performance issues.  I'm ok with lazy
>>>>>>>> loading.
>>>>>>>> It should be a reasonably simple change since that's how the symbol
>>>>>>>> library table already works but that may problematic for the library
>>>>>>>> editor.  I can tell you that it is much more noticeable on
>>>>>>>> windows than
>>>>>>>> linux on my desktop computer.  We really need to get this resolved
>>>>>>>> before the stable release.
>>>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> 
>>>>>>>> Wayne
>>>>>>>> 
>>>>>>>> On 1/24/2018 9:01 AM, Jeff Young wrote:
>>>>>>>>> I have my symbol libraries pared down to some 3900 symbols.
>>>>>>>>> 
>>>>>>>>> Loading the first time takes roughly 3-1/2 seconds.
>>>>>>>>> 
>>>>>>>>> The second time it will say “Loading symbols”, but what it’s
>>>>>>>>> actually doing is building the component tree.  This takes about
>>>>>>>>> 1-1/2 seconds.
>>>>>>>>> 
>>>>>>>>> I tried making the component tree static and doing a sync, but
>>>>>>>>> this only shaves about 1/2 a second off the time.
>>>>>>>>> 
>>>>>>>>> Maybe we need to show the dialog with just the libraries in the
>>>>>>>>> tree, and then load the symbols on demand?  (We’d probably need
>>>>>>>>> to load the first symbol of each library up-front to get the
>>>>>>>>> turn-down arrows, but that should be doable.)
>>>>>>>>> 
>>>>>>>>> Thoughts?
>>>>>>>>> _______________________________________________
>>>>>>>>> Mailing list: https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>>>>>>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>>>>>>>>> <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>>
>>>>>>>>> Unsubscribe : https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>>>>>>>> More help   : https://help.launchpad.net/ListHelp <https://help.launchpad.net/ListHelp>
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> Mailing list: https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>>>>>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>>>>>>>> <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>>
>>>>>>>> Unsubscribe : https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>>>>>>> More help   : https://help.launchpad.net/ListHelp <https://help.launchpad.net/ListHelp>
>>>>>>> 
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> Mailing list: https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>>>>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>>>>>>> <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>>
>>>>>>> Unsubscribe : https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>>>>>> More help   : https://help.launchpad.net/ListHelp <https://help.launchpad.net/ListHelp>
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>>>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>>>>>> <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>>
>>>>>> Unsubscribe : https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>>>>> More help   : https://help.launchpad.net/ListHelp <https://help.launchpad.net/ListHelp>
>>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>>>>> <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>>
>>>>> Unsubscribe : https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>>>> More help   : https://help.launchpad.net/ListHelp <https://help.launchpad.net/ListHelp>
>>>> 
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>>>> <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>>
>>>> Unsubscribe : https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>>> More help   : https://help.launchpad.net/ListHelp <https://help.launchpad.net/ListHelp>
>>> 
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>>> <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>>
>>> Unsubscribe : https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>> More help   : https://help.launchpad.net/ListHelp <https://help.launchpad.net/ListHelp>
>> 
>> 
>> 
>> _______________________________________________
>> Mailing list: https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>> Unsubscribe : https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>> More help   : https://help.launchpad.net/ListHelp <https://help.launchpad.net/ListHelp>
>> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
> Unsubscribe : https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
> More help   : https://help.launchpad.net/ListHelp <https://help.launchpad.net/ListHelp>