kicad-developers team mailing list archive

[Wayne Stambaugh <stambaughw@xxxxxxxxx>]

On 12/14/2017 7:46 PM, Oliver Walters wrote:
>     I noticed that your symbol library table uses absolute paths rather than
>     using the SYMBOL_LIB_DIR path expansion.  This should not make any
>     difference as long as the path you used is the same path as old library
>     list look up path.  If it's not, that is your problem.  The remapping
>     will not look up by symbol name in the symbol library table if the
>     symbol is not found in the original library in the old library list that
>     it was loaded from.  I will attempt to explain this again so everyone
>     who is having this issue can understand what is happening.
> 
> 
> The sym-lib-table was created by the project-update tool, and I had not
> defined SYMBOL_LIB_DIR. However the generated sym-lib-table did point to
> the libraries I was using previously in this project (v4)
> 
> SYMBOL_LIB_DIR was defined as "C:\msys64\mingw64\share\kicad\library". I
> don't have a build setup on this machine and the signed windows were
> unavailable. This is a jenkins nightly.
> 
> The process makes sense (I think) but the failures are quite transparent.
> 
> Currently the "rescue orphans" procedure in the other dialog works very
> well. Perhaps we could:

Since I didn't write it, I do not know how this code works but if it
looks up by symbol name only and library search order, then it is just
as broken as the previous design and it's only a matter of time before
there is bug report filed against it.

> 
> a) Run this operation after "rescue" if the rescue did not work?

I do not know what rescue didn't work means.  All the rescuer does is
creates a new library from symbols in the cache library that are
different than the library they where loaded from or symbols that are no
longer found in the original library list.  The only way the rescue does
not work is if the cache library file is corrupt or missing in which
case your out of luck.

> b) Explicitly point users to make use of this dialog.
> 
> On Fri, Dec 15, 2017 at 10:42 AM, Wayne Stambaugh <stambaughw@xxxxxxxxx
> <mailto:stambaughw@xxxxxxxxx>> wrote:
> 
>     I noticed that your symbol library table uses absolute paths rather than
>     using the SYMBOL_LIB_DIR path expansion.  This should not make any
>     difference as long as the path you used is the same path as old library
>     list look up path.  If it's not, that is your problem.  The remapping
>     will not look up by symbol name in the symbol library table if the
>     symbol is not found in the original library in the old library list that
>     it was loaded from.  I will attempt to explain this again so everyone
>     who is having this issue can understand what is happening.
> 
>     Here are the steps that occur for remapping each symbol (not including
>     the final rescue since that is just another library in the old project
>     library list):
> 
>     Find the full path and file name of the old library list lookup method
>     used obtain the symbol link.  This is found in LIB_PART::m_library
>     member.  This will point to the same library as if you were using stable
>     version 4.
> 
>     Look for the library by path and full file name in the global symbol
>     library table.  If the library cannot be found in the global symbol
>     library table and the file exists, add this library to the project
>     symbol library table and give it a unique name if necessary.
> 
>     Set the symbol library nickname to the entry in the symbol library
>     table.
> 
>     Once all symbols are remapped, write the project symbol library table
>     and refresh if necessary and remove all of the libraries from the old
>     project library list.
> 
>     I suspect that you are running a development version of kicad that
>     cannot find the original symbol libraries in the project library list
>     because they are not in a path that it expects them to be in.  You can
>     fix this by manually editing the project file to include this path so
>     the remapping will work correctly.
> 
>     Once the remapping is complete,
>     On 12/14/2017 06:05 PM, Oliver Walters wrote:
>     > Wayne,
>     >
>     > Here is the issue I am facing. Maybe I have misunderstood the process,
>     > or maybe I have found a bug.
>     >
>     > 1. Load simple legacy project and attempt to remap. Failures across the
>     > board.
>     > Inline image 1
>     >
>     > 2. Check the sym-lib-table
>     >
>     > Inline image 2
>     >
>     >
>     > 3. Confirm that the symbols do actually exist in the loaded libs
>     >
>     >
>     > Inline image 3
>     >
>     > 4. No dice
>     >
>     > Inline image 4
>     >
>     > 5. Try again?
>     >
>     > Inline image 5
>     >
>     > 6. Weep, etc
>     >
>     > What am I doing wrong?
>     >
>     > Oliver
>     >
>     > On Thu, Dec 14, 2017 at 10:41 AM, Wayne Stambaugh <stambaughw@xxxxxxxxx <mailto:stambaughw@xxxxxxxxx>
>     > <mailto:stambaughw@xxxxxxxxx <mailto:stambaughw@xxxxxxxxx>>> wrote:
>     >
>     >     On 12/13/2017 01:56 AM, Oliver Walters wrote:
>     >     > I finally had a chance to try the symbol remapping on a real
>     project,
>     >     > and I have a few thoughts on how I think it could be
>     improved from a
>     >     > users perspective (with code to back up my thoughts!)
>     >     >
>     >     > 1. "Remap" should replace "rescue"
>     >     >
>     >     > The remap tool, with some work, should replace the "symbol
>     rescue" tool.
>     >     > There are two major cases which it should cover:
>     >     >
>     >     > a) Handling "legacy" (v4 and prior) symbols which do not
>     specify a
>     >     > library nickname
>     >     > b) Handling missing libraries or symbols
>     >     >  i) A library has been moved or deleted
>     >     > ii) An item has been renamed in a library
>     >
>     >
>     >     These are two distinctly different processes and need to stay
>     separate.
>     >     Please do not merge them.
>     >
>     >     >
>     >     > The rescue tool does not seem to work at all now, with the new
>     >     > sym-lib-table approach. I think I have come up with a way to
>     combine the
>     >     > two tools that provides a better approach anyhow.
>     >
>     >     Please explain.  The last testing I did, both the legacy and fully
>     >     defined LIB_ID rescue works as required.  Are you saying if
>     you delete
>     >     or modify a library symbol used in a schematic with a fully
>     defined
>     >     LIB_ID that is does not get rescued properly from the cache
>     library?  If
>     >     so, please file a bug report so I can fix it.
>     >
>     >     >
>     >     > 2. Let users "remap" multiple times.
>     >     >
>     >     > The current implementation only allows remap if *all* the
>     symbols are
>     >     > un-mapped. If a partial remap occurs then the users are left
>     high and
>     >     > dry. Partial remaps should be allowed, this gives the users
>     chance to
>     >     > fix their library tables or whatever.
>     >
>     >     I'm not opposed to this but only for symbols that do not
>     already have a
>     >     library nickname assigned to the schematic symbol LIB_ID. 
>     Once a symbol
>     >     is remapped and mapping is correct, the only other valid
>     operation is to
>     >     choose a new symbol.  I would reject any change that fell back to
>     >     looking up symbols by name alone (no library).  This was the
>     entire
>     >     point of the remapping and would reintroduce the bug that has been
>     >     around since the beginning of the project.
>     >
>     >     >
>     >     > 3. Separate project-remap from symbol-remap
>     >     >
>     >     > The regeneration of library tables should be separate from
>     symbol
>     >     > remapping. Rescue the project once, (with a visible warning
>     to user at
>     >     > first project load, as is currently the case). But, allow
>     symbol remap
>     >     > from the menu at any stage (even if there are no symbols to
>     remap!)
>     >
>     >     This will likely not work because more often than not, successful
>     >     remapping depends on the project specific symbol libraries not
>     just the
>     >     global symbol libraries.  This is why the remapping utility first
>     >     creates a project specific symbol library table based upon the
>     libraries
>     >     it needs (contains symbols used in the schematic) from the old
>     library
>     >     list.  Keep in mind, the remapping does not just look for the
>     first
>     >     occurrence of a symbol with a given name in the symbol library
>     table
>     >     (that would be broken), it actually matches the full file path
>     and file
>     >     name from the symbol library table to do the mapping.  There
>     is a bug
>     >     where symlinks will get mapped as a different library.  I have
>     a fix for
>     >     that but I have had time to test it.
>     >
>     >     >
>     >     > 4. Group similar symbols together when remapping
>     >     >
>     >     > If a schematic has 100+ 'R' symbols, they should be grouped
>     together so
>     >     > that they can *all* be remapped at once. Collect similar symbols
>     >     > together. (See demonstrator image below).
>     >
>     >     Nice but not crucial to the remapping.  Sorting could be added
>     to the
>     >     existing remapping feature as well.
>     >
>     >     >
>     >     > 5. Allow manual remap selection
>     >     >
>     >     > If auto-remapping fails for a particular symbol, allow the
>     user to
>     >     > select a new symbol manually
>     >
>     >     This isn't really remapping.  This is reassigning a library
>     symbol.  If
>     >     a symbol cannot be located in any library in the old library
>     list, there
>     >     is no way to recover it.  This was true before the switch to fully
>     >     defined LIB_IDs.
>     >
>     >     >
>     >     > 6. Allow skipping of remapped symbols
>     >     >
>     >     > Some may be not able to be remaped at the current time.
>     Allow user to
>     >     > skip and come back later.
>     >
>     >     Do you really want listen to users scream about broken symbol
>     links?
>     >     Keep in mind, once the remapping is complete, all of the old
>     libraries
>     >     from the symbol library table are removed from the project
>     files so they
>     >     are no longer available for future remapping which will almost
>     certainly
>     >     lead to issues.  I will reject keeping the old library list in
>     tact as
>     >     it was part of the original issue.
>     >
>     >     >
>     >     > 7. Provide a dry-run first and let the user see what is
>     happening.
>     >
>     >     I'm fine with a dry run feature.  I had it on my todo list but
>     if you
>     >     get to it before I do.  My feelings wont be hurt.
>     >
>     >     >
>     >     >
>     >     > I have actually implemented all of the points above. See
>     demonstrator
>     >     > images here:
>     >     >
>     >     > A) Dry run with explicit output messages, and information on
>     what will
>     >     > be changed
>     >     >
>     >     > Inline image 1
>     >     >
>     >     > B) Success!
>     >     >
>     >     > Inline image 2
>     >     >
>     >     > If this is a good approach I will spend some more time on
>     this, there
>     >     > are still some issues to work out. However, It is (to me) a
>     much cleaner
>     >     > approach.
>     >     >
>     >     > Thoughts?
>     >     >
>     >     > Oliver
>     >     >
>     >     >
>     >     > _______________________________________________
>     >     > Mailing list: https://launchpad.net/~kicad-developers
>     <https://launchpad.net/~kicad-developers>
>     >     <https://launchpad.net/~kicad-developers
>     <https://launchpad.net/~kicad-developers>>
>     >     > Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>     <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>     >     <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx
>     <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>>
>     >     > Unsubscribe : https://launchpad.net/~kicad-developers
>     <https://launchpad.net/~kicad-developers>
>     >     <https://launchpad.net/~kicad-developers
>     <https://launchpad.net/~kicad-developers>>
>     >     > More help   : https://help.launchpad.net/ListHelp
>     <https://help.launchpad.net/ListHelp>
>     >     <https://help.launchpad.net/ListHelp
>     <https://help.launchpad.net/ListHelp>>
>     >     >
>     >
>     >     _______________________________________________
>     >     Mailing list: https://launchpad.net/~kicad-developers
>     <https://launchpad.net/~kicad-developers>
>     >     <https://launchpad.net/~kicad-developers
>     <https://launchpad.net/~kicad-developers>>
>     >     Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>     <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>     >     <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx
>     <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>>
>     >     Unsubscribe : https://launchpad.net/~kicad-developers
>     <https://launchpad.net/~kicad-developers>
>     >     <https://launchpad.net/~kicad-developers
>     <https://launchpad.net/~kicad-developers>>
>     >     More help   : https://help.launchpad.net/ListHelp
>     <https://help.launchpad.net/ListHelp>
>     >     <https://help.launchpad.net/ListHelp
>     <https://help.launchpad.net/ListHelp>>
>     >
>     >
> 
>