kicad-developers team mailing list archive
-
kicad-developers team
-
Mailing list archive
-
Message #44441
Re: Placing real components out of a database instead of: "Automatic assignment of footprint with a database"
I think this thread has kind of blown up to use cases beyond the original
subject.
While I don't think it's wrong to consider *other* proposals for how people
want to get library data into KiCad, I want to limit this particular thread
(and the linked spec doc to one in particular):
An external ODBC-compatible database, managed outside of KiCad, containing
links to symbols/footprints in existing KiCad library files.
The use case for this is teams collaborating on designs in "private"
Not to be a global community-wide database of parts for all KiCad users,
and not to be an additional library option that adds value to individual
hobbyists using KiCad alone.
It's possible that both of those use-cases can be addressed by building
additional tools (inside or outside of KiCad) on top of this proposed
feature, but I don't want to try to address all those use cases at once.
Best,
Jon
On Sat, Aug 29, 2020 at 9:40 AM Wayne Stambaugh <stambaughw@xxxxxxxxx>
wrote:
>
> On 8/29/20 9:21 AM, Jeff Young wrote:
> > Surely there must be an open source impl of an ODBC interface on a CSV
> file?
>
> I'm pretty sure this already exists.
>
> >
> > Although I’m not sure of the desire to avoid MySQL. It’s remarkably
> > easy to set up an instance (or auto-deploy one with an app).
>
> MySQL has an ODBC plug in. So does PostgreSQL. I can't imagine any of
> the commercial databases not having an ODBC plug-in. The goal here is
> to maximize database coverage.
>
> >
> > Apologies if we’ve already talked about that; I’ll confess to not having
> > followed this thread 100%….
> >
> >> On 29 Aug 2020, at 14:11, Wayne Stambaugh <stambaughw@xxxxxxxxx
> >> <mailto:stambaughw@xxxxxxxxx>> wrote:
> >>
> >> I would most likely reject any solution that was tied to a particular
> >> database. All this would do is open up a Pandora's box of complaints as
> >> to why we didn't use database A over database B. ODBC is the most
> >> flexible solution that I am aware of and allows users to choose their
> >> preferred database.
> >>
> >> Cheers,
> >>
> >> Wayne
> >>
> >> On 8/29/20 8:29 AM, Jon Evans wrote:
> >>> Putting aside the fact that I think this feature isn't really aimed at
> >>> hobbyists, I would not be opposed to people wanting to extend it beyond
> >>> ODBC but that comes with extra complexity that must be handled.
> >>>
> >>> With ODBC, KiCad just needs to know about the interface to retrieve the
> >>> data.
> >>>
> >>> With a CSV file, KiCad actually needs to read that file in and keep it
> >>> in memory. Watch for modifications on disk, or else lock it
> >>> exclusively. Things like that.
> >>>
> >>> I'm not sure I really see the advantage of a CSV-backed "database" over
> >>> the existing KiCad library system, if we're talking about a single
> user.
> >>>
> >>> -Jon
> >>>
> >>> On Sat, Aug 29, 2020 at 8:19 AM Mark Roszko <mark.roszko@xxxxxxxxx
> >>> <mailto:mark.roszko@xxxxxxxxx>
> >>> <mailto:mark.roszko@xxxxxxxxx>> wrote:
> >>>
> >>>> Sqlite was a quick&dirty way to test if my ideas would work.
> >>> There are ODBC wrappers for SQLite........
> >>>
> >>>> I mean libreoffice would do for the management.
> >>> Yes, and you know what you would use? Not CSV files.
> >>> LibreOffice Base / Microsoft Access.
> >>> This is the office suite database that's basically SQLite and there
> >>> are ODBC wrappers as well :D
> >>>
> >>>> Also, why would a hobbyist fire up a sql database when a CSV file
> >>> would be sufficient? I mean libreoffice would do for the management.
> >>>
> >>> KiCad's uses aren't limited to hobbyists...
> >>> And, you assume there aren't hobbyists like me who will gladly take
> >>> that ODBC link and spin up an frontend in a few hours to the whole
> >>> system :D
> >>>
> >>>
> >>> On Sat, Aug 29, 2020 at 4:22 AM Johann Wilhelm
> >>> <johann.wilhelm@wilhelm.consulting
> >>> <mailto:johann.wilhelm@wilhelm.consulting>> wrote:
> >>>
> >>> Hi there!
> >>>
> >>> Well, then my comment was not completely wrong.
> >>>
> >>> Sqlite was a quick&dirty way to test if my ideas would work. For
> >>> my future productive system I really want to use a mix of
> >>> couchdb and maybe postgres. i.e. a JSON document storage for the
> >>> component information and sql for inventory management.
> >>>
> >>> So ODBC would not work well for me.
> >>>
> >>> Also, why would a hobbyist fire up a sql database when a CSV
> >>> file would be sufficient? I mean libreoffice would do for the
> >>> management.
> >>>
> >>> Additionally, I'd suggest looking at the BOM creation. There,
> >>> external programs are called.
> >>>
> >>> So why not define a dataformat (xml, json, csv,...) and just
> >>> call an external app or read from a file/uri?
> >>>
> >>> Anyways, I would volunteer for implementing some alternatives
> >>> (read from file/uri and output of executable) to the ODBC
> >>> interface if someone guides me through the KiCad procedures.
> >>>
> >>> Regards,
> >>> Johann
> >>>
> >>>
> >>> Jon Evans <jon@xxxxxxxxxxxxx
> >>> <mailto:jon@xxxxxxxxxxxxx> <mailto:jon@xxxxxxxxxxxxx>> schrieb
> >>> am Fr., 28. Aug. 2020, 19:54:
> >>>
> >>> My idea is to make it possible for KiCad to talk to an
> >>> external database.
> >>> The database itself (and its schema) will not be defined as
> >>> part of this spec, and will not be part of KiCad.
> >>>
> >>> The only requirement is that you have some columns in your
> >>> schema that KiCad understands (for example, to point to the
> >>> right symbols and footprints in the KiCad libraries).
> >>> The planned interface to connect to the database is ODBC.
> >>> This would in theory allow using sqlite to create a database
> >>> as a file on disk, although I anticipate that most users
> >>> will be using something like MariaDB or Postgres.
> >>>
> >>> There has been some discussion of supporting talking to web
> >>> interfaces via some REST API, or even talking to arbitrary
> >>> interfaces via Python scripting, but that discussion should
> >>> stay separate.
> >>> The original thread was about getting component information
> >>> out of a database and I just wanted to let people know that
> >>> I am working on this.
> >>>
> >>> People are welcome to also discuss the ideas of getting
> >>> component information from a web API or from some Python
> >>> script.
> >>> But, I am not working on that right now, and there hasn't
> >>> even been a conversation started on what that spec would
> >>> look like.
> >>>
> >>> Best,
> >>> Jon
> >>>
> >>> On Fri, Aug 28, 2020 at 1:42 PM Johann Wilhelm
> >>> <johann.wilhelm@wilhelm.consulting
> >>> <mailto:johann.wilhelm@wilhelm.consulting>> wrote:
> >>>
> >>> Hi Jon,
> >>>
> >>> Well, you're idea was to define a database or at least a
> >>> database schema if I understood this correctly:
> >>>
> >>> What I plan on implementing is not a "full"
> >>> database management system, but
> >>>
> >>> rather an interface to grab info out of a
> >>> database, just like you say.
> >>>
> >>> The only difference is, there are no plans to
> >>> store actual symbols or
> >>>
> >>> footprints in the database.
> >>>
> >>> The symbols and footprints will still be stored in
> >>> "normal" KiCad library
> >>>
> >>> files; the database will just contain "pointers"
> >>> telling KiCad which symbol
> >>>
> >>> to use (and what library it can be found in).
> >>>
> >>>
> >>>
> >>> I was pointing toward a specification of a data
> >>> format.So either read the data from a file or
> >>> webinterface (that's why I used URI).
> >>> My script-set currently gives me that information via
> >>> http://localhost:8000/API/Component/getMatchWisdom
> >>>
> >>> Others would maybe like to save a file in ~/.KiCad (so
> >>> the URI would be file://~/.KiCad)
> >>>
> >>> If "database"/"database interface" would include
> >>> something which could read from files and/or web
> >>> resources, well, never mind my comment :)
> >>>
> >>> With "global parameter" I mean something which could be
> >>> part of a .pro file.
> >>>
> >>> Regards,
> >>> Johann
> >>>
> >>>
> >>> Am Fr., 28. Aug. 2020 um 19:04 Uhr schrieb Jon Evans
> >>> <jon@xxxxxxxxxxxxx
> >>> <mailto:jon@xxxxxxxxxxxxx> <mailto:jon@xxxxxxxxxxxxx>>:
> >>>
> >>> Hi Johann,
> >>>
> >>> I am not sure exactly what you are arguing against.
> >>> I don't see any difference between what you are
> >>> looking for and what is in my spec.
> >>> I am not sure what you mean by "global URI
> >>> parameter" but the part picker will be able to
> >>> filter using any of the external data present.
> >>>
> >>> Best,
> >>> Jon
> >>>
> >>> On Fri, Aug 28, 2020 at 12:56 PM Johann Wilhelm
> >>> <johann.wilhelm@wilhelm.consulting
> >>> <mailto:johann.wilhelm@wilhelm.consulting>> wrote:
> >>>
> >>> Hi everyone!
> >>>
> >>> I'm new here in the mailing list but I'm
> >>> currently building my own electronics business
> >>> around KiCad which I use and love for years now.
> >>>
> >>> I spent the last weeks, trying to tie together
> >>> procurement, PCB design, and fabrication in a
> >>> single script-set.
> >>> Just to have it mentioned - before getting
> >>> self-employed, I worked for 10 years in the
> >>> electronics industry in a huge enterprise.
> >>> So I had some strict requirements for my tooling
> >>> and I have some very strong opinions on how my
> >>> perfect workflow should look like :)
> >>>
> >>> I'm very sorry but I need to say: PLEASE, don't
> >>> just throw a part database at the community!
> >>> Why? Because everyone has different
> >>> approaches how to do procurement and inventory
> >>> management!
> >>> It's ok (and actually good!) if you try to come
> >>> up with something but PLEASE, go for a defined
> >>> filter interface!
> >>>
> >>> What I would suggest implementing - actually, I
> >>> have plans to implement it myself in mid-term -
> >>> is a simple checkbox in the component dialog
> >>> named "Apply Parts Filter" and a global URI
> >>> parameter that provides the filter (or better:
> >>> the source of "wisdom" used by the filter).
> >>> Maybe another parameter defining the default for
> >>> the checkbox would be nice as well...
> >>>
> >>> I think even a simple CSV with columns for
> >>> Symbol, Footprint, and Value could provide
> >>> sufficient information for effective
> >>> filtering/adaption of the symbol-tree.
> >>>
> >>> A CSV like this:
> >>>
> >>>
> >>>
> "Audio:TLV320AIC23BPW", "TLV320AIC23BPW", "Package_SO:TSSOP-28_4.4x9.7mm_P0.65mm"
> >>>
> "Device:R_Small", "1k", "Resistor_SMD:R_0805_2012Metric"
> >>> "Device:R*", "12k4",
> >>> "Resistor_SMD:R_0805_2012Metric"
> >>>
> "Device:R*", "12k4", "Resistor_SMD:R_1206_3216Metric"
> >>>
> "Device:R*", "10R", "Resistor_SMD:R_0805_2012Metric"
> >>>
> >>>
> >>> should result in a component tree like this:
> >>>
> >>> Audio
> >>>
> >>> TLV320AIC23BPW <-- this symbol has a
> >>> matching default footprint and value
> >>>
> >>> Device
> >>>
> >>> R <-- this symbol is included in the
> >>> CSV data
> >>>
> >>> 12k4 <-- a value of 12k4 would
> >>> match Device:R and a corresponding
> >>> symbol is added by the filter
> >>>
> >>> Resistor_SMD:R_0805_2012Metric
> >>> <--- both footprints would
> >>> match Device:R 12k4 so the
> >>> filter adds both symbols with
> >>> the footprint and values fields
> >>> complemented
> >>> Resistor_SMD:R_0805_2012Metric
> >>> <--- ....
> >>>
> >>> 10R
> >>>
> >>> Resistor_SMD:R_0805_2012Metric
> >>>
> >>> R_Small
> >>>
> >>> 1k <-- 1k only matches
> >>> Device:R_Small
> >>>
> >>> Resistor_SMD:R_0805_2012Metric
> >>>
> >>> 12k4
> >>>
> >>> Resistor_SMD:R_0805_2012Metric
> >>> Resistor_SMD:R_0805_2012Metric
> >>>
> >>> 10R
> >>>
> >>> Resistor_SMD:R_0805_2012Metric
> >>>
> >>>
> >>> So the filter should add all symbols referenced
> >>> in the CSV.
> >>> If a symbol has no matching footprint and/or
> >>> value => "create" them on fly
> >>>
> >>> With this, you can use the whole symbol library
> >>> or the existing parts in your inventory.
> >>> The best of both worlds!
> >>>
> >>> Regards,
> >>> Johann
> >>>
> >>>
> >>> Am Fr., 28. Aug. 2020 um 16:38 Uhr schrieb Brian
> >>> <lotharyx@xxxxxxxxx
> >>> <mailto:lotharyx@xxxxxxxxx> <mailto:lotharyx@xxxxxxxxx>>:
> >>>
> >>> Just want to add my +1 for the interface
> >>> approach. I am glad to hear that is the
> >>> intent (I’ve not had time to read the
> >>> proposal). With such an interface, as has
> >>> been pointed out already,
> >>> data-source-specific implementations should
> >>> be relatively straightforward, and then I
> >>> don’t have to potentially throw away my
> >>> years of privately curated data or figure
> >>> out how to cram it into some alternate
> >>> schema. All I would need to do is write the
> >>> code to answer the questions presented by
> >>> the interface.
> >>>
> >>> Hopefully in the next few days I’ll be able
> >>> to read Jon’s draft spec and comment from a
> >>> better-informed position.
> >>>
> >>> Cheers,
> >>> -Brian
> >>>
> >>>> On Aug 28, 2020, at 9:43 AM, Jon Evans
> >>>> <jon@xxxxxxxxxxxxx <mailto:
> jon@xxxxxxxxxxxxx>
> >>>> <mailto:jon@xxxxxxxxxxxxx>> wrote:
> >>>>
> >>>>
> >>>> Hi Clemens,
> >>>>
> >>>> On Fri, Aug 28, 2020 at 9:34 AM Clemens
> >>>> Koller <cko@xxxxxxxxx <mailto:
> cko@xxxxxxxxx>
> >>>> <mailto:cko@xxxxxxxxx>> wrote:
> >>>>
> >>>> Hello!
> >>>>
> >>>> This is related to the previous
> >>>> thread: "Automatic assignment of
> >>>> footprint with a database"
> >>>>
> >>>> I would generally prefer assemble real
> >>>> components on a real PCB right from
> >>>> the beginning instead of first placing
> >>>> generic components and then assign
> >>>> footprints + manufacturers + types + x
> >>>> manually. This seems extra work for
> >>>> each component which could possibly be
> >>>> avoided.
> >>>>
> >>>>
> >>>> Me too!
> >>>>
> >>>>
> >>>> So, regarding on the Kicad codebase, I
> >>>> would very likely not recomment to
> >>>> embed a full component management /
> >>>> database system, since this might vary
> >>>> from site to site and even from
> >>>> project/assembly house to
> >>>> project/house. But it would be great
> >>>> to be able to have an interface to
> >>>> grab a component out of a database and
> >>>> Kicad grabs all desired / a selection
> >>>> of individual columns out of the
> >>>> database as needed.
> >>>> This might include the actual
> >>>> footprints stored in the database as
> >>>> well.
> >>>>
> >>>>
> >>>> What I plan on implementing is not a
> >>>> "full" database management system, but
> >>>> rather an interface to grab info out of a
> >>>> database, just like you say.
> >>>> The only difference is, there are no plans
> >>>> to store actual symbols or footprints in
> >>>> the database.
> >>>> The symbols and footprints will still be
> >>>> stored in "normal" KiCad library files;
> >>>> the database will just contain "pointers"
> >>>> telling KiCad which symbol to use (and
> >>>> what library it can be found in).
> >>>>
> >>>> BTW, the example schema in your email
> >>>> looks very familiar to me. This is the
> >>>> kind of data source that can be used with
> >>>> the feature I am talking about: just add
> >>>> columns to the schema for tracking which
> >>>> KiCad symbol and footprint(s) should be
> >>>> used for a part.
> >>>>
> >>>> Regarding the advanced features you
> >>>> mention, some of them sound like they
> >>>> would be handled by a PLM tool.
> >>>> Some of the PLM tools I have worked with
> >>>> can interface to external databases for
> >>>> managing this kind of component library
> >>>> for an EDA or mechanical CAD tool.
> >>>>
> >>>> Best,
> >>>> Jon
> >>>>
> _______________________________________________
> >>>> Mailing list:
> >>>> https://launchpad.net/~kicad-developers
> >>>> Post to :
> >>>> kicad-developers@xxxxxxxxxxxxxxxxxxx
> >>>> <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
> >>>> <mailto:
> kicad-developers@xxxxxxxxxxxxxxxxxxx>
> >>>> Unsubscribe :
> >>>> https://launchpad.net/~kicad-developers
> >>>> More help :
> >>>> https://help.launchpad.net/ListHelp
> >>>
> _______________________________________________
> >>> Mailing list:
> >>> https://launchpad.net/~kicad-developers
> >>> Post to :
> >>> kicad-developers@xxxxxxxxxxxxxxxxxxx
> >>> <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
> >>> <mailto:
> kicad-developers@xxxxxxxxxxxxxxxxxxx>
> >>> Unsubscribe :
> >>> https://launchpad.net/~kicad-developers
> >>> More help :
> >>> https://help.launchpad.net/ListHelp
> >>>
> >>> _______________________________________________
> >>> Mailing list:
> >>> https://launchpad.net/~kicad-developers
> >>> Post to :
> >>> kicad-developers@xxxxxxxxxxxxxxxxxxx
> >>> <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
> >>> <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
> >>> Unsubscribe :
> >>> https://launchpad.net/~kicad-developers
> >>> More help :
> https://help.launchpad.net/ListHelp
> >>>
> >>> _______________________________________________
> >>> Mailing list: https://launchpad.net/~kicad-developers
> >>> Post to : kicad-developers@xxxxxxxxxxxxxxxxxxx
> >>> <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
> >>> <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
> >>> Unsubscribe : https://launchpad.net/~kicad-developers
> >>> More help : https://help.launchpad.net/ListHelp
> >>>
> >>>
> >>>
> >>> --
> >>> Mark
> >>>
> >>>
> >>> _______________________________________________
> >>> Mailing list: https://launchpad.net/~kicad-developers
> >>> Post to : kicad-developers@xxxxxxxxxxxxxxxxxxx
> >>> <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
> >>> Unsubscribe : https://launchpad.net/~kicad-developers
> >>> More help : https://help.launchpad.net/ListHelp
> >>>
> >>
> >> _______________________________________________
> >> Mailing list: https://launchpad.net/~kicad-developers
> >> Post to : kicad-developers@xxxxxxxxxxxxxxxxxxx
> >> <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
> >> Unsubscribe : https://launchpad.net/~kicad-developers
> >> More help : https://help.launchpad.net/ListHelp
> >
>
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help : https://help.launchpad.net/ListHelp
>
References
-
Re: Placing real components out of a database instead of: "Automatic assignment of footprint with a database"
From: Jon Evans, 2020-08-28
-
Re: Placing real components out of a database instead of: "Automatic assignment of footprint with a database"
From: Brian, 2020-08-28
-
Re: Placing real components out of a database instead of: "Automatic assignment of footprint with a database"
From: Johann Wilhelm, 2020-08-28
-
Re: Placing real components out of a database instead of: "Automatic assignment of footprint with a database"
From: Jon Evans, 2020-08-28
-
Re: Placing real components out of a database instead of: "Automatic assignment of footprint with a database"
From: Johann Wilhelm, 2020-08-28
-
Re: Placing real components out of a database instead of: "Automatic assignment of footprint with a database"
From: Jon Evans, 2020-08-28
-
Re: Placing real components out of a database instead of: "Automatic assignment of footprint with a database"
From: Johann Wilhelm, 2020-08-29
-
Re: Placing real components out of a database instead of: "Automatic assignment of footprint with a database"
From: Mark Roszko, 2020-08-29
-
Re: Placing real components out of a database instead of: "Automatic assignment of footprint with a database"
From: Jon Evans, 2020-08-29
-
Re: Placing real components out of a database instead of: "Automatic assignment of footprint with a database"
From: Wayne Stambaugh, 2020-08-29
-
Re: Placing real components out of a database instead of: "Automatic assignment of footprint with a database"
From: Jeff Young, 2020-08-29
-
Re: Placing real components out of a database instead of: "Automatic assignment of footprint with a database"
From: Wayne Stambaugh, 2020-08-29
