kicad-developers team mailing list archive

[Jon Evans <jon@xxxxxxxxxxxxx>]

I have never worked on Blender but just poked around the source a bit.
They write custom modules in CPython rather than using a wrapper like SWIG.

To expose internal data, they have a Python wrapper over a C data access
API.
The C API seems to have portions that are auto-generated by the build, and
portions manually done for each structure to be exposed.

More info:
http://dustycloud.org/misc/pycon_2011_blender_talk.pdf
https://wiki.blender.org/index.php/Dev:2.5/Source/Architecture/RNA
https://wiki.blender.org/index.php/Dev:2.5/Source/Python/PyRNA

If we can get an object properties system installed throughout KiCad,
perhaps we can write a lightweight "universal wrapper" over properties
get/set.
Then we can define some standard mechanism for calling methods where
necessary, but I'm guessing the bulk of Python scripts will eventually be
able to be reduced to property get/set operations (assuming we write the
"magic" in the backend that takes care of notifying other parts of the code
that things have been changed, etc).

That way, there is no longer any direct link between Python objects and C++
objects; but rather an indirect link that we control.
This should make it easier to handle cases like new/delete properly.

I'm interested in helping brainstorm this more at some point; Python/C++
interaction is something I'd like to learn more about how to do "right"
both for KiCad and for some personal projects.

-Jon

On Sat, Mar 10, 2018 at 3:59 AM, miles mccoo <mail@xxxxxxxxxx> wrote:

> thank you all.
>
> Yes, looking at other projects, blender in particular, would be wise. my
> perception is they have a really nice scripting capability. You can do
> anything there in scripting including commands that are indistinguishable
> from the others.
>
> I don't suppose anyone knows a blender contributor? (or some other open
> project with strong scripting support).
>
>
> Miles
>
> personal reference: patch 6/7
>
>
>
>
> On Thu, Mar 8, 2018 at 5:35 PM, Wayne Stambaugh <stambaughw@xxxxxxxxx>
> wrote:
>
>> As a short term fix, I'll allow this patch to prevent the segfaults.
>> However, I do not want to continue to allowing these types of ad hoc
>> fixes.  We should take Dick's suggestion and look at what other projects
>> are doing.  We cannot be the only project wrestling with this issue.
>>
>> On 3/8/2018 3:59 AM, Maciej Sumiński wrote:
>> > In my opinion #3 is acceptable. It does not require any extra changes in
>> > the existing scripts and it fixes the problem until we come up with a
>> > better scripting interface. If nobody has objections, I am going to
>> > commit the patch.
>> >
>> > Regards,
>> > Orson
>> >
>> > On 03/06/2018 10:02 AM, miles mccoo wrote:
>> >> Thanks all for your replies.
>> >>
>> >> I like the plugin mechanism. It does some nice things for python folks.
>> >> Refresh, undo, garbage collection (I think). I want to see it succeed.
>> >>
>> >> From reading Orson's mail, I think I wasn't entirely clear, so let me
>> try
>> >> to fix that first.
>> >>
>> >>
>> >> The plugin mechanism tries to track the changes of a python plugin. In
>> the
>> >> c++ world, developers are expected to inform BOARD_COMMIT that
>> something
>> >> has changed. The plugin infrastructure does this for you. One could
>> compare
>> >> it to garbage collection. My interpretation of how this is done is
>> simple:
>> >> before a plugin runs, make a list of all design objects. zones, tracks,
>> >> modules, nets... after the plugin completes, do a "diff".
>> >>
>> >> This is mostly fine with the exception of removed items.
>> >>
>> >> The problem with removed items is how do you do a diff on a deleted,
>> memory
>> >> managed, cleaned up object? Perhaps that memory now contains a
>> different
>> >> kind of object. Perhaps it's been reallocated to a similar object as
>> before.
>> >>
>> >>
>> >>
>> >> *as it stands now, any plugin that removes items from a board item
>> >> container will likely fail*. It's not quite a crash, but it has a
>> similar
>> >> feel to the user.
>> >>
>> >>
>> >>
>> >> Solutions. I can think of four.
>> >>
>> >> solution 1. Why is the plugin mechanism in the business of tracking
>> >> changes? Shouldn't BOARD_COMMIT just magically happen whenever a c++
>> >> instance is modified. I brought this up in a previous thread[1] and
>> various
>> >> reasons were given why this isn't desirable.
>> >>
>> >> solution 2. Same as #1 except BC magic only happens on python API
>> calls.
>> >> The plugin mechanism would no longer do diffs. Just add BC checkpoints.
>> >> This is probably a lot of work. I might be willing to do this work but
>> it
>> >> would take time. [2]
>> >>
>> >> solution 3. easy to implement. turn off deletion on calls to remove if
>> >> currently running a plugin
>> >>
>> >>    - plugin gets a boolean: isPluginActive. set/unset around the call
>> to
>> >>    actual python run code.
>> >>    - add pcbnew.isPluginActive() to python api. (I could imagine this
>> could
>> >>    have other uses)
>> >>    - the remove code looks at isPluginActive to decide whether to set
>> >>    isown. (that code is actually python, not swig)
>> >>
>> >> I have #3 implemented and my plugins no longer crash. *See attached
>> patch
>> >> for if folks agree it is an acceptable stopgap.*
>> >>
>> >> solution 4. probably not the direction I would go but a way to enable
>> >> python memory management and do the plugin diff thing. Basically, give
>> each
>> >> object some sort of unique id. (could be useful for other things). In
>> >> addition to holding a list of object pointers, plugin infrastructure
>> would
>> >> hold a list of object ids.
>> >>
>> >>
>> >> *Given the ease with which a plugin can hit this issue and given how
>> long
>> >> it would take to get #2 right, I guess I'm recommending the changes of
>> that
>> >> attached patch for #3.*
>> >>
>> >> Miles
>> >>
>> >> [1] https://lists.launchpad.net/kicad-developers/msg32063.html
>> >> [2] I think my personal opinion is that #1 is superior over #2. Python
>> >> people shouldn't have to care but why is it that c++ should or want to?
>> >> Yes, I read the arguments from the previous thread but I'm not
>> convinced.
>> >> I'm also just a kicad spectator, so there's a lot I don't know.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Fri, Mar 2, 2018 at 2:47 PM, Wayne Stambaugh <stambaughw@xxxxxxxxx>
>> >> wrote:
>> >>
>> >>> LOL, I just replied to Miles.  Thanks Orson for helping out!
>> >>>
>> >>> On 3/2/2018 8:36 AM, Maciej Sumiński wrote:
>> >>>> Hi Miles,
>> >>>>
>> >>>> I suppose the silence in the thread indicates there are not many
>> >>>> developers knowing the Python scripting interface inside out. Since
>> you
>> >>>> are both studying the scripting interface and developing own
>> scripts, it
>> >>>> is quite possible you are the most competent person to give us an
>> advice
>> >>>> on how to proceed. See some comments below, but I am neither a Python
>> >>>> script developer nor a scripting interface maintainer, so I might be
>> >>>> lacking some basic knowledge here.
>> >>>>
>> >>>> On 02/28/2018 05:12 PM, miles mccoo wrote:
>> >>>>> So I'm plugin-ifying my python scripts (the mechanism is awesome).
>> One
>> >>> of
>> >>>>> the plugins deletes some stuff and that is causing trouble.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> I'm not sure how to fix the root cause. Hence this mail.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> The plugin just deletes Edge.Cuts[1]:
>> >>>>> for d in board.GetDrawings():
>> >>>>>             if (d.GetLayerName() == 'Edge.Cuts'):
>> >>>>>                 board.Remove(d)
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> in board_item_container.i, I see this (with stuff deleted):
>> >>>>> %rename(RemoveNative)    BOARD_ITEM_CONTAINER::Remove;
>> >>>>>     def Remove(self,item):
>> >>>>>         self.RemoveNative(item)
>> >>>>>         item.thisown=1
>> >>>>>
>> >>>>>
>> >>>>> Setting thisown tells, python "you own it". Delete it when you're
>> done.
>> >>>>> Which it does.
>> >>>>>
>> >>>>>
>> >>>>> The problem this causes is that the plugin mechanism saves a list
>> of all
>> >>>>> objects before running the plugin and then checks if any of them
>> has a
>> >>> null
>> >>>>> list after (ie is it still in the design).
>> >>>>
>> >>>> Is this mechanism implemented to handle memory leaks? If so, would
>> not
>> >>>> it be sufficient to stick to 'thisown' flag on Remove() calls or is
>> >>>> there another way objects might be destroyed using Python scripts?
>> >>>>
>> >>>>> Since the object has been deleted by python, the plugin stuff gets
>> >>> confused.
>> >>>>>
>> >>>>>
>> >>>>> *So, the question is how to fix this?*
>> >>>>>
>> >>>>>
>> >>>>> It appears that the plugin infrastructure will delete for you
>> (that's
>> >>> what
>> >>>>> I'm guessing), so the thisown setting shouldn't be done.
>> >>>>>
>> >>>>>
>> >>>>> On the other hand, if running code from within a standalone script
>> (ie
>> >>> from
>> >>>>> regular python shell), now thisown'ing it would yield a memory leak.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> Perhaps the plugin stuff should have some sort of flag indicating
>> >>> "you're
>> >>>>> in a plugin". Then the thisown setting could be conditional.
>> >>>>
>> >>>> If the object listing mechanism is required for other reasons, then I
>> >>>> suppose it is second best idea. Generally speaking, I would like to
>> make
>> >>>> the scripting interface convenient for the users, so they do not
>> need to
>> >>>> worry about whether their scripts are run standalone or as a plugin.
>> >>>> Let's hide the dirty magic from them and make the coding process
>> >>> enjoyable.
>> >>>>
>> >>>> Regards,
>> >>>> Orson
>> >>>>
>> >>>>> But I'm just a spectator. *I'm happy to put in the time to fix this
>> but
>> >>>>> need guidance on what approach to take.*
>> >>>>>
>> >>>>>
>> >>>>> Miles
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> [1] full plugin text
>> >>>>> import pcbnew
>> >>>>>
>> >>>>> class RemoveBoundaryPlugin(pcbnew.ActionPlugin):
>> >>>>>     def defaults(self):
>> >>>>>         self.name = "Remove boundary"
>> >>>>>         self.category = "A descriptive category name"
>> >>>>>         self.description = "This plugin reads a dxf file and
>> converts
>> >>> it to
>> >>>>> a zone"
>> >>>>>
>> >>>>>     def Run(self):
>> >>>>>         board = pcbnew.GetBoard()
>> >>>>>
>> >>>>>         for d in board.GetDrawings():
>> >>>>>             print("{}".format(str(d)))
>> >>>>>             #print("on layer {} {} {}".format(d.GetLayerName(),
>> >>>>>             #                                 str(d.GetStart()),
>> >>>>>             #                                 str(d.GetEnd())))
>> >>>>>             if (d.GetLayerName() == 'Edge.Cuts'):
>> >>>>>                 board.Remove(d)
>> >>>>>
>> >>>>> RemoveBoundaryPlugin().register()
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> Mailing list: https://launchpad.net/~kicad-developers
>> >>>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>> >>>>> Unsubscribe : https://launchpad.net/~kicad-developers
>> >>>>> More help   : https://help.launchpad.net/ListHelp
>> >>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> Mailing list: https://launchpad.net/~kicad-developers
>> >>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>> >>>> Unsubscribe : https://launchpad.net/~kicad-developers
>> >>>> More help   : https://help.launchpad.net/ListHelp
>> >>>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> Mailing list: https://launchpad.net/~kicad-developers
>> >>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>> >>> Unsubscribe : https://launchpad.net/~kicad-developers
>> >>> More help   : https://help.launchpad.net/ListHelp
>> >>>
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> Mailing list: https://launchpad.net/~kicad-developers
>> >> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>> >> Unsubscribe : https://launchpad.net/~kicad-developers
>> >> More help   : https://help.launchpad.net/ListHelp
>> >>
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > Mailing list: https://launchpad.net/~kicad-developers
>> > Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>> > Unsubscribe : https://launchpad.net/~kicad-developers
>> > More help   : https://help.launchpad.net/ListHelp
>> >
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers
> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
>