← Back to team overview

kicad-developers team mailing list archive

Re: Kicad + V8/NodeJS/Swig [split from kicad ideas mail]

 

Thanks Miguel.  Can you clearly state the the SWIG support for V8, which is still external
to the SWIG project, is ready for production use?  Or do you still have doubts?

That was the original question. 

I also appreciate the use cases being identified up front.  I noticed you also did not say
we should try and drive the UI from a scripting language.

This is interesting work, and I would eventually like to be able to offer more of my time
on this particular topic.  However at this moment, my time is limited.  I do feel that I
am very qualified to offer help.  I have written an entire Java Virtual Machine myself,
including a JNI layer.  Sold it on the open market for a couple of years, back in the
90's, pre-JIT period.  This was the 2nd Java Virtual Machine ever written, and it has a
real time incremental garbage collector in it.   I have embedded it into C++, call it from
C++, it calls C++, and all this.  I know what it takes to package something like this,
support it, and debug it.   It still runs in factories all over the world.  All this to
say that I could offer qualified help with this if I had time or financial incentive.

If I were to be able to spend more time on this.  I certainly would not be in a hurry to
rush to a solution, since any quality implementation requires enough up front research to
first establish the best path, and in today's world with so many open source projects to
look at, that is a non-trivial effort.

Quite a few questions need to be answered and here are only a couple:

a) should we be concerned about supporting more than one scripting language, or will it be
sufficient to support only one?  This decision has a big impact on how you build
infrastructure.

b) Is the scripting language sufficiently easier to use than C++?  Because another path
would be to put a C++ compiler and build environment out in the cloud, where a guy could
write his script in C++, HTTP POST it to a website, and get a DLL/DSO back, ready to run. 
This is your "script" in binary.  (This one is a setup for the next question.)

c) What kind of security holes to you open up when you down load (scripting) code from
third parties, and is it our job to be concerned about those?

d) ..


Scripting is not at the top of my priority list at this time.  Therefore I will not be
able to offer any assistance at this time, but probably will be able to later, and might
have time to offer some advice from time to time now, but even that will be limited.

Thanks for your enthusiasm.

Dick





> Ok, I changed the topic of this thread, to match the discussion, which is about making
> Kicad fully scriptable in Javascript.
>
> For speed reasons, and wide adoption, it seems that V8 must be our preferred engine.
>
> V8 comes just with the javascript JIT interpreter, and no more. It's ready to link
> agains your library or applications, it can be debugged with a remote connection, but it
> doesn't include *any* system access library, networking, just nothing.
>
> In V8, you register functions which can be accessed from javascript to C++, and you can
> also call Javascript code from C++
>
> Here is where NodeJS seems attractive: it adds many system and network access layers.
> It's not ready out of the box to be embedded in other applications, but it could be
> easily patched for that (already checked the code). 
>
> Anyway nodejs's implementation (I think) wouldn't match just any architecture, it's
> something we should study carefully:
>
>        1) It's ready for asynchronous operations (like most javascript code).  [
> see http://nodejs.org/docs/latest/api/fs.html#fs.writeFile ]
>        2) It's internal flow (in one thread) is done like this (simplified):
>                a) Init V8
>                b) Add all nodejs C++ functions.
>                c) Load "node.js" part (some % of the code is just in javascript, which
> makes sense).
>                d) Enter the event loop   "while(notexit) { process_events(); }" <--- not
> exactly like this, but you can get the underlaying idea.
>
>
> With this architecture, if we wanted to work with nodeJS , we may need to launch a
> separate thread for the node.js interpreter (note, this is now V8 handles threaded host
> apps: http://markmail.org/message/uxqpvsbc7sw7qntl) 
>
> Then we may need to define clearly how to work in this situation:
>
> 0) The ideas of Dick, about making a big library, would be nice, becaouse we could use
> nodejs to work directly with kicad schematics, pcbs, etc, even without the interface
> itself, in this case I don't find any problem.
>
> 1) now, when the interface is around: when we needed to call some JS programmed function
> from kicad, we could simply pass the code to V8, and setup a callback that would unlock
> us back...., and lock ourselves waiting for the callback  [danger: we could lock forever
> if the js code never unlock us back, solution: timeout?]
>
> 2) more danger: The user could leave timer operations or any other operation, which
> could call us back later from the nodejs thread (when the timer expires). In this case,
> our BOARD or BOARD_ACTION objects would be called (even if we're doing any other thing).
> Some questions come to my mind:
>         a) How it's the event loop handled in kicad, it comes from the wx toolkit?
>         b) Are our operations thread safe?
>         c) Would a "big lock" on the accessed objects (like V8 implementation for this)
> make sense?, this could be a problem, if for example, our operations (at any moment)
> could have callbacks to javascript
>
> Other option could be using http://code.google.com/p/v8-juice/wiki/Plugins  (or
> equivalent), which doesn't seem to rely on an event loop. But, there we would loose all
> the nodejs power.
>
>
>
> There is also something quite important, we may decide which are our use cases, I see
> some clear ones, but many other could be interesting:
>
>
> 1) access to kicad objects from external scripting (kicad UI is off and we only access
> the PCB, schema, etc ... all the objects from our wonderful script), this could lead to
> many usefult operations:
>
>         A) automated/parametric creation of schematics and circuits (think of Scad, but
> in javascript): imagine a javscript that takes an image and builds a PCB/schematic full
> of interconnected leds (even RGB), that when you power, you get the image :-)
>
>         B) Automated test/extra DRC checks for our pcbs/schematics.
>  
>         C) Automated fixing.
>
>          D) options are endless....
>
> 2) Into the kicad UI, assigning hotkeys (or toolbar buttons) to our own scripts (like
> google sketchup would do with ruby scripts). The user could install new scripts
> (somebody could make them), and they would add new functionalities to kicad (with no
> need for direct integration in the main source code).
>
> 3) Scripts that are installed, could register callbacks for certain operations, examples:
>
>         A) when we finish drawing a track, the "Kicad.pcbnew.routedTrack"  callback
> could trigger, and then the user may have registered some algorithm to check the PCB for
> some kind of special DRC operations.
>
>          B) "pcbChanged", more generic.
>
>          C) I'm out of more ideas, but I have the feeling that many useful things could
> be done with that.
>
>
> As summary:
>
>     Scripting can be very powerful, JS/V8 makes a lot of sense with the help of swig (or
> even may be without... have to test...).
>
>
>      We must check how feasible is to put a nodeJS stack inside kicad, V8 would have a
> lot to do: calls from kicad to V8 and V8 back to kicad via the objects interface. But
> the nodejs implementation for evented I/O could be a problem if we don't handle it right.
>
>
>    That's all, thanks for reading, I wait for feedback :-)
>
>         
>       
>
>
>
>
>
>        
>
>
>        
>
>
>
>
>  
>
>
>
>
>
>
> 2012/2/22 Miguel Angel Ajo Pelayo <miguelangel@xxxxxxx <mailto:miguelangel@xxxxxxx>>
>
>     Ok, I cleaned up a little, and forked the project of swig-js
>
>     https://github.com/espardino/swig-js
>
>     It should require heavy testing, I will try to make it work with some of the Kicad
>     classes, and we will see 
>     how well or bad does it work :-)
>
>
>     PS: 
>
>     Espardino is the project where I'm using swig too:
>     https://github.com/espardino/xpress/tree/master/wrappers
>
>     I will reuse the work of this swig-js for both: Kicad, and Espardino.
>
>
>
>
>     2012/2/22 Miguel Angel Ajo Pelayo <miguelangel@xxxxxxx <mailto:miguelangel@xxxxxxx>>
>
>         I need to spend some time understanding your idea of design, right now I'm a
>         little bit lost,
>         but I started by the "simple" part, which was building the swig-js branch, and
>         it seems
>         that I managed to do it (it was only ready for WIN32) but I have a patch for
>         swig-js that
>         seems to work in linux (partially)...
>
>         I was able to generate the example for:
>
>         https://github.com/oliver----/swig-js/blob/master/src/Examples/jsv8/class/example.h
>
>         and it seems to link to standard libv8 from ubuntu, and run. (It compiled it's
>         own v8, but it was standard one, no patches).
>
>         I think I will try to cleanup the build, and then publishing it. 
>
>         The swig-js is built using CMake (I'm quite experienced on it) so I could need
>         some help ':-)
>
>
>
>
>         2012/2/22 Dick Hollenbeck <dick@xxxxxxxxxxx <mailto:dick@xxxxxxxxxxx>>
>
>             On 02/22/2012 09:11 AM, Miguel Angel Ajo Pelayo wrote:
>             > I think I could put some efforts on testing it, and see how well the "JS"
>             branch of swig
>             > does work.
>             >
>             > Do we have an idea of which interface we may want to give to the scripting
>             layers?,
>             > access to PCB objects, modules, tracks, etc?
>
>             To keep it simple, let me talk about PCBNEW only *for now* (duplicate the
>             discussion in
>             your mind for EESCHEMA).
>
>             class BOARD, and the needed significant objects contained within it.  I say
>             needed,
>             because I think there is some flexibility as to where you build this bridge,
>             either in the
>             class BOARD by accessor augmentation in C++ go give access to everything
>             there, or by
>             SWIGing out the contained classes.
>
>             We also talked about moving the non-UI member functions that are now in
>             PCB_BASE_FRAME,
>             PCB_EDIT_FRAME, into a class which is purely for significant actions but
>             with NO UI, like
>             plotting, annotating, etc.  The UI portion (meaning the preceding dialog
>             invocation) would
>             stay out of this class.  conceptually lets call this class BOARD_ACTIONS.
>              All the
>             functions in BOARD_ACTIONS take enough parameters to operation *without any
>             UI*, by
>             definition.
>
>
>
>             BOARD_ACTIONS could then be transported into two realms:
>
>             a) back into PCB_*_FRAME by multiple inheritance, so you are back to where
>             you started
>             from with respect to PCB_EDIT_FRAME.
>
>             b) into the top of a scripting entry point DLL/DSO.
>
>
>             Along with this, it might be useful to try and build a PCBNEW DLL/DSO that
>             has the public
>             BOARD_ACTIONS, and BOARD entry points exposed.
>             The actual PCBNEW program (where main() is), could be a thin layer with
>             linked with the UI
>             (mostly dialogs and wxFrames).
>
>             The plotting and drawing functions themselves might have to be yet a
>             separate DLL/DSO.
>             Since your BOARD_ACTIONS might actually have to plot or print.
>
>             We are finally getting in a position to be able to do this, reduction of
>             globals has been
>             an enabler in this effort.  There is some more work to do.  We still have a
>             number of
>             globals, and getting the separation into BOARD_ACTIONS would be some work.
>
>             This design I think deals with both the paradigms:
>
>             a) scripting calls C++ actions, and
>             b) C++ calls scripting, since these calls from C++ would be from the upper
>             main() area,
>             and you could still call BOARD_ACTIONS and BOARD from the script.
>
>             It just does not however deal with the letting the script control the UI.  I
>             think that is
>             significantly more work, and is probably best done in C++ forever.
>
>             I don't anticipate that this would take under two days to do.
>
>             :) read carefully now.
>
>             Dick
>
>
>
>             >
>             >
>             >
>             > 2012/2/22 Dick Hollenbeck <dick@xxxxxxxxxxx <mailto:dick@xxxxxxxxxxx>
>             <mailto:dick@xxxxxxxxxxx <mailto:dick@xxxxxxxxxxx>>>
>             >
>             >     On 02/17/2012 08:56 AM, Miguel Angel Ajo Pelayo wrote:
>             >     > Hi Dick,
>             >     >
>             >     >    Thanks for your feedback on the ideas, it's really appreciated, I
>             fully understand
>             >     > all of your points, as I'm on the same situation with some open
>             projects and my little
>             >     > company. I supposed that most of the ideas were already in your mind,
>             but if I didn't
>             >     > share them I wouldn't know now where do they converge with yours :-)
>             >     >
>             >     >     I really love the V8 engine, just played a little bit on embedded
>             systems, and it
>             >     > works incredibly fast (it does JIT of the code and you get all the
>             flexibility of
>             >     > javascript), and nodeJS gives all the functionality needed to access
>             the underlaying OS
>             >     > and network, although I'm not sure if it's internal design makes it
>             suitable to get
>             >     > integrated in other apps.
>             >     >
>             >     >      I'm also using SWIG to build interfaces to many languages, and
>             it works incredibly
>             >     > well. I must test the swig-js and see how mature is.
>             >
>             >     We will be interested in finding out what you learn.  JavaScript seems
>             to have a lot of
>             >     momentum now, and it needs to be looked at in detail.
>             >
>             >     With HTML5 giving it new life on the client, NodeJS giving it further
>             life on the
>             >     server,
>             >     its C++ like syntax, its standardization by a standards body, it has a
>             lot going for it.
>             >
>             >     And for any one not already knowing a language, one is comforted by the
>             fact that
>             >     learning
>             >     it might serve a real purpose in other areas, (like when you go design
>             your next
>             >     webpage.)
>             >
>             >     Dick
>             >
>             >
>             >     >
>             >     >     Personally, I'd like to give "some blood" on the project as soon
>             as I can and get
>             >     > familiar with kicad's code/design :-)
>             >     >
>             >     >     Cheers,
>             >     > Mike
>             >
>             >
>             >     > Two weeks ago I looked into adding support for V8.  V8 is google's
>             javascript
>             >     engine and
>             >     >
>             >     >     they use it under the chrome browser.  (You may already have a
>             copy on your disk
>             >     >     now.)  V8
>             >     >     is very fast.  There is another project that sits on top of V8
>             called node.js.
>             >     >      node.js is
>             >     >     great at writing asynchronous webservers, and has been shown to
>             actually be
>             >     faster than
>             >     >     apache in some circumstances.
>             >     >
>             >     >     My measurement of doing it manually said that it is a lot of
>             work.  However
>             >     since then I
>             >     >     see some folks are trying to add the binding generation
>             functionality to SWIG.
>             >      SWIG can
>             >     >     generate bindings for a number of languages, in theory.
>             >     >
>             >     >
>             >     >     http://comments.gmane.org/gmane.comp.programming.swig.devel/20373
>             >     >     https://github.com/oliver----/swig-js
>             >     >
>             >     >     This is an ancillary project, and not integrated into core SWIG
>             from what I can
>             >     tell.
>             >     >
>             >     >     node.js has gathered so much momentum that it is actually
>             changing the choice of
>             >     >     languages
>             >     >     used on the server side for many web applications.  There are two
>             reasons for this
>             >     >     that I
>             >     >     can see:  a) node.js on top of V8 is fast.  b) being able to use
>             javascript
>             >     both on the
>             >     >     client side and server side is attractive to many developers.
>             >     >
>             >
>             >
>             >
>             >
>             >
>             > --
>             >
>             > Miguel Angel Ajo Pelayo
>             > http://www.nbee.es
>             > +34 636 52 25 69 <tel:%2B34%20636%2052%2025%2069>
>             > skype: ajoajoajo
>
>
>
>
>         -- 
>
>         Miguel Angel Ajo Pelayo
>         http://www.nbee.es
>         +34 636 52 25 69 <tel:%2B34%20636%2052%2025%2069>
>         skype: ajoajoajo
>
>
>
>
>     -- 
>
>     Miguel Angel Ajo Pelayo
>     http://www.nbee.es
>     +34 636 52 25 69 <tel:%2B34%20636%2052%2025%2069>
>     skype: ajoajoajo
>
>
>
>
> -- 
>
> Miguel Angel Ajo Pelayo
> http://www.nbee.es
> +34 636 52 25 69
> skype: ajoajoajo



Follow ups

References