fenics team mailing list archive

[Johan Hake <johan.hake@xxxxxxxxx>]

On Tuesday September 7 2010 14:05:07 Kristian Ølgaard wrote:
> On 7 September 2010 19:23, Anders Logg <logg@xxxxxxxxx> wrote:
> > On Tue, Sep 07, 2010 at 10:19:15AM -0700, Johan Hake wrote:
> >> On Tuesday September 7 2010 10:08:34 Kristian Ølgaard wrote:
> >> > On 7 September 2010 18:51, Johan Hake <johan.hake@xxxxxxxxx> wrote:
> >> > > On Tuesday September 7 2010 09:24:40 Kristian Ølgaard wrote:
> >> > >> On 7 September 2010 17:59, Johan Hake <johan.hake@xxxxxxxxx> wrote:
> >> > >> > [snip]
> >> > >> > 
> >> > >> >> > But how do we extract the different arguments? I suppose this
> >> > >> >> > is collected by Doxygen, and we just need to parse these and
> >> > >> >> > output them in a correct way?
> >> > >> >> 
> >> > >> >> I don't think we need to parse the arguments and output them. We
> >> > >> >> just get the function name and if we have more than one set of
> >> > >> >> arguments i.e., a different signature we know that we have an
> >> > >> >> overloaded method and how to handle it.
> >> > >> > 
> >> > >> > And I guess the almighty generate_cpp_documentation.py script are
> >> > >> > able to extract the argument information?
> >> > >> 
> >> > >> No, but we should be able to figure this out from the signature
> >> > >> (split ',' in '()').
> >> > > 
> >> > > Ok! Anders mentioned this too.
> >> > > 
> >> > >> >> The arguments should be described in the *Arguments* section of
> >> > >> >> the individual docstring with links to classes formatted like
> >> > >> >> _MeshEntity_, which we will substitute with
> >> > >> >> :py:class:`MeshEntity` or
> >> > >> >> 
> >> > >> >> :cpp:class:`MeshEntity` depending on which interface we
> >> > >> >> :document.
> >> > >> > 
> >> > >> > Ok, but we only want that once for each method in python, even if
> >> > >> > it is overloaded?
> >> > >> 
> >> > >> No, I think we need to document the argument list for every
> >> > >> overloaded version like it is done in docstrings.dolfin.cpp.Mesh.
> >> > > 
> >> > > Agree, I think I misunderstood you.
> >> > > 
> >> > >> >> Although I just realized that standard C++ stuff like double*
> >> > >> >> which end up as numpy.array etc. should probably be handled.
> >> > >> > 
> >> > >> > Yes this part I am a little worried about... But maybe a god
> >> > >> > handwritten lookup table will do the trick? At least for 99% of
> >> > >> > the cases ;)
> >> > >> 
> >> > >> I share your concern, but if, as you suggest, we'll be able to get
> >> > >> God to hand write our documentation I think we should be OK. :)
> >> > > 
> >> > > Good to have God on our side!
> >> > > 
> >> > >> (a lookup table would be my best bet at the moment)
> >> > > 
> >> > > Ok!
> >> > > 
> >> > >> >> On a related note:
> >> > >> >> int some_func()
> >> > >> >> and
> >> > >> >> const int some_func() const
> >> > >> >> are different in C++, but in Python we don't have const right?
> >> > >> >> This will simplify the documentation a lot.
> >> > >> > 
> >> > >> > Yes, we tend to %ignore all const versions of different methods.
> >> > >> > 
> >> > >> > [snap]
> >> > >> > 
> >> > >> >> >> >  * Extended methods needs to be handled in one of three
> >> > >> >> >> > ways: 1) Write the docstring directly into the foo_post.i
> >> > >> >> >> > file
> >> > >> >> 
> >> > >> >> I like this option, if this is where we have the code for a
> >> > >> >> function, then this is where the docstring should be as it
> >> > >> >> increases the probability of the docstring being up to date.
> >> > >> > 
> >> > >> > Ok, lets settle on this one. We also need to make sure that all
> >> > >> > %extended methods in the C++ layer gets a proper docstring.
> >> > >> > However I am not really sure how this can be done :P
> >> > >> 
> >> > >> I'm not sure I follow this, which %extended methods do you mean?
> >> > > 
> >> > > There are two ways to extend a class.
> >> > > 
> >> > >  1) The C++ layer
> >> > >  2) The Python layer
> >> > > 
> >> > > often we use 1) to create a protected helper method which is called
> >> > > using an extended method in the Python layer, 2). The latter can be
> >> > > properly documented directly.
> >> > > 
> >> > > But some cases excists where we just extend the C++ layer, see for
> >> > > example the IMPLEMENT_VARIABLE_INTERFACE macro in
> >> > > shared_ptr_classes.i. These methods gets no docstrings and I am not
> >> > > sure it is possible to add them later.
> >> > 
> >> > OK, docstrings for 2) should go in the code as we agreed, and I guess
> >> > 1) will fall under the 1% category which we may/may not be able to
> >> > handle in a clever way later.
> >> 
> >> Ok.
> >> 
> >> > >> > [snup]
> >> > >> > 
> >> > >> >> > Why do we need to assign to these methods? They already get
> >> > >> >> > their docstrings from the docstrings.i file. However if we
> >> > >> >> > want to get rid of the new_instancemethod assignment above,
> >> > >> >> > we can just remove the
> >> > >> >> 
> >> > >> >> Some history.
> >> > >> >> Initially, we wanted to have all docstrings separated from the
> >> > >> >> DOLFIN code and collected in the fenics-doc module. The easiest
> >> > >> >> way to get the >>> help(dolfin) docstring correct is to assign
> >> > >> >> to __doc__ dynamically.
> >> > >> >> If we could do this we wouldn't even need the docstrings.i file
> >> > >> >> and things would be simple.
> >> > >> >> However, we discovered that this was not possible, and because
> >> > >> >> of that we still need to generate the docstrings.i file.
> >> > >> >> Then, still assuming we wanted to separate docs from code and
> >> > >> >> keeping docstrings in fenics-doc, I thought it would be easier
> >> > >> >> to generate the docstrings.i file from the handwritten
> >> > >> >> docstrings module in fenics-doc.
> >> > >> >> Some methods don't get their docstrings from the docstrings.i
> >> > >> >> file though, so we still need to assign to __doc__ which is the
> >> > >> >> easiest thing to do.
> >> > >> >> Just recently we decided to extract the docstrings from the C++
> >> > >> >> implementation thus moving the docs back into DOLFIN. This makes
> >> > >> >> the docstrings module almost superfluous with the only
> >> > >> >> practical usage is to have documentation for the extended
> >> > >> >> methods defined in the _post.i files but if we put the
> >> > >> >> docstrings directly in the _post.i files we no longer need it.
> >> > >> > 
> >> > >> > Ok, then I do not see any reason for a separate docstring module,
> >> > >> > makes life a lite bit easier...
> >> > >> 
> >> > >> Agree.
> >> > > 
> >> > > Ok.
> >> > > 
> >> > >> > [snep]
> >> > >> > 
> >> > >> >> > I am confused. Do you suggest that we just document the
> >> > >> >> > extended Python layer directly in the python module as it is
> >> > >> >> > today? Why should we then dumpt the docstrings in a separate
> >> > >> >> > docstring module? So autodoc can have something to shew on?
> >> > >> >> > Couldn't autodoc just shew on the dolfin module directly?
> >> > >> >> 
> >> > >> >> I'm confused too. :) I guess my head has not been properly reset
> >> > >> >> between the changes in documentation strategies.
> >> > >> >> The Sphinx autodoc can only handle one dolfin module, so we need
> >> > >> >> to either import the 'real' one or the docstrings dolfin
> >> > >> >> module. If we can completely remove the need for the docstrings
> >> > >> >> module, then we should of course include the 'real' one.
> >> > >> > 
> >> > >> > Ok!
> >> > >> > 
> >> > >> >> >> Then programmer's writing the Python
> >> > >> >> >> layer just need to document while they're coding, where they
> >> > >> >> >> are coding just like they do (or should anyways) for the C++
> >> > >> >> >> part.
> >> > >> >> > 
> >> > >> >> > Still confused why we need a certain docstring module.
> >> > >> >> 
> >> > >> >> Maybe we don't.
> >> > >> >> 
> >> > >> >> >> >  2) for the extended Python layer in the cpp.py
> >> > >> >> >> > 
> >> > >> >> >> > For the rest, and this will be the main part, we rely on
> >> > >> >> >> > parsed docstrings from the headers.
> >> > >> >> >> > 
> >> > >> >> >> > The python programmers reference will then be generated
> >> > >> >> >> > based on the actual dolfin module using sphinx and
> >> > >> >> >> > autodoc.
> >> > >> >> >> 
> >> > >> >> >> We could/should probably use either the dolfin module or the
> >> > >> >> >> generated docstring module to generate the relevant reST
> >> > >> >> >> files. Although we might need to run some cross-checks with
> >> > >> >> >> the Doxygen xml to get the correct file names where the
> >> > >> >> >> classes are defined in DOLFIN such that we retain the
> >> > >> >> >> original DOLFIN source tree structure. Otherwise all our
> >> > >> >> >> documentation will end up in cpp.rst which I would hate to
> >> > >> >> >> navigate through as a user.
> >> > >> >> > 
> >> > >> >> > This one got to technical for me. Do you say that there is no
> >> > >> >> > way to split the documentation into smaller parts without
> >> > >> >> > relying on the c++ module/file structure?
> >> > >> >> 
> >> > >> >> But how would you split it?
> >> > >> > 
> >> > >> > I do not know. But then I do not know what the generation step
> >> > >> > can take as different inputs.
> >> > >> 
> >> > >> The write_python_documentation step should probably take the dolfin
> >> > >> module and the intermediate representation.
> >> > > 
> >> > > What is the intermediate representation?
> >> > 
> >> > It is whatever output we get from the extract_documentation script
> >> > which we'll add to the dolfin module. How it will look depends a bit
> >> > on what we need in the write_cpp_documentation and
> >> > write_python_documentation functions in fenics-doc.
> >> 
> >> Ok.
> >> 
> >> > >> >> It makes sense to keep the classes Mesh
> >> > >> >> and MeshEntity in the mesh/ part of the documentation.
> >> > >> >> Unfortunately, Swig doesn't add info to the classes in the
> >> > >> >> cpp.py module about where they were originally defined. This is
> >> > >> >> why we need to pair it with info from the xml output.
> >> > >> > 
> >> > >> > Ok, but say we keep all documentation in one module. If you are
> >> > >> > able to pair the different classes or functions with a module
> >> > >> > name, or file name you are able to create documentation which is
> >> > >> > structured after this hierarchy?
> >> > >> 
> >> > >> We need to figure out something, having everything in the cpp.py
> >> > >> module would create one big mess and it makes sense to follow the
> >> > >> DOLFIN C++ structure even for the Python interface.
> >> > > 
> >> > > Ok, but we do not have everything in a big cpp file. Types get
> >> > > imported into the Dolfin namespace in __init__ mostly from cpp.py.
> >> > 
> >> > My point is, there's no telling where the cpp.Mesh class was
> >> > originally defined. Everything from la to mesh to fem is dumped in the
> >> > cpp.py module.
> >> 
> >> Ok, but don't you just need a way to associate the classes to different
> >> modules? I thought this was what you used the doxygen output to. If we
> >> instead use the module representation we should be able to do this
> >> association directly with just the dolfin tree as the assosiated types
> >> should reside in:
> >> 
> >>   dolfin.submodule.__dict__
> >> 
> >> > > Would it help to add the cpp imports to submodules instead of the
> >> > > main __init__ file? We already have the submodules:
> >> > > 
> >> > >  mesh, common, compilemodules, fem, adaptivity and function
> >> > > 
> >> > > We could add:
> >> > > 
> >> > >  io, log, nls, ode, parameter, ale
> >> > > 
> >> > > and drag in stuff from cpp.py in these modules instead. In this way
> >> > > we structure the main __init__ file better, and you might be able
> >> > > to structure the pythons reference manual better?
> >> > 
> >> > I'm quite sure I tried something like this, and the problem was that
> >> > even if you do:
> >> > 
> >> > from dolfin.cpp import Mesh
> >> > 
> >> > in dolfin/mesh/__init__.py
> >> > 
> >> > the Mesh class will still point to dolfin.cpp when you inspect it -->
> >> > difficult to create the proper structure for the *.rst files.
> >> 
> >> Would something that I sceted above work?
> >> 
> >> > I'll need to double check though that this is really the case.
> >> > And even if we can use the approach you outline above it means that we
> >> > have more stuff we need to manually maintain.
> >> 
> >> Sure, but now everything is throwed into dolfin/__init__.py which is
> >> really a mess now.
> >> 
> >> Johan
> > 
> > This discussion is getting long and complex. I'm tempted to just let
> > the two of you sort it out between you and then ask for a 10-line
> > summary.
> > 
> > And then, of course, suggest a completely different solution. :-)
> 
> That's what we're afraid of. In fact this is the reason we're still
> discussing, to avoid implementing something which you'll tell us to
> redo :)

Argee

> I think we're quite close to a converged solution were the last
> details concerns the structure of the Python documentation i.e., how
> to retrieve the original DOLFIN library structure from the cpp.py
> module.
> 
> Just a crazy thought, is it possible to tell Swig to split the modules
> for us? 

He, he, if we would have stuck to SCons this would have been an easy shift, 
but now, I am not so sure...

However, I have structured the SWIG interface files, with the foo_post.i and 
foo_pre.i files with this in mind! So it should be possible... But we need to 
sort out internal dependencies of such modules first, and that can turn out to 
be difficult, as there might be some circular dependencies. But if we do not 
modularize libdolfin it might be possible.

> Such that we get
> dolfin/mesh --> site-packages/dolfin/mesh.
> This will also speed up the compilation of the Python module if more
> than one process is used.

But we eventually need to import everything in the dolfin namespace as we do 
today. 

Johan

> Kristian
> 
> > --
> > Anders
> > 
> > _______________________________________________
> > Mailing list: https://launchpad.net/~fenics
> > Post to     : fenics@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~fenics
> > More help   : https://help.launchpad.net/ListHelp