fenics team mailing list archive

[Johan Hake <johan.hake@xxxxxxxxx>]

On Tuesday September 7 2010 10:08:34 Kristian Ølgaard wrote:
> On 7 September 2010 18:51, Johan Hake <johan.hake@xxxxxxxxx> wrote:
> > On Tuesday September 7 2010 09:24:40 Kristian Ølgaard wrote:
> >> On 7 September 2010 17:59, Johan Hake <johan.hake@xxxxxxxxx> wrote:
> >> > [snip]
> >> > 
> >> >> > But how do we extract the different arguments? I suppose this is
> >> >> > collected by Doxygen, and we just need to parse these and output
> >> >> > them in a correct way?
> >> >> 
> >> >> I don't think we need to parse the arguments and output them. We just
> >> >> get the function name and if we have more than one set of arguments
> >> >> i.e., a different signature we know that we have an overloaded method
> >> >> and how to handle it.
> >> > 
> >> > And I guess the almighty generate_cpp_documentation.py script are able
> >> > to extract the argument information?
> >> 
> >> No, but we should be able to figure this out from the signature (split
> >> ',' in '()').
> > 
> > Ok! Anders mentioned this too.
> > 
> >> >> The arguments should be described in the *Arguments* section of the
> >> >> individual docstring with links to classes formatted like
> >> >> _MeshEntity_, which we will substitute with :py:class:`MeshEntity` or
> >> >> 
> >> >> :cpp:class:`MeshEntity` depending on which interface we document.
> >> > 
> >> > Ok, but we only want that once for each method in python, even if it
> >> > is overloaded?
> >> 
> >> No, I think we need to document the argument list for every overloaded
> >> version like it is done in docstrings.dolfin.cpp.Mesh.
> > 
> > Agree, I think I misunderstood you.
> > 
> >> >> Although I just realized that standard C++ stuff like double* which
> >> >> end up as numpy.array etc. should probably be handled.
> >> > 
> >> > Yes this part I am a little worried about... But maybe a god
> >> > handwritten lookup table will do the trick? At least for 99% of the
> >> > cases ;)
> >> 
> >> I share your concern, but if, as you suggest, we'll be able to get God
> >> to hand write our documentation I think we should be OK. :)
> > 
> > Good to have God on our side!
> > 
> >> (a lookup table would be my best bet at the moment)
> > 
> > Ok!
> > 
> >> >> On a related note:
> >> >> int some_func()
> >> >> and
> >> >> const int some_func() const
> >> >> are different in C++, but in Python we don't have const right?
> >> >> This will simplify the documentation a lot.
> >> > 
> >> > Yes, we tend to %ignore all const versions of different methods.
> >> > 
> >> > [snap]
> >> > 
> >> >> >> >  * Extended methods needs to be handled in one of three ways:
> >> >> >> >    1) Write the docstring directly into the foo_post.i file
> >> >> 
> >> >> I like this option, if this is where we have the code for a function,
> >> >> then this is where the docstring should be as it increases the
> >> >> probability of the docstring being up to date.
> >> > 
> >> > Ok, lets settle on this one. We also need to make sure that all
> >> > %extended methods in the C++ layer gets a proper docstring. However I
> >> > am not really sure how this can be done :P
> >> 
> >> I'm not sure I follow this, which %extended methods do you mean?
> > 
> > There are two ways to extend a class.
> > 
> >  1) The C++ layer
> >  2) The Python layer
> > 
> > often we use 1) to create a protected helper method which is called using
> > an extended method in the Python layer, 2). The latter can be properly
> > documented directly.
> > 
> > But some cases excists where we just extend the C++ layer, see for
> > example the IMPLEMENT_VARIABLE_INTERFACE macro in shared_ptr_classes.i.
> > These methods gets no docstrings and I am not sure it is possible to add
> > them later.
> 
> OK, docstrings for 2) should go in the code as we agreed, and I guess
> 1) will fall under the 1% category which we may/may not be able to
> handle in a clever way later.

Ok.

> >> > [snup]
> >> > 
> >> >> > Why do we need to assign to these methods? They already get their
> >> >> > docstrings from the docstrings.i file. However if we want to get
> >> >> > rid of the new_instancemethod assignment above, we can just remove
> >> >> > the
> >> >> 
> >> >> Some history.
> >> >> Initially, we wanted to have all docstrings separated from the DOLFIN
> >> >> code and collected in the fenics-doc module. The easiest way to get
> >> >> the >>> help(dolfin) docstring correct is to assign to __doc__
> >> >> dynamically.
> >> >> If we could do this we wouldn't even need the docstrings.i file and
> >> >> things would be simple.
> >> >> However, we discovered that this was not possible, and because of
> >> >> that we still need to generate the docstrings.i file.
> >> >> Then, still assuming we wanted to separate docs from code and keeping
> >> >> docstrings in fenics-doc, I thought it would be easier to generate
> >> >> the docstrings.i file from the handwritten docstrings module in
> >> >> fenics-doc.
> >> >> Some methods don't get their docstrings from the docstrings.i file
> >> >> though, so we still need to assign to __doc__ which is the easiest
> >> >> thing to do.
> >> >> Just recently we decided to extract the docstrings from the C++
> >> >> implementation thus moving the docs back into DOLFIN. This makes the
> >> >> docstrings module almost superfluous with the only practical usage is
> >> >> to have documentation for the extended methods defined in the _post.i
> >> >> files but if we put the docstrings directly in the _post.i files we
> >> >> no longer need it.
> >> > 
> >> > Ok, then I do not see any reason for a separate docstring module,
> >> > makes life a lite bit easier...
> >> 
> >> Agree.
> > 
> > Ok.
> > 
> >> > [snep]
> >> > 
> >> >> > I am confused. Do you suggest that we just document the extended
> >> >> > Python layer directly in the python module as it is today? Why
> >> >> > should we then dumpt the docstrings in a separate docstring
> >> >> > module? So autodoc can have something to shew on? Couldn't autodoc
> >> >> > just shew on the dolfin module directly?
> >> >> 
> >> >> I'm confused too. :) I guess my head has not been properly reset
> >> >> between the changes in documentation strategies.
> >> >> The Sphinx autodoc can only handle one dolfin module, so we need to
> >> >> either import the 'real' one or the docstrings dolfin module.
> >> >> If we can completely remove the need for the docstrings module, then
> >> >> we should of course include the 'real' one.
> >> > 
> >> > Ok!
> >> > 
> >> >> >> Then programmer's writing the Python
> >> >> >> layer just need to document while they're coding, where they are
> >> >> >> coding just like they do (or should anyways) for the C++ part.
> >> >> > 
> >> >> > Still confused why we need a certain docstring module.
> >> >> 
> >> >> Maybe we don't.
> >> >> 
> >> >> >> >  2) for the extended Python layer in the cpp.py
> >> >> >> > 
> >> >> >> > For the rest, and this will be the main part, we rely on parsed
> >> >> >> > docstrings from the headers.
> >> >> >> > 
> >> >> >> > The python programmers reference will then be generated based on
> >> >> >> > the actual dolfin module using sphinx and autodoc.
> >> >> >> 
> >> >> >> We could/should probably use either the dolfin module or the
> >> >> >> generated docstring module to generate the relevant reST files.
> >> >> >> Although we might need to run some cross-checks with the Doxygen
> >> >> >> xml to get the correct file names where the classes are defined
> >> >> >> in DOLFIN such that we retain the original DOLFIN source tree
> >> >> >> structure. Otherwise all our documentation will end up in cpp.rst
> >> >> >> which I would hate to navigate through as a user.
> >> >> > 
> >> >> > This one got to technical for me. Do you say that there is no way
> >> >> > to split the documentation into smaller parts without relying on
> >> >> > the c++ module/file structure?
> >> >> 
> >> >> But how would you split it?
> >> > 
> >> > I do not know. But then I do not know what the generation step can
> >> > take as different inputs.
> >> 
> >> The write_python_documentation step should probably take the dolfin
> >> module and the intermediate representation.
> > 
> > What is the intermediate representation?
> 
> It is whatever output we get from the extract_documentation script
> which we'll add to the dolfin module. How it will look depends a bit
> on what we need in the write_cpp_documentation and
> write_python_documentation functions in fenics-doc.

Ok.

> >> >> It makes sense to keep the classes Mesh
> >> >> and MeshEntity in the mesh/ part of the documentation. Unfortunately,
> >> >> Swig doesn't add info to the classes in the cpp.py module about where
> >> >> they were originally defined. This is why we need to pair it with
> >> >> info from the xml output.
> >> > 
> >> > Ok, but say we keep all documentation in one module. If you are able
> >> > to pair the different classes or functions with a module name, or
> >> > file name you are able to create documentation which is structured
> >> > after this hierarchy?
> >> 
> >> We need to figure out something, having everything in the cpp.py
> >> module would create one big mess and it makes sense to follow the
> >> DOLFIN C++ structure even for the Python interface.
> > 
> > Ok, but we do not have everything in a big cpp file. Types get imported
> > into the Dolfin namespace in __init__ mostly from cpp.py.
> 
> My point is, there's no telling where the cpp.Mesh class was
> originally defined. Everything from la to mesh to fem is dumped in the
> cpp.py module.

Ok, but don't you just need a way to associate the classes to different 
modules? I thought this was what you used the doxygen output to. If we instead 
use the module representation we should be able to do this association 
directly with just the dolfin tree as the assosiated types should reside in:

  dolfin.submodule.__dict__

> > Would it help to add the cpp imports to submodules instead of the main
> > __init__ file? We already have the submodules:
> > 
> >  mesh, common, compilemodules, fem, adaptivity and function
> > 
> > We could add:
> > 
> >  io, log, nls, ode, parameter, ale
> > 
> > and drag in stuff from cpp.py in these modules instead. In this way we
> > structure the main __init__ file better, and you might be able to
> > structure the pythons reference manual better?
> 
> I'm quite sure I tried something like this, and the problem was that
> even if you do:
> 
> from dolfin.cpp import Mesh
> 
> in dolfin/mesh/__init__.py
> 
> the Mesh class will still point to dolfin.cpp when you inspect it -->
> difficult to create the proper structure for the *.rst files.

Would something that I sceted above work?

> I'll need to double check though that this is really the case.
> And even if we can use the approach you outline above it means that we
> have more stuff we need to manually maintain.

Sure, but now everything is throwed into dolfin/__init__.py which is really a 
mess now. 

Johan