fenics team mailing list archive

[Kristian Ølgaard <k.b.oelgaard@xxxxxxxxx>]

On 7 September 2010 19:19, Johan Hake <johan.hake@xxxxxxxxx> wrote:
> On Tuesday September 7 2010 10:08:34 Kristian Ølgaard wrote:
>> On 7 September 2010 18:51, Johan Hake <johan.hake@xxxxxxxxx> wrote:
>> > On Tuesday September 7 2010 09:24:40 Kristian Ølgaard wrote:
>> >> On 7 September 2010 17:59, Johan Hake <johan.hake@xxxxxxxxx> wrote:
>> >> > [snip]
>> >> >
>> >> >> > But how do we extract the different arguments? I suppose this is
>> >> >> > collected by Doxygen, and we just need to parse these and output
>> >> >> > them in a correct way?
>> >> >>
>> >> >> I don't think we need to parse the arguments and output them. We just
>> >> >> get the function name and if we have more than one set of arguments
>> >> >> i.e., a different signature we know that we have an overloaded method
>> >> >> and how to handle it.
>> >> >
>> >> > And I guess the almighty generate_cpp_documentation.py script are able
>> >> > to extract the argument information?
>> >>
>> >> No, but we should be able to figure this out from the signature (split
>> >> ',' in '()').
>> >
>> > Ok! Anders mentioned this too.
>> >
>> >> >> The arguments should be described in the *Arguments* section of the
>> >> >> individual docstring with links to classes formatted like
>> >> >> _MeshEntity_, which we will substitute with :py:class:`MeshEntity` or
>> >> >>
>> >> >> :cpp:class:`MeshEntity` depending on which interface we document.
>> >> >
>> >> > Ok, but we only want that once for each method in python, even if it
>> >> > is overloaded?
>> >>
>> >> No, I think we need to document the argument list for every overloaded
>> >> version like it is done in docstrings.dolfin.cpp.Mesh.
>> >
>> > Agree, I think I misunderstood you.
>> >
>> >> >> Although I just realized that standard C++ stuff like double* which
>> >> >> end up as numpy.array etc. should probably be handled.
>> >> >
>> >> > Yes this part I am a little worried about... But maybe a god
>> >> > handwritten lookup table will do the trick? At least for 99% of the
>> >> > cases ;)
>> >>
>> >> I share your concern, but if, as you suggest, we'll be able to get God
>> >> to hand write our documentation I think we should be OK. :)
>> >
>> > Good to have God on our side!
>> >
>> >> (a lookup table would be my best bet at the moment)
>> >
>> > Ok!
>> >
>> >> >> On a related note:
>> >> >> int some_func()
>> >> >> and
>> >> >> const int some_func() const
>> >> >> are different in C++, but in Python we don't have const right?
>> >> >> This will simplify the documentation a lot.
>> >> >
>> >> > Yes, we tend to %ignore all const versions of different methods.
>> >> >
>> >> > [snap]
>> >> >
>> >> >> >> >  * Extended methods needs to be handled in one of three ways:
>> >> >> >> >    1) Write the docstring directly into the foo_post.i file
>> >> >>
>> >> >> I like this option, if this is where we have the code for a function,
>> >> >> then this is where the docstring should be as it increases the
>> >> >> probability of the docstring being up to date.
>> >> >
>> >> > Ok, lets settle on this one. We also need to make sure that all
>> >> > %extended methods in the C++ layer gets a proper docstring. However I
>> >> > am not really sure how this can be done :P
>> >>
>> >> I'm not sure I follow this, which %extended methods do you mean?
>> >
>> > There are two ways to extend a class.
>> >
>> >  1) The C++ layer
>> >  2) The Python layer
>> >
>> > often we use 1) to create a protected helper method which is called using
>> > an extended method in the Python layer, 2). The latter can be properly
>> > documented directly.
>> >
>> > But some cases excists where we just extend the C++ layer, see for
>> > example the IMPLEMENT_VARIABLE_INTERFACE macro in shared_ptr_classes.i.
>> > These methods gets no docstrings and I am not sure it is possible to add
>> > them later.
>>
>> OK, docstrings for 2) should go in the code as we agreed, and I guess
>> 1) will fall under the 1% category which we may/may not be able to
>> handle in a clever way later.
>
> Ok.
>
>> >> > [snup]
>> >> >
>> >> >> > Why do we need to assign to these methods? They already get their
>> >> >> > docstrings from the docstrings.i file. However if we want to get
>> >> >> > rid of the new_instancemethod assignment above, we can just remove
>> >> >> > the
>> >> >>
>> >> >> Some history.
>> >> >> Initially, we wanted to have all docstrings separated from the DOLFIN
>> >> >> code and collected in the fenics-doc module. The easiest way to get
>> >> >> the >>> help(dolfin) docstring correct is to assign to __doc__
>> >> >> dynamically.
>> >> >> If we could do this we wouldn't even need the docstrings.i file and
>> >> >> things would be simple.
>> >> >> However, we discovered that this was not possible, and because of
>> >> >> that we still need to generate the docstrings.i file.
>> >> >> Then, still assuming we wanted to separate docs from code and keeping
>> >> >> docstrings in fenics-doc, I thought it would be easier to generate
>> >> >> the docstrings.i file from the handwritten docstrings module in
>> >> >> fenics-doc.
>> >> >> Some methods don't get their docstrings from the docstrings.i file
>> >> >> though, so we still need to assign to __doc__ which is the easiest
>> >> >> thing to do.
>> >> >> Just recently we decided to extract the docstrings from the C++
>> >> >> implementation thus moving the docs back into DOLFIN. This makes the
>> >> >> docstrings module almost superfluous with the only practical usage is
>> >> >> to have documentation for the extended methods defined in the _post.i
>> >> >> files but if we put the docstrings directly in the _post.i files we
>> >> >> no longer need it.
>> >> >
>> >> > Ok, then I do not see any reason for a separate docstring module,
>> >> > makes life a lite bit easier...
>> >>
>> >> Agree.
>> >
>> > Ok.
>> >
>> >> > [snep]
>> >> >
>> >> >> > I am confused. Do you suggest that we just document the extended
>> >> >> > Python layer directly in the python module as it is today? Why
>> >> >> > should we then dumpt the docstrings in a separate docstring
>> >> >> > module? So autodoc can have something to shew on? Couldn't autodoc
>> >> >> > just shew on the dolfin module directly?
>> >> >>
>> >> >> I'm confused too. :) I guess my head has not been properly reset
>> >> >> between the changes in documentation strategies.
>> >> >> The Sphinx autodoc can only handle one dolfin module, so we need to
>> >> >> either import the 'real' one or the docstrings dolfin module.
>> >> >> If we can completely remove the need for the docstrings module, then
>> >> >> we should of course include the 'real' one.
>> >> >
>> >> > Ok!
>> >> >
>> >> >> >> Then programmer's writing the Python
>> >> >> >> layer just need to document while they're coding, where they are
>> >> >> >> coding just like they do (or should anyways) for the C++ part.
>> >> >> >
>> >> >> > Still confused why we need a certain docstring module.
>> >> >>
>> >> >> Maybe we don't.
>> >> >>
>> >> >> >> >  2) for the extended Python layer in the cpp.py
>> >> >> >> >
>> >> >> >> > For the rest, and this will be the main part, we rely on parsed
>> >> >> >> > docstrings from the headers.
>> >> >> >> >
>> >> >> >> > The python programmers reference will then be generated based on
>> >> >> >> > the actual dolfin module using sphinx and autodoc.
>> >> >> >>
>> >> >> >> We could/should probably use either the dolfin module or the
>> >> >> >> generated docstring module to generate the relevant reST files.
>> >> >> >> Although we might need to run some cross-checks with the Doxygen
>> >> >> >> xml to get the correct file names where the classes are defined
>> >> >> >> in DOLFIN such that we retain the original DOLFIN source tree
>> >> >> >> structure. Otherwise all our documentation will end up in cpp.rst
>> >> >> >> which I would hate to navigate through as a user.
>> >> >> >
>> >> >> > This one got to technical for me. Do you say that there is no way
>> >> >> > to split the documentation into smaller parts without relying on
>> >> >> > the c++ module/file structure?
>> >> >>
>> >> >> But how would you split it?
>> >> >
>> >> > I do not know. But then I do not know what the generation step can
>> >> > take as different inputs.
>> >>
>> >> The write_python_documentation step should probably take the dolfin
>> >> module and the intermediate representation.
>> >
>> > What is the intermediate representation?
>>
>> It is whatever output we get from the extract_documentation script
>> which we'll add to the dolfin module. How it will look depends a bit
>> on what we need in the write_cpp_documentation and
>> write_python_documentation functions in fenics-doc.
>
> Ok.
>
>> >> >> It makes sense to keep the classes Mesh
>> >> >> and MeshEntity in the mesh/ part of the documentation. Unfortunately,
>> >> >> Swig doesn't add info to the classes in the cpp.py module about where
>> >> >> they were originally defined. This is why we need to pair it with
>> >> >> info from the xml output.
>> >> >
>> >> > Ok, but say we keep all documentation in one module. If you are able
>> >> > to pair the different classes or functions with a module name, or
>> >> > file name you are able to create documentation which is structured
>> >> > after this hierarchy?
>> >>
>> >> We need to figure out something, having everything in the cpp.py
>> >> module would create one big mess and it makes sense to follow the
>> >> DOLFIN C++ structure even for the Python interface.
>> >
>> > Ok, but we do not have everything in a big cpp file. Types get imported
>> > into the Dolfin namespace in __init__ mostly from cpp.py.
>>
>> My point is, there's no telling where the cpp.Mesh class was
>> originally defined. Everything from la to mesh to fem is dumped in the
>> cpp.py module.
>
> Ok, but don't you just need a way to associate the classes to different
> modules? I thought this was what you used the doxygen output to.

Yes, this is the plan. I thought I could use doxygen to tell me that
the class Mesh is defined in dolfin/mesh/Mesh.h, and then associate
the dolfin.cpp.Mesh class with the dolfin/mesh directory to generate
prog-ref/python/mesh/Mesh.rst.
But come to think of it, the potential renaming will screw this up arrrrgh.

If we instead
> use the module representation we should be able to do this association
> directly with just the dolfin tree as the assosiated types should reside in:
>
>  dolfin.submodule.__dict__

This has some potential, we just need to be careful what the
submodules import. The big drawback as I see it is that if someone
adds a class to say dolfin/mesh/Foo.h, we'll need to manually import
it in dolfin/mesh/__init__.py.
In the long run, I think that the doxygen approach will yield the most
stable result, but we'll most likely have to look in the *.i files for
%rename.

>> > Would it help to add the cpp imports to submodules instead of the main
>> > __init__ file? We already have the submodules:
>> >
>> >  mesh, common, compilemodules, fem, adaptivity and function
>> >
>> > We could add:
>> >
>> >  io, log, nls, ode, parameter, ale
>> >
>> > and drag in stuff from cpp.py in these modules instead. In this way we
>> > structure the main __init__ file better, and you might be able to
>> > structure the pythons reference manual better?
>>
>> I'm quite sure I tried something like this, and the problem was that
>> even if you do:
>>
>> from dolfin.cpp import Mesh
>>
>> in dolfin/mesh/__init__.py
>>
>> the Mesh class will still point to dolfin.cpp when you inspect it -->
>> difficult to create the proper structure for the *.rst files.
>
> Would something that I sceted above work?
>
>> I'll need to double check though that this is really the case.
>> And even if we can use the approach you outline above it means that we
>> have more stuff we need to manually maintain.
>
> Sure, but now everything is throwed into dolfin/__init__.py which is really a
> mess now.

No disagreeing you there.

Kristian

> Johan
>