fenics team mailing list archive

[Johan Hake <johan.hake@xxxxxxxxx>]

On Saturday July 24 2010 09:48:44 Kristian Ølgaard wrote:
> On 22 July 2010 20:10, Johan Hake <johan.hake@xxxxxxxxx> wrote:
> > On Monday July 19 2010 12:54:44 Kristian Oelgaard wrote:
> >> On 23 June 2010 15:35, Kristian Oelgaard <k.b.oelgaard@xxxxxxxxx> wrote:
> >> > On 23 June 2010 10:55, Kristian Oelgaard <k.b.oelgaard@xxxxxxxxx> 
wrote:
> >> >> On 22 June 2010 19:03, Johan Hake <johan.hake@xxxxxxxxx> wrote:
> >> >>> On Tuesday June 22 2010 08:28:37 Kristian Oelgaard wrote:
> >> >>>> I've started writing the programmer's reference for FEniCS.
> >> >>>> One of the features that we decided on was that doc-strings for
> >> >>>> PyDOLFIN should be written and maintained as part of the
> >> >>>> documentation project and then added to the dolfin module on
> >> >>>> import.
> >> >>>> 
> >> >>>> I thought about doing this in the following way:
> >> >>>> 
> >> >>>> 1) Create a pseudo module 'dolfin-doc' which is a copy of the
> >> >>>> classes and functions in the 'real' dolfin module only it contains
> >> >>>> no code at all, just doc-strings. (This approach will also make it
> >> >>>> easy to create a script to check if all functions are documented
> >> >>>> or if any docs are obsolete).
> >> >>> 
> >> >>> Sounds good. I first thought of a structure (other than a dummy
> >> >>> class) that just mimics the class hierarchy, but in some way that
> >> >>> is what you actually suggests and it is probably as easy as
> >> >>> anything else.
> >> >>> 
> >> >>>> 2) Use the autodoc functionality of Sphinx to create parts of the
> >> >>>> documentation for functions and classes
> >> >>>> 
> >> >>>> 3) Manually add additional information (in the reST file) and links
> >> >>>> to other parts of the documentation like demos etc. This will not
> >> >>>> be available using help() in the Python interpreter.
> >> >>>> 
> >> >>>> 4) In the dolfin.__init__.py function import the 'dolfin-doc'
> >> >>>> module and copy the doc-strings from all classes and functions to
> >> >>>> the classes and functions in the real dolfin module as was
> >> >>>> suggested by Johan Hake.
> >> >>>> 
> >> >>>> The problem with this approach is that assigning to __doc__ is not
> >> >>>> permitted for objects of 'type' type.
> >> >>>> 
> >> >>> :(
> >> >>> 
> >> >>> I did not anticipate this. Not sure why this is. I have got the
> >> >>> impression that numpy get around this. They use numpydoc to
> >> >>> dynamically add their documentation. It makes heavy use of sphinx,
> >> >>> but I couldn't figure how they get around that __doc__ is
> >> >>> read-only.
> >> >> 
> >> >> To me it looks like numpydoc is a Sphinx extension that translates
> >> >> the Numpy docstrings into something that Sphinx can understand, not
> >> >> the other way around which is what we want.
> >> >> 
> >> >> http://projects.scipy.org/numpy/browser/trunk/doc/sphinxext/README.tx
> >> >> t
> >> >> 
> >> >> So I think our best bet is to proceed with your suggestions below.
> >> >> 
> >> >> Kristian
> >> >> 
> >> >>> While it might be cool to look into what NumPy have done, (they also
> >> >>> define a pseudo classes, which they populate with docstrings, (look
> >> >>> into phantom_import.py), and they also define some nice format for
> >> >>> the reST used in the docstrings), I suggest two things we can do:
> >> >>> 
> >> >>> 1) SWIG can generate docstrings. We do that allready using parsed
> >> >>> doxygen documentation. All of this is gathered in docstrings.i. I
> >> >>> suggest generating such a file from our documentation. We need to
> >> >>> turn of %feature("autodoc","1") in dolfin.i to get rid of the long
> >> >>> and sometimes faulty generated signatures.
> >> > 
> >> > I turns out that it's only the __doc__ of the class I can't assign to,
> >> > not the __doc__ of member functions (and regular functions).
> >> > A simpler solution (at least for me) is to parse the cpp.py module
> >> > once generated and substitute all docstrings of classes with the
> >> > docstrings from the dolfindoc module rather than creating the
> >> > docstrings.i file.
> >> > 
> >> > Then for the classes that we manually add we use the method you
> >> > described below, but only for class.__doc__ .
> >> > 
> >> > class Foo(object):
> >> >    __doc__ = dolfindoc.Foo.__doc__
> >> >    def bar(self):
> >> >        "this doc string will be substituted with the
> >> > dolfindoc.Foo.__dict__["bar"].__doc__."
> >> >        pass
> >> 
> >> Unfortunately it turned out I was too quick to make this conclusion.
> >> For instance,
> >> for the Swig generated class Mesh in the module cpp I can't assign to
> >> 
> >> Mesh.__doc__
> >> 
> >> because it is a 'type' object as pointed our earlier, I can assign to:
> >> 
> >> Mesh.__dict__["__init__"].__doc__
> >> 
> >> but not
> >> 
> >> Mesh.__dict__["size"].__doc__
> >> 
> >> along with practically all other member functions.
> >> 
> >> The reason is that Swig, after the class definitions does this:
> >> 
> >> Mesh.size = new_instancemethod(_cpp.Mesh_size,None,Mesh)
> >> 
> >> to which no assignment is possible.
> >> 'AttributeError: attribute '__doc__' of 'instancemethod' objects is
> >> not writable'
> >> The code in Mesh.size is already
> >> 
> >>     def size(self, *args):
> >>         return _cpp.Mesh_size(self, *args)
> >> 
> >> so why Swig overrides the method I don't know. If it didn't I could
> >> assign to the docstring.
> >> 
> >> Recall that my initial plan was to simply parse the cpp.py file and
> >> substitute the docstrings of all classes with what the pseudo module
> >> dolfindoc would define. However, that doesn't seem so practical
> >> anymore since I would need to also comment out all lines where
> >> instancemethods are being assigned to class members such that on
> >> import (in dolfin/__init__.py) I can loop classes and assign to
> >> function docstrings. This is of course possible but seems a bit
> >> awkward and I don't know what implications the commenting out of
> >> instancemethod assignment will have.
> >> As I see it, two solutions remain.
> >> 
> >> 1) Take the pseudo module dolfindoc from the FEniCS documentation and
> >> create a docstring.i generator. The script
> >> dolfin/dolfin/swig/generator.py should then try to import the
> >> dolfindoc module, if successful use that to create docstring.i,
> >> otherwise use the Doxy2SWIG generator.
> > 
> > I thought that you landed on this alternative. The point is to generate a
> > dosctrings.i file from the FEniCS documentation. Then for the extended
> > Python layer we can use a generated Python module to assign the
> > docstrings:
> > 
> >  class VariationalProblem(...):
> >     ...
> >     __doc__ generated_docstring_module.VariationalProblem.__doc__
> 
> This approach is getting more and more complicated; it turns out we
> also need to generate all the
> _post.i files to get the docstrings correct. 

Do yo meen the extended python functions that resides in these files?

> At this point it seems a
> lot easier to me to simply extend the Python layer with everything we
> want to have detailed documentation for and distribute the
> dolfindocstrings module with DOLFIN so we can do:
> 
> class VariationalProblem(...):
>      ...
>    __doc__ generated_docstring_module.VariationalProblem.__doc__
> 
> this way we handle the docstrings in a uniform way, else we need to
> generate docstrings.i for some functions, *_post.i files for other
> functions and STILL do the assign to __doc__ trick.
> 
> For functions that we don't extend in the Python layer we still have
> the oneliner docs from the header files extended with some random
> output from Swig.

How do we handle the documentation of the pure cpp classes, for example 
cpp.Mesh?

Johan

> Kristian
> 
> >> One question w.r.t this approach, when is docstrings.i being
> >> generated? I see it is distributed with DOLFIN but shouldn't it be
> >> generated everytime DOLFIN is rebuild with enablePython, or does
> >> enableDocs have to be switched on as well?
> > 
> > It is generated by running dolfin/swig/generate.py. It takes quite a long
> > time so I think it is good to pre-generate this in the distribution.
> > 
> >> 2) Extend our Python layer with all functions and classes that we want
> >> to use the dolfindoc docstrings for.
> >> These should just be empty and redirect calls to classes and functions
> >> of the cpp. In dolfin/mesh/mesh.py:
> >> 
> >> class Mesh(cpp.Mesh):
> >>     def size(self):
> >>         try:
> >>             import dolfindoc.mesh.mesh
> >>             __doc__ = dolfindoc.mesh.mesh.Mesh.__doc__
> >>         except:
> >>             __doc__ = cpp.Mesh.__doc__
> >>         return cpp.Mesh.size()
> >> 
> >> An additional benefit of this approach is that the module structure
> >> can be identical to what we have in the _real_ DOLFIN, not as it is
> >> now where everything is dumped in the dolfin.cpp module.
> >> I don't know how much overhead this will create, alternatively we can
> >> skip the try/except clause and simply have the documentation as a
> >> dependency, or not add the docstrings for memberfunctions and add them
> >> later on import as was the original idea.
> > 
> > This sounds cumbersome and in the example above will the try: except
> > clause be called everytime size is called.
> > 
> > Johan
> > 
> >> Suggestions and comments are more than welcome!
> >> 
> >> Kristian
> >> 
> >> > then in dolfin/__init__.py we load the classes as we do now from the
> >> > dolfin module, and then iterate over all functions and member
> >> > functions and substitute docstrings from the dolfindoc module.
> >> > 
> >> > Kristian
> >> > 
> >> >>> 2) The added python classes and methods can be documented using your
> >> >>> suggested approach, but instead of adding the docstring after class
> >> >>> creation, do it during class (method or function) creation, a la:
> >> >>> 
> >> >>>  class Foo(object):
> >> >>>      __doc__ = docstrings.Foo.__doc__
> >> >>>      ...
> >> >>> 
> >> >>> where docstrings is the generated module containing the docstrings.
> >> >>> 
> >> >>> Johan
> >> >>> 
> >> >>>> In other words we can't assign to the __doc__ of
> >> >>>> 
> >> >>>> class Foo(object):
> >> >>>>     "Foo doc"
> >> >>>>     pass
> >> >>>> 
> >> >>>> Which is a new-style class and found in UFL and the SWIG code in
> >> >>>> DOLFIN.
> >> >>>> 
> >> >>>> It works fine for
> >> >>>> 
> >> >>>> def some_function(v):
> >> >>>>     "function doc"
> >> >>>>     return 2*v
> >> >>>> 
> >> >>>> and
> >> >>>> 
> >> >>>> class Bar:
> >> >>>>     "Bar doc"
> >> >>>>     pass
> >> >>>> 
> >> >>>> which is the old-style class often found in FFC.
> >> >>>> 
> >> >>>> Does anyone have a solution or comments to the above approach, or
> >> >>>> maybe we can do it in a completely different way.
> >> >>>> 
> >> >>>> I read about some workaround for the 'assign to __doc__' problem,
> >> >>>> but it doesn't seem that nice and it might be a problem to
> >> >>>> incorporate in the SWIG generated code?
> >> >>>> 
> >> >>>> http://stackoverflow.com/questions/71817/using-the-docstring-from-o
> >> >>>> ne- metho d-to-automatically-overwrite-that-of-another-me
> >> >>>> 
> >> >>>> 
> >> >>>> Kristian
> >> >>>> 
> >> >>>> _______________________________________________
> >> >>>> Mailing list: https://launchpad.net/~fenics
> >> >>>> Post to     : fenics@xxxxxxxxxxxxxxxxxxx
> >> >>>> Unsubscribe : https://launchpad.net/~fenics
> >> >>>> More help   : https://help.launchpad.net/ListHelp
> >> 
> >> _______________________________________________
> >> Mailing list: https://launchpad.net/~fenics
> >> Post to     : fenics@xxxxxxxxxxxxxxxxxxx
> >> Unsubscribe : https://launchpad.net/~fenics
> >> More help   : https://help.launchpad.net/ListHelp
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~fenics
> Post to     : fenics@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~fenics
> More help   : https://help.launchpad.net/ListHelp