fenics team mailing list archive

[Kristian Ølgaard <k.b.oelgaard@xxxxxxxxx>]

On 22 July 2010 20:10, Johan Hake <johan.hake@xxxxxxxxx> wrote:
> On Monday July 19 2010 12:54:44 Kristian Oelgaard wrote:
>> On 23 June 2010 15:35, Kristian Oelgaard <k.b.oelgaard@xxxxxxxxx> wrote:
>> > On 23 June 2010 10:55, Kristian Oelgaard <k.b.oelgaard@xxxxxxxxx> wrote:
>> >> On 22 June 2010 19:03, Johan Hake <johan.hake@xxxxxxxxx> wrote:
>> >>> On Tuesday June 22 2010 08:28:37 Kristian Oelgaard wrote:
>> >>>> I've started writing the programmer's reference for FEniCS.
>> >>>> One of the features that we decided on was that doc-strings for
>> >>>> PyDOLFIN should be written and maintained as part of the documentation
>> >>>> project and then added to the dolfin module on import.
>> >>>>
>> >>>> I thought about doing this in the following way:
>> >>>>
>> >>>> 1) Create a pseudo module 'dolfin-doc' which is a copy of the classes
>> >>>> and functions in the 'real' dolfin module only it contains no code at
>> >>>> all, just doc-strings. (This approach will also make it easy to create
>> >>>> a script to check if all functions are documented or if any docs are
>> >>>> obsolete).
>> >>>
>> >>> Sounds good. I first thought of a structure (other than a dummy class)
>> >>> that just mimics the class hierarchy, but in some way that is what you
>> >>> actually suggests and it is probably as easy as anything else.
>> >>>
>> >>>> 2) Use the autodoc functionality of Sphinx to create parts of the
>> >>>> documentation for functions and classes
>> >>>>
>> >>>> 3) Manually add additional information (in the reST file) and links to
>> >>>> other parts of the documentation like demos etc. This will not be
>> >>>> available using help() in the Python interpreter.
>> >>>>
>> >>>> 4) In the dolfin.__init__.py function import the 'dolfin-doc' module
>> >>>> and copy the doc-strings from all classes and functions to the classes
>> >>>> and functions in the real dolfin module as was suggested by Johan
>> >>>> Hake.
>> >>>>
>> >>>> The problem with this approach is that assigning to __doc__ is not
>> >>>> permitted for objects of 'type' type.
>> >>>>
>> >>> :(
>> >>>
>> >>> I did not anticipate this. Not sure why this is. I have got the
>> >>> impression that numpy get around this. They use numpydoc to
>> >>> dynamically add their documentation. It makes heavy use of sphinx, but
>> >>> I couldn't figure how they get around that __doc__ is read-only.
>> >>
>> >> To me it looks like numpydoc is a Sphinx extension that translates the
>> >> Numpy docstrings into something that Sphinx can understand, not the
>> >> other way around which is what we want.
>> >>
>> >> http://projects.scipy.org/numpy/browser/trunk/doc/sphinxext/README.txt
>> >>
>> >> So I think our best bet is to proceed with your suggestions below.
>> >>
>> >> Kristian
>> >>
>> >>> While it might be cool to look into what NumPy have done, (they also
>> >>> define a pseudo classes, which they populate with docstrings, (look
>> >>> into phantom_import.py), and they also define some nice format for the
>> >>> reST used in the docstrings), I suggest two things we can do:
>> >>>
>> >>> 1) SWIG can generate docstrings. We do that allready using parsed
>> >>> doxygen documentation. All of this is gathered in docstrings.i. I
>> >>> suggest generating such a file from our documentation. We need to turn
>> >>> of %feature("autodoc","1") in dolfin.i to get rid of the long and
>> >>> sometimes faulty generated signatures.
>> >
>> > I turns out that it's only the __doc__ of the class I can't assign to,
>> > not the __doc__ of member functions (and regular functions).
>> > A simpler solution (at least for me) is to parse the cpp.py module
>> > once generated and substitute all docstrings of classes with the
>> > docstrings from the dolfindoc module rather than creating the
>> > docstrings.i file.
>> >
>> > Then for the classes that we manually add we use the method you
>> > described below, but only for class.__doc__ .
>> >
>> > class Foo(object):
>> >    __doc__ = dolfindoc.Foo.__doc__
>> >    def bar(self):
>> >        "this doc string will be substituted with the
>> > dolfindoc.Foo.__dict__["bar"].__doc__."
>> >        pass
>>
>> Unfortunately it turned out I was too quick to make this conclusion.
>> For instance,
>> for the Swig generated class Mesh in the module cpp I can't assign to
>>
>> Mesh.__doc__
>>
>> because it is a 'type' object as pointed our earlier, I can assign to:
>>
>> Mesh.__dict__["__init__"].__doc__
>>
>> but not
>>
>> Mesh.__dict__["size"].__doc__
>>
>> along with practically all other member functions.
>>
>> The reason is that Swig, after the class definitions does this:
>>
>> Mesh.size = new_instancemethod(_cpp.Mesh_size,None,Mesh)
>>
>> to which no assignment is possible.
>> 'AttributeError: attribute '__doc__' of 'instancemethod' objects is
>> not writable'
>> The code in Mesh.size is already
>>
>>     def size(self, *args):
>>         return _cpp.Mesh_size(self, *args)
>>
>> so why Swig overrides the method I don't know. If it didn't I could
>> assign to the docstring.
>>
>> Recall that my initial plan was to simply parse the cpp.py file and
>> substitute the docstrings of all classes with what the pseudo module
>> dolfindoc would define. However, that doesn't seem so practical
>> anymore since I would need to also comment out all lines where
>> instancemethods are being assigned to class members such that on
>> import (in dolfin/__init__.py) I can loop classes and assign to
>> function docstrings. This is of course possible but seems a bit
>> awkward and I don't know what implications the commenting out of
>> instancemethod assignment will have.
>> As I see it, two solutions remain.
>>
>> 1) Take the pseudo module dolfindoc from the FEniCS documentation and
>> create a docstring.i generator. The script
>> dolfin/dolfin/swig/generator.py should then try to import the
>> dolfindoc module, if successful use that to create docstring.i,
>> otherwise use the Doxy2SWIG generator.
>
> I thought that you landed on this alternative. The point is to generate a
> dosctrings.i file from the FEniCS documentation. Then for the extended Python
> layer we can use a generated Python module to assign the docstrings:

I landed on a slightly modified version of that yes. OK, the
docstrings.i part should be relatively easy to generate and probably
very fast too since I don't have to generate the html pages and parse.

>  class VariationalProblem(...):
>     ...
>     __doc__ generated_docstring_module.VariationalProblem.__doc__

This I don't understand, if we generate a module which we will
distribute with DOLFIN why not simply distribute the module from the
FEniCS documentation?

>> One question w.r.t this approach, when is docstrings.i being
>> generated? I see it is distributed with DOLFIN but shouldn't it be
>> generated everytime DOLFIN is rebuild with enablePython, or does
>> enableDocs have to be switched on as well?
>
> It is generated by running dolfin/swig/generate.py. It takes quite a long time
> so I think it is good to pre-generate this in the distribution.

OK, so if a developer actually happens to update or even write
docstrings, the docstrings.i should be updated before commit.
(at least in theory ;))

>> 2) Extend our Python layer with all functions and classes that we want
>> to use the dolfindoc docstrings for.
>> These should just be empty and redirect calls to classes and functions
>> of the cpp. In dolfin/mesh/mesh.py:
>>
>> class Mesh(cpp.Mesh):
>>     def size(self):
>>         try:
>>             import dolfindoc.mesh.mesh
>>             __doc__ = dolfindoc.mesh.mesh.Mesh.__doc__
>>         except:
>>             __doc__ = cpp.Mesh.__doc__
>>         return cpp.Mesh.size()
>>
>> An additional benefit of this approach is that the module structure
>> can be identical to what we have in the _real_ DOLFIN, not as it is
>> now where everything is dumped in the dolfin.cpp module.
>> I don't know how much overhead this will create, alternatively we can
>> skip the try/except clause and simply have the documentation as a
>> dependency, or not add the docstrings for memberfunctions and add them
>> later on import as was the original idea.
>
> This sounds cumbersome and in the example above will the try: except clause be
> called everytime size is called.

Yes, I thought about having no docstrings as default (just need them
for the class docstring) and the dynamically add them on import for
all member functions.
I know it's a bit more work, but we will have the same layout of
modules as we have for the directories in DOLFIN.

Kristian

> Johan
>
>> Suggestions and comments are more than welcome!
>>
>> Kristian
>>
>> > then in dolfin/__init__.py we load the classes as we do now from the
>> > dolfin module, and then iterate over all functions and member
>> > functions and substitute docstrings from the dolfindoc module.
>> >
>> > Kristian
>> >
>> >>> 2) The added python classes and methods can be documented using your
>> >>> suggested approach, but instead of adding the docstring after class
>> >>> creation, do it during class (method or function) creation, a la:
>> >>>
>> >>>  class Foo(object):
>> >>>      __doc__ = docstrings.Foo.__doc__
>> >>>      ...
>> >>>
>> >>> where docstrings is the generated module containing the docstrings.
>> >>>
>> >>> Johan
>> >>>
>> >>>> In other words we can't assign to the __doc__ of
>> >>>>
>> >>>> class Foo(object):
>> >>>>     "Foo doc"
>> >>>>     pass
>> >>>>
>> >>>> Which is a new-style class and found in UFL and the SWIG code in
>> >>>> DOLFIN.
>> >>>>
>> >>>> It works fine for
>> >>>>
>> >>>> def some_function(v):
>> >>>>     "function doc"
>> >>>>     return 2*v
>> >>>>
>> >>>> and
>> >>>>
>> >>>> class Bar:
>> >>>>     "Bar doc"
>> >>>>     pass
>> >>>>
>> >>>> which is the old-style class often found in FFC.
>> >>>>
>> >>>> Does anyone have a solution or comments to the above approach, or
>> >>>> maybe we can do it in a completely different way.
>> >>>>
>> >>>> I read about some workaround for the 'assign to __doc__' problem, but
>> >>>> it doesn't seem that nice and it might be a problem to incorporate in
>> >>>> the SWIG generated code?
>> >>>>
>> >>>> http://stackoverflow.com/questions/71817/using-the-docstring-from-one-
>> >>>> metho d-to-automatically-overwrite-that-of-another-me
>> >>>>
>> >>>>
>> >>>> Kristian
>> >>>>
>> >>>> _______________________________________________
>> >>>> Mailing list: https://launchpad.net/~fenics
>> >>>> Post to     : fenics@xxxxxxxxxxxxxxxxxxx
>> >>>> Unsubscribe : https://launchpad.net/~fenics
>> >>>> More help   : https://help.launchpad.net/ListHelp
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~fenics
>> Post to     : fenics@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~fenics
>> More help   : https://help.launchpad.net/ListHelp
>