fenics team mailing list archive

[Kristian Ølgaard <k.b.oelgaard@xxxxxxxxx>]

On 28 July 2010 20:14, Johan Hake <johan.hake@xxxxxxxxx> wrote:
> On Sunday July 25 2010 13:25:25 Kristian Ølgaard wrote:
>> On 25 July 2010 20:29, Johan Hake <johan.hake@xxxxxxxxx> wrote:
>> > On Saturday July 24 2010 09:48:44 Kristian Ølgaard wrote:
>> >> On 22 July 2010 20:10, Johan Hake <johan.hake@xxxxxxxxx> wrote:
>> >> > On Monday July 19 2010 12:54:44 Kristian Oelgaard wrote:
>> >> >> On 23 June 2010 15:35, Kristian Oelgaard <k.b.oelgaard@xxxxxxxxx>
> wrote:
>> >> >> > On 23 June 2010 10:55, Kristian Oelgaard <k.b.oelgaard@xxxxxxxxx>
>> >
>> > wrote:
>> >> >> >> On 22 June 2010 19:03, Johan Hake <johan.hake@xxxxxxxxx> wrote:
>> >> >> >>> On Tuesday June 22 2010 08:28:37 Kristian Oelgaard wrote:
>> >> >> >>>> I've started writing the programmer's reference for FEniCS.
>> >> >> >>>> One of the features that we decided on was that doc-strings for
>> >> >> >>>> PyDOLFIN should be written and maintained as part of the
>> >> >> >>>> documentation project and then added to the dolfin module on
>> >> >> >>>> import.
>> >> >> >>>>
>> >> >> >>>> I thought about doing this in the following way:
>> >> >> >>>>
>> >> >> >>>> 1) Create a pseudo module 'dolfin-doc' which is a copy of the
>> >> >> >>>> classes and functions in the 'real' dolfin module only it
>> >> >> >>>> contains no code at all, just doc-strings. (This approach will
>> >> >> >>>> also make it easy to create a script to check if all functions
>> >> >> >>>> are documented or if any docs are obsolete).
>> >> >> >>>
>> >> >> >>> Sounds good. I first thought of a structure (other than a dummy
>> >> >> >>> class) that just mimics the class hierarchy, but in some way that
>> >> >> >>> is what you actually suggests and it is probably as easy as
>> >> >> >>> anything else.
>> >> >> >>>
>> >> >> >>>> 2) Use the autodoc functionality of Sphinx to create parts of
>> >> >> >>>> the documentation for functions and classes
>> >> >> >>>>
>> >> >> >>>> 3) Manually add additional information (in the reST file) and
>> >> >> >>>> links to other parts of the documentation like demos etc. This
>> >> >> >>>> will not be available using help() in the Python interpreter.
>> >> >> >>>>
>> >> >> >>>> 4) In the dolfin.__init__.py function import the 'dolfin-doc'
>> >> >> >>>> module and copy the doc-strings from all classes and functions
>> >> >> >>>> to the classes and functions in the real dolfin module as was
>> >> >> >>>> suggested by Johan Hake.
>> >> >> >>>>
>> >> >> >>>> The problem with this approach is that assigning to __doc__ is
>> >> >> >>>> not permitted for objects of 'type' type.
>> >> >> >>>>
>> >> >> >>> :(
>> >> >> >>>
>> >> >> >>> I did not anticipate this. Not sure why this is. I have got the
>> >> >> >>> impression that numpy get around this. They use numpydoc to
>> >> >> >>> dynamically add their documentation. It makes heavy use of
>> >> >> >>> sphinx, but I couldn't figure how they get around that __doc__
>> >> >> >>> is read-only.
>> >> >> >>
>> >> >> >> To me it looks like numpydoc is a Sphinx extension that translates
>> >> >> >> the Numpy docstrings into something that Sphinx can understand,
>> >> >> >> not the other way around which is what we want.
>> >> >> >>
>> >> >> >> http://projects.scipy.org/numpy/browser/trunk/doc/sphinxext/README
>> >> >> >> .tx t
>> >> >> >>
>> >> >> >> So I think our best bet is to proceed with your suggestions below.
>> >> >> >>
>> >> >> >> Kristian
>> >> >> >>
>> >> >> >>> While it might be cool to look into what NumPy have done, (they
>> >> >> >>> also define a pseudo classes, which they populate with
>> >> >> >>> docstrings, (look into phantom_import.py), and they also define
>> >> >> >>> some nice format for the reST used in the docstrings), I suggest
>> >> >> >>> two things we can do:
>> >> >> >>>
>> >> >> >>> 1) SWIG can generate docstrings. We do that allready using parsed
>> >> >> >>> doxygen documentation. All of this is gathered in docstrings.i. I
>> >> >> >>> suggest generating such a file from our documentation. We need to
>> >> >> >>> turn of %feature("autodoc","1") in dolfin.i to get rid of the
>> >> >> >>> long and sometimes faulty generated signatures.
>> >> >> >
>> >> >> > I turns out that it's only the __doc__ of the class I can't assign
>> >> >> > to, not the __doc__ of member functions (and regular functions). A
>> >> >> > simpler solution (at least for me) is to parse the cpp.py module
>> >> >> > once generated and substitute all docstrings of classes with the
>> >> >> > docstrings from the dolfindoc module rather than creating the
>> >> >> > docstrings.i file.
>> >> >> >
>> >> >> > Then for the classes that we manually add we use the method you
>> >> >> > described below, but only for class.__doc__ .
>> >> >> >
>> >> >> > class Foo(object):
>> >> >> >    __doc__ = dolfindoc.Foo.__doc__
>> >> >> >    def bar(self):
>> >> >> >        "this doc string will be substituted with the
>> >> >> > dolfindoc.Foo.__dict__["bar"].__doc__."
>> >> >> >        pass
>> >> >>
>> >> >> Unfortunately it turned out I was too quick to make this conclusion.
>> >> >> For instance,
>> >> >> for the Swig generated class Mesh in the module cpp I can't assign to
>> >> >>
>> >> >> Mesh.__doc__
>> >> >>
>> >> >> because it is a 'type' object as pointed our earlier, I can assign
>> >> >> to:
>> >> >>
>> >> >> Mesh.__dict__["__init__"].__doc__
>> >> >>
>> >> >> but not
>> >> >>
>> >> >> Mesh.__dict__["size"].__doc__
>> >> >>
>> >> >> along with practically all other member functions.
>> >> >>
>> >> >> The reason is that Swig, after the class definitions does this:
>> >> >>
>> >> >> Mesh.size = new_instancemethod(_cpp.Mesh_size,None,Mesh)
>> >> >>
>> >> >> to which no assignment is possible.
>> >> >> 'AttributeError: attribute '__doc__' of 'instancemethod' objects is
>> >> >> not writable'
>> >> >> The code in Mesh.size is already
>> >> >>
>> >> >>     def size(self, *args):
>> >> >>         return _cpp.Mesh_size(self, *args)
>> >> >>
>> >> >> so why Swig overrides the method I don't know. If it didn't I could
>> >> >> assign to the docstring.
>> >> >>
>> >> >> Recall that my initial plan was to simply parse the cpp.py file and
>> >> >> substitute the docstrings of all classes with what the pseudo module
>> >> >> dolfindoc would define. However, that doesn't seem so practical
>> >> >> anymore since I would need to also comment out all lines where
>> >> >> instancemethods are being assigned to class members such that on
>> >> >> import (in dolfin/__init__.py) I can loop classes and assign to
>> >> >> function docstrings. This is of course possible but seems a bit
>> >> >> awkward and I don't know what implications the commenting out of
>> >> >> instancemethod assignment will have.
>> >> >> As I see it, two solutions remain.
>> >> >>
>> >> >> 1) Take the pseudo module dolfindoc from the FEniCS documentation and
>> >> >> create a docstring.i generator. The script
>> >> >> dolfin/dolfin/swig/generator.py should then try to import the
>> >> >> dolfindoc module, if successful use that to create docstring.i,
>> >> >> otherwise use the Doxy2SWIG generator.
>> >> >
>> >> > I thought that you landed on this alternative. The point is to
>> >> > generate a dosctrings.i file from the FEniCS documentation. Then for
>> >> > the extended Python layer we can use a generated Python module to
>> >> > assign the docstrings:
>> >> >
>> >> >  class VariationalProblem(...):
>> >> >     ...
>> >> >     __doc__ generated_docstring_module.VariationalProblem.__doc__
>> >>
>> >> This approach is getting more and more complicated; it turns out we
>> >> also need to generate all the
>> >> _post.i files to get the docstrings correct.
>> >
>> > Do yo meen the extended python functions that resides in these files?
>>
>> Yes, they will need a docstring too. But since Swig doesn't mess with
>> these particular (member-) functions, it is actually possible to
>> assign to __doc__ dynamically (on import) so maybe that would be an
>> easier option.
>
> How would you do that? I tried
>
> %extend dolfin::GenericVector
> {
>  %pythoncode
>  %{
>    def data(self):
>        generated_docstring_module.GenericVector.data.__doc__
>        return self._data()
>  %}
> }
>
> But it was only possible to have a "real" string, not a variable that was a
> str. Not sure why...

Me neither, what I thought about doing was to generate the contents of
the *_post.i files from some source and then insert the docstring as a
literal string in the *_post.i file. But I decided that was too much
work.

So what I will end up doing is to assign to the __doc__ members of
these functions in the dolfin/__init__.py (on import). This is
possible because Swig doesn't mess around with these functions but
just adds them to the module. If the *_post.i contains other than
simple functions/member functions we'll have to generate the *_post.i
files from some source.

Kristian

>> >> At this point it seems a
>> >> lot easier to me to simply extend the Python layer with everything we
>> >> want to have detailed documentation for and distribute the
>> >> dolfindocstrings module with DOLFIN so we can do:
>> >>
>> >> class VariationalProblem(...):
>> >>      ...
>> >>    __doc__ generated_docstring_module.VariationalProblem.__doc__
>> >>
>> >> this way we handle the docstrings in a uniform way, else we need to
>> >> generate docstrings.i for some functions, *_post.i files for other
>> >> functions and STILL do the assign to __doc__ trick.
>> >>
>> >> For functions that we don't extend in the Python layer we still have
>> >> the oneliner docs from the header files extended with some random
>> >> output from Swig.
>> >
>> > How do we handle the documentation of the pure cpp classes, for example
>> > cpp.Mesh?
>>
>> For all functions/classes we put a docstring in the docstringsmodule
>> and distribute this module with DOLFIN.
>
> Ok
>
>> The current approach that we're pursuing is:
>> For the classes in cpp, we generate the docstrings.i,
>> For add on functions in *_post.i we need to either generate *_post.i
>> or assign to __doc__ of those particular functions on import.
>> For classes functions in the Python layer we just assign to __doc__ in
>> the definition of classes/functions.
>
> Ok
>
>> or as I suggested:
>>
>> Extend the Python layer with whatever classes/functions that we want
>> to have documented in detail and assign to __doc__ in the definitions.
>
> Ok
>
> Johan
>
>> Kristian
>>
>> > Johan
>> >
>> >> Kristian
>> >>
>> >> >> One question w.r.t this approach, when is docstrings.i being
>> >> >> generated? I see it is distributed with DOLFIN but shouldn't it be
>> >> >> generated everytime DOLFIN is rebuild with enablePython, or does
>> >> >> enableDocs have to be switched on as well?
>> >> >
>> >> > It is generated by running dolfin/swig/generate.py. It takes quite a
>> >> > long time so I think it is good to pre-generate this in the
>> >> > distribution.
>> >> >
>> >> >> 2) Extend our Python layer with all functions and classes that we
>> >> >> want to use the dolfindoc docstrings for.
>> >> >> These should just be empty and redirect calls to classes and
>> >> >> functions of the cpp. In dolfin/mesh/mesh.py:
>> >> >>
>> >> >> class Mesh(cpp.Mesh):
>> >> >>     def size(self):
>> >> >>         try:
>> >> >>             import dolfindoc.mesh.mesh
>> >> >>             __doc__ = dolfindoc.mesh.mesh.Mesh.__doc__
>> >> >>         except:
>> >> >>             __doc__ = cpp.Mesh.__doc__
>> >> >>         return cpp.Mesh.size()
>> >> >>
>> >> >> An additional benefit of this approach is that the module structure
>> >> >> can be identical to what we have in the _real_ DOLFIN, not as it is
>> >> >> now where everything is dumped in the dolfin.cpp module.
>> >> >> I don't know how much overhead this will create, alternatively we can
>> >> >> skip the try/except clause and simply have the documentation as a
>> >> >> dependency, or not add the docstrings for memberfunctions and add
>> >> >> them later on import as was the original idea.
>> >> >
>> >> > This sounds cumbersome and in the example above will the try: except
>> >> > clause be called everytime size is called.
>> >> >
>> >> > Johan
>> >> >
>> >> >> Suggestions and comments are more than welcome!
>> >> >>
>> >> >> Kristian
>> >> >>
>> >> >> > then in dolfin/__init__.py we load the classes as we do now from
>> >> >> > the dolfin module, and then iterate over all functions and member
>> >> >> > functions and substitute docstrings from the dolfindoc module.
>> >> >> >
>> >> >> > Kristian
>> >> >> >
>> >> >> >>> 2) The added python classes and methods can be documented using
>> >> >> >>> your suggested approach, but instead of adding the docstring
>> >> >> >>> after class creation, do it during class (method or function)
>> >> >> >>> creation, a la:
>> >> >> >>>
>> >> >> >>>  class Foo(object):
>> >> >> >>>      __doc__ = docstrings.Foo.__doc__
>> >> >> >>>      ...
>> >> >> >>>
>> >> >> >>> where docstrings is the generated module containing the
>> >> >> >>> docstrings.
>> >> >> >>>
>> >> >> >>> Johan
>> >> >> >>>
>> >> >> >>>> In other words we can't assign to the __doc__ of
>> >> >> >>>>
>> >> >> >>>> class Foo(object):
>> >> >> >>>>     "Foo doc"
>> >> >> >>>>     pass
>> >> >> >>>>
>> >> >> >>>> Which is a new-style class and found in UFL and the SWIG code in
>> >> >> >>>> DOLFIN.
>> >> >> >>>>
>> >> >> >>>> It works fine for
>> >> >> >>>>
>> >> >> >>>> def some_function(v):
>> >> >> >>>>     "function doc"
>> >> >> >>>>     return 2*v
>> >> >> >>>>
>> >> >> >>>> and
>> >> >> >>>>
>> >> >> >>>> class Bar:
>> >> >> >>>>     "Bar doc"
>> >> >> >>>>     pass
>> >> >> >>>>
>> >> >> >>>> which is the old-style class often found in FFC.
>> >> >> >>>>
>> >> >> >>>> Does anyone have a solution or comments to the above approach,
>> >> >> >>>> or maybe we can do it in a completely different way.
>> >> >> >>>>
>> >> >> >>>> I read about some workaround for the 'assign to __doc__'
>> >> >> >>>> problem, but it doesn't seem that nice and it might be a
>> >> >> >>>> problem to incorporate in the SWIG generated code?
>> >> >> >>>>
>> >> >> >>>> http://stackoverflow.com/questions/71817/using-the-docstring-fro
>> >> >> >>>> m-o ne- metho d-to-automatically-overwrite-that-of-another-me
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>> Kristian
>> >> >> >>>>
>> >> >> >>>> _______________________________________________
>> >> >> >>>> Mailing list: https://launchpad.net/~fenics
>> >> >> >>>> Post to     : fenics@xxxxxxxxxxxxxxxxxxx
>> >> >> >>>> Unsubscribe : https://launchpad.net/~fenics
>> >> >> >>>> More help   : https://help.launchpad.net/ListHelp
>> >> >>
>> >> >> _______________________________________________
>> >> >> Mailing list: https://launchpad.net/~fenics
>> >> >> Post to     : fenics@xxxxxxxxxxxxxxxxxxx
>> >> >> Unsubscribe : https://launchpad.net/~fenics
>> >> >> More help   : https://help.launchpad.net/ListHelp
>> >>
>> >> _______________________________________________
>> >> Mailing list: https://launchpad.net/~fenics
>> >> Post to     : fenics@xxxxxxxxxxxxxxxxxxx
>> >> Unsubscribe : https://launchpad.net/~fenics
>> >> More help   : https://help.launchpad.net/ListHelp
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~fenics
>> Post to     : fenics@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~fenics
>> More help   : https://help.launchpad.net/ListHelp
>