fenics team mailing list archive

[Anders Logg <logg@xxxxxxxxx>]

On Thu, Jul 22, 2010 at 10:10:01PM -0700, Johan Hake wrote:
> On Thursday July 22 2010 12:37:26 Kristian Ølgaard wrote:
> > On 22 July 2010 20:10, Johan Hake <johan.hake@xxxxxxxxx> wrote:
> > > On Monday July 19 2010 12:54:44 Kristian Oelgaard wrote:
> > >> On 23 June 2010 15:35, Kristian Oelgaard <k.b.oelgaard@xxxxxxxxx> wrote:
> > >> > On 23 June 2010 10:55, Kristian Oelgaard <k.b.oelgaard@xxxxxxxxx>
> wrote:
> > >> >> On 22 June 2010 19:03, Johan Hake <johan.hake@xxxxxxxxx> wrote:
> > >> >>> On Tuesday June 22 2010 08:28:37 Kristian Oelgaard wrote:
> > >> >>>> I've started writing the programmer's reference for FEniCS.
> > >> >>>> One of the features that we decided on was that doc-strings for
> > >> >>>> PyDOLFIN should be written and maintained as part of the
> > >> >>>> documentation project and then added to the dolfin module on
> > >> >>>> import.
> > >> >>>>
> > >> >>>> I thought about doing this in the following way:
> > >> >>>>
> > >> >>>> 1) Create a pseudo module 'dolfin-doc' which is a copy of the
> > >> >>>> classes and functions in the 'real' dolfin module only it contains
> > >> >>>> no code at all, just doc-strings. (This approach will also make it
> > >> >>>> easy to create a script to check if all functions are documented
> > >> >>>> or if any docs are obsolete).
> > >> >>>
> > >> >>> Sounds good. I first thought of a structure (other than a dummy
> > >> >>> class) that just mimics the class hierarchy, but in some way that
> > >> >>> is what you actually suggests and it is probably as easy as
> > >> >>> anything else.
> > >> >>>
> > >> >>>> 2) Use the autodoc functionality of Sphinx to create parts of the
> > >> >>>> documentation for functions and classes
> > >> >>>>
> > >> >>>> 3) Manually add additional information (in the reST file) and links
> > >> >>>> to other parts of the documentation like demos etc. This will not
> > >> >>>> be available using help() in the Python interpreter.
> > >> >>>>
> > >> >>>> 4) In the dolfin.__init__.py function import the 'dolfin-doc'
> > >> >>>> module and copy the doc-strings from all classes and functions to
> > >> >>>> the classes and functions in the real dolfin module as was
> > >> >>>> suggested by Johan Hake.
> > >> >>>>
> > >> >>>> The problem with this approach is that assigning to __doc__ is not
> > >> >>>> permitted for objects of 'type' type.
> > >> >>>>
> > >> >>> :(
> > >> >>>
> > >> >>> I did not anticipate this. Not sure why this is. I have got the
> > >> >>> impression that numpy get around this. They use numpydoc to
> > >> >>> dynamically add their documentation. It makes heavy use of sphinx,
> > >> >>> but I couldn't figure how they get around that __doc__ is
> > >> >>> read-only.
> > >> >>
> > >> >> To me it looks like numpydoc is a Sphinx extension that translates
> > >> >> the Numpy docstrings into something that Sphinx can understand, not
> > >> >> the other way around which is what we want.
> > >> >>
> > >> >> http://projects.scipy.org/numpy/browser/trunk/doc/sphinxext/README.tx
> > >> >> t
> > >> >>
> > >> >> So I think our best bet is to proceed with your suggestions below.
> > >> >>
> > >> >> Kristian
> > >> >>
> > >> >>> While it might be cool to look into what NumPy have done, (they also
> > >> >>> define a pseudo classes, which they populate with docstrings, (look
> > >> >>> into phantom_import.py), and they also define some nice format for
> > >> >>> the reST used in the docstrings), I suggest two things we can do:
> > >> >>>
> > >> >>> 1) SWIG can generate docstrings. We do that allready using parsed
> > >> >>> doxygen documentation. All of this is gathered in docstrings.i. I
> > >> >>> suggest generating such a file from our documentation. We need to
> > >> >>> turn of %feature("autodoc","1") in dolfin.i to get rid of the long
> > >> >>> and sometimes faulty generated signatures.
> > >> >
> > >> > I turns out that it's only the __doc__ of the class I can't assign to,
> > >> > not the __doc__ of member functions (and regular functions).
> > >> > A simpler solution (at least for me) is to parse the cpp.py module
> > >> > once generated and substitute all docstrings of classes with the
> > >> > docstrings from the dolfindoc module rather than creating the
> > >> > docstrings.i file.
> > >> >
> > >> > Then for the classes that we manually add we use the method you
> > >> > described below, but only for class.__doc__ .
> > >> >
> > >> > class Foo(object):
> > >> >    __doc__ = dolfindoc.Foo.__doc__
> > >> >    def bar(self):
> > >> >        "this doc string will be substituted with the
> > >> > dolfindoc.Foo.__dict__["bar"].__doc__."
> > >> >        pass
> > >>
> > >> Unfortunately it turned out I was too quick to make this conclusion.
> > >> For instance,
> > >> for the Swig generated class Mesh in the module cpp I can't assign to
> > >>
> > >> Mesh.__doc__
> > >>
> > >> because it is a 'type' object as pointed our earlier, I can assign to:
> > >>
> > >> Mesh.__dict__["__init__"].__doc__
> > >>
> > >> but not
> > >>
> > >> Mesh.__dict__["size"].__doc__
> > >>
> > >> along with practically all other member functions.
> > >>
> > >> The reason is that Swig, after the class definitions does this:
> > >>
> > >> Mesh.size = new_instancemethod(_cpp.Mesh_size,None,Mesh)
> > >>
> > >> to which no assignment is possible.
> > >> 'AttributeError: attribute '__doc__' of 'instancemethod' objects is
> > >> not writable'
> > >> The code in Mesh.size is already
> > >>
> > >>     def size(self, *args):
> > >>         return _cpp.Mesh_size(self, *args)
> > >>
> > >> so why Swig overrides the method I don't know. If it didn't I could
> > >> assign to the docstring.
> > >>
> > >> Recall that my initial plan was to simply parse the cpp.py file and
> > >> substitute the docstrings of all classes with what the pseudo module
> > >> dolfindoc would define. However, that doesn't seem so practical
> > >> anymore since I would need to also comment out all lines where
> > >> instancemethods are being assigned to class members such that on
> > >> import (in dolfin/__init__.py) I can loop classes and assign to
> > >> function docstrings. This is of course possible but seems a bit
> > >> awkward and I don't know what implications the commenting out of
> > >> instancemethod assignment will have.
> > >> As I see it, two solutions remain.
> > >>
> > >> 1) Take the pseudo module dolfindoc from the FEniCS documentation and
> > >> create a docstring.i generator. The script
> > >> dolfin/dolfin/swig/generator.py should then try to import the
> > >> dolfindoc module, if successful use that to create docstring.i,
> > >> otherwise use the Doxy2SWIG generator.
> > >
> > > I thought that you landed on this alternative. The point is to generate a
> > > dosctrings.i file from the FEniCS documentation. Then for the extended
> > > Python
> >
> > > layer we can use a generated Python module to assign the docstrings:
> > I landed on a slightly modified version of that yes. OK, the
> > docstrings.i part should be relatively easy to generate and probably
> > very fast too since I don't have to generate the html pages and parse.
>
> Ok.
>
> > >  class VariationalProblem(...):
> > >     ...
> > >     __doc__ generated_docstring_module.VariationalProblem.__doc__
> >
> > This I don't understand, if we generate a module which we will
> > distribute with DOLFIN why not simply distribute the module from the
> > FEniCS documentation?
>
> Yes, this should be the generated_docstring_module, which is imported before
> the statement above.
>
> > >> One question w.r.t this approach, when is docstrings.i being
> > >> generated? I see it is distributed with DOLFIN but shouldn't it be
> > >> generated everytime DOLFIN is rebuild with enablePython, or does
> > >> enableDocs have to be switched on as well?
> > >
> > > It is generated by running dolfin/swig/generate.py. It takes quite a long
> > > time so I think it is good to pre-generate this in the distribution.
> >
> > OK, so if a developer actually happens to update or even write
> > docstrings, the docstrings.i should be updated before commit.
> > (at least in theory ;))
>
> This has been the procedure for now. It is probably possible to change this
> workflow, to a more automatic one.

Would it be possible to integrate it in the build system? Or too much
noise every time we build?

It should at least be integrated in the release script. I have added
it to the pre-release hook of the release script now. We might want to
have a top-level script that does all the import/autogeneration magic
Kristian is working on, including calling generate.py in dolfin/swig.
Look at the file release.conf.

--
Anders


>
> Johan
>
> > >> 2) Extend our Python layer with all functions and classes that we want
> > >> to use the dolfindoc docstrings for.
> > >> These should just be empty and redirect calls to classes and functions
> > >> of the cpp. In dolfin/mesh/mesh.py:
> > >>
> > >> class Mesh(cpp.Mesh):
> > >>     def size(self):
> > >>         try:
> > >>             import dolfindoc.mesh.mesh
> > >>             __doc__ = dolfindoc.mesh.mesh.Mesh.__doc__
> > >>         except:
> > >>             __doc__ = cpp.Mesh.__doc__
> > >>         return cpp.Mesh.size()
> > >>
> > >> An additional benefit of this approach is that the module structure
> > >> can be identical to what we have in the _real_ DOLFIN, not as it is
> > >> now where everything is dumped in the dolfin.cpp module.
> > >> I don't know how much overhead this will create, alternatively we can
> > >> skip the try/except clause and simply have the documentation as a
> > >> dependency, or not add the docstrings for memberfunctions and add them
> > >> later on import as was the original idea.
> > >
> > > This sounds cumbersome and in the example above will the try: except
> > > clause be called everytime size is called.
> >
> > Yes, I thought about having no docstrings as default (just need them
> > for the class docstring) and the dynamically add them on import for
> > all member functions.
> > I know it's a bit more work, but we will have the same layout of
> > modules as we have for the directories in DOLFIN.
> >
> > Kristian
> >
> > > Johan
> > >
> > >> Suggestions and comments are more than welcome!
> > >>
> > >> Kristian
> > >>
> > >> > then in dolfin/__init__.py we load the classes as we do now from the
> > >> > dolfin module, and then iterate over all functions and member
> > >> > functions and substitute docstrings from the dolfindoc module.
> > >> >
> > >> > Kristian
> > >> >
> > >> >>> 2) The added python classes and methods can be documented using your
> > >> >>> suggested approach, but instead of adding the docstring after class
> > >> >>> creation, do it during class (method or function) creation, a la:
> > >> >>>
> > >> >>>  class Foo(object):
> > >> >>>      __doc__ = docstrings.Foo.__doc__
> > >> >>>      ...
> > >> >>>
> > >> >>> where docstrings is the generated module containing the docstrings.
> > >> >>>
> > >> >>> Johan
> > >> >>>
> > >> >>>> In other words we can't assign to the __doc__ of
> > >> >>>>
> > >> >>>> class Foo(object):
> > >> >>>>     "Foo doc"
> > >> >>>>     pass
> > >> >>>>
> > >> >>>> Which is a new-style class and found in UFL and the SWIG code in
> > >> >>>> DOLFIN.
> > >> >>>>
> > >> >>>> It works fine for
> > >> >>>>
> > >> >>>> def some_function(v):
> > >> >>>>     "function doc"
> > >> >>>>     return 2*v
> > >> >>>>
> > >> >>>> and
> > >> >>>>
> > >> >>>> class Bar:
> > >> >>>>     "Bar doc"
> > >> >>>>     pass
> > >> >>>>
> > >> >>>> which is the old-style class often found in FFC.
> > >> >>>>
> > >> >>>> Does anyone have a solution or comments to the above approach, or
> > >> >>>> maybe we can do it in a completely different way.
> > >> >>>>
> > >> >>>> I read about some workaround for the 'assign to __doc__' problem,
> > >> >>>> but it doesn't seem that nice and it might be a problem to
> > >> >>>> incorporate in the SWIG generated code?
> > >> >>>>
> > >> >>>> http://stackoverflow.com/questions/71817/using-the-docstring-from-o
> > >> >>>> ne- metho d-to-automatically-overwrite-that-of-another-me
> > >> >>>>
> > >> >>>>
> > >> >>>> Kristian
> > >> >>>>
> > >> >>>> _______________________________________________
> > >> >>>> Mailing list: https://launchpad.net/~fenics
> > >> >>>> Post to     : fenics@xxxxxxxxxxxxxxxxxxx
> > >> >>>> Unsubscribe : https://launchpad.net/~fenics
> > >> >>>> More help   : https://help.launchpad.net/ListHelp
> > >>
> > >> _______________________________________________
> > >> Mailing list: https://launchpad.net/~fenics
> > >> Post to     : fenics@xxxxxxxxxxxxxxxxxxx
> > >> Unsubscribe : https://launchpad.net/~fenics
> > >> More help   : https://help.launchpad.net/ListHelp
>
> _______________________________________________
> Mailing list: https://launchpad.net/~fenics
> Post to     : fenics@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~fenics
> More help   : https://help.launchpad.net/ListHelp

Attachment: signature.asc
Description: Digital signature