fenics team mailing list archive

[Hans Petter Langtangen <hpl@xxxxxxxxx>]

Sat, 20 Mar Anders Logg wrote:

> > >We have also discussed using a standard format for the Python Docstrings. So
> > >nice programmers references can be generated from these. We haven't decided
> > >which one we are going to use (epydoc or others). I couldn't find doconce (too
> > >close to many other meanings of doc once :P) at the net so I do not know what
> > >this software does or how well adapted it is.
> 
> Perhaps Hans Petter could say something about doconce?

I wrote doconce during the FEniCS'06 (?) meeting in Delft :-) as a
solution to the problem that I wanted to document software in one
place only, and still be able to put that single piece of text into
pure source code, latex manuals, html manuals, other formats like
docbook, etc. There is and was a relevant tool, docutils and the reST
(restructured text) markup, for doing this. The problem with docutils
was that I think reST has too much markup compared to the plain text
you want to see in the source code and with pydoc in a
terminal. Writing a new backend in docutils that filters reST to plain
(almost) untagged text is the obvious solution, but that seemed
very complicated. I also wanted better support for source code and
mathematics in documents. The attached tutorial explains what doconce is.

> It might have some good features we can use. But one issue would be
> the status of doconce. To be useful, it needs to be actively
> maintained and widely available (as in having a Debian package).

Right now it's living in Simula's svn repository only. It has been
frequently used by myself and others (Johannes, Ilmar). We have thought
of putting it out on googlecode, but that requires maintenance...

For example, my latest book has half a chapter where the source is in
doconce.  That source is automatically copied into a doc string in the
Python code, so that we have the book material also as documentation
of a Python module (via pydoc), the source is filtered to Epytext and
included in the Python module to produce an Epydoc manual of the
module, the source is also filtered to HTML for an online tutorial,
and to latex for fancy typesetting in the book (and in fact also
Johannes Ring's master thesis). The idea is to document once and only
once: if the tutorial is updated, it's the doconce source, and I have
automatic update of the book, the Python source code, and other
manuals :-)

Personally, I use doconce for almost everything I write (except slides
and complete books) since that enables me to postpone the choice of
format.

Johannes (who have made a wiki filter to doconce and done many other
adjustments) and I could maintain a more official version and maybe
try to get it into Debian. However, we should carefully consider the
work associated with a more official version. Doconce is very limited
(no footnotes, no citations and bibliography, for instance) and as
soon as it gets users, they will demand extensions.  The point was to
make it minimal. Debugging errors can also be somewhat frustrating
and challenging.

FEniCS should definitely adopt the doconce *strategy*, i.e., write a
piece of text only once, but maybe distutils and reST is a better
technical choice?  The downside is no support for math, you cannot
copy code directly from a source file and into a manual, and the reST
tagged text (sometimes a bit ugly) appears in Doxygen text or Python
doc strings. The upside is strong support.

> I think we need to separate demos from tutorial/example programs. The
> demos should be as they are now: short, simple and self-explaining.
> 
> Then we can have a different set of programs which are explained very
> carefully. It would be natural for these to be part of the tutorial
> (based on the existing tutorial written by Hans Petter).

I share this view. You copy demos and modify to solve a new problem.
Tutorial examples are made for teaching ideas, syntax, etc. The code
therefore often becomes quite different from a self-explaining demo.
 

> > >>2. Both the user manual and tutorial will come in two different
> > >>flavors, one C++ and Python. With some clever use of LaTex \input, it
> > >>should be possible to handle with not too much extra work.

I use a program preprocess (in Python :-) to allow for #ifdef and other
C preprocessor type of commands in latex, shell, doconce, and almost
whatever. Then it's easy to compile a tutorial in one mode. (At the
University of Oslo this is used for compiling course manuals with
Python or Matlab.)

But I doubt that #ifdef or clever \input is the way to go with Python
and C++ in a tutorial. The text will naturally be quite different for
the Python and C++ versions. I think it's better to explain most of
the things in Python first and then provide add-on material for
C++. This can be made quite short with a focus on syntax, and reference
back to where the example is introduced in the Python part. However,
if you have a substantial amount of users who don't want to use or
read about the Python interface, you need a stand-alone C++
tutorial. In my view, FEniCS users should normally use Python for
rapid development, but turn to the C++ interface for speed in a more
comprehensive code. That is, the C++ FEniCS programmers are quite
advanced users already and you can take advantage of that when writing
tutorial material. (For the Python tutorial, I forsee very novice
users who don't know finite elements and Python, but they have a PDE
to solve...it seems that FEniCS can, maybe uexpectedly, work well
for this group of users.)

> > >>6. We need well-documented demos. It's currently unclear what the
> > >>relation is between the manual, documented demos, current demos in
> > >>DOLFIN and the examples in Hans Petter's tutorial so we need to work
> > >>out a model for this.

I suggest to view the manual, tutorial and demos as three different
types of documentation. The tutorial has its example programs, so may
the manual (but I guess there will be more of short code snippets in a
manual), and the demo selection is a third source of information. To
me it seems difficult to maintain these three types of documentation
if they are closely related to each other.

Hans Petter

Attachment: tutorial.pdf
Description: Adobe PDF document

TITLE: Doconce: Document Once, Include Everywhere
AUTHOR: H. P. Langtangen at Simula Research Laboratory and University of Oslo
DATE: July 15, 2009


# lines beginning with # are comment lines

 * When writing a note, report, manual, etc., do you find it difficult
   to choose the typesetting format? That is, to choose between plain
   (email-like) text, Word/OpenOffice, LaTeX, HTML, reStructuredText,
   XML, wiki, etc.  Would it be convenient to start with some very
   simple text-like format that easily converts to the formats listed
   above, and at some later stage eventually go with a particular
   format?

 * Do you find it problematic that you have the same information
   scattered around in different documents in different typesetting
   formats? Would it be a good idea to write things once, in one
   place, and include it everywhere?

If any of these questions are of interest, you should keep on reading.


===== The Doconce Concept  =====

Doconce is two things:

  o Doconce is a working strategy for documenting software in a single
    place and avoiding duplication of information. The slogan is:
    "Write once, include everywhere". This requires that what you
    write can be transformed to many different formats for a variety
    of documents (manuals, tutorials, books, doc strings, source code
    comments, etc.).
    
  o Doconce is a simple and minimally tagged markup language that can
    be used for the above purpose. The Doconce format look
    like ordinary ASCII text (much like what you would use in an
    email), but the text can be transformed to numerous other formats,
    including HTML, wiki, LaTeX, reStructuredText, XML,
    OpenOffice/Word, Epytext, PDF, XML - and even plain text (with
    tags removed for clearer reading).


===== What Does Doconce Look Like? =====

Doconce text looks like ordinary text, but there are some almost invisible
text constructions that allow you to control the formating. For example,

  * bullet lists arise from lines starting with an asterix,

  * *emphasized words* are surrounded by an asterix, 

  * _words in boldface_ are surrounded by underscores, 

  * words from computer code are enclosed in back quotes and 
    then typeset verbatim,

  * blocks of computer code can easily be included, also from source files,

  * blocks of LaTeX mathematics can easily be included,
 
  * there is support oforboth LaTeX and text-like inline mathematics,

  * figures with captions, URLs with links, labels and references
    are supported,

  * comments can be inserted throughout the text,

  * a preprocessor (much like the C preprocessor) is integrated so
    other documents (files) can be included and large portions of text
    can be defined in or out of the text.

Here is an example of some simple text written in the Doconce format:
# We typeset this as a block of verbatim text ("code"), using !bc/!ec tags
# (for begin code / end code).
!bc
===== A Subsection with Sample Text =====

Ordinary text looks like ordinary text, and the tags used for
_boldface_ words, *emphasized* words, and `computer` words look
natural in plain text.  Lists are typeset as you would do in an email,

  * item 1
  * item 2
  * item 3

Lists can also have automatically numbered items instead of bullets,

  o item 1
  o item 2
  o item 3

URLs with a link word are possible, as in http://folk.uio.no/hpl<hpl>.
!ec
The Doconce text above results in the following little document:

===== A Subsection with Sample Text =====

Ordinary text looks like ordinary text, and the tags used for
_boldface_ words, *emphasized* words, and `computer` words look
natural in plain text.  Lists are typeset as you would do in an email,

  * item 1
  * item 2
  * item 3

Lists can also have numbered items instead of bullets, just use an
o (for ordered) instead of the asterix:

  o item 1
  o item 2
  o item 3

URLs with a link word are possible, as in http://folk.uio.no/hpl<hpl>.

===== Mathematics and Computer Code =====

Inline mathematics, such as $\nu = \sin(x)$|$v = sin(x)$,
allows the formula to be specified both as LaTeX and as plain text.
This results in a professional LaTeX typesetting, but in other formats
the text version normally looks better than raw LaTeX mathematics with
backslashes. An inline formula like $\nu = \sin(x)$|$v = sin(x)$ is
typeset as
!bc
$\nu = \sin(x)$|$v = sin(x)$
!ec
The pipe symbol acts as a delimiter between LaTeX code and the plain text
version of the formula.

Blocks of mathematics are better typeset with raw LaTeX, inside
`!bt` and `!et` (begin tex / end tex) instructions. 
The result looks like this:
!bt
\begin{eqnarray}
{\partial u\over\partial t} &=& \nabla^2 u + f,\label{myeq1}\\
{\partial v\over\partial t} &=& \nabla\cdot(q(u)\nabla v) + g
\end{eqnarray}
!et
Of course, such blocks only looks nice in LaTeX. The raw
LaTeX syntax appears in all other formats (but can still be useful
for those who can read LaTeX syntax).

You can have blocks of computer code, starting and ending with
`!bc` and `!ec` instructions, respectively. Such blocks look like
!bc
from math import sin, pi
def myfunc(x):
    return sin(pi*x)

import integrate
I = integrate.trapezoidal(myfunc, 0, pi, 100)
!ec

One can also copy computer code directly from files, either the
complete file or specified parts.  Computer code is then never
duplicated in the documentation (important for the principle
"document once"!).

Another document can be included by writing `#include "mynote.do.txt"`
on a line starting with (another) hash sign.  Doconce documents have
extension `do.txt`. The `do` part stands for doconce, while the
trailing `.txt` denotes a text document so that editors gives you the
right writing enviroment for plain text.


===== Seeing More of What Doconce Is ===== 

After the quick syntax tour above, we recommend to read the Doconce
source of the current tutorial and compare it with what you see in
a browser, a PDF document, in plain text, and so forth.
The Doconce source is found in the folder `doc/tutorial.do.txt` in the
source code tree of Doconce. The Doconce example documentation
displays both the source `tutorial.do.txt` and the result of many other
formats.

A more complete documentation of and motivation for Doconce appears
in the file `lib/doconce/doc/doconce.do.txt` in the Doconce
source code tree. The same documentation appears in the doc string of
the `doconce` module.


===== From Doconce to Other Formats ===== 

Transformation of a Doconce document to various other
formats applies the script `doconce2format`:
!bc
Unix/DOS> doconce2format format doconce-file
!ec
For example, making an HTML version of a Doconce file `mydoc.do.txt`
is performed by
!bc
Unix/DOS> doconce2format HTML mydoc.do.txt
!ec
The resulting file `mydoc.html` can be loaded into any web browser for viewing.

Making a LaTeX and PDF file from `mydoc.do.txt` is done in two steps:

   o Filter the doconce text to a pre-LaTeX form `mydoc.p.tex` for
     `ptex2tex`:
!bc
Unix/DOS> doconce2format LaTeX mydoc.do.txt
!ec
   o Run `ptex2tex` (if you have it) to make a standard LaTeX file,
!bc
Unix/DOS> ptex2tex mydoc
!ec
     or just perform a plain copy,
!bc
Unix/DOS> cp mydoc.p.tex mydoc.tex
!ec
     The `ptex2tex` tool makes it possible to easily switch between many
     different fancy formattings of computer or verbatim code in LaTeX
     documents.
   o Compile `mydoc.tex` the usual way and create the PDF file.

We can go from Doconce "back to" plain untagged text suitable for viewing
in terminal windows, inclusion in email text, or for insertion in
computer source code:
!bc
Unix/DOS> doconce2format plain mydoc.do.txt  # results in mydoc.txt
!ec
Going from Doconce to reStructuredText gives a lot of possibilities to
go to other formats. First we filter the Doconce text to a
reStructuredText file `mydoc.rst`:
!bc
Unix/DOS> doconce2format rst mydoc.do.txt
!ec
We may now produce various other formats:
!bc
Unix/DOS> rst2html.py  mydoc.rst > mydoc.html # HTML
Unix/DOS> rst2latex.py mydoc.rst > mydoc.tex  # LaTeX
Unix/DOS> rst2xml.py   mydoc.rst > mydoc.xml  # XML
Unix/DOS> rst2odt.py   mydoc.rst > mydoc.odt  # OpenOffice
!ec
The OpenOffice file `mydoc.odt` can be loaded into OpenOffice and
saved in, among other things, the RTF format or the Microsoft Word format.
That is, one can easily go from Doconce to Microsoft Word, if desired.

The file `make.sh` in the same directory as the `doconce.do.txt` file
shows how to run `doconce2format`
on the `doconce.do.txt` file to obtain documents in various formats.  To
go from the LaTeX format to PDF, see `latex.sh`.
Running this demo (`make.sh` and `latex.sh`) and studying the various
generated files and comparing them with the original `doconce.do.txt`
file, gives a quick introduction to how Doconce is used in a real case.


===== The Doconce Documentation Strategy for User Manuals ===== 

Doconce was particularly made for writing tutorials or user manuals
associated with computer codes. The text is written in Doconce format
in separate files. LaTeX, HTML, XML, and other versions of the text
is easily produced by the `doconce2format` script and standard tools.
A plain text version is often wanted for the computer source code,
this is easy to make, and then one can use
`#include` statements in the computer source code to automatically
get the manual or tutorial text in comments or doc strings.
Below is a worked example.

Consider a Python module in a `basename.p.py` file.  The `.p.py`
extension identifies this as a file that has to be preprocessed by the
`preprocess` program (`preprocess` is much like the standard C
preprocessor, but it works for TeX/LaTeX, Bash, Python, Perl, Ruby,
Java, etc.).  In a doc string in `basename.p.py` we do a preprocessor
include in a comment line, say
#
# Note: we insert an error here as the the right quote is missing.
# Then preprocess skips the statement, otherwise it gives an error
# message about a missing file doc/doc1.dst.txt (which we don't
# have, it's just a sample file name).
#
!bc
#    #include "doc/doc1.dst.txt
!ec
The file `doc/doc1.dst.txt` is a file filtered to a specific format
(typically plain text or Epytext) from an original "singleton"
documentation file named `doc/doc1.do.txt`. The `.dst.txt` is the 
extension of a file filtered ready for being included in a doc string 
(`d` for doc, `st` for string).

For making an Epydoc manual, the `doc/doc1.do.txt` file is filtered to
`doc/doc1.epytext` and renamed to `doc/doc1.dst.txt`.  Then we run the
preprocessor on the `basename.p.py` file and create a real Python file
`basename.py`. Finally, we run Epydoc on this file.

The next step is to produce the final pure Python source code. For this
purpose we filter `doc/doc1.do.txt` to plain text format (`doc/doc1.txt`)
and rename to `doc/doc1.dst.txt`. The preprocessor transforms the
`basename.p.py` file to a standard Python file `basename.py`. The
doc strings are now in plain text and well suited for Pydoc or reading
by humans. All these steps are automated by the `insertdocstr.py` script.
Here are the corresponding Unix commands:

!bc
# make Epydoc API manual of basename module:
cd doc
doconce2format epytext doc1.do.txt
mv doc1.epytext doc1.dst.txt
cd ..
preprocess basename.p.py > basename.py
epydoc basename

# make ordinary Python module files with doc strings:
cd doc
doconce2format plain doc1.do.txt
mv doc1.txt doc1.dst.txt
cd ..
preprocess basename.p.py > basename.py

# can automate inserting doc strings in all .p.py files:
insertdocstr.py plain .
# (runs through all .do.txt files and filters them to plain format and
# renames to .dst.txt extension, then the script runs through all 
# .p.py files and runs the preprocessor, which includes the .dst.txt
# files)
!ec


======= Warning =======

Doconce can be viewed is a unified interface to a variety of
typesetting formats.  This interface is minimal in the sense that a
lot of typesetting features are not supported, for example, footnotes
and bibliography. For many documents the simple Doconce format is
sufficient, while in other cases you need more sophisticated
formats. Then you can just filter the Doconce text to a more
approprite format and continue working in this format only
(reStructuredText is a good alternative: it is more tagged than
Doconce and cannot be filtered to plain, untagged text, but it also
has a lot more typesetting and tagging features than Doconce).