pyreport-dev team mailing list archive

[Gael Varoquaux <gael.varoquaux@xxxxxxxxxxxxxx>]

Hi,

You have showed more or less recently interest in pyreport. After a
recent conversation with Andrew Straw, we decided to set up a mailing
list for pyreport, and use it so that discussions are not lost. This is
the first e-mail sent to the mailling list, and Bcc to people to invite
them to join the mail list.

The mailing list address in pyreport-dev@xxxxxxxxxxxxxxxxxxx. The archive
URL is http://lists.launchpad.net/pyreport-dev/ (though it is currently
empty). I am not too sure how you can join it, but Andrew Straw might
know, and will hopefully post the info on the mailing list, after
joining.

I here copy the e-mail I sent to Andrew a while ago. The short version is
that I am having a lot of hard time maintaining pyreport, because of my
involvmeent in other projects (namely Mayavi2), and these projects seem
to me more important than pyreport. I do have plans for pyreport, as you
can read in the full e-mail I paste below. Whether they will one day come
true remains to be seen. However, the code is on launchpad, in a BSD
license. The latest version of the code on my box (with the refactors not
terribly well tested) is on the chainsaw branch, and can be picked up by
anyone. I don't consider this code as sacred, and would love just anybody
picking it up, and doing something with it.

Thanks to Andrew Straw for pushing me to do this.

********* Full and long original e-mail *************

On Mon, Apr 28, 2008 at 05:30:50PM -0700, Andrew Straw wrote:
> hey, thought i'd let you know i'm hacking:
> http://bazaar.launchpad.net/~astraw/pyreport/devel

Fantastic. That's why shared bzr branches are for.

One piece of warning about pyreport: it was my first project in Python.
It was coded in a real bad way and evolved. It accumulated cruft and I
started refactoring it, but my test suite (retrofitted) didn't cover the
whole program. In addition I don't really use it any more. So there are
big design mistakes (I have idea on how to change them, just not finding
the time), and possible bugs lying around in the bzr branch.

About the future of pyreport: most of the code in pyreport deserves to
die. I want to merge pyreport with Ipython. I want to use Ipython's
execution engine, instead of mine. Ipython will have at some point a
notebook interface. I want to use the same internal representation to
represent the execution block, and rest block, and to share the output
rendering engine, so that the notebook and pyreport can output the same
file formats, using the same codepath. What should be kept of pyreport is
the parsing layer (the code hasher, I call it). Some of the ideas in it
are good, and it is something both very hard to get right, and that no
other project has tackled. One big requirement for it is to accept
unmodified Python input, and to split independant execution block as
small as possible, so that any output of the script appears at the right
place.

About the file format. Currently special comments (rst and control
sequences) are stored in comments. This has proved to be a very bad
choice, mainly because any kind of parser throws away this information.
It is thus hard to reuse modules like the AST module. In pyreport2 (aka
Ipython) I want to change this. The special comments will be replaced by
strings. They will have to be "level 0" blocks, ie in the root of the AST
of the file. This requirement makes things orders of magnitude easier to
code. In addition, it makes it easier to implement a syntax higlighter
for an editor. An example of file format would be something like:

"""--pyreport--
A command string (although I am toying with the idea of killing all the
command strings).
"""
if foo:
    bar
"""--comment--
This is a **rst** comment, if it is not valid rest, it is displayed as
plain text.
"""
print "bar"
# Bar should appear just above this line in the output file.
"""--latex--
This is \LaTeX.
"""

We had a big brainstorming discussion about this at last SciPy with Brian
Granger and Fernando Perez. We pretty much agreed that the above file
format could be a minimalistic way to save a notebook. They want to keep
a lot of information in the notebook, like plots and numerical results.
We decided this should not go in the source file, because it would make
it unreadable. We decided one option was to have a zip file, with a
source file like the above, that had hook to insert more complexe
objects, the objects themselves being pickle in other files in the zip
file. As a result the notebook file (the complete zip file) would also be
a valid Python importable zipped module.

Of course this is far future. I haven't had time to implement these ideas
because I have been busy retrofitting tests to pyreport, and breaking the
monolythic design in smaller bits. But this gives you an idea of what I
would like to do with pyreport.

Unfortunately I am very busy with Mayavi. I am focusing more on Mayavi
because I am afraid that if something doesn't come out with a good
project to do 3D plottinh (ie documented, installable, and that does what
the user wants), people are going to start a new one, and that would be
such a waste. Mayavi has a lot of good things, it is just lacking details
that will make the user happy, because Prabhu doesn't really have
usecases for it. In addition I would really like to establish the ETS as
the standard way to do interactiv scientific computing in Python. We need
to get trust behind a project, an order to be able to build an IDE, and
actually share the work many different projects have been doing with GUIs
(I was stunned to see at last SciPy people reinventing Traits but not as
well). This is also why I am slowly working on a Wx frontend for ipython.
One day all this will fit together.

I am not sure where you got the branch you are working on. Probably on
the launchpad main branch (gael-hosted, I called it). I had local changes
that I wasn't too sure were actually working. I just created a chainsaw
branch that allows me to share them. I have learned working in a group
and the proper tools and best principles only recently (I guess I am
still learning), so the structure of the project isn't the best. Feel
free to hack along, I am not too strongly attached to anything in this
project, whatever makes it move forward is great.

I hope I am not breaking your enthusiasm. I want to be honest about the
state and the future of pyreport. The code is out there, it is BSD, do
what ever you want with it, I'll be happy.

Cheers,

Gaël