openstack team mailing list archive

[Joshua Harlow <harlowja@xxxxxxxxxxxxx>]

Hmmm, so the RNG schemas aren't stable? Is that basically the problem there (or part of it)? That seems not so good, since I thought the whole point of publishing schemas was for people to use them, darn :(

This libvirt-gconfig does sound good though, is there anyway we can get the dependencies relaxed to versions that other distributions can actually handle (without having more dependencies that need to be custom built). Is there a need for the glib dependency to be that recent (or is the gobject introspection-stuff just that new?).

Still though I think what the above does is still just provide the lower level or the idea I was thinking:

The three levels were:


 1.  Object format that contains methods/properties for exactly what we use with libvirt
    *   Not connected to #2 or #3 in any way
 2.  Formatter layer that takes in #1 and outputs a string/file (or something similar) using various #3 lower level formats
    *   One formatter could be a TempitaLibvirtFormatter
    *   Second could be RngLibvirtFormatter (or GconfigLibvirtFormatter when that happens...)
 3.  Lower level objects/libraries
    *   This would be where RNG->python objects would live or the libvirt-gconfig objects
    *   This could also use a tempita library

Right now basically there is libvirt/connection.py which interacts with #3 (tempita), instead of interacting with #1. So this could be phased, get #1, #2, #3 working with the current stuff (actually a simplified tempita since I really want to get rid of the usage of tempita as a mini-scripting language, since the last time I checked we are in python to begin with). #3 could then use this simplified tempita template, until this libvirt-gconfig comes along (is there a timeline for that?).

Thoughts?

-Josh

On 1/5/12 6:46 AM, "Daniel P. Berrange" <berrange@xxxxxxxxxx> wrote:

On Tue, Jan 03, 2012 at 11:17:45AM -0800, Joshua Harlow wrote:
> I was wondering if there has been any thought or consideration of removing tempita and replacing it with "just python".
> Personally the current tempita usage (libvirt.xml.template) seems to be heading down a hairy path and I wanted to see others opinions on say replacing this with something that doesn't require a whole templating language to use. Some of this may just be my bias against templating languages from experience in different projects @ yahoo (they always start to get hairy, especially when u start to code in them).
>
> Some thoughts:
>
>
>  1.  Assuming we can get a libvirt.domain.xsd (?) we can use a xsd->object model
>  utility to transform that xsd into a python object model (there seem to be a couple of these?)
>     *   http://www.rexx.com/~dkuhlman/generateDS.html or http://pyxsd.org/ (or something else?)

libvirt uses RNG for describing its schemas [1] rather than XSD. That aside though,
I'm not really convinced that this is a good idea. Regardless of which schema
langauge is used, there are always multiple different ways to describe the
same overall concept. In libvirt we have often re-structed our schemas to
express things in a different way. If you are generating APIs / object models
from the schema then, AFAICT, your generated API is liable change in a non
backwards compatible manner.

>  2.  Create a exposed "tree" representation of the sections of the libvirt domain
>  xml that we are interested in (and only those that we are interested in) as python
> objects and have current code create these objects (which right now is basically a
> set of python hashes getting sent to the tempita library)
>  3.  Pass the root element of this exposed "tree" representation to a formatter class (which itself could use pyxsd objects, or tempita - for backward compatibility, or something else, but I have a strong preference for keeping a single language in use, instead of a tempita language and a python language).
>  4.  Write output created by formatter class to domain.xml file (and continue as normal).
>  5.  Profit!
>
> Some of the benefits I think exist with this:
>
>
>  1.  XML escaping will actually happen (does this happen right now?)

AFAICT, no, and this is a security exploit just waiting to happen, if
indeed the code isn't already vulnerable.

>  2.  We can have a underlying object layer which comes directly from the libvirt.domain.xsd (and possibly have versions of this to work with different libvirt versions)
>  3.  We can have an exposed object layer which will attempt to be version independent of the underlying layer and only contain methods/properties that we will use with libvirt (ie the xsd will have many properties/fields we will not use, thus we should not expose them).
>  4.  We can have a formatter layer that will know how to use this exposed layer and return a object that can convert the exposed layer into a string, thus allowing for different implementations (or at least a separation of what is exposed, how its formatted and what the formatter internally uses).
>  5.  We can have the if statements and loops and such that are starting to get put in the template code in python code (thus u don't have to context switch into a templating language to make changes, thus making it easier to work with libvirt).
>  6.  Possible remove a dependency (always good).
>
> Thoughts?

In general, I agree with your suggestion that Nova should not generate
XML docs directly, but have object based API to ensure correctly structured
and escape XML.

The upstream libvirt community is working on a new library (libvirt-gconfig [2])
which directly exposes libvirt XML formats via an object oriented API. This
allows apps to read & write libvirt XML configuration documents, without
having to know anything about XML.

The library is written in C, and uses the GObject library as its base. Via
the GObject introspection support, this trivially provides access to the
API from Python, Perl, Php, JavaScript, Java, Vala, and many more, without
having to manually write bindings for each language. It does not directly
depend on the libvirt library itself, so you can use libvirt-gconfig even
in situations where you don't have a connection to libvirt. eg, if you're
using libvirt indirectly via CIM, SNMP or AMQP, you can still use the
API to deal with libvirt XML documents.

We're using this API in libvirt-sandbox, GNOME Boxes, and plan to eventually
port virt-install & virt-manager too. Our intent is that any app which deals
with libvirt that wants to read/write XML can use this API.

I was in fact planning to suggest use of libvirt-gconfig for Nova in the
near future. The main unknown here is around dependancies.

The GObject introspection code, required to use libvirt-gconfig from python,
requires a fairly new version of GLib, which most enterprise distros will
not have yet.  For Fedora this would be Fedora >= 15

Regards,
Daniel


[1] http://libvirt.org/git/?p=libvirt.git;a=tree;f=docs/schemas;hb=HEAD
[2] https://www.redhat.com/archives/libvirt-announce/2011-December/msg00003.html
--
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|