launchpad-dev team mailing list archive

[John Arbash Meinel <john@xxxxxxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


...

>> Thats a very interesting analysis; I don't see anything there about
>> using a general purpose graph DB  - did you consider that? (e.g.
>> flockdb).
> 
> No.  I'm not familiar with such things, and we weren't thinking outside
> the box.  Everything we considered is listed there.

It would be interesting to evaluate. But I think it does get into the
realm of:
 a) How does this integrate with the rest of Launchpad
 b) Where do these graph dbs live
 c) How are they replicated, scaled, etc.

> 
>> History db seems like a very specific preprocessing answer to me; its
>> nice (very nice!) but I'm wondering if we'll get more leverage in
>> Launchpad by bringing in a slightly heavier tool which can answer all
>> the things we need for bzr straight away
> 
> It is always hard to judge what will be most effective in these cases.
> Such tech would be new and unfamiliar, and would probably be harder to
> deploy initially.  history_db is just SQL, and we know how to handle
> SQL.  But history_db is perfect either, and there are some changes I
> would make if I were working on replacing the BranchRevision table with it.
> 

I assume you mean "isn't perfect", since certainly I don't think it is
perfect either. It seems to be a pretty good starting point, though.

>> , replace our
>> TeamParticipation construct (1.3M rows) , let us do the graph logic
>> needed in derivative distributions, blueprint dependencies (a
>> transitive data structure when the issue tracker arrives), and
>> similarly bug dependencies and project relationships.
> 
> We also have a deadline with the BranchRevision table, as I believe we
> are halfway to exceeding maximum index value.  Increasing the scope of
> this project runs the risk of delaying it to the point that we run out
> of BranchRevision index numbers.
> 
> Aaron

Jeroen did some good work towards getting rid of the index numbers
entirely. AIUI they are now fully removed from the Launchpad code
itself. And the primary blocker is that you can't trivially drop a
primary key column and change it to a new one. (At least on a 500M row
table with a downtime window of <90minutes.)

However, again Jeroen looked at what it would take to modify the
postgres meta tables to simply say "the primary index is now this other
index that we already have on the table", and then you can delete the
primary column and have no-downtime VACUUM runs reclaim the disk space, etc.

As such it gives a much bigger window of time that could be used to
switch to a different tech, rather than just redoing the table design
itself.


My personal experience has been that operational issues like deploying a
new technology into production is *significantly* harder than just
tweaking the tables in a given database.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1AgEIACgkQJdeBCYSNAAOW3wCgqKIgrDov9H4gaVZiYsxj3/Ey
P48An2KQFUzXD6orBDwCx9525Tt1FWNv
=FMTX
-----END PGP SIGNATURE-----