launchpad-dev team mailing list archive

[John Arbash Meinel <john@xxxxxxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


...
> We have queries on it that bring back 200000 rows to do set
> differences; thats 10 times the data for the fact table - and its
> being joined on awkwardly; making those do a graphtraversal in the
> query (SELECT WITH RECURSIVE) should get it processing a few thousand
> rows most of the time - which will be a lot faster.

I'm not sure if you can do set intersection via recursive queries in
postgres, I would imagine it is possible, but probably runs into the
same issues we've run into in bzr (race conditions in simple graph
traversal can be ugly). (Stuff like X being found as an ancestor of one
branch before the other, and you stop walking some of the common
ancestors to find that it is really common to both.)

history-db taught be a lot about possibly layering here. Adding
'generation' (greatest-distance-from-origin) can help a ton, since if
you've gone far enough back, you *know* you've found everything that can
be uncommon, etc.

Anyway, my point is that to do accurate set intersection, postgres may
still need to touch those 200k rows as the data exists today, even if it
only returns 10 rows when it is done.

The fact that you actually shrink to 1/3rd the size instead of grow to
3x the size is a great hint that this will work. The actual "matrix" is
huge, but there are a lot of zeros, so having it as a sparse matrix
works great.

I don't really see people taking something with 100k bugs, and actually
milestoning all of them and tagging all of them. Unless we get much
better at automatic scripting for it. As long as a human is actually
making the decision, I think your scaling factors are probably under
control.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk2msrEACgkQJdeBCYSNAAOiVwCfY4qD4/gn9Jyo6P7vsyM2w79p
YeEAoJQNIC6pqA7oBTy+y5k3Nl53vcZY
=YDH0
-----END PGP SIGNATURE-----