maas-devel team mailing list archive

[John Arbash Meinel <john@xxxxxxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

With 16 Cluster Controllers, we can achieve CPU saturation on the
Database with a shared Appserver (19.2s rebuild). Moving the appserver
can relieve some of this overhead, but it adds bandwidth
considerations between the DB and Appserver. With 'hostnossl'
separating the appserver actually slows down rebuilding tags because
the network becomes saturated at 900+Mbps (24.6s). Switching to
'hostssl' increases some CPU overhead on the Database machine, but is
a net win (17.7s).

Enabling a second Appserver is not directly a benefit (19.1s). As a
quirk of our topology, you can run 1 appserver in 'low cpu high
bandwidth nossl' mode, and one in 'higher cpu low bandwidth ssl' mode,
and maximize the DB's available resources (15.1s).

In all cases, enabling Gzip between the appserver and the cluster
controllers is a big win (26s for 2 appservers in ssl mode, and 35s
for 2 appservers in nossl mode when gzip is disabled.)


So at least for *tags*, we really only get a 1:1 scaleout of
Appservers to Databases (putting them on different hosts is a win, but
the DB is already busy enough that sticking more appservers in front
of it does not improve things significantly).

Note that the best case where we aren't CPU or Bandwidth constrained,
the appserver is colocated with the db, and we are only rebuilding 1
cluster controller's data, it takes 12.5s. Scaling up to 64k nodes and
16 cluster controllers and splitting the appserver onto another
machine, we drop down to 17.7s rebuilding all of those nodes.

So we are able to throw a little bit more hardware at the system, but
we aren't able to scale out the region controller appservers much more
horizontally than the DB itself. And I don't think MaaS currently
would 'just work' with a master & slave postgres setup.

So the next thing to potentially try would be moving the DB server up
from a c1.xlarge into a cluster computing node with 16 CPUs.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlCHzecACgkQJdeBCYSNAANjxACfT7kVCeCdGF8Xcl5EUcpfaHD/
k44AoIVSI9BMA4hGt5fFIyb7E9oBnSA0
=LfGb
-----END PGP SIGNATURE-----