maas-devel team mailing list archive

[Julian Edwards <julian.edwards@xxxxxxxxxxxxx>]

Hi

Gavin also has a proposal to come, but this is the basis of what I discussed 
with Robert earlier.

The main aim is hyperscale. That is, we want to support hundreds of thousands 
of nodes.  Cobbler simply doesn't allow us to do that.  In addition we are 
currently tied into driving it synchronously so that we can be sure it synced 
its own database, which causes bugs like: 
https://bugs.launchpad.net/maas/+bug/989355

The other main issue surrounds DHCP/TFTP and PXE booting.  When we write out 
config files for these services we need to make sure there are no conflicts 
with multiple processes/threads doing the same thing.

Finally, we need to separate appserver requests from provisioning tasks 
because the latter can be long running (as per the above bug).

To this end we discussed:

 * Use Celery as a task queue (can use Rabbit or Django's DB as a broker)
  → Django has a Celery plugin which makes this very easy
 * Have one queue per pserv process
 * Each pserv is responsible for servicing the tasks on that queue
 * It scales by having multiple queues each with another pserv, or just 
multiple pservs pulling from one queue.
 * The pserv can read the database by using Django's ORM, and write to it by 
sending API requests back to Django's appserver threads (again, scalable).

We also talked about batching requests on the Rabbit task queues because we 
might want to accumulate several requests that change the configs for DHCP 
etc.  For now though we can process serially and optimise later.  We'd need to 
think about how to implement locking though if there are multiple writers to 
the same files.

J