launchpad-reviewers team mailing list archive

[John A Meinel <john@xxxxxxxxxxxxxxxxx>]

John A Meinel has proposed merging lp:~jameinel/maas/manually-serialize-api into lp:maas.

Commit message:
Change nodegroups/UUID/?op=node_hardware_details=XXX to directly serialize the response, rather than having Piston do it.

It turns out that Piston's default stream is using encode_ascii=False, which has a 10x performance overhead. Presumably it is because it is passing bad regex's over the data to check for characters it doesn't like. This one change essentially halves the time to rebuild a tag (from ~27s for 10,000 nodes down to 14s).
This now properly swings the balance so that the MAAS appserver consumes less CPU in a given request than the celeryd workers processing the content. (With 2 tags being updated concurrently, I get 2 celeryd at 60% cpu, and one maas at 32%cpu).

The other tweak is to change the number of hardware details we request in one batch. We can return 100 nodes in < 100ms. That means that the connection and request overheads start to be a large portion of the conversation. At a batch size of 1000, the celeryd workers go up to 70-80% and the maas server drops to 15-30%. Rebuilding a single tag is down to 12s. Going up to a batch of 5000 still takes 12s to build a tag, though the CPU consumption does hit 100% for the celery worker. However, the extra memory consumed in the content starts showing up (24MB at 1k nodes, 120MB at 5k nodes). 1k nodes seems to be the sweet spot.


Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~jameinel/maas/manually-serialize-api/+merge/129147

This gets rid of another major CPU overhead as part of tag rebuilding.

-- 
https://code.launchpad.net/~jameinel/maas/manually-serialize-api/+merge/129147
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~jameinel/maas/manually-serialize-api into lp:maas.

=== modified file 'src/maasserver/api.py'
--- src/maasserver/api.py	2012-10-10 13:46:34 +0000
+++ src/maasserver/api.py	2012-10-11 10:42:22 +0000
@@ -87,7 +87,7 @@
 from functools import partial
 import httplib
 from inspect import getdoc
-import json
+import simplejson as json
 import sys
 from textwrap import dedent
 
@@ -1209,10 +1209,11 @@
         nodegroup = get_object_or_404(NodeGroup, uuid=uuid)
         if not request.user.is_superuser:
             check_nodegroup_access(request, nodegroup)
-        nodes = Node.objects.filter(
-            system_id__in=system_ids, nodegroup=nodegroup)
-        details = [(node.system_id, node.hardware_details) for node in nodes]
-        return details
+        value_list = Node.objects.filter(
+            system_id__in=system_ids, nodegroup=nodegroup
+            ).values_list('system_id', 'hardware_details')
+        return HttpResponse(
+            json.dumps(list(value_list)), content_type='application/json')
 
 
 DISPLAYED_NODEGROUP_FIELDS = (

=== modified file 'src/provisioningserver/tags.py'
--- src/provisioningserver/tags.py	2012-10-10 15:58:45 +0000
+++ src/provisioningserver/tags.py	2012-10-11 10:42:22 +0000
@@ -41,7 +41,7 @@
     """The MAAS URL or credentials are not yet set."""
 
 
-DEFAULT_BATCH_SIZE = 100
+DEFAULT_BATCH_SIZE = 1000
 
 
 def get_cached_knowledge():