launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #12789
[Merge] lp:~gz/maas/populate_hardware_details into lp:maas
Martin Packman has proposed merging lp:~gz/maas/populate_hardware_details into lp:maas.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
For more details, see:
https://code.launchpad.net/~gz/maas/populate_hardware_details/+merge/127551
Migrates existing lshw output from commissioned nodes into hardware_details on Node and fills in cpu_count and memory which depend on its value.
As this is a little unusual, I've stuck some detailed notes in the migration module's docstring rather than just explaining here.
Testing
=======
Migrations don't seem to be part of the automated testing. To test manually, apply:
=== modified file 'src/maasserver/fixtures/dev_fixture.yaml'
--- src/maasserver/fixtures/dev_fixture.yaml 2012-10-02 10:16:52 +0000
+++ src/maasserver/fixtures/dev_fixture.yaml 2012-10-02 12:28:46 +0000
@@ -171,3 +171,28 @@
username: test
model: auth.user
pk: 106
+- model: metadataserver.nodecommissionresult
+ pk: 1015
+ fields:
+ node: 15
+ name: 01-lshw.out
+ data: |
+ <node>
+ <node id="core">
+ <node id="cpu:0" class="processor">
+ <size units="Hz">2000000000</size>
+ </node>
+ <node id="cpu:1" class="processor">
+ <size units="Hz">2000000000</size>
+ </node>
+ <node id="memory" class="memory">
+ <size units="bytes">4294967296</size>
+ </node>
+ </node>
+ </node>
+- model: metadataserver.nodecommissionresult
+ pk: 1115
+ fields: {node: 15, name: 01-lshw.err, data: ''}
+- model: metadataserver.nodecommissionresult
+ pk: 1016
+ fields: {node: 16, name: 01-lshw.out, data: ''}
This is not worth keeping in tree as the general process for filling in sampledata only happens after all migrations are run, so only creating the sampledata without this migration, then running it afterwards exercises the code.
Then test the forwards migration by removing this migration and running it on the sampledata:
$ rm -rf db
$ rm src/metadataserver/migrations/0003_populate_hardware_details.py
$ make sampledata
$ bzr revert src/metadataserver/migrations/0003_populate_hardware_details.py
$ make syncdb
$ bin/database shell
# \c maas
# SELECT id, hardware_details, cpu_count, memory FROM maasserver_node;
# \q
Then test creation of db + sampledata from scratch:
$ rm -rf db
$ make sampledata
Not sure about testing the reverse migration?
--
https://code.launchpad.net/~gz/maas/populate_hardware_details/+merge/127551
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~gz/maas/populate_hardware_details into lp:maas.
=== modified file 'HACKING.txt'
--- HACKING.txt 2012-09-19 05:20:02 +0000
+++ HACKING.txt 2012-10-02 17:35:23 +0000
@@ -429,7 +429,7 @@
to run South's `datamigration`_ command. For instance, if you want to perform
changes to the ``maasserver`` application, run::
- $ ./bin/maas datamigration maasserver --auto description_of_the_change
+ $ ./bin/maas datamigration maasserver description_of_the_change
.. _datamigration: http://south.aeracode.org/docs/commands.html#datamigration
=== modified file 'src/maasserver/fixtures/dev_fixture.yaml'
--- src/maasserver/fixtures/dev_fixture.yaml 2012-10-02 10:16:52 +0000
+++ src/maasserver/fixtures/dev_fixture.yaml 2012-10-02 17:35:23 +0000
@@ -59,7 +59,7 @@
pk: 6
- fields:
name: big
- definition: "LARGER THAN earth"
+ definition: "\"LARGER THAN earth\""
comment: ''
created: 2012-10-02
updated: 2012-10-02
@@ -67,7 +67,7 @@
pk: 30
- fields:
name: planet
- definition: "ORBITS star AND it's complicated"
+ definition: "\"ORBITS star AND it's complicated\""
comment: 'Seriously, "clearing the neighbourhood"...'
created: 2012-10-02
updated: 2012-10-02
@@ -75,7 +75,7 @@
pk: 31
- fields:
name: ringed
- definition: "HAS prominent ring system"
+ definition: "\"HAS prominent ring system\""
comment: 'Excludes the rings of Jupiter.'
created: 2012-10-02
updated: 2012-10-02
=== modified file 'src/maasserver/models/node.py'
--- src/maasserver/models/node.py 2012-10-02 09:30:25 +0000
+++ src/maasserver/models/node.py 2012-10-02 17:35:23 +0000
@@ -323,20 +323,15 @@
_xpath_processor_count = "count(//node[@id='core']/node[@class='processor'])"
-_xpath_memory_bytes = ("//node[@id='memory']/size[@units='bytes']/text()"
- " div 1048576")
-
-
-def update_hardware_details(node, xmlbytes):
+_xpath_memory_bytes = ("//node[@id='memory']/size[@units='bytes'] div 1048576")
+
+
+def update_hardware_details(node, xmlbytes, tag_manager):
"""Set node hardware_details from lshw output and update related fields
- This is designed to be called with individual nodes on commissioning and
- is not optimised for batch updates.
-
- There are a bunch of suboptimal things here:
- * Is a function rather than method in hope south migration can reuse.
- * Doing UPDATE then transaction.commit_unless_managed doesn't work?
- * Scalar returns from xpath() work in postgres 9.2 or later only.
+ This is a helper function just so it can be used in the south migration
+ to do the correct updates to mem and cpu_count when hardware_details is
+ first set.
"""
try:
doc = etree.XML(xmlbytes)
@@ -352,7 +347,7 @@
memory = 0
node.cpu_count = cpu_count or 0
node.memory = memory
- for tag in Tag.objects.all():
+ for tag in tag_manager.all():
has_tag = evaluator(tag.definition)
if has_tag:
node.tags.add(tag)
@@ -690,4 +685,4 @@
def set_hardware_details(self, xmlbytes):
"""Set the `lshw -xml` output"""
- update_hardware_details(self, xmlbytes)
+ update_hardware_details(self, xmlbytes, Tag.objects)
=== added file 'src/metadataserver/migrations/0003_populate_hardware_details.py'
--- src/metadataserver/migrations/0003_populate_hardware_details.py 1970-01-01 00:00:00 +0000
+++ src/metadataserver/migrations/0003_populate_hardware_details.py 2012-10-02 17:35:23 +0000
@@ -0,0 +1,172 @@
+"""Migration for populating new Node fields related to constraints
+
+A shared helper function is used that both the real Node model in maasserver
+and the stubs used by south can work with. This saves writing the logic to
+populate cpu_count and memory in two places. On migration it is likely that
+tags do not exist yet.
+
+If for whatever reason previously stored lshw output is not well formed XML,
+the fields in Node are not populated.
+"""
+from south.v2 import DataMigration
+from django.core.exceptions import ValidationError
+
+from maasserver.models.node import update_hardware_details
+
+
+class Migration(DataMigration):
+
+ _lshw_name = "01-lshw.out"
+
+ def forwards(self, orm):
+ """Move lshw text blob in metadataserver to xml field in Node"""
+ Tag = orm['maasserver.tag']
+ for commission_result in orm.NodeCommissionResult.objects.filter(
+ name=self._lshw_name):
+ node = commission_result.node
+ data = commission_result.data
+ try:
+ update_hardware_details(node, data, Tag.objects)
+ except ValidationError:
+ pass
+ else:
+ node.save()
+
+ def backwards(self, orm):
+ """Move xml field in Node to lshw text blob in metadataserver"""
+ Node = orm['maasserver.Node']
+ for node in Node.objects.all():
+ lshw_output = node.hardware_details
+ if lshw_output:
+ orm.NodeCommissionResult.objects.store_data(
+ node, self._lshw_name, lshw_output)
+
+ models = {
+ 'auth.group': {
+ 'Meta': {'object_name': 'Group'},
+ 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'name': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '80'}),
+ 'permissions': ('django.db.models.fields.related.ManyToManyField', [], {'to': "orm['auth.Permission']", 'symmetrical': 'False', 'blank': 'True'})
+ },
+ 'auth.permission': {
+ 'Meta': {'ordering': "('content_type__app_label', 'content_type__model', 'codename')", 'unique_together': "(('content_type', 'codename'),)", 'object_name': 'Permission'},
+ 'codename': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'content_type': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['contenttypes.ContentType']"}),
+ 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'name': ('django.db.models.fields.CharField', [], {'max_length': '50'})
+ },
+ 'auth.user': {
+ 'Meta': {'object_name': 'User'},
+ 'date_joined': ('django.db.models.fields.DateTimeField', [], {'default': 'datetime.datetime.now'}),
+ 'email': ('django.db.models.fields.EmailField', [], {'unique': 'True', 'max_length': '75', 'blank': 'True'}),
+ 'first_name': ('django.db.models.fields.CharField', [], {'max_length': '30', 'blank': 'True'}),
+ 'groups': ('django.db.models.fields.related.ManyToManyField', [], {'to': "orm['auth.Group']", 'symmetrical': 'False', 'blank': 'True'}),
+ 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'is_active': ('django.db.models.fields.BooleanField', [], {'default': 'True'}),
+ 'is_staff': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
+ 'is_superuser': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
+ 'last_login': ('django.db.models.fields.DateTimeField', [], {'default': 'datetime.datetime.now'}),
+ 'last_name': ('django.db.models.fields.CharField', [], {'max_length': '30', 'blank': 'True'}),
+ 'password': ('django.db.models.fields.CharField', [], {'max_length': '128'}),
+ 'user_permissions': ('django.db.models.fields.related.ManyToManyField', [], {'to': "orm['auth.Permission']", 'symmetrical': 'False', 'blank': 'True'}),
+ 'username': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '30'})
+ },
+ 'contenttypes.contenttype': {
+ 'Meta': {'ordering': "('name',)", 'unique_together': "(('app_label', 'model'),)", 'object_name': 'ContentType', 'db_table': "'django_content_type'"},
+ 'app_label': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'model': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'name': ('django.db.models.fields.CharField', [], {'max_length': '100'})
+ },
+ u'maasserver.node': {
+ 'Meta': {'object_name': 'Node'},
+ 'after_commissioning_action': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
+ 'architecture': ('django.db.models.fields.CharField', [], {'default': "u'i386/generic'", 'max_length': '31'}),
+ 'cpu_count': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
+ 'distro_series': ('django.db.models.fields.CharField', [], {'default': 'None', 'max_length': '10', 'null': 'True', 'blank': 'True'}),
+ 'error': ('django.db.models.fields.CharField', [], {'default': "u''", 'max_length': '255', 'blank': 'True'}),
+ 'hardware_details': ('maasserver.fields.XMLField', [], {'default': 'None', 'null': 'True', 'blank': 'True'}),
+ 'hostname': ('django.db.models.fields.CharField', [], {'default': "u''", 'max_length': '255', 'blank': 'True'}),
+ 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'memory': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
+ 'netboot': ('django.db.models.fields.BooleanField', [], {'default': 'True'}),
+ 'nodegroup': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['maasserver.NodeGroup']", 'null': 'True'}),
+ 'owner': ('django.db.models.fields.related.ForeignKey', [], {'default': 'None', 'to': "orm['auth.User']", 'null': 'True', 'blank': 'True'}),
+ 'power_parameters': ('maasserver.fields.JSONObjectField', [], {'default': "u''", 'blank': 'True'}),
+ 'power_type': ('django.db.models.fields.CharField', [], {'default': "u''", 'max_length': '10', 'blank': 'True'}),
+ 'status': ('django.db.models.fields.IntegerField', [], {'default': '0', 'max_length': '10'}),
+ 'system_id': ('django.db.models.fields.CharField', [], {'default': "u'node-dc867442-0ca0-11e2-9ff0-fa163e46ecd0'", 'unique': 'True', 'max_length': '41'}),
+ 'tags': ('django.db.models.fields.related.ManyToManyField', [], {'to': u"orm['maasserver.Tag']", 'symmetrical': 'False'}),
+ 'token': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['piston.Token']", 'null': 'True'}),
+ 'updated': ('django.db.models.fields.DateTimeField', [], {})
+ },
+ u'maasserver.nodegroup': {
+ 'Meta': {'object_name': 'NodeGroup'},
+ 'api_key': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '18'}),
+ 'api_token': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['piston.Token']", 'unique': 'True'}),
+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
+ 'dhcp_key': ('django.db.models.fields.CharField', [], {'default': "u''", 'max_length': '255', 'blank': 'True'}),
+ 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'name': ('django.db.models.fields.CharField', [], {'max_length': '80', 'blank': 'True'}),
+ 'status': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
+ 'updated': ('django.db.models.fields.DateTimeField', [], {}),
+ 'uuid': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '36'})
+ },
+ u'maasserver.tag': {
+ 'Meta': {'object_name': 'Tag'},
+ 'comment': ('django.db.models.fields.TextField', [], {'blank': 'True'}),
+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
+ 'definition': ('django.db.models.fields.TextField', [], {}),
+ 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'name': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '256'}),
+ 'updated': ('django.db.models.fields.DateTimeField', [], {})
+ },
+ u'metadataserver.nodecommissionresult': {
+ 'Meta': {'unique_together': "((u'node', u'name'),)", 'object_name': 'NodeCommissionResult'},
+ 'data': ('django.db.models.fields.CharField', [], {'max_length': '1048576'}),
+ 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'name': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
+ 'node': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['maasserver.Node']"})
+ },
+ u'metadataserver.nodekey': {
+ 'Meta': {'object_name': 'NodeKey'},
+ 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'key': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '18'}),
+ 'node': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['maasserver.Node']", 'unique': 'True'}),
+ 'token': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['piston.Token']", 'unique': 'True'})
+ },
+ u'metadataserver.nodeuserdata': {
+ 'Meta': {'object_name': 'NodeUserData'},
+ 'data': ('metadataserver.fields.BinaryField', [], {}),
+ 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'node': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['maasserver.Node']", 'unique': 'True'})
+ },
+ 'piston.consumer': {
+ 'Meta': {'object_name': 'Consumer'},
+ 'description': ('django.db.models.fields.TextField', [], {}),
+ 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'key': ('django.db.models.fields.CharField', [], {'max_length': '18'}),
+ 'name': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
+ 'secret': ('django.db.models.fields.CharField', [], {'max_length': '32'}),
+ 'status': ('django.db.models.fields.CharField', [], {'default': "'pending'", 'max_length': '16'}),
+ 'user': ('django.db.models.fields.related.ForeignKey', [], {'blank': 'True', 'related_name': "'consumers'", 'null': 'True', 'to': "orm['auth.User']"})
+ },
+ 'piston.token': {
+ 'Meta': {'object_name': 'Token'},
+ 'callback': ('django.db.models.fields.CharField', [], {'max_length': '255', 'null': 'True', 'blank': 'True'}),
+ 'callback_confirmed': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
+ 'consumer': ('django.db.models.fields.related.ForeignKey', [], {'to': "orm['piston.Consumer']"}),
+ 'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
+ 'is_approved': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
+ 'key': ('django.db.models.fields.CharField', [], {'max_length': '18'}),
+ 'secret': ('django.db.models.fields.CharField', [], {'max_length': '32'}),
+ 'timestamp': ('django.db.models.fields.IntegerField', [], {'default': '1349189580L'}),
+ 'token_type': ('django.db.models.fields.IntegerField', [], {}),
+ 'user': ('django.db.models.fields.related.ForeignKey', [], {'blank': 'True', 'related_name': "'tokens'", 'null': 'True', 'to': "orm['auth.User']"}),
+ 'verifier': ('django.db.models.fields.CharField', [], {'max_length': '10'})
+ }
+ }
+
+ complete_apps = ['metadataserver']
+ symmetrical = True
Follow ups
