yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1281748] [NEW] nova-compute crash due to missing DB column 'compute_nodes_1.metrics'

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Adam Jodock <ajodock@xxxxxxxxx>
Date: Tue, 18 Feb 2014 19:26:58 -0000
Reply-to: Bug 1281748 <1281748@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Doing a fresh install from http://docs.openstack.org/trunk/install-
guide/install/apt/content/index.html  Running Ubuntu 12.04.4 LTS +
cloud-archive:havana PPA as per guide.  The only variation from the
guide is that I have setup a RBD store for Glance.

Here are the package versions:
# dpkg -l | grep nova
ii  nova-ajax-console-proxy          1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - AJAX console proxy - transitional package
ii  nova-api                         1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - API frontend
ii  nova-cert                        1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - certificate management
ii  nova-common                      1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - common files
ii  nova-compute                     1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - compute node
ii  nova-compute-kvm                 1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - compute node (KVM)
ii  nova-conductor                   1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - conductor service
ii  nova-consoleauth                 1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - Console Authenticator
ii  nova-doc                         1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - documentation
ii  nova-network                     1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - Network manager
ii  nova-novncproxy                  1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - NoVNC proxy
ii  nova-scheduler                   1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - virtual machine scheduler
ii  python-nova                      1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute Python libraries
ii  python-novaclient                1:2.15.0-0ubuntu1~cloud0                            client library for OpenStack Compute API

Everything is smooth up to setting up nova-compute.  Other nova services
seem to be working correctly, but when I got to deploying the first
image the instance didn't start.  I am attaching the nova-compute.log
file.  The nova-compute process crashes immediately when you attempt to
start it.

According to the errors it appears that there is a missing field
(metrics) in the compute_nodes table.  The error doesn't mention it, but
the table also appears to be missing the extra_resources column as well.

Here is what my MySQL says the schema is for compute_nodes
CREATE TABLE `compute_nodes` (
  `created_at` datetime DEFAULT NULL,
  `updated_at` datetime DEFAULT NULL,
  `deleted_at` datetime DEFAULT NULL,
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `service_id` int(11) NOT NULL,
  `vcpus` int(11) NOT NULL,
  `memory_mb` int(11) NOT NULL,
  `local_gb` int(11) NOT NULL,
  `vcpus_used` int(11) NOT NULL,
  `memory_mb_used` int(11) NOT NULL,
  `local_gb_used` int(11) NOT NULL,
  `hypervisor_type` mediumtext NOT NULL,
  `hypervisor_version` int(11) NOT NULL,
  `cpu_info` mediumtext NOT NULL,
  `disk_available_least` int(11) DEFAULT NULL,
  `free_ram_mb` int(11) DEFAULT NULL,
  `free_disk_gb` int(11) DEFAULT NULL,
  `current_workload` int(11) DEFAULT NULL,
  `running_vms` int(11) DEFAULT NULL,
  `hypervisor_hostname` varchar(255) DEFAULT NULL,
  `deleted` int(11) DEFAULT NULL,
  `host_ip` varchar(39) DEFAULT NULL,
  `supported_instances` text,
  `pci_stats` text,
  PRIMARY KEY (`id`),
  KEY `fk_compute_nodes_service_id` (`service_id`),
  CONSTRAINT `fk_compute_nodes_service_id` FOREIGN KEY (`service_id`) REFERENCES `services` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8$$

So it would appear that there is no metrics column to select as the
error indicates.  I thought this might have been an issue with `nova-
manage db sync`, so I dropped, recreated, ran `nova-manage db sync`
again on the DB, and restarted the services but I am getting the same
issue.  The nova-manage log for the db schema updates doesn't have
anything that would indicate any issues (schema appears to be at version
216).

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "nova-compute.log"
   https://bugs.launchpad.net/bugs/1281748/+attachment/3985757/+files/nova-compute.log

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1281748

Title:
  nova-compute crash due to missing DB column 'compute_nodes_1.metrics'

Status in OpenStack Compute (Nova):
  New

Bug description:
  Doing a fresh install from http://docs.openstack.org/trunk/install-
  guide/install/apt/content/index.html  Running Ubuntu 12.04.4 LTS +
  cloud-archive:havana PPA as per guide.  The only variation from the
  guide is that I have setup a RBD store for Glance.

  Here are the package versions:
  # dpkg -l | grep nova
  ii  nova-ajax-console-proxy          1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - AJAX console proxy - transitional package
  ii  nova-api                         1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - API frontend
  ii  nova-cert                        1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - certificate management
  ii  nova-common                      1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - common files
  ii  nova-compute                     1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - compute node
  ii  nova-compute-kvm                 1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - compute node (KVM)
  ii  nova-conductor                   1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - conductor service
  ii  nova-consoleauth                 1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - Console Authenticator
  ii  nova-doc                         1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - documentation
  ii  nova-network                     1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - Network manager
  ii  nova-novncproxy                  1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - NoVNC proxy
  ii  nova-scheduler                   1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute - virtual machine scheduler
  ii  python-nova                      1:2013.2.1-0ubuntu1~cloud0                          OpenStack Compute Python libraries
  ii  python-novaclient                1:2.15.0-0ubuntu1~cloud0                            client library for OpenStack Compute API

  Everything is smooth up to setting up nova-compute.  Other nova
  services seem to be working correctly, but when I got to deploying the
  first image the instance didn't start.  I am attaching the nova-
  compute.log file.  The nova-compute process crashes immediately when
  you attempt to start it.

  According to the errors it appears that there is a missing field
  (metrics) in the compute_nodes table.  The error doesn't mention it,
  but the table also appears to be missing the extra_resources column as
  well.

  Here is what my MySQL says the schema is for compute_nodes
  CREATE TABLE `compute_nodes` (
    `created_at` datetime DEFAULT NULL,
    `updated_at` datetime DEFAULT NULL,
    `deleted_at` datetime DEFAULT NULL,
    `id` int(11) NOT NULL AUTO_INCREMENT,
    `service_id` int(11) NOT NULL,
    `vcpus` int(11) NOT NULL,
    `memory_mb` int(11) NOT NULL,
    `local_gb` int(11) NOT NULL,
    `vcpus_used` int(11) NOT NULL,
    `memory_mb_used` int(11) NOT NULL,
    `local_gb_used` int(11) NOT NULL,
    `hypervisor_type` mediumtext NOT NULL,
    `hypervisor_version` int(11) NOT NULL,
    `cpu_info` mediumtext NOT NULL,
    `disk_available_least` int(11) DEFAULT NULL,
    `free_ram_mb` int(11) DEFAULT NULL,
    `free_disk_gb` int(11) DEFAULT NULL,
    `current_workload` int(11) DEFAULT NULL,
    `running_vms` int(11) DEFAULT NULL,
    `hypervisor_hostname` varchar(255) DEFAULT NULL,
    `deleted` int(11) DEFAULT NULL,
    `host_ip` varchar(39) DEFAULT NULL,
    `supported_instances` text,
    `pci_stats` text,
    PRIMARY KEY (`id`),
    KEY `fk_compute_nodes_service_id` (`service_id`),
    CONSTRAINT `fk_compute_nodes_service_id` FOREIGN KEY (`service_id`) REFERENCES `services` (`id`)
  ) ENGINE=InnoDB DEFAULT CHARSET=utf8$$

  So it would appear that there is no metrics column to select as the
  error indicates.  I thought this might have been an issue with `nova-
  manage db sync`, so I dropped, recreated, ran `nova-manage db sync`
  again on the DB, and restarted the services but I am getting the same
  issue.  The nova-manage log for the db schema updates doesn't have
  anything that would indicate any issues (schema appears to be at
  version 216).

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1281748/+subscriptions
Follow ups

[Bug 1281748] Re: nova-compute crash due to missing DB column 'compute_nodes_1.metrics'
From: Sean Dague, 2014-09-03
[Bug 1281748] [NEW] nova-compute crash due to missing DB column 'compute_nodes_1.metrics'
From: Adam Jodock, 2014-02-18
References

[Bug 1281748] [NEW] nova-compute crash due to missing DB column 'compute_nodes_1.metrics'
From: Adam Jodock, 2014-02-18