yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

*** This bug is a duplicate of bug 1255624 ***
    https://bugs.launchpad.net/bugs/1255624

** This bug is no longer a duplicate of bug 1257644
   gate-tempest-dsvm-postgres-full fails - unable to schedule instance
** This bug has been marked a duplicate of bug 1255624
   libvirtError: Unable to read from monitor: Connection reset by peer

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1244024

Title:
  devstack gate fail "is disabled or has not been heard from in a while"
  but last_heartbeat was 0.2 seconds ago

Status in OpenStack Compute (Nova):
  New

Bug description:
  From n-sch.log:

  enant] Attempting to build 1 instance(s) uuids: [u'e1bef59e-d005-4ec2-bfcf-222f97a035e0']
  2013-10-24 02:22:03.711 DEBUG nova.scheduler.filter_scheduler [req-9d2d3dc3-1376-4214-8c2d-8ab2c1740c43 ServersTestJSON-tempest-197827455-user ServersTestJSON-tempest-197827455-tenant] Request Spec: {u'num_instances': 1, u'block_device_mapping': [{u'instance_uuid': u'e1bef59e-d005-4ec2-bfcf-222f97a035e0', u'guest_format': None, u'boot_index': 0, u'delete_on_termination': True, u'no_device': None, u'connection_info': None, u'snapshot_id': None, u'device_name': None, u'disk_bus': None, u'image_id': u'6e6f217d-3b49-4cf6-80ec-b87e6987d498', u'source_type': u'image', u'device_type': u'disk', u'volume_id': None, u'destination_type': u'local', u'volume_size': None}], u'image': {u'status': u'active', u'name': u'cirros-0.3.1-x86_64-uec', u'deleted': False, u'container_format': u'ami', u'created_at': u'2013-10-24T02:18:58.141185', u'disk_format': u'ami', u'updated_at': u'2013-10-24T02:18:58.702888', u'id': u'6e6f217d-3b49-4cf6-80ec-b87e6987d498', u'owner': u'd207ccda9dce4b769577b52da53853ae', u'min_ram': 0, u'checksum': u'f8a2eeee2dc65b3d9b6e63678955bd83', u'min_disk': 0, u'is_public': True, u'deleted_at': None, u'properties': {u'kernel_id': u'e63a677f-bac1-499e-849a-6378a01946bf', u'ramdisk_id': u'0c6ff9e9-318f-42c5-a873-d8e005fe9a33'}, u'size': 25165824}, u'instance_type': {u'root_gb': 0, u'name': u'm1.nano', u'ephemeral_gb': 0, u'memory_mb': 64, u'vcpus': 1, u'extra_specs': {}, u'swap': 0, u'rxtx_factor': 1.0, u'flavorid': u'42', u'vcpu_weight': None, u'id': 6}, u'instance_properties': {u'vm_state': u'building', u'availability_zone': None, u'terminated_at': None, u'ephemeral_gb': 0, u'instance_type_id': 6, u'user_data': None, u'cleaned': False, u'vm_mode': None, u'deleted_at': None, u'reservation_id': u'r-aox9119r', u'id': 26, u'security_groups': {u'objects': [{u'project_id': u'c7ad4d764ad3478995b8c5d07ed61e53', u'user_id': u'5090f7e679224462b3b122639af89627', u'description': u'default', u'deleted': False, u'created_at': u'2013-10-24T02:22:03.517832', u'updated_at': None, u'deleted_at': None, u'id': 20, u'name': u'default'}]}, u'disable_terminate': False, u'display_name': u'server-tempest-354874675', u'uuid': u'e1bef59e-d005-4ec2-bfcf-222f97a035e0', u'default_swap_device': None, u'info_cache': {u'instance_uuid': u'e1bef59e-d005-4ec2-bfcf-222f97a035e0', u'network_info': []}, u'hostname': u'server-tempest-354874675', u'launched_on': None, u'display_description': u'server-tempest-354874675', u'key_data': None, u'kernel_id': u'e63a677f-bac1-499e-849a-6378a01946bf', u'power_state': 0, u'default_ephemeral_device': None, u'progress': 0, u'project_id': u'c7ad4d764ad3478995b8c5d07ed61e53', u'launched_at': None, u'config_drive': u'', u'node': None, u'ramdisk_id': u'0c6ff9e9-318f-42c5-a873-d8e005fe9a33', u'access_ip_v6': u'::babe:dc0c:1602', u'access_ip_v4': u'1.1.1.1', u'deleted': False, u'key_name': None, u'updated_at': None, u'host': None, u'architecture': None, u'user_id': u'5090f7e679224462b3b122639af89627', u'system_metadata': {u'image_kernel_id': u'e63a677f-bac1-499e-849a-6378a01946bf', u'image_min_disk': u'0', u'instance_type_memory_mb': u'64', u'instance_type_swap': u'0', u'instance_type_vcpu_weight': None, u'instance_type_root_gb': u'0', u'instance_type_name': u'm1.nano', u'image_ramdisk_id': u'0c6ff9e9-318f-42c5-a873-d8e005fe9a33', u'instance_type_id': u'6', u'instance_type_ephemeral_gb': u'0', u'instance_type_rxtx_factor': u'1.0', u'instance_type_flavorid': u'42', u'instance_type_vcpus': u'1', u'image_container_format': u'ami', u'image_min_ram': u'0', u'image_disk_format': u'ami', u'image_base_image_ref': u'6e6f217d-3b49-4cf6-80ec-b87e6987d498'}, u'task_state': u'scheduling', u'shutdown_terminate': False, u'cell_name': None, u'root_gb': 0, u'locked': False, u'name': u'instance-0000001a', u'created_at': u'2013-10-24T02:22:03.529998', u'locked_by': None, u'launch_index': 0, u'metadata': {u'hello': u'world'}, u'memory_mb': 64, u'vcpus': 1, u'image_ref': u'6e6f217d-3b49-4cf6-80ec-b87e6987d498', u'root_device_name': None, u'auto_disk_config': True, u'os_type': None, u'scheduled_at': None}, u'security_group': [u'default'], u'instance_uuids': [u'e1bef59e-d005-4ec2-bfcf-222f97a035e0']} schedule_run_instance /opt/stack/new/nova/nova/scheduler/filter_scheduler.py:82
  2013-10-24 02:22:03.720 DEBUG nova.filters [req-9d2d3dc3-1376-4214-8c2d-8ab2c1740c43 ServersTestJSON-tempest-197827455-user ServersTestJSON-tempest-197827455-tenant] Starting with 1 host(s) get_filtered_objects /opt/stack/new/nova/nova/filters.py:70
  2013-10-24 02:22:03.720 DEBUG nova.scheduler.filters.retry_filter [req-9d2d3dc3-1376-4214-8c2d-8ab2c1740c43 ServersTestJSON-tempest-197827455-user ServersTestJSON-tempest-197827455-tenant] Host [u'devstack-precise-hpcloud-az2-603172', u'devstack-precise-hpcloud-az2-603172'] passes.  Previously tried hosts: [] host_passes /opt/stack/new/nova/nova/scheduler/filters/retry_filter.py:45
  2013-10-24 02:22:03.720 DEBUG nova.filters [req-9d2d3dc3-1376-4214-8c2d-8ab2c1740c43 ServersTestJSON-tempest-197827455-user ServersTestJSON-tempest-197827455-tenant] Filter RetryFilter returned 1 host(s) get_filtered_objects /opt/stack/new/nova/nova/filters.py:88
  2013-10-24 02:22:03.721 DEBUG nova.filters [req-9d2d3dc3-1376-4214-8c2d-8ab2c1740c43 ServersTestJSON-tempest-197827455-user ServersTestJSON-tempest-197827455-tenant] Filter AvailabilityZoneFilter returned 1 host(s) get_filtered_objects /opt/stack/new/nova/nova/filters.py:88
  2013-10-24 02:22:03.721 DEBUG nova.filters [req-9d2d3dc3-1376-4214-8c2d-8ab2c1740c43 ServersTestJSON-tempest-197827455-user ServersTestJSON-tempest-197827455-tenant] Filter RamFilter returned 1 host(s) get_filtered_objects /opt/stack/new/nova/nova/filters.py:88
  2013-10-24 02:22:03.721 DEBUG nova.servicegroup.api [req-9d2d3dc3-1376-4214-8c2d-8ab2c1740c43 ServersTestJSON-tempest-197827455-user ServersTestJSON-tempest-197827455-tenant] Check if the given member [{u'binary': u'nova-compute', u'deleted': 0, u'created_at': datetime.datetime(2013, 10, 24, 2, 18, 32, 298728), u'updated_at': datetime.datetime(2013, 10, 24, 2, 22, 3, 692980), u'report_count': 21, u'topic': u'compute', u'host': u'devstack-precise-hpcloud-az2-603172', u'disabled': True, u'deleted_at': None, u'disabled_reason': None, u'id': 5}] is part of the ServiceGroup, is up service_is_up /opt/stack/new/nova/nova/servicegroup/api.py:94
  2013-10-24 02:22:03.722 DEBUG nova.servicegroup.drivers.db [req-9d2d3dc3-1376-4214-8c2d-8ab2c1740c43 ServersTestJSON-tempest-197827455-user ServersTestJSON-tempest-197827455-tenant] DB_Driver.is_up last_heartbeat = 2013-10-24 02:22:03.692980 elapsed = 0.028868 is_up /opt/stack/new/nova/nova/servicegroup/drivers/db.py:70
  2013-10-24 02:22:03.722 DEBUG nova.scheduler.filters.compute_filter [req-9d2d3dc3-1376-4214-8c2d-8ab2c1740c43 ServersTestJSON-tempest-197827455-user ServersTestJSON-tempest-197827455-tenant] (devstack-precise-hpcloud-az2-603172, devstack-precise-hpcloud-az2-603172) ram:6770 disk:208896 io_ops:1 instances:11 is disabled or has not been heard from in a while host_passes /opt/stack/new/nova/nova/scheduler/filters/compute_filter.py:44
  2013-10-24 02:22:03.722 INFO nova.filters [req-9d2d3dc3-1376-4214-8c2d-8ab2c1740c43 ServersTestJSON-tempest-197827455-user ServersTestJSON-tempest-197827455-tenant] Filter ComputeFilter returned 0 hosts
  2013-10-24 02:22:03.723 WARNING nova.scheduler.driver [req-9d2d3dc3-1376-4214-8c2d-8ab2c1740c43 ServersTestJSON-tempest-197827455-user ServersTestJSON-tempest-197827455-tenant] [instance: e1bef59e-d005-4ec2-bfcf-222f97a035e0] Setting instance to ERROR state.

  It is not at all clear to me why this compute node returned as
  "disabled or not heard from in a while". Anyway, it caused a gate
  failure here:

  http://logs.openstack.org/35/52835/4/gate/gate-tempest-devstack-vm-
  postgres-full/f93e20b/

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1244024/+subscriptions