yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1634925] Re: nova api error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Thu, 20 Oct 2016 22:00:53 -0000
Reply-to: Bug 1634925 <1634925@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Looks like this is your problem:

2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters [req-328a55d5-b96d-46e7-9d77-90bd1c03eb3a 58931d6e7ca247a2be2dd7fac6ee9927 869d2cede19d483e8565a6984202a37a - - -] DBAPIError exception wrapped from (pymysql.err.InternalError) (23, u"Out of resources when opening file '/tmp/#sql_87a4_0.MAI' (Errcode: 24)") [SQL: u'SELECT anon_1.instances_created_at AS anon_1_instances_created_at, anon_1.instances_updated_at AS anon_1_instances_updated_at, anon_1.instances_deleted_at AS anon_1_instances_deleted_at, anon_1.instances_deleted AS anon_1_instances_deleted, anon_1.instances_id AS anon_1_instances_id, anon_1.instances_user_id AS anon_1_instances_user_id, anon_1.instances_project_id AS anon_1_instances_project_id, anon_1.instances_image_ref AS anon_1_instances_image_ref, anon_1.instances_kernel_id AS anon_1_instances_kernel_id, anon_1.instances_ramdisk_id AS anon_1_instances_ramdisk_id, anon_1.instances_hostname AS anon_1_instances_hostname, anon_1.instances_launch_index AS anon_1_instances_launch_index, anon_1.instances_key_name AS anon_1_instances_key_name, anon_1.instances_key_data AS anon_1_instances_key_data, anon_1.instances_power_state AS anon_1_instances_power_state, anon_1.instances_vm_state AS anon_1_instances_vm_state, anon_1.instances_task_state AS anon_1_instances_task_state, anon_1.instances_memory_mb AS anon_1_instances_memory_mb, anon_1.instances_vcpus AS anon_1_instances_vcpus, anon_1.instances_root_gb AS anon_1_instances_root_gb, anon_1.instances_ephemeral_gb AS anon_1_instances_ephemeral_gb, anon_1.instances_ephemeral_key_uuid AS anon_1_instances_ephemeral_key_uuid, anon_1.instances_host AS anon_1_instances_host, anon_1.instances_node AS anon_1_instances_node, anon_1.instances_instance_type_id AS anon_1_instances_instance_type_id, anon_1.instances_user_data AS anon_1_instances_user_data, anon_1.instances_reservation_id AS anon_1_instances_reservation_id, anon_1.instances_launched_at AS anon_1_instances_launched_at, anon_1.instances_terminated_at AS anon_1_instances_terminated_at, anon_1.instances_availability_zone AS anon_1_instances_availability_zone, anon_1.instances_display_name AS anon_1_instances_display_name, anon_1.instances_display_description AS anon_1_instances_display_description, anon_1.instances_launched_on AS anon_1_instances_launched_on, anon_1.instances_locked AS anon_1_instances_locked, anon_1.instances_locked_by AS anon_1_instances_locked_by, anon_1.instances_os_type AS anon_1_instances_os_type, anon_1.instances_architecture AS anon_1_instances_architecture, anon_1.instances_vm_mode AS anon_1_instances_vm_mode, anon_1.instances_uuid AS anon_1_instances_uuid, anon_1.instances_root_device_name AS anon_1_instances_root_device_name, anon_1.instances_default_ephemeral_device AS anon_1_instances_default_ephemeral_device, anon_1.instances_default_swap_device AS anon_1_instances_default_swap_device, anon_1.instances_config_drive AS anon_1_instances_config_drive, anon_1.instances_access_ip_v4 AS anon_1_instances_access_ip_v4, anon_1.instances_access_ip_v6 AS anon_1_instances_access_ip_v6, anon_1.instances_auto_disk_config AS anon_1_instances_auto_disk_config, anon_1.instances_progress AS anon_1_instances_progress, anon_1.instances_shutdown_terminate AS anon_1_instances_shutdown_terminate, anon_1.instances_disable_terminate AS anon_1_instances_disable_terminate, anon_1.instances_cell_name AS anon_1_instances_cell_name, anon_1.instances_internal_id AS anon_1_instances_internal_id, anon_1.instances_cleaned AS anon_1_instances_cleaned, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, security_group_rules_1.created_at AS security_group_rules_1_created_at, security_group_rules_1.updated_at AS security_group_rules_1_updated_at, security_group_rules_1.deleted_at AS security_group_rules_1_deleted_at, security_group_rules_1.deleted AS security_group_rules_1_deleted, security_group_rules_1.id AS security_group_rules_1_id, security_group_rules_1.parent_group_id AS security_group_rules_1_parent_group_id, security_group_rules_1.protocol AS security_group_rules_1_protocol, security_group_rules_1.from_port AS security_group_rules_1_from_port, security_group_rules_1.to_port AS security_group_rules_1_to_port, security_group_rules_1.cidr AS security_group_rules_1_cidr, security_group_rules_1.group_id AS security_group_rules_1_group_id, instance_extra_1.numa_topology AS instance_extra_1_numa_topology, instance_extra_1.flavor AS instance_extra_1_flavor, instance_extra_1.created_at AS instance_extra_1_created_at, instance_extra_1.updated_at AS instance_extra_1_updated_at, instance_extra_1.deleted_at AS instance_extra_1_deleted_at, instance_extra_1.deleted AS instance_extra_1_deleted, instance_extra_1.id AS instance_extra_1_id, instance_extra_1.instance_uuid AS instance_extra_1_instance_uuid, pci_devices_1.created_at AS pci_devices_1_created_at, pci_devices_1.updated_at AS pci_devices_1_updated_at, pci_devices_1.deleted_at AS pci_devices_1_deleted_at, pci_devices_1.deleted AS pci_devices_1_deleted, pci_devices_1.id AS pci_devices_1_id, pci_devices_1.compute_node_id AS pci_devices_1_compute_node_id, pci_devices_1.address AS pci_devices_1_address, pci_devices_1.vendor_id AS pci_devices_1_vendor_id, pci_devices_1.product_id AS pci_devices_1_product_id, pci_devices_1.dev_type AS pci_devices_1_dev_type, pci_devices_1.dev_id AS pci_devices_1_dev_id, pci_devices_1.label AS pci_devices_1_label, pci_devices_1.status AS pci_devices_1_status, pci_devices_1.request_id AS pci_devices_1_request_id, pci_devices_1.extra_info AS pci_devices_1_extra_info, pci_devices_1.instance_uuid AS pci_devices_1_instance_uuid, pci_devices_1.numa_node AS pci_devices_1_numa_node, pci_devices_1.parent_addr AS pci_devices_1_parent_addr, instance_info_caches_1.created_at AS instance_info_caches_1_created_at, instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at, instance_info_caches_1.deleted_at AS instance_info_caches_1_deleted_at, instance_info_caches_1.deleted AS instance_info_caches_1_deleted, instance_info_caches_1.id AS instance_info_caches_1_id, instance_info_caches_1.network_info AS instance_info_caches_1_network_info, instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid, instance_metadata_1.created_at AS instance_metadata_1_created_at, instance_metadata_1.updated_at AS instance_metadata_1_updated_at, instance_metadata_1.deleted_at AS instance_metadata_1_deleted_at, instance_metadata_1.deleted AS instance_metadata_1_deleted, instance_metadata_1.id AS instance_metadata_1_id, instance_metadata_1.`key` AS instance_metadata_1_key, instance_metadata_1.value AS instance_metadata_1_value, instance_metadata_1.instance_uuid AS instance_metadata_1_instance_uuid, instance_system_metadata_1.created_at AS instance_system_metadata_1_created_at, instance_system_metadata_1.updated_at AS instance_system_metadata_1_updated_at, instance_system_metadata_1.deleted_at AS instance_system_metadata_1_deleted_at, instance_system_metadata_1.deleted AS instance_system_metadata_1_deleted, instance_system_metadata_1.id AS instance_system_metadata_1_id, instance_system_metadata_1.`key` AS instance_system_metadata_1_key, instance_system_metadata_1.value AS instance_system_metadata_1_value, instance_system_metadata_1.instance_uuid AS instance_system_metadata_1_instance_uuid \nFROM (SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.hostname AS instances_hostname, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.ephemeral_key_uuid AS instances_ephemeral_key_uuid, instances.host AS instances_host, instances.node AS instances_node, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.locked_by AS instances_locked_by, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instances.cell_name AS instances_cell_name, instances.internal_id AS instances_internal_id, instances.cleaned AS instances_cleaned \nFROM instances \nWHERE instances.deleted = %s AND instances.uuid = %s \n LIMIT %s) AS anon_1 LEFT OUTER JOIN (security_group_instance_association AS security_group_instance_association_1 INNER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s) ON security_group_instance_association_1.instance_uuid = anon_1.instances_uuid AND anon_1.instances_deleted = %s LEFT OUTER JOIN security_group_rules AS security_group_rules_1 ON security_group_rules_1.parent_group_id = security_groups_1.id AND security_group_rules_1.deleted = %s LEFT OUTER JOIN instance_extra AS instance_extra_1 ON instance_extra_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN pci_devices AS pci_devices_1 ON pci_devices_1.instance_uuid = anon_1.instances_uuid AND pci_devices_1.deleted = %s LEFT OUTER JOIN instance_info_caches AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN instance_metadata AS instance_metadata_1 ON instance_metadata_1.instance_uuid = anon_1.instances_uuid AND instance_metadata_1.deleted = %s LEFT OUTER JOIN instance_system_metadata AS instance_system_metadata_1 ON anon_1.instances_uuid = instance_system_metadata_1.instance_uuid'] [parameters: (0, u'd64ad8d8-9397-4da4-a8cc-57371654720a', 1, 0, 0, 0, 0, 0, 0)]
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters     context)
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 146, in execute
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters     result = self._query(query)
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 296, in _query
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters     conn.query(q)
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 781, in query
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 942, in _read_query_result
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters     result.read()
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1138, in read
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters     first_packet = self.connection._read_packet()
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 906, in _read_packet
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters     packet.check_error()
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 367, in check_error
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters     err.raise_mysql_exception(self._data)
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters     _check_mysql_exception(errinfo)
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters     raise InternalError(errno, errorvalue)
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (23, u"Out of resources when opening file '/tmp/#sql_87a4_0.MAI' (Errcode: 24)")
2016-10-19 16:29:19.011 39516 ERROR oslo_db.sqlalchemy.exc_filters 
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions [req-328a55d5-b96d-46e7-9d77-90bd1c03eb3a 58931d6e7ca247a2be2dd7fac6ee9927 869d2cede19d483e8565a6984202a37a - - -] Unexpected exception in API method
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 523, in show
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     instance = self._get_server(context, req, id, is_detail=True)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 442, in _get_server
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     expected_attrs=expected_attrs)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/common.py", line 516, in get_instance
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     expected_attrs=expected_attrs)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2046, in get
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     context, instance_id, expected_attrs=expected_attrs)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 181, in wrapper
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 399, in get_by_uuid
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     use_slave=use_slave)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 285, in wrapper
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 391, in _db_instance_get_by_uuid
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     columns_to_join=columns_to_join)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 696, in instance_get_by_uuid
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 229, in wrapper
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 330, in wrapped
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     return f(context, *args, **kwargs)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1970, in instance_get_by_uuid
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     columns_to_join=columns_to_join)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1975, in _instance_get_by_uuid
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     filter_by(uuid=uuid).\
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2634, in first
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     ret = list(self[0:1])
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2457, in __getitem__
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     return list(res)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     return self._execute_and_instances(context)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     result = conn.execute(querycontext.statement, self._params)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     return meth(self, multiparams, params)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     return connection._execute_clauseelement(self, multiparams, params)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     compiled_sql, distilled_params
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     context)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     util.raise_from_cause(newraise, exc_info)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     reraise(type(exception), exception, tb=exc_tb)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     context)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     cursor.execute(statement, parameters)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 146, in execute
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     result = self._query(query)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 296, in _query
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     conn.query(q)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 781, in query
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 942, in _read_query_result
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     result.read()
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1138, in read
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     first_packet = self.connection._read_packet()
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 906, in _read_packet
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     packet.check_error()
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 367, in check_error
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     err.raise_mysql_exception(self._data)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     _check_mysql_exception(errinfo)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions     raise InternalError(errno, errorvalue)
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions DBError: (pymysql.err.InternalError) (23, u"Out of resources when opening file '/tmp/#sql_87a4_0.MAI' (Errcode: 24)") [SQL: u'SELECT anon_1.instances_created_at AS anon_1_instances_created_at, anon_1.instances_updated_at AS anon_1_instances_updated_at, anon_1.instances_deleted_at AS anon_1_instances_deleted_at, anon_1.instances_deleted AS anon_1_instances_deleted, anon_1.instances_id AS anon_1_instances_id, anon_1.instances_user_id AS anon_1_instances_user_id, anon_1.instances_project_id AS anon_1_instances_project_id, anon_1.instances_image_ref AS anon_1_instances_image_ref, anon_1.instances_kernel_id AS anon_1_instances_kernel_id, anon_1.instances_ramdisk_id AS anon_1_instances_ramdisk_id, anon_1.instances_hostname AS anon_1_instances_hostname, anon_1.instances_launch_index AS anon_1_instances_launch_index, anon_1.instances_key_name AS anon_1_instances_key_name, anon_1.instances_key_data AS anon_1_instances_key_data, anon_1.instances_power_state AS anon_1_instances_power_state, anon_1.instances_vm_state AS anon_1_instances_vm_state, anon_1.instances_task_state AS anon_1_instances_task_state, anon_1.instances_memory_mb AS anon_1_instances_memory_mb, anon_1.instances_vcpus AS anon_1_instances_vcpus, anon_1.instances_root_gb AS anon_1_instances_root_gb, anon_1.instances_ephemeral_gb AS anon_1_instances_ephemeral_gb, anon_1.instances_ephemeral_key_uuid AS anon_1_instances_ephemeral_key_uuid, anon_1.instances_host AS anon_1_instances_host, anon_1.instances_node AS anon_1_instances_node, anon_1.instances_instance_type_id AS anon_1_instances_instance_type_id, anon_1.instances_user_data AS anon_1_instances_user_data, anon_1.instances_reservation_id AS anon_1_instances_reservation_id, anon_1.instances_launched_at AS anon_1_instances_launched_at, anon_1.instances_terminated_at AS anon_1_instances_terminated_at, anon_1.instances_availability_zone AS anon_1_instances_availability_zone, anon_1.instances_display_name AS anon_1_instances_display_name, anon_1.instances_display_description AS anon_1_instances_display_description, anon_1.instances_launched_on AS anon_1_instances_launched_on, anon_1.instances_locked AS anon_1_instances_locked, anon_1.instances_locked_by AS anon_1_instances_locked_by, anon_1.instances_os_type AS anon_1_instances_os_type, anon_1.instances_architecture AS anon_1_instances_architecture, anon_1.instances_vm_mode AS anon_1_instances_vm_mode, anon_1.instances_uuid AS anon_1_instances_uuid, anon_1.instances_root_device_name AS anon_1_instances_root_device_name, anon_1.instances_default_ephemeral_device AS anon_1_instances_default_ephemeral_device, anon_1.instances_default_swap_device AS anon_1_instances_default_swap_device, anon_1.instances_config_drive AS anon_1_instances_config_drive, anon_1.instances_access_ip_v4 AS anon_1_instances_access_ip_v4, anon_1.instances_access_ip_v6 AS anon_1_instances_access_ip_v6, anon_1.instances_auto_disk_config AS anon_1_instances_auto_disk_config, anon_1.instances_progress AS anon_1_instances_progress, anon_1.instances_shutdown_terminate AS anon_1_instances_shutdown_terminate, anon_1.instances_disable_terminate AS anon_1_instances_disable_terminate, anon_1.instances_cell_name AS anon_1_instances_cell_name, anon_1.instances_internal_id AS anon_1_instances_internal_id, anon_1.instances_cleaned AS anon_1_instances_cleaned, security_groups_1.created_at AS security_groups_1_created_at, security_groups_1.updated_at AS security_groups_1_updated_at, security_groups_1.deleted_at AS security_groups_1_deleted_at, security_groups_1.deleted AS security_groups_1_deleted, security_groups_1.id AS security_groups_1_id, security_groups_1.name AS security_groups_1_name, security_groups_1.description AS security_groups_1_description, security_groups_1.user_id AS security_groups_1_user_id, security_groups_1.project_id AS security_groups_1_project_id, security_group_rules_1.created_at AS security_group_rules_1_created_at, security_group_rules_1.updated_at AS security_group_rules_1_updated_at, security_group_rules_1.deleted_at AS security_group_rules_1_deleted_at, security_group_rules_1.deleted AS security_group_rules_1_deleted, security_group_rules_1.id AS security_group_rules_1_id, security_group_rules_1.parent_group_id AS security_group_rules_1_parent_group_id, security_group_rules_1.protocol AS security_group_rules_1_protocol, security_group_rules_1.from_port AS security_group_rules_1_from_port, security_group_rules_1.to_port AS security_group_rules_1_to_port, security_group_rules_1.cidr AS security_group_rules_1_cidr, security_group_rules_1.group_id AS security_group_rules_1_group_id, instance_extra_1.numa_topology AS instance_extra_1_numa_topology, instance_extra_1.flavor AS instance_extra_1_flavor, instance_extra_1.created_at AS instance_extra_1_created_at, instance_extra_1.updated_at AS instance_extra_1_updated_at, instance_extra_1.deleted_at AS instance_extra_1_deleted_at, instance_extra_1.deleted AS instance_extra_1_deleted, instance_extra_1.id AS instance_extra_1_id, instance_extra_1.instance_uuid AS instance_extra_1_instance_uuid, pci_devices_1.created_at AS pci_devices_1_created_at, pci_devices_1.updated_at AS pci_devices_1_updated_at, pci_devices_1.deleted_at AS pci_devices_1_deleted_at, pci_devices_1.deleted AS pci_devices_1_deleted, pci_devices_1.id AS pci_devices_1_id, pci_devices_1.compute_node_id AS pci_devices_1_compute_node_id, pci_devices_1.address AS pci_devices_1_address, pci_devices_1.vendor_id AS pci_devices_1_vendor_id, pci_devices_1.product_id AS pci_devices_1_product_id, pci_devices_1.dev_type AS pci_devices_1_dev_type, pci_devices_1.dev_id AS pci_devices_1_dev_id, pci_devices_1.label AS pci_devices_1_label, pci_devices_1.status AS pci_devices_1_status, pci_devices_1.request_id AS pci_devices_1_request_id, pci_devices_1.extra_info AS pci_devices_1_extra_info, pci_devices_1.instance_uuid AS pci_devices_1_instance_uuid, pci_devices_1.numa_node AS pci_devices_1_numa_node, pci_devices_1.parent_addr AS pci_devices_1_parent_addr, instance_info_caches_1.created_at AS instance_info_caches_1_created_at, instance_info_caches_1.updated_at AS instance_info_caches_1_updated_at, instance_info_caches_1.deleted_at AS instance_info_caches_1_deleted_at, instance_info_caches_1.deleted AS instance_info_caches_1_deleted, instance_info_caches_1.id AS instance_info_caches_1_id, instance_info_caches_1.network_info AS instance_info_caches_1_network_info, instance_info_caches_1.instance_uuid AS instance_info_caches_1_instance_uuid, instance_metadata_1.created_at AS instance_metadata_1_created_at, instance_metadata_1.updated_at AS instance_metadata_1_updated_at, instance_metadata_1.deleted_at AS instance_metadata_1_deleted_at, instance_metadata_1.deleted AS instance_metadata_1_deleted, instance_metadata_1.id AS instance_metadata_1_id, instance_metadata_1.`key` AS instance_metadata_1_key, instance_metadata_1.value AS instance_metadata_1_value, instance_metadata_1.instance_uuid AS instance_metadata_1_instance_uuid, instance_system_metadata_1.created_at AS instance_system_metadata_1_created_at, instance_system_metadata_1.updated_at AS instance_system_metadata_1_updated_at, instance_system_metadata_1.deleted_at AS instance_system_metadata_1_deleted_at, instance_system_metadata_1.deleted AS instance_system_metadata_1_deleted, instance_system_metadata_1.id AS instance_system_metadata_1_id, instance_system_metadata_1.`key` AS instance_system_metadata_1_key, instance_system_metadata_1.value AS instance_system_metadata_1_value, instance_system_metadata_1.instance_uuid AS instance_system_metadata_1_instance_uuid \nFROM (SELECT instances.created_at AS instances_created_at, instances.updated_at AS instances_updated_at, instances.deleted_at AS instances_deleted_at, instances.deleted AS instances_deleted, instances.id AS instances_id, instances.user_id AS instances_user_id, instances.project_id AS instances_project_id, instances.image_ref AS instances_image_ref, instances.kernel_id AS instances_kernel_id, instances.ramdisk_id AS instances_ramdisk_id, instances.hostname AS instances_hostname, instances.launch_index AS instances_launch_index, instances.key_name AS instances_key_name, instances.key_data AS instances_key_data, instances.power_state AS instances_power_state, instances.vm_state AS instances_vm_state, instances.task_state AS instances_task_state, instances.memory_mb AS instances_memory_mb, instances.vcpus AS instances_vcpus, instances.root_gb AS instances_root_gb, instances.ephemeral_gb AS instances_ephemeral_gb, instances.ephemeral_key_uuid AS instances_ephemeral_key_uuid, instances.host AS instances_host, instances.node AS instances_node, instances.instance_type_id AS instances_instance_type_id, instances.user_data AS instances_user_data, instances.reservation_id AS instances_reservation_id, instances.launched_at AS instances_launched_at, instances.terminated_at AS instances_terminated_at, instances.availability_zone AS instances_availability_zone, instances.display_name AS instances_display_name, instances.display_description AS instances_display_description, instances.launched_on AS instances_launched_on, instances.locked AS instances_locked, instances.locked_by AS instances_locked_by, instances.os_type AS instances_os_type, instances.architecture AS instances_architecture, instances.vm_mode AS instances_vm_mode, instances.uuid AS instances_uuid, instances.root_device_name AS instances_root_device_name, instances.default_ephemeral_device AS instances_default_ephemeral_device, instances.default_swap_device AS instances_default_swap_device, instances.config_drive AS instances_config_drive, instances.access_ip_v4 AS instances_access_ip_v4, instances.access_ip_v6 AS instances_access_ip_v6, instances.auto_disk_config AS instances_auto_disk_config, instances.progress AS instances_progress, instances.shutdown_terminate AS instances_shutdown_terminate, instances.disable_terminate AS instances_disable_terminate, instances.cell_name AS instances_cell_name, instances.internal_id AS instances_internal_id, instances.cleaned AS instances_cleaned \nFROM instances \nWHERE instances.deleted = %s AND instances.uuid = %s \n LIMIT %s) AS anon_1 LEFT OUTER JOIN (security_group_instance_association AS security_group_instance_association_1 INNER JOIN security_groups AS security_groups_1 ON security_groups_1.id = security_group_instance_association_1.security_group_id AND security_group_instance_association_1.deleted = %s AND security_groups_1.deleted = %s) ON security_group_instance_association_1.instance_uuid = anon_1.instances_uuid AND anon_1.instances_deleted = %s LEFT OUTER JOIN security_group_rules AS security_group_rules_1 ON security_group_rules_1.parent_group_id = security_groups_1.id AND security_group_rules_1.deleted = %s LEFT OUTER JOIN instance_extra AS instance_extra_1 ON instance_extra_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN pci_devices AS pci_devices_1 ON pci_devices_1.instance_uuid = anon_1.instances_uuid AND pci_devices_1.deleted = %s LEFT OUTER JOIN instance_info_caches AS instance_info_caches_1 ON instance_info_caches_1.instance_uuid = anon_1.instances_uuid LEFT OUTER JOIN instance_metadata AS instance_metadata_1 ON instance_metadata_1.instance_uuid = anon_1.instances_uuid AND instance_metadata_1.deleted = %s LEFT OUTER JOIN instance_system_metadata AS instance_system_metadata_1 ON anon_1.instances_uuid = instance_system_metadata_1.instance_uuid'] [parameters: (0, u'd64ad8d8-9397-4da4-a8cc-57371654720a', 1, 0, 0, 0, 0, 0, 0)]
2016-10-19 16:29:19.013 39516 ERROR nova.api.openstack.extensions 

Basically out of memory on your nodes....not a nova issue.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1634925

Title:
  nova api error

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Description:

  Nova API on Director failed

  Steps to reproduce

  After first deployment overcloud which failed I was delete failed stack:
  openstack stack delete stackID

  After finish I deploy overcloud one again and resources was failed with error :
  resources.Compute: Property error: resources[1].properties.Image: 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) |

  So I have check openstack-status on director and such error occurs:
  == Nova instances ==
  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_db.exception.DBError'> (HTTP 500) (Request-ID: req-d7280012-6832-4c33-b910-c3f15c32019d)

  Expected result:

  Nova api should be run and I can deploy Overcloud

  Environment:
  nova - version: 13.1.1

  Neutron with OpenVSwitch and Ceph

  Logs & Configs:
  See attachment file

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1634925/+subscriptions
References

[Bug 1634925] [NEW] nova api error
From: Malgorzata Rembas, 2016-10-19