yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1754661] [NEW] Unexpected API Error.

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Updesh Bhadoriya <1754661@xxxxxxxxxxxxxxxxxx>
Date: Fri, 09 Mar 2018 13:40:28 -0000
Reply-to: Bug 1754661 <1754661@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

launching an instance in openstack newton =====
openstack server create --flavor m1.tiny --image cirros  --nic net-id=8c1fa730-fe35-4e01-a0cf-774c1f417df3 --security-group default  selfservice-instance
error==
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500) (Request-ID: req-3b5ca259-0c97-4dac-bec2-1b19876108cf)


nova-api-log==

2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 631, in create
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     **create_kwargs)
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1535, in create
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1128, in _create_instance
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     reservation_id, max_count)
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 824, in _validate_and_build_base_options
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     requested_networks, max_count)
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 439, in _check_requested_networks
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     max_count)
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 391, in validate_networks
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     requested_networks)
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/rpcapi.py", line 214, in validate_networks
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     return self.client.call(ctxt, 'validate_networks', networks=networks)
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 428, in call
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     return self.prepare().call(ctxt, method, **kwargs)
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     retry=self.retry)
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     retry=retry)
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     result = self._waiter.wait(msg_id, timeout)
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 456, in wait
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     message = self.waiters.get(msg_id, timeout=timeout)
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 347, in get
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     'to message ID %s' % msg_id)
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID 2c212cca27ad4597a1a41378e60c1898
2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions
2018-03-09 13:32:31.666 28121 INFO nova.api.openstack.wsgi [req-3b5ca259-0c97-4dac-bec2-1b19876108cf 7e880f3357fa41d2a43e7c9af5c6a0f4 b7d66937364e4af8866f4d7b26519a6d - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'>
2018-03-09 13:32:31.667 28121 INFO nova.osapi_compute.wsgi.server [req-3b5ca259-0c97-4dac-bec2-1b19876108cf 7e880f3357fa41d2a43e7c9af5c6a0f4 b7d66937364e4af8866f4d7b26519a6d - default default] 172.31.2.197 "POST /v2.1/b7d66937364e4af8866f4d7b26519a6d/servers HTTP/1.1" status: 500 len: 583 time: 60.8083072


nova.conf==

[default]
enabled_apis = osapi_compute,metadata
transport_url = rabbit://openstack:admin@172.31.2.197
auth_strategy = keystone
my_ip = 172.31.2.197
use_neutron = True
firewall_driver = nova.virt.firewall.NoopFirewallDriver
[api_database]
connection = mysql+pymysql://nova:admin@controller/nova_api
[barbican]
[cache]
[cells]
[cinder]
os_region_name = RegionOne
[cloudpipe]
[conductor]
[cors]
[cors.subdomain]
[crypto]
[database]
connection = mysql+pymysql://nova:admin@controller/nova
[ephemeral_storage_encryption]
[glance]
api_servers = http://controller:9292
[guestfs]
[hyperv]
[image_file_url]
[ironic]
[key_manager]
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = Default
user_domain_name =  Default
project_name =  service
username = nova
password = admin
[libvirt]
[matchmaker_redis]
[metrics]
[mks]
[neutron]
url = http://controller:9696
auth_url = http://controller:35357
auth_type = password
project_domain_name = Default
user_domain_name = Default
region_name = RegionOne
project_name = service
username = neutron
password = admin
service_metadata_proxy = True
metadata_proxy_shared_secret = metadata
[osapi_v21]
[oslo_concurrency]
lock_path = /var/lib/nova/tmp
[oslo_messaging_amqp]
#allow_insecure_clients = False
[oslo_messaging_notifications]
#transport_url = rabbit://openstack:admin@172.31.2.197
[oslo_messaging_rabbit]
[oslo_messaging_zmq]
[oslo_middleware]
[oslo_policy]
[placement]
[placement_database]
[rdp]
[remote_debug]
[serial_console]
[spice]
[ssl]
[trusted_computing]
[upgrade_levels]
[vmware]
[vnc]
vncserver_listen = 172.31.2.197
vncserver_proxyclient_address = 172.31.2.197
[workarounds]
[wsgi]
[xenserver]
[xvp]

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1754661

Title:
  Unexpected API Error.

Status in OpenStack Compute (nova):
  New

Bug description:
  launching an instance in openstack newton =====
  openstack server create --flavor m1.tiny --image cirros  --nic net-id=8c1fa730-fe35-4e01-a0cf-774c1f417df3 --security-group default  selfservice-instance
  error==
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500) (Request-ID: req-3b5ca259-0c97-4dac-bec2-1b19876108cf)

  
  nova-api-log==

  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 631, in create
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     **create_kwargs)
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1535, in create
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1128, in _create_instance
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     reservation_id, max_count)
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 824, in _validate_and_build_base_options
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     requested_networks, max_count)
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 439, in _check_requested_networks
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     max_count)
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 391, in validate_networks
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     requested_networks)
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/rpcapi.py", line 214, in validate_networks
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     return self.client.call(ctxt, 'validate_networks', networks=networks)
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 428, in call
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     return self.prepare().call(ctxt, method, **kwargs)
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     retry=self.retry)
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     retry=retry)
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     result = self._waiter.wait(msg_id, timeout)
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 456, in wait
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     message = self.waiters.get(msg_id, timeout=timeout)
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 347, in get
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions     'to message ID %s' % msg_id)
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID 2c212cca27ad4597a1a41378e60c1898
  2018-03-09 13:32:31.663 28121 ERROR nova.api.openstack.extensions
  2018-03-09 13:32:31.666 28121 INFO nova.api.openstack.wsgi [req-3b5ca259-0c97-4dac-bec2-1b19876108cf 7e880f3357fa41d2a43e7c9af5c6a0f4 b7d66937364e4af8866f4d7b26519a6d - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  2018-03-09 13:32:31.667 28121 INFO nova.osapi_compute.wsgi.server [req-3b5ca259-0c97-4dac-bec2-1b19876108cf 7e880f3357fa41d2a43e7c9af5c6a0f4 b7d66937364e4af8866f4d7b26519a6d - default default] 172.31.2.197 "POST /v2.1/b7d66937364e4af8866f4d7b26519a6d/servers HTTP/1.1" status: 500 len: 583 time: 60.8083072

  
  nova.conf==

  [default]
  enabled_apis = osapi_compute,metadata
  transport_url = rabbit://openstack:admin@172.31.2.197
  auth_strategy = keystone
  my_ip = 172.31.2.197
  use_neutron = True
  firewall_driver = nova.virt.firewall.NoopFirewallDriver
  [api_database]
  connection = mysql+pymysql://nova:admin@controller/nova_api
  [barbican]
  [cache]
  [cells]
  [cinder]
  os_region_name = RegionOne
  [cloudpipe]
  [conductor]
  [cors]
  [cors.subdomain]
  [crypto]
  [database]
  connection = mysql+pymysql://nova:admin@controller/nova
  [ephemeral_storage_encryption]
  [glance]
  api_servers = http://controller:9292
  [guestfs]
  [hyperv]
  [image_file_url]
  [ironic]
  [key_manager]
  [keystone_authtoken]
  auth_uri = http://controller:5000
  auth_url = http://controller:35357
  memcached_servers = controller:11211
  auth_type = password
  project_domain_name = Default
  user_domain_name =  Default
  project_name =  service
  username = nova
  password = admin
  [libvirt]
  [matchmaker_redis]
  [metrics]
  [mks]
  [neutron]
  url = http://controller:9696
  auth_url = http://controller:35357
  auth_type = password
  project_domain_name = Default
  user_domain_name = Default
  region_name = RegionOne
  project_name = service
  username = neutron
  password = admin
  service_metadata_proxy = True
  metadata_proxy_shared_secret = metadata
  [osapi_v21]
  [oslo_concurrency]
  lock_path = /var/lib/nova/tmp
  [oslo_messaging_amqp]
  #allow_insecure_clients = False
  [oslo_messaging_notifications]
  #transport_url = rabbit://openstack:admin@172.31.2.197
  [oslo_messaging_rabbit]
  [oslo_messaging_zmq]
  [oslo_middleware]
  [oslo_policy]
  [placement]
  [placement_database]
  [rdp]
  [remote_debug]
  [serial_console]
  [spice]
  [ssl]
  [trusted_computing]
  [upgrade_levels]
  [vmware]
  [vnc]
  vncserver_listen = 172.31.2.197
  vncserver_proxyclient_address = 172.31.2.197
  [workarounds]
  [wsgi]
  [xenserver]
  [xvp]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1754661/+subscriptions
Follow ups

[Bug 1754661] Re: Unexpected API Error.
From: Matt Riedemann, 2018-03-21