yahoo-eng-team team mailing list archive

[yangjianfeng <2125034@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

I occasionally got error when I live migrage a instance. After I check
the nova-conductor logs, I found the errors as below:

Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [None req-9fa0ab78-a993-47d4-bfe5-7869a499f3fa None None] [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d] Binding failed for port 774a5c63-1e9f-4f7c-bc37-f038f2330338 and host master-node2.: neutronclient.common.exceptions.Conflict: Binding for port 774a5c63-1e9f-4f7c-bc37-f038f2330338 on host master-node2 already exists.
Sep 18 05:53:07 master-node1 nova-conductor[199668]: Neutron server returns request_ids: ['req-de9620ea-62fb-4246-9463-80d2f46b3c65']
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d] Traceback (most recent call last):
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]   File "/opt/stack/nova/nova/network/neutron.py", line 1601, in bind_ports_to_host
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]     binding = client.create_port_binding(port_id, data)['binding']
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]   File "/opt/stack/nova/nova/network/neutron.py", line 198, in wrapper
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]     ret = obj(*args, **kwargs)
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]           ^^^^^^^^^^^^^^^^^^^^
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]   File "/opt/stack/data/venv/lib/python3.12/site-packages/neutronclient/v2_0/client.py", line 840, in create_port_binding
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]     return self.post(self.port_bindings_path % port_id, body=body)
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]   File "/opt/stack/data/venv/lib/python3.12/site-packages/neutronclient/v2_0/client.py", line 364, in post
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]     return self.do_request("POST", action, body=body,
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]   File "/opt/stack/data/venv/lib/python3.12/site-packages/neutronclient/v2_0/client.py", line 300, in do_request
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]     self._handle_fault_response(status_code, replybody, resp)
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]   File "/opt/stack/data/venv/lib/python3.12/site-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]     exception_handler_v20(status_code, error_body)
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]   File "/opt/stack/data/venv/lib/python3.12/site-packages/neutronclient/v2_0/client.py", line 90, in exception_handler_v20
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]     raise client_exc(message=error_message,
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d] neutronclient.common.exceptions.Conflict: Binding for port 774a5c63-1e9f-4f7c-bc37-f038f2330338 on host master-node2 already exists.
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d] Neutron server returns request_ids: ['req-de9620ea-62fb-4246-9463-80d2f46b3c65']
Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]
Sep 18 05:53:07 master-node1 nova-conductor[199668]: DEBUG nova.conductor.tasks.live_migrate [None req-9fa0ab78-a993-47d4-bfe5-7869a499f3fa None None] Skipping host: master-node2 because: Migration pre-check error: Binding failed for port 774a5c63-1e9f-4f7c-bc37-f038f2330338, please check neutron logs for more information. {{(pid=199668) _find_destination /opt/stack/nova/nova/conductor/tasks/live_migrate.py:576}}
Sep 18 05:53:08 master-node1 nova-conductor[199668]: DEBUG nova.conductor.tasks.migrate [None req-9fa0ab78-a993-47d4-bfe5-7869a499f3fa None None] Created allocations for instance 999b1fc9-389b-4a78-b041-da0556e9e42d on 134cc6fb-d42c-4513-9114-1e105de6663a {{(pid=199668) revert_allocation_for_migration /opt/stack/nova/nova/conductor/tasks/migrate.py:111}}
Sep 18 05:53:08 master-node1 nova-conductor[199668]: WARNING nova.scheduler.utils [None req-9fa0ab78-a993-47d4-bfe5-7869a499f3fa None None] Failed to compute_task_migrate_server: No valid host was found. There are not enough hosts available.
Sep 18 05:53:08 master-node1 nova-conductor[199668]: Traceback (most recent call last):
Sep 18 05:53:08 master-node1 nova-conductor[199668]:   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/server.py", line 269, in inner
Sep 18 05:53:08 master-node1 nova-conductor[199668]:     return func(*args, **kwargs)
Sep 18 05:53:08 master-node1 nova-conductor[199668]:            ^^^^^^^^^^^^^^^^^^^^^
Sep 18 05:53:08 master-node1 nova-conductor[199668]:   File "/opt/stack/nova/nova/scheduler/manager.py", line 269, in select_destinations
Sep 18 05:53:08 master-node1 nova-conductor[199668]:     selections = self._select_destinations(
Sep 18 05:53:08 master-node1 nova-conductor[199668]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
Sep 18 05:53:08 master-node1 nova-conductor[199668]:   File "/opt/stack/nova/nova/scheduler/manager.py", line 296, in _select_destinations
Sep 18 05:53:08 master-node1 nova-conductor[199668]:     selections = self._schedule(
Sep 18 05:53:08 master-node1 nova-conductor[199668]:                  ^^^^^^^^^^^^^^^
Sep 18 05:53:08 master-node1 nova-conductor[199668]:   File "/opt/stack/nova/nova/scheduler/manager.py", line 497, in _schedule
Sep 18 05:53:08 master-node1 nova-conductor[199668]:     self._ensure_sufficient_hosts(
Sep 18 05:53:08 master-node1 nova-conductor[199668]:   File "/opt/stack/nova/nova/scheduler/manager.py", line 544, in _ensure_sufficient_hosts
Sep 18 05:53:08 master-node1 nova-conductor[199668]:     raise exception.NoValidHost(reason=reason)
Sep 18 05:53:08 master-node1 nova-conductor[199668]: nova.exception.NoValidHost: No valid host was found. There are not enough hosts available.
Sep 18 05:53:08 master-node1 nova-conductor[199668]: : nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available.
Sep 18 05:53:08 master-node1 nova-conductor[199668]: WARNING nova.scheduler.utils [None req-9fa0ab78-a993-47d4-bfe5-7869a499f3fa None None] [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d] Setting instance to ACTIVE state.: nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available.

After investigation, I found this was due to previous failures of the
instance live migration, which resulted in a redundant port binding.
There is a related bug in neutron:
https://bugs.launchpad.net/neutron/+bug/1979072

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2125034

Title:
  During live-migration nova-conductor report 'No valid host was found'
  due to duplicated port bindings

Status in neutron:
  New

Bug description:
  I occasionally got error when I live migrage a instance. After I check
  the nova-conductor logs, I found the errors as below:

  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [None req-9fa0ab78-a993-47d4-bfe5-7869a499f3fa None None] [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d] Binding failed for port 774a5c63-1e9f-4f7c-bc37-f038f2330338 and host master-node2.: neutronclient.common.exceptions.Conflict: Binding for port 774a5c63-1e9f-4f7c-bc37-f038f2330338 on host master-node2 already exists.
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: Neutron server returns request_ids: ['req-de9620ea-62fb-4246-9463-80d2f46b3c65']
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d] Traceback (most recent call last):
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]   File "/opt/stack/nova/nova/network/neutron.py", line 1601, in bind_ports_to_host
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]     binding = client.create_port_binding(port_id, data)['binding']
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]   File "/opt/stack/nova/nova/network/neutron.py", line 198, in wrapper
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]     ret = obj(*args, **kwargs)
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]           ^^^^^^^^^^^^^^^^^^^^
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]   File "/opt/stack/data/venv/lib/python3.12/site-packages/neutronclient/v2_0/client.py", line 840, in create_port_binding
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]     return self.post(self.port_bindings_path % port_id, body=body)
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]   File "/opt/stack/data/venv/lib/python3.12/site-packages/neutronclient/v2_0/client.py", line 364, in post
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]     return self.do_request("POST", action, body=body,
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]   File "/opt/stack/data/venv/lib/python3.12/site-packages/neutronclient/v2_0/client.py", line 300, in do_request
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]     self._handle_fault_response(status_code, replybody, resp)
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]   File "/opt/stack/data/venv/lib/python3.12/site-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]     exception_handler_v20(status_code, error_body)
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]   File "/opt/stack/data/venv/lib/python3.12/site-packages/neutronclient/v2_0/client.py", line 90, in exception_handler_v20
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]     raise client_exc(message=error_message,
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d] neutronclient.common.exceptions.Conflict: Binding for port 774a5c63-1e9f-4f7c-bc37-f038f2330338 on host master-node2 already exists.
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d] Neutron server returns request_ids: ['req-de9620ea-62fb-4246-9463-80d2f46b3c65']
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: ERROR nova.network.neutron [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d]
  Sep 18 05:53:07 master-node1 nova-conductor[199668]: DEBUG nova.conductor.tasks.live_migrate [None req-9fa0ab78-a993-47d4-bfe5-7869a499f3fa None None] Skipping host: master-node2 because: Migration pre-check error: Binding failed for port 774a5c63-1e9f-4f7c-bc37-f038f2330338, please check neutron logs for more information. {{(pid=199668) _find_destination /opt/stack/nova/nova/conductor/tasks/live_migrate.py:576}}
  Sep 18 05:53:08 master-node1 nova-conductor[199668]: DEBUG nova.conductor.tasks.migrate [None req-9fa0ab78-a993-47d4-bfe5-7869a499f3fa None None] Created allocations for instance 999b1fc9-389b-4a78-b041-da0556e9e42d on 134cc6fb-d42c-4513-9114-1e105de6663a {{(pid=199668) revert_allocation_for_migration /opt/stack/nova/nova/conductor/tasks/migrate.py:111}}
  Sep 18 05:53:08 master-node1 nova-conductor[199668]: WARNING nova.scheduler.utils [None req-9fa0ab78-a993-47d4-bfe5-7869a499f3fa None None] Failed to compute_task_migrate_server: No valid host was found. There are not enough hosts available.
  Sep 18 05:53:08 master-node1 nova-conductor[199668]: Traceback (most recent call last):
  Sep 18 05:53:08 master-node1 nova-conductor[199668]:   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/server.py", line 269, in inner
  Sep 18 05:53:08 master-node1 nova-conductor[199668]:     return func(*args, **kwargs)
  Sep 18 05:53:08 master-node1 nova-conductor[199668]:            ^^^^^^^^^^^^^^^^^^^^^
  Sep 18 05:53:08 master-node1 nova-conductor[199668]:   File "/opt/stack/nova/nova/scheduler/manager.py", line 269, in select_destinations
  Sep 18 05:53:08 master-node1 nova-conductor[199668]:     selections = self._select_destinations(
  Sep 18 05:53:08 master-node1 nova-conductor[199668]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  Sep 18 05:53:08 master-node1 nova-conductor[199668]:   File "/opt/stack/nova/nova/scheduler/manager.py", line 296, in _select_destinations
  Sep 18 05:53:08 master-node1 nova-conductor[199668]:     selections = self._schedule(
  Sep 18 05:53:08 master-node1 nova-conductor[199668]:                  ^^^^^^^^^^^^^^^
  Sep 18 05:53:08 master-node1 nova-conductor[199668]:   File "/opt/stack/nova/nova/scheduler/manager.py", line 497, in _schedule
  Sep 18 05:53:08 master-node1 nova-conductor[199668]:     self._ensure_sufficient_hosts(
  Sep 18 05:53:08 master-node1 nova-conductor[199668]:   File "/opt/stack/nova/nova/scheduler/manager.py", line 544, in _ensure_sufficient_hosts
  Sep 18 05:53:08 master-node1 nova-conductor[199668]:     raise exception.NoValidHost(reason=reason)
  Sep 18 05:53:08 master-node1 nova-conductor[199668]: nova.exception.NoValidHost: No valid host was found. There are not enough hosts available.
  Sep 18 05:53:08 master-node1 nova-conductor[199668]: : nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available.
  Sep 18 05:53:08 master-node1 nova-conductor[199668]: WARNING nova.scheduler.utils [None req-9fa0ab78-a993-47d4-bfe5-7869a499f3fa None None] [instance: 999b1fc9-389b-4a78-b041-da0556e9e42d] Setting instance to ACTIVE state.: nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available.

  After investigation, I found this was due to previous failures of the
  instance live migration, which resulted in a redundant port binding.
  There is a related bug in neutron:
  https://bugs.launchpad.net/neutron/+bug/1979072

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2125034/+subscriptions