← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #30936

[Bug 1439302] [NEW] "FixedIpNotFoundForAddress: Fixed ip not found for address None." traces in gate runs

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Seeing this quite a bit in normal gate runs:

http://logs.openstack.org/53/169753/2/check/check-tempest-dsvm-full-
ceph/07dcae0/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-04-01_14_34_37_110

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRml4ZWRJcE5vdEZvdW5kRm9yQWRkcmVzczogRml4ZWQgaXAgbm90IGZvdW5kIGZvciBhZGRyZXNzIE5vbmUuXCIgQU5EIHRhZ3M6XCJzY3JlZW4tbi1jcHUudHh0XCIgQU5EIHRhZ3M6XCJtdWx0aWxpbmVcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQyNzkwMjQ0NTg4OSwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d] FixedIpNotFoundForAddress: Fixed ip not found for address None.
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d] Traceback (most recent call last):
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     executor_callback))
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     executor_callback)
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     result = func(ctxt, **new_args)
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/opt/stack/new/nova/nova/network/floating_ips.py", line 186, in deallocate_for_instance
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     super(FloatingIP, self).deallocate_for_instance(context, **kwargs)
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/opt/stack/new/nova/nova/network/manager.py", line 558, in deallocate_for_instance
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     instance=instance)
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/opt/stack/new/nova/nova/network/manager.py", line 214, in deallocate_fixed_ip
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     context, address, expected_attrs=['network'])
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/opt/stack/new/nova/nova/objects/base.py", line 161, in wrapper
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     args, kwargs)
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/opt/stack/new/nova/nova/conductor/rpcapi.py", line 329, in object_class_action
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     objver=objver, args=args, kwargs=kwargs)
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     retry=self.retry)
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     timeout=timeout, retry=retry)
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     retry=retry)
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     raise result

In nova.compute.manager._do_build_and_run_instance we're hitting an
UnexpectedDeletingTaskStateError exception:

http://logs.openstack.org/53/169753/2/check/check-tempest-dsvm-full-
ceph/07dcae0/logs/screen-n-cpu.txt.gz#_2015-04-01_14_34_36_898

Then we start tearing down here:

http://logs.openstack.org/53/169753/2/check/check-tempest-dsvm-full-
ceph/07dcae0/logs/screen-n-cpu.txt.gz#_2015-04-01_14_34_36_905

And for some reason there are no fixed_ips with no address once we try
to deallocate in the network manager.

** Affects: nova
     Importance: High
         Status: Confirmed


** Tags: compute network

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1439302

Title:
  "FixedIpNotFoundForAddress: Fixed ip not found for address None."
  traces in gate runs

Status in OpenStack Compute (Nova):
  Confirmed

Bug description:
  Seeing this quite a bit in normal gate runs:

  http://logs.openstack.org/53/169753/2/check/check-tempest-dsvm-full-
  ceph/07dcae0/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-04-01_14_34_37_110

  http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRml4ZWRJcE5vdEZvdW5kRm9yQWRkcmVzczogRml4ZWQgaXAgbm90IGZvdW5kIGZvciBhZGRyZXNzIE5vbmUuXCIgQU5EIHRhZ3M6XCJzY3JlZW4tbi1jcHUudHh0XCIgQU5EIHRhZ3M6XCJtdWx0aWxpbmVcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQyNzkwMjQ0NTg4OSwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d] FixedIpNotFoundForAddress: Fixed ip not found for address None.
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d] Traceback (most recent call last):
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     executor_callback))
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     executor_callback)
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     result = func(ctxt, **new_args)
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/opt/stack/new/nova/nova/network/floating_ips.py", line 186, in deallocate_for_instance
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     super(FloatingIP, self).deallocate_for_instance(context, **kwargs)
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/opt/stack/new/nova/nova/network/manager.py", line 558, in deallocate_for_instance
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     instance=instance)
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/opt/stack/new/nova/nova/network/manager.py", line 214, in deallocate_fixed_ip
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     context, address, expected_attrs=['network'])
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/opt/stack/new/nova/nova/objects/base.py", line 161, in wrapper
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     args, kwargs)
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/opt/stack/new/nova/nova/conductor/rpcapi.py", line 329, in object_class_action
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     objver=objver, args=args, kwargs=kwargs)
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     retry=self.retry)
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     timeout=timeout, retry=retry)
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     retry=retry)
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
  2015-04-01 14:34:37.110 674 TRACE nova.compute.manager [instance: 2fc5caf4-8ff7-45bc-940f-c13d696a1d9d]     raise result

  In nova.compute.manager._do_build_and_run_instance we're hitting an
  UnexpectedDeletingTaskStateError exception:

  http://logs.openstack.org/53/169753/2/check/check-tempest-dsvm-full-
  ceph/07dcae0/logs/screen-n-cpu.txt.gz#_2015-04-01_14_34_36_898

  Then we start tearing down here:

  http://logs.openstack.org/53/169753/2/check/check-tempest-dsvm-full-
  ceph/07dcae0/logs/screen-n-cpu.txt.gz#_2015-04-01_14_34_36_905

  And for some reason there are no fixed_ips with no address once we try
  to deallocate in the network manager.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1439302/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next