yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1134604] Re: New IPs not available after subnet changes

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Wed, 17 Sep 2014 11:56:04 -0000
Reply-to: Bug 1134604 <1134604@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This is sort of working as designed. It might be nice to get rid of the
static pre allocation of addresses, but we're not there yet.

** Changed in: nova
       Status: Incomplete => Won't Fix

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1134604

Title:
  New IPs not available after subnet changes

Status in OpenStack Compute (Nova):
  Won't Fix

Bug description:
  We recently run out IP address in the subnet used for our instances
  and we had to extend out subnet from /24 to /23.

  We edited /etc/nova/nova.conf, restared nova-network using /etc/init.d/nova-network restart.
  "ifconfig bf100" was showing correct (new) netmask so was "nova-manage network list"

  When new instances were about to use new range they failed to get IPs:

  On compute node:
  2013-02-27 10:46:38 ERROR nova.compute.manager [req-xxxx xxxx xxxx] [instance: xxxx] Instance failed network setup

  And corresponding on our controller:
  2013-02-27 10:46:38 DEBUG nova.network.manager [req-xxxx xxxx xxxx] [instance: xxxx] networks retrieved for instance: |[<nova.db.sqlalchemy.models.Network object at 0x4997850>]| from (pid=2687) allocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:982
  2013-02-27 10:46:38 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 275, in _process_data
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 257, in wrapped
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp     return func(self, context, *args, **kwargs)
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 320, in allocate_for_instance
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp     **kwargs)
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 257, in wrapped
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp     return func(self, context, *args, **kwargs)
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 986, in allocate_for_instance
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp     requested_networks=requested_networks)
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 213, in _allocate_fixed_ips
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp     vpn=vpn, address=address)
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1265, in allocate_fixed_ip
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp     instance_ref['uuid'])
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 442, in fixed_ip_associate_pool
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp     instance_uuid, host)
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 111, in wrapper
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp     return f(*args, **kwargs)
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1089, in fixed_ip_associate_pool
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp     raise exception.NoMoreFixedIps()
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp NoMoreFixedIps: Zero fixed ips available.
  2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp 
  2013-02-27 10:46:38 ERROR nova.openstack.common.rpc.common [-] Returning exception Zero fixed ips available. to caller

  After some investigation we realised that fixed_ips table in nova DB only had IPs from the old range. We added new IPs using:
  INSERT INTO fixed_ips (created_at, deleted, allocated, leased, reserved, network_id, address) VALUES (CURDATE(), 0,0,0,0, 1, "xx.xx.xx.xx");
  which allowed our instances to get IPs from new range.

  On compute node we are running:
  ii  nova-common                      2012.2-0ubuntu5~cloud0                        OpenStack Compute - common files
  ii  nova-compute                     2012.2-0ubuntu5~cloud0                        OpenStack Compute - compute node
  ii  nova-compute-kvm                 2012.2-0ubuntu5~cloud0                        OpenStack Compute - compute node (KVM)
  ii  python-nova                      2012.2-0ubuntu5~cloud0                        OpenStack Compute Python libraries
  ii  python-novaclient                1:2.9.0-0ubuntu1~cloud0                       client library for OpenStack Compute API

  and on the controller:
  ii  nova-api                         2012.2-0ubuntu5~cloud0                        OpenStack Compute - API frontend
  ii  nova-cert                        2012.2-0ubuntu5~cloud0                        OpenStack Compute - certificate management
  ii  nova-common                      2012.2-0ubuntu5~cloud0                        OpenStack Compute - common files
  ii  nova-doc                         2012.2-0ubuntu5~cloud0                        OpenStack Compute - documentation
  ii  nova-network                     2012.2-0ubuntu5~cloud0                        OpenStack Compute - Network manager
  ii  nova-objectstore                 2012.2-0ubuntu5~cloud0                        OpenStack Compute - object store
  ii  nova-scheduler                   2012.2-0ubuntu5~cloud0                        OpenStack Compute - virtual machine scheduler
  ii  python-nova                      2012.2-0ubuntu5~cloud0                        OpenStack Compute Python libraries
  ii  python-novaclient                1:2.9.0-0ubuntu1~cloud0                       client library for OpenStack Compute API

  Both running:
  Description:	Ubuntu 12.04.2 LTS
  Release:	12.04

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1134604/+subscriptions