yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #00963
[Bug 1134604] [NEW] New IPs not available after subnet changes
You have been subscribed to a public bug:
We recently run out IP address in the subnet used for our instances and
we had to extend out subnet from /24 to /23.
We edited /etc/nova/nova.conf, restared nova-network using /etc/init.d/nova-network restart.
"ifconfig bf100" was showing correct (new) netmask so was "nova-manage network list"
When new instances were about to use new range they failed to get IPs:
On compute node:
2013-02-27 10:46:38 ERROR nova.compute.manager [req-xxxx xxxx xxxx] [instance: xxxx] Instance failed network setup
And corresponding on our controller:
2013-02-27 10:46:38 DEBUG nova.network.manager [req-xxxx xxxx xxxx] [instance: xxxx] networks retrieved for instance: |[<nova.db.sqlalchemy.models.Network object at 0x4997850>]| from (pid=2687) allocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:982
2013-02-27 10:46:38 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 275, in _process_data
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 257, in wrapped
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp return func(self, context, *args, **kwargs)
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 320, in allocate_for_instance
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp **kwargs)
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 257, in wrapped
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp return func(self, context, *args, **kwargs)
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 986, in allocate_for_instance
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp requested_networks=requested_networks)
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 213, in _allocate_fixed_ips
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp vpn=vpn, address=address)
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1265, in allocate_fixed_ip
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp instance_ref['uuid'])
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 442, in fixed_ip_associate_pool
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp instance_uuid, host)
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 111, in wrapper
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs)
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1089, in fixed_ip_associate_pool
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp raise exception.NoMoreFixedIps()
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp NoMoreFixedIps: Zero fixed ips available.
2013-02-27 10:46:38 TRACE nova.openstack.common.rpc.amqp
2013-02-27 10:46:38 ERROR nova.openstack.common.rpc.common [-] Returning exception Zero fixed ips available. to caller
After some investigation we realised that fixed_ips table in nova DB only had IPs from the old range. We added new IPs using:
INSERT INTO fixed_ips (created_at, deleted, allocated, leased, reserved, network_id, address) VALUES (CURDATE(), 0,0,0,0, 1, "xx.xx.xx.xx");
which allowed our instances to get IPs from new range.
On compute node we are running:
ii nova-common 2012.2-0ubuntu5~cloud0 OpenStack Compute - common files
ii nova-compute 2012.2-0ubuntu5~cloud0 OpenStack Compute - compute node
ii nova-compute-kvm 2012.2-0ubuntu5~cloud0 OpenStack Compute - compute node (KVM)
ii python-nova 2012.2-0ubuntu5~cloud0 OpenStack Compute Python libraries
ii python-novaclient 1:2.9.0-0ubuntu1~cloud0 client library for OpenStack Compute API
and on the controller:
ii nova-api 2012.2-0ubuntu5~cloud0 OpenStack Compute - API frontend
ii nova-cert 2012.2-0ubuntu5~cloud0 OpenStack Compute - certificate management
ii nova-common 2012.2-0ubuntu5~cloud0 OpenStack Compute - common files
ii nova-doc 2012.2-0ubuntu5~cloud0 OpenStack Compute - documentation
ii nova-network 2012.2-0ubuntu5~cloud0 OpenStack Compute - Network manager
ii nova-objectstore 2012.2-0ubuntu5~cloud0 OpenStack Compute - object store
ii nova-scheduler 2012.2-0ubuntu5~cloud0 OpenStack Compute - virtual machine scheduler
ii python-nova 2012.2-0ubuntu5~cloud0 OpenStack Compute Python libraries
ii python-novaclient 1:2.9.0-0ubuntu1~cloud0 client library for OpenStack Compute API
Both running:
Description: Ubuntu 12.04.2 LTS
Release: 12.04
** Affects: nova
Importance: Undecided
Status: New
--
New IPs not available after subnet changes
https://bugs.launchpad.net/bugs/1134604
You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova).
