yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1489576] [NEW] key error in del l3_agent.pd.routers[router['id']]['subnets']

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Adolfo Duarte <adolfo.duarte@xxxxxx>
Date: Thu, 27 Aug 2015 19:25:01 -0000
Reply-to: Bug 1489576 <1489576@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

currently the following erorr is seen in the q-l3 or q-vpn agents when
deleting a router:  (steps to reproduce below):

Exit code: 0
Stdin:
Stdout: lo
Stderr:  execute /opt/stack/neutron/neutron/agent/linux/utils.py:150
2015-08-27 19:08:38.007 12027 DEBUG neutron.agent.linux.utils [-]
Command: ['ip', 'netns', 'delete', u'qrouter-f53025cc-5159-480e-a7f3-19122aa330a3']
Exit code: 0
Stdin:
Stdout:
Stderr:  execute /opt/stack/neutron/neutron/agent/linux/utils.py:150
2015-08-27 19:08:38.009 12027 DEBUG neutron.callbacks.manager [-] Notify callbacks for router, after_delete _notify_loop /opt/stack/neutron/neutron/callbacks/manager.py:133
2015-08-27 19:08:38.009 12027 DEBUG neutron.callbacks.manager [-] Calling callback neutron.agent.linux.pd.remove_router _notify_loop /opt/stack/neutron/neutron/callbacks/manager.py:140
2015-08-27 19:08:38.010 12027 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" acquired by "remove_router" :: waited 0.000s inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:251
2015-08-27 19:08:38.010 12027 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" released by "remove_router" :: held 0.000s inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:262
2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager [-] Error during notification for neutron.agent.linux.pd.remove_router router, after_delete
2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager Traceback (most recent call last):
2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager   File "/opt/stack/neutron/neutron/callbacks/manager.py", line 141, in _notify_loop
2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager     callback(resource, event, trigger, **kwargs)
2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 252, in inner
2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager     return f(*args, **kwargs)
2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager   File "/opt/stack/neutron/neutron/agent/linux/pd.py", line 307, in remove_router
2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager     del l3_agent.pd.routers[router['id']]['subnets']
2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager KeyError: 'id'
2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager
2015-08-27 19:08:38.015 12027 DEBUG neutron.callbacks.manager [-] Calling callback neutron_vpnaas.services.vpn.vpn_service.router_removed_actions _notify_loop /opt/stack/neutron/neutron/callbacks/manager.py:140
2015-08-27 19:08:47.697 12027 DEBUG oslo_service.loopingcall [-] Fixed interval looping call 'neutron_vpnaas.services.vpn.agent.VPNAgent._report_state' sleeping for 30.00 seconds _run_loop /usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py:121
2015-08-27 19:08:47.756 12027 DEBUG oslo_service.loopingcall [-] Fixed interval looping call 'neutron.service.Service.report_state' sleeping for 30.00 seconds _run_loop /usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py:121


STEPS TO REPRODUCE: 
create a stack with a neutron server (q-vpn) node,  "network" server (q-l3), and a compute node. 
Then execut the following: 

neutron net-create public --router:external
neutron subnet-create public 123.0.0.0/24 --disable-dhcp
for id in $(neutron security-group-list | grep -v " id " | grep -v "\-\-" | awk '{print $2}'); do neutron security-group-delete $id; done 
neutron security-group-rule-create --protocol icmp --direction ingress default
neutron security-group-rule-create --protocol tcp --port-range-min 1 --port-range-max 65535 --direction ingress default 
neutron security-group-rule-create --protocol udp --port-range-min 1 --port-range-max 65535 --direction ingress default 
neutron net-create private
neutron subnet-create private103.0.0.0/24 --name private


neutron router-create r1 --distributed=False --ha=False
neutron router-gateway-set r1 public 

then execute: 
neutron router-delete r1

you will see the error above.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1489576

Title:
  key error in del l3_agent.pd.routers[router['id']]['subnets']

Status in neutron:
  New

Bug description:
  currently the following erorr is seen in the q-l3 or q-vpn agents when
  deleting a router:  (steps to reproduce below):

  Exit code: 0
  Stdin:
  Stdout: lo
  Stderr:  execute /opt/stack/neutron/neutron/agent/linux/utils.py:150
  2015-08-27 19:08:38.007 12027 DEBUG neutron.agent.linux.utils [-]
  Command: ['ip', 'netns', 'delete', u'qrouter-f53025cc-5159-480e-a7f3-19122aa330a3']
  Exit code: 0
  Stdin:
  Stdout:
  Stderr:  execute /opt/stack/neutron/neutron/agent/linux/utils.py:150
  2015-08-27 19:08:38.009 12027 DEBUG neutron.callbacks.manager [-] Notify callbacks for router, after_delete _notify_loop /opt/stack/neutron/neutron/callbacks/manager.py:133
  2015-08-27 19:08:38.009 12027 DEBUG neutron.callbacks.manager [-] Calling callback neutron.agent.linux.pd.remove_router _notify_loop /opt/stack/neutron/neutron/callbacks/manager.py:140
  2015-08-27 19:08:38.010 12027 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" acquired by "remove_router" :: waited 0.000s inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:251
  2015-08-27 19:08:38.010 12027 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" released by "remove_router" :: held 0.000s inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:262
  2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager [-] Error during notification for neutron.agent.linux.pd.remove_router router, after_delete
  2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager Traceback (most recent call last):
  2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager   File "/opt/stack/neutron/neutron/callbacks/manager.py", line 141, in _notify_loop
  2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager     callback(resource, event, trigger, **kwargs)
  2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 252, in inner
  2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager     return f(*args, **kwargs)
  2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager   File "/opt/stack/neutron/neutron/agent/linux/pd.py", line 307, in remove_router
  2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager     del l3_agent.pd.routers[router['id']]['subnets']
  2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager KeyError: 'id'
  2015-08-27 19:08:38.012 12027 ERROR neutron.callbacks.manager
  2015-08-27 19:08:38.015 12027 DEBUG neutron.callbacks.manager [-] Calling callback neutron_vpnaas.services.vpn.vpn_service.router_removed_actions _notify_loop /opt/stack/neutron/neutron/callbacks/manager.py:140
  2015-08-27 19:08:47.697 12027 DEBUG oslo_service.loopingcall [-] Fixed interval looping call 'neutron_vpnaas.services.vpn.agent.VPNAgent._report_state' sleeping for 30.00 seconds _run_loop /usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py:121
  2015-08-27 19:08:47.756 12027 DEBUG oslo_service.loopingcall [-] Fixed interval looping call 'neutron.service.Service.report_state' sleeping for 30.00 seconds _run_loop /usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py:121

  
  STEPS TO REPRODUCE: 
  create a stack with a neutron server (q-vpn) node,  "network" server (q-l3), and a compute node. 
  Then execut the following: 

  neutron net-create public --router:external
  neutron subnet-create public 123.0.0.0/24 --disable-dhcp
  for id in $(neutron security-group-list | grep -v " id " | grep -v "\-\-" | awk '{print $2}'); do neutron security-group-delete $id; done 
  neutron security-group-rule-create --protocol icmp --direction ingress default
  neutron security-group-rule-create --protocol tcp --port-range-min 1 --port-range-max 65535 --direction ingress default 
  neutron security-group-rule-create --protocol udp --port-range-min 1 --port-range-max 65535 --direction ingress default 
  neutron net-create private
  neutron subnet-create private103.0.0.0/24 --name private

  
  neutron router-create r1 --distributed=False --ha=False
  neutron router-gateway-set r1 public 

  then execute: 
  neutron router-delete r1

  you will see the error above.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1489576/+subscriptions
Follow ups

[Bug 1489576] Re: key error in del l3_agent.pd.routers[router['id']]['subnets']
From: Thierry Carrez, 2015-09-03