yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1712795] [NEW] Fail to startup neutron-l3-agent

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Zhengwei Gao <gaozhengwei1@xxxxxxxx>
Date: Thu, 24 Aug 2017 11:06:06 -0000
Reply-to: Bug 1712795 <1712795@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

2017-08-17 08:00:09.601 2381 ERROR oslo.messaging._drivers.impl_rabbit [req-aa7132bf-38d3-4e1f-9158-8743e2c5d163 - - - - -] AMQP server on 192.168.25.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2017-08-17 08:00:10.610 2381 ERROR oslo.messaging._drivers.impl_rabbit [req-aa7132bf-38d3-4e1f-9158-8743e2c5d163 - - - - -] AMQP server on 192.168.25.2:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds.
2017-08-17 08:00:30.640 2381 INFO oslo.messaging._drivers.impl_rabbit [req-aa7132bf-38d3-4e1f-9158-8743e2c5d163 - - - - -] Reconnected to AMQP server on 192.168.25.1:5672 via [amqp] client
2017-08-17 08:00:30.724 2381 INFO eventlet.wsgi.server [-] (2381) wsgi starting up on http:/var/lib/neutron/keepalived-state-change
2017-08-17 08:00:30.766 2381 INFO neutron.agent.l3.agent [-] L3 agent started
2017-08-17 08:00:30.770 2381 INFO neutron.agent.l3.agent [-] Agent has just been revived. Doing a full sync.
2017-08-17 08:00:35.352 2381 INFO oslo_rootwrap.client [-] Spawned new rootwrap daemon process with pid=25789
2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task [req-104a5ce9-3d9d-4367-8bb1-edb0880ef96f - - - - -] Error during L3NATAgentWithStateReport.periodic_sync_routers_task
2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task Traceback (most recent call last):
2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task     task(self, context)
2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 545, in periodic_sync_routers_task
2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task     self.fetch_and_sync_all_routers(context, ns_manager)
2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 579, in fetch_and_sync_all_routers
2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task     r['id'], r.get(l3_constants.HA_ROUTER_STATE_KEY))
2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha.py", line 120, in check_ha_state_for_router
2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task     if ri and current_state != TRANSLATION_MAP[ri.ha_state]:
2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 76, in ha_state
2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task     ha_state_path = self.keepalived_manager.get_full_config_file_path(
2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task AttributeError: 'NoneType' object has no attribute 'get_full_config_file_path'
2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task 
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "snat-9ed03dce-1c07-4b65-abe1-ca4f0e8f5d04": No such file or directory

2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent [-] Failed to process compatible router '9ed03dce-1c07-4b65-abe1-ca4f0e8f5d04'
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 501, in _process_router_update
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 438, in _process_router_if_compatible
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     self._process_added_router(router)
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 443, in _process_added_router
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     self._router_added(router['id'], router)
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 350, in _router_added
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     ri.initialize(self.process_monitor)
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_edge_ha_router.py", line 87, in initialize
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     self._create_snat_namespace()
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_edge_router.py", line 166, in _create_snat_namespace
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     self.snat_namespace.create()
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_snat_ns.py", line 33, in create
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     super(SnatNamespace, self).create()
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/namespaces.py", line 95, in create
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     ip_wrapper.netns.execute(cmd)
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 957, in execute
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     log_fail_as_error=log_fail_as_error, **kwargs)
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 146, in execute
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     raise ProcessExecutionError(msg, returncode=returncode)
2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "snat-9ed03dce-1c07-4b65-abe1-ca4f0e8f5d04": No such file or directory

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1712795

Title:
  Fail to startup neutron-l3-agent

Status in neutron:
  New

Bug description:
  2017-08-17 08:00:09.601 2381 ERROR oslo.messaging._drivers.impl_rabbit [req-aa7132bf-38d3-4e1f-9158-8743e2c5d163 - - - - -] AMQP server on 192.168.25.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
  2017-08-17 08:00:10.610 2381 ERROR oslo.messaging._drivers.impl_rabbit [req-aa7132bf-38d3-4e1f-9158-8743e2c5d163 - - - - -] AMQP server on 192.168.25.2:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds.
  2017-08-17 08:00:30.640 2381 INFO oslo.messaging._drivers.impl_rabbit [req-aa7132bf-38d3-4e1f-9158-8743e2c5d163 - - - - -] Reconnected to AMQP server on 192.168.25.1:5672 via [amqp] client
  2017-08-17 08:00:30.724 2381 INFO eventlet.wsgi.server [-] (2381) wsgi starting up on http:/var/lib/neutron/keepalived-state-change
  2017-08-17 08:00:30.766 2381 INFO neutron.agent.l3.agent [-] L3 agent started
  2017-08-17 08:00:30.770 2381 INFO neutron.agent.l3.agent [-] Agent has just been revived. Doing a full sync.
  2017-08-17 08:00:35.352 2381 INFO oslo_rootwrap.client [-] Spawned new rootwrap daemon process with pid=25789
  2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task [req-104a5ce9-3d9d-4367-8bb1-edb0880ef96f - - - - -] Error during L3NATAgentWithStateReport.periodic_sync_routers_task
  2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task Traceback (most recent call last):
  2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
  2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task     task(self, context)
  2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 545, in periodic_sync_routers_task
  2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task     self.fetch_and_sync_all_routers(context, ns_manager)
  2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 579, in fetch_and_sync_all_routers
  2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task     r['id'], r.get(l3_constants.HA_ROUTER_STATE_KEY))
  2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha.py", line 120, in check_ha_state_for_router
  2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task     if ri and current_state != TRANSLATION_MAP[ri.ha_state]:
  2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 76, in ha_state
  2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task     ha_state_path = self.keepalived_manager.get_full_config_file_path(
  2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task AttributeError: 'NoneType' object has no attribute 'get_full_config_file_path'
  2017-08-17 08:00:35.486 2381 ERROR oslo_service.periodic_task 
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "snat-9ed03dce-1c07-4b65-abe1-ca4f0e8f5d04": No such file or directory

  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent [-] Failed to process compatible router '9ed03dce-1c07-4b65-abe1-ca4f0e8f5d04'
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 501, in _process_router_update
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 438, in _process_router_if_compatible
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     self._process_added_router(router)
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 443, in _process_added_router
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     self._router_added(router['id'], router)
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 350, in _router_added
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     ri.initialize(self.process_monitor)
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_edge_ha_router.py", line 87, in initialize
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     self._create_snat_namespace()
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_edge_router.py", line 166, in _create_snat_namespace
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     self.snat_namespace.create()
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_snat_ns.py", line 33, in create
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     super(SnatNamespace, self).create()
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/namespaces.py", line 95, in create
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     ip_wrapper.netns.execute(cmd)
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 957, in execute
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     log_fail_as_error=log_fail_as_error, **kwargs)
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 146, in execute
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent     raise ProcessExecutionError(msg, returncode=returncode)
  2017-08-17 08:00:35.495 2381 ERROR neutron.agent.l3.agent ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "snat-9ed03dce-1c07-4b65-abe1-ca4f0e8f5d04": No such file or directory

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1712795/+subscriptions
Follow ups

[Bug 1712795] Re: Fail to startup neutron-l3-agent
From: Swaminathan Vasudevan, 2017-10-03