← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #57899

[Bug 1634123] [NEW] neutron openvswitch agent exists if unix:/var/run/openvswitch/db.sock is not yet created

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

In kubernetes environment is very difficult to make sure that neutron-
openvswitch-agent pod starts after openvswitch db and vswitchd pods
during kubernetes cluster startup process. As a result it does not find
socket, fails and stay in failed state. If instead of just failing it
could retry several times then result would be different as by that time
openvswitch would have come up and created db socket. It would be great
to add retry mechanism to neutron-openvswitch-agent to make it more
robust in kubernetes environment.

neutron (9.0.0)
neutron-lib (0.4.0)
python-neutronclient (6.0.0)

  
2016-10-17 13:05:57.922 36 INFO ryu.base.app_manager [-] loading app ryu.app.ofctl.service
2016-10-17 13:05:57.984 36 INFO ryu.base.app_manager [-] loading app ryu.controller.ofp_handler
2016-10-17 13:05:57.989 36 INFO ryu.base.app_manager [-] instantiating app neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp of OVSNeutronAgentRyuApp
2016-10-17 13:05:57.990 36 INFO ryu.base.app_manager [-] instantiating app ryu.controller.ofp_handler of OFPHandler
2016-10-17 13:05:57.991 36 INFO ryu.base.app_manager [-] instantiating app ryu.app.ofctl.service of OfctlService
2016-10-17 13:05:57.994 36 DEBUG neutron.callbacks.manager [-] Subscribe: <function init_handler at 0x4549ed8> Open vSwitch agent after_init subscribe /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/callbacks/manager.py:42
2016-10-17 13:05:57.995 36 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', 'addr', 'show', 'to', '172.29.75.24'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:83
2016-10-17 13:05:58.181 36 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:140
2016-10-17 13:05:58.225 36 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', 'set-manager', 'ptcp:6640:0.0.0.0'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:83
2016-10-17 13:05:58.792 36 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)

2016-10-17 13:05:58.793 36 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
 Agent terminated!
2016-10-17 13:05:58.796 36 ERROR ryu.lib.hub [-] hub: uncaught exception: Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/ryu/lib/hub.py", line 54, in _launch
    return func(*args, **kwargs)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py", line 37, in agent_main_wrapper
    ovs_agent.main(bridge_classes)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2174, in main
    sys.exit(1)
SystemExit: 1

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1634123

Title:
  neutron openvswitch agent exists if unix:/var/run/openvswitch/db.sock
  is not yet created

Status in neutron:
  New

Bug description:
  In kubernetes environment is very difficult to make sure that neutron-
  openvswitch-agent pod starts after openvswitch db and vswitchd pods
  during kubernetes cluster startup process. As a result it does not
  find socket, fails and stay in failed state. If instead of just
  failing it could retry several times then result would be different as
  by that time openvswitch would have come up and created db socket. It
  would be great to add retry mechanism to neutron-openvswitch-agent to
  make it more robust in kubernetes environment.

  neutron (9.0.0)
  neutron-lib (0.4.0)
  python-neutronclient (6.0.0)

    
  2016-10-17 13:05:57.922 36 INFO ryu.base.app_manager [-] loading app ryu.app.ofctl.service
  2016-10-17 13:05:57.984 36 INFO ryu.base.app_manager [-] loading app ryu.controller.ofp_handler
  2016-10-17 13:05:57.989 36 INFO ryu.base.app_manager [-] instantiating app neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp of OVSNeutronAgentRyuApp
  2016-10-17 13:05:57.990 36 INFO ryu.base.app_manager [-] instantiating app ryu.controller.ofp_handler of OFPHandler
  2016-10-17 13:05:57.991 36 INFO ryu.base.app_manager [-] instantiating app ryu.app.ofctl.service of OfctlService
  2016-10-17 13:05:57.994 36 DEBUG neutron.callbacks.manager [-] Subscribe: <function init_handler at 0x4549ed8> Open vSwitch agent after_init subscribe /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/callbacks/manager.py:42
  2016-10-17 13:05:57.995 36 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', 'addr', 'show', 'to', '172.29.75.24'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:83
  2016-10-17 13:05:58.181 36 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:140
  2016-10-17 13:05:58.225 36 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', 'set-manager', 'ptcp:6640:0.0.0.0'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:83
  2016-10-17 13:05:58.792 36 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)

  2016-10-17 13:05:58.793 36 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
   Agent terminated!
  2016-10-17 13:05:58.796 36 ERROR ryu.lib.hub [-] hub: uncaught exception: Traceback (most recent call last):
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/ryu/lib/hub.py", line 54, in _launch
      return func(*args, **kwargs)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py", line 37, in agent_main_wrapper
      ovs_agent.main(bridge_classes)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2174, in main
      sys.exit(1)
  SystemExit: 1

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1634123/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next