yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1675151] [NEW] Switch to haproxy fails dhcp request

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Rajini Ram <rajini_ram@xxxxxxxx>
Date: Wed, 22 Mar 2017 19:58:33 -0000
Reply-to: Bug 1675151 <1675151@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Looks like the patch to Switch ns-metadata-proxy to haproxy is breaking our Dell Ironic CI builds.
haproxy doesn't like the config file. 


The patch https://review.openstack.org/#/c/431691/ broke our Ironic CI builds, can you help look at the logs and help me fix?
Full logs
https://stash.opencrowbar.org/logs/71/446571/12/check/dell-hw-tempest-dsvm-ironic-pxe_ipmitool/3a6e628/
 
f6ecb-068d-4c89-8fe9-c4c14f0228ee.pid get_value_from_file /opt/stack/new/neutron/neutron/agent/linux/utils.py:252
2017-03-22 16:30:10.365 15528 DEBUG neutron.agent.metadata.driver [req-19ee385c-6764-4d9f-bf50-eab47d4bab38 - -] haproxy_cfg =
global
    log         /dev/log local0 debug
    user        stack
    group       stack
    maxconn     1024
    pidfile     /opt/stack/data/neutron/external/pids/075f6ecb-068d-4c89-8fe9-c4c14f0228ee.pid
    daemon
 
defaults
    log global
    mode http
    option httplog
    option dontlognull
    option http-server-close
    option forwardfor
    retries                 3
    timeout http-request    30s
    timeout connect         30s
    timeout client          32s
    timeout server          32s
    timeout http-keep-alive 30s
 
listen listener
    bind 0.0.0.0:80
    server metadata /opt/stack/data/neutron/metadata_proxy
    http-request add-header X-Neutron-Network-ID 075f6ecb-068d-4c89-8fe9-c4c14f0228ee
create_config_file /opt/stack/new/neutron/neutron/agent/metadata/driver.py:126
2017-03-22 16:30:10.366 15528 DEBUG neutron.agent.linux.utils [req-19ee385c-6764-4d9f-bf50-eab47d4bab38 - -] Running command (rootwrap daemon): ['ip', 'netns', 'exec'
, 'qdhcp-075f6ecb-068d-4c89-8fe9-c4c14f0228ee', 'haproxy', '-f', '/opt/stack/data/neutron/ns-metadata-proxy/075f6ecb-068d-4c89-8fe9-c4c14f0228ee.conf'] execute_rootwr
ap_daemon /opt/stack/new/neutron/neutron/agent/linux/utils.py:108
2017-03-22 16:30:10.440 15528 ERROR neutron.agent.linux.utils [req-19ee385c-6764-4d9f-bf50-eab47d4bab38 - -] Exit code: 1; Stdin: ; Stdout: ; Stderr: [ALERT] 080/1630
10 (16482) : parsing [/opt/stack/data/neutron/ns-metadata-proxy/075f6ecb-068d-4c89-8fe9-c4c14f0228ee.conf:26] : Unknown host in '/opt/stack/data/neutron/metadata_prox
y'
[ALERT] 080/163010 (16482) : parsing [/opt/stack/data/neutron/ns-metadata-proxy/075f6ecb-068d-4c89-8fe9-c4c14f0228ee.conf:27]: unknown parameter 'X-Neutron-Network-ID
', expects 'allow', 'deny', 'auth'.
[ALERT] 080/163010 (16482) : Error(s) found in configuration file : /opt/stack/data/neutron/ns-metadata-proxy/075f6ecb-068d-4c89-8fe9-c4c14f0228ee.conf
 
2017-03-22 16:30:10.440 15528 DEBUG oslo_concurrency.lockutils [req-19ee385c-6764-4d9f-bf50-eab47d4bab38 - -] Releasing semaphore "dhcp-agent-network-lock-075f6ecb-06
8d-4c89-8fe9-c4c14f0228ee" lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server [req-19ee385c-6764-4d9f-bf50-eab47d4bab38 - -] Exception during message handling
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/o
017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server [req-19ee385c-6764-4d9f-bf50-eab47d4bab38 - -] Exception during message handling
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_inco
ming
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispa
tch
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 153, in wrapper
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/dhcp/agent.py", line 61, in wrapped
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/dhcp/agent.py", line 363, in network_create_end
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     self.enable_dhcp_helper(network_id)
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 153, in wrapper
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/dhcp/agent.py", line 282, in enable_dhcp_helper
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     self.configure_dhcp_for_network(network)
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 153, in wrapper
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/dhcp/agent.py", line 303, in configure_dhcp_for_network
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     self.update_isolated_metadata_proxy(network)
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 153, in wrapper
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/dhcp/agent.py", line 484, in update_isolated_metadata_proxy
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     self.enable_isolated_metadata_proxy(network)
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 153, in wrapper
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/dhcp/agent.py", line 515, in enable_isolated_metadata_proxy
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     self.conf, **kwargs)
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/metadata/driver.py", line 241, in spawn_monitored_metadata_
proxy
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     pm.enable()
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/linux/external_process.py", line 96, in enable
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     run_as_root=self.run_as_root)
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 900, in execute
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     log_fail_as_error=log_fail_as_error, **kwargs)
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 147, in execute
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     raise ProcessExecutionError(msg, returncode=returncode)
2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: [ALERT] 080/163010 (16482) : parsing [/o
pt/stack/data/neutron/ns-metadata-proxy/075f6ecb-068d-4c89-8fe9-c4c14f0228ee.conf:26] : Unknown host in '/opt/stack/data/

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1675151

Title:
  Switch to haproxy fails dhcp request

Status in neutron:
  New

Bug description:
  Looks like the patch to Switch ns-metadata-proxy to haproxy is breaking our Dell Ironic CI builds.
  haproxy doesn't like the config file. 

  
  The patch https://review.openstack.org/#/c/431691/ broke our Ironic CI builds, can you help look at the logs and help me fix?
  Full logs
  https://stash.opencrowbar.org/logs/71/446571/12/check/dell-hw-tempest-dsvm-ironic-pxe_ipmitool/3a6e628/
   
  f6ecb-068d-4c89-8fe9-c4c14f0228ee.pid get_value_from_file /opt/stack/new/neutron/neutron/agent/linux/utils.py:252
  2017-03-22 16:30:10.365 15528 DEBUG neutron.agent.metadata.driver [req-19ee385c-6764-4d9f-bf50-eab47d4bab38 - -] haproxy_cfg =
  global
      log         /dev/log local0 debug
      user        stack
      group       stack
      maxconn     1024
      pidfile     /opt/stack/data/neutron/external/pids/075f6ecb-068d-4c89-8fe9-c4c14f0228ee.pid
      daemon
   
  defaults
      log global
      mode http
      option httplog
      option dontlognull
      option http-server-close
      option forwardfor
      retries                 3
      timeout http-request    30s
      timeout connect         30s
      timeout client          32s
      timeout server          32s
      timeout http-keep-alive 30s
   
  listen listener
      bind 0.0.0.0:80
      server metadata /opt/stack/data/neutron/metadata_proxy
      http-request add-header X-Neutron-Network-ID 075f6ecb-068d-4c89-8fe9-c4c14f0228ee
  create_config_file /opt/stack/new/neutron/neutron/agent/metadata/driver.py:126
  2017-03-22 16:30:10.366 15528 DEBUG neutron.agent.linux.utils [req-19ee385c-6764-4d9f-bf50-eab47d4bab38 - -] Running command (rootwrap daemon): ['ip', 'netns', 'exec'
  , 'qdhcp-075f6ecb-068d-4c89-8fe9-c4c14f0228ee', 'haproxy', '-f', '/opt/stack/data/neutron/ns-metadata-proxy/075f6ecb-068d-4c89-8fe9-c4c14f0228ee.conf'] execute_rootwr
  ap_daemon /opt/stack/new/neutron/neutron/agent/linux/utils.py:108
  2017-03-22 16:30:10.440 15528 ERROR neutron.agent.linux.utils [req-19ee385c-6764-4d9f-bf50-eab47d4bab38 - -] Exit code: 1; Stdin: ; Stdout: ; Stderr: [ALERT] 080/1630
  10 (16482) : parsing [/opt/stack/data/neutron/ns-metadata-proxy/075f6ecb-068d-4c89-8fe9-c4c14f0228ee.conf:26] : Unknown host in '/opt/stack/data/neutron/metadata_prox
  y'
  [ALERT] 080/163010 (16482) : parsing [/opt/stack/data/neutron/ns-metadata-proxy/075f6ecb-068d-4c89-8fe9-c4c14f0228ee.conf:27]: unknown parameter 'X-Neutron-Network-ID
  ', expects 'allow', 'deny', 'auth'.
  [ALERT] 080/163010 (16482) : Error(s) found in configuration file : /opt/stack/data/neutron/ns-metadata-proxy/075f6ecb-068d-4c89-8fe9-c4c14f0228ee.conf
   
  2017-03-22 16:30:10.440 15528 DEBUG oslo_concurrency.lockutils [req-19ee385c-6764-4d9f-bf50-eab47d4bab38 - -] Releasing semaphore "dhcp-agent-network-lock-075f6ecb-06
  8d-4c89-8fe9-c4c14f0228ee" lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server [req-19ee385c-6764-4d9f-bf50-eab47d4bab38 - -] Exception during message handling
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/o
  017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server [req-19ee385c-6764-4d9f-bf50-eab47d4bab38 - -] Exception during message handling
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_inco
  ming
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispa
  tch
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 153, in wrapper
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/dhcp/agent.py", line 61, in wrapped
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/dhcp/agent.py", line 363, in network_create_end
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     self.enable_dhcp_helper(network_id)
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 153, in wrapper
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/dhcp/agent.py", line 282, in enable_dhcp_helper
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     self.configure_dhcp_for_network(network)
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 153, in wrapper
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/dhcp/agent.py", line 303, in configure_dhcp_for_network
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     self.update_isolated_metadata_proxy(network)
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 153, in wrapper
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/dhcp/agent.py", line 484, in update_isolated_metadata_proxy
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     self.enable_isolated_metadata_proxy(network)
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 153, in wrapper
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/dhcp/agent.py", line 515, in enable_isolated_metadata_proxy
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     self.conf, **kwargs)
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/metadata/driver.py", line 241, in spawn_monitored_metadata_
  proxy
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     pm.enable()
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/linux/external_process.py", line 96, in enable
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     run_as_root=self.run_as_root)
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 900, in execute
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     log_fail_as_error=log_fail_as_error, **kwargs)
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server   File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 147, in execute
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server     raise ProcessExecutionError(msg, returncode=returncode)
  2017-03-22 16:30:10.441 15528 ERROR oslo_messaging.rpc.server ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: [ALERT] 080/163010 (16482) : parsing [/o
  pt/stack/data/neutron/ns-metadata-proxy/075f6ecb-068d-4c89-8fe9-c4c14f0228ee.conf:26] : Unknown host in '/opt/stack/data/

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1675151/+subscriptions
Follow ups

[Bug 1675151] Re: Switch to haproxy fails dhcp request
From: Ihar Hrachyshka, 2017-03-27