yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1249609] Re: Metadata agent fails with "Connection to neutron failed: Maximum attempts reached"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Maru Newby <marun@xxxxxxxxxx>
Date: Tue, 12 Nov 2013 02:35:33 -0000
Reply-to: Bug 1249609 <1249609@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: neutron
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1249609

Title:
  Metadata agent fails with "Connection to neutron failed: Maximum
  attempts reached"

Status in OpenStack Neutron (virtual network service):
  Invalid

Bug description:
  Using the current Havana release from the Cloud Archive
  (2013.2-0ubuntu1~cloud0).

  I have applied the patches that fix
  https://bugs.launchpad.net/neutron/+bug/1211915 and
  https://bugs.launchpad.net/nova/+bug/1229475

  Still getting the error every time I try to access the metadata
  service. This will trigger the error below:

  ip netns exec qdhcp-e164fe97-f599-4b2e-81c9-ebca991a8af1 curl http://169.254.169.254/latest/meta-data/
  <html>
   <head>
    <title>500 Internal Server Error</title>
   </head>
   <body>
    <h1>500 Internal Server Error</h1>
    Remote metadata server experienced an internal server error.<br /><br />


   </body>

  in /var/log/neutron/metadata-agent.log:

  2013-11-09 06:11:40.513 42522 ERROR neutron.agent.metadata.agent [-] Unexpected error.
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent Traceback (most recent call last):
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 94, in __call__
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent     instance_id = self._get_instance_id(req)
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 124, in _get_instance_id
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent     fixed_ips=['ip_address=%s' % remote_address])['ports']
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 108, in with_params
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent     ret = self.function(instance, *args, **kwargs)
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 298, in list_ports
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent     **_params)
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1197, in list
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent     for r in self._pagination(collection, path, **params):
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1210, in _pagination
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent     res = self.get(path, params=params)
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1183, in get
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent     headers=headers, params=params)
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1175, in retry_request
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent     raise exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent ConnectionFailed: Connection to neutron failed: Maximum attempts reached
  2013-11-09 06:11:40.513 42522 TRACE neutron.agent.metadata.agent

  I have a neutron flat network configured:

  /etc/neutron/neutron.conf:

  [DEFAULT]
  state_path = /var/lib/neutron
  lock_path = $state_path/lock
  core_plugin = neutron.plugins.openvswitch.ovs_neutron_plugin.OVSNeutronPluginV2
  auth_strategy = keystone
  rpc_backend = neutron.openstack.common.rpc.impl_kombu
  rabbit_host = 10.0.0.61
  notification_driver = neutron.openstack.common.notifier.rpc_notifier
  [quotas]
  [agent]
  root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
  [keystone_authtoken]
  auth_host = 10.0.0.61
  auth_port = 35357
  auth_protocol = http
  admin_tenant_name = service
  admin_user = neutron
  admin_password = service_pass
  signing_dir = $state_path/keystone-signing
  [database]
  sql_connection = mysql://neutronUser:neutronPass@10.0.0.61/neutron
  connection_debug = 0
  [service_providers]
  service_provider=LOADBALANCER:Haproxy:neutron.services.loadbalancer.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default

  /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini :
  [ovs]
  network_vlan_ranges = physnet1
  bridge_mappings = physnet1:br-eth2
  [agent]
  [securitygroup]
  firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
  [database]
  connection = mysql://neutronUser:neutronPass@10.0.0.61/neutron

  # neutron net-show flat-provider-network
  +---------------------------+--------------------------------------+
  | Field                     | Value                                |
  +---------------------------+--------------------------------------+
  | admin_state_up            | True                                 |
  | id                        | e164fe97-f599-4b2e-81c9-ebca991a8af1 |
  | name                      | flat-provider-network                |
  | provider:network_type     | flat                                 |
  | provider:physical_network | physnet1                             |
  | provider:segmentation_id  |                                      |
  | router:external           | False                                |
  | shared                    | True                                 |
  | status                    | ACTIVE                               |
  | subnets                   | 87a0b91e-1875-4849-8c57-25f51ecdd87f |
  | tenant_id                 | fefce81749874bcb9b2030b1f6ade9dc     |
  +---------------------------+--------------------------------------+

  /etc/neutron/metadata_agent.ini:
  [DEFAULT]
  debug = False
  auth_url = http://10.0.0.61:5000/v2.0
  auth_region = RegionOne
  admin_tenant_name = service
  admin_user = neutron
  admin_password = service_pass

  /etc/neutron# egrep -v "^$|^#" dhcp_agent.ini
  [DEFAULT]
  resync_interval = 5
  interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
  dhcp_driver = neutron.agent.linux.dhcp.Dnsmasq
  use_namespaces = True
  enable_isolated_metadata = True
  enable_metadata_network = True

  /etc/neutron/l3_agent.ini
  [DEFAULT]
  interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
  use_namespaces = True

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1249609/+subscriptions