yahoo-eng-team team mailing list archive

[Huang Cheng <1717707@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Related to bug #1696094.

An 'Timed out waiting for nova-conductor.  Is it running? Or did this
service start before nova-conductor?  Reattempting establishment of
nova-conductor connection...' error occurs in nova-compute.log when:

on compute node
1. no usable nameserver in /etc/resolv.conf
2. only ipv4 or only ipv6 address of 'controller' (as rabbitmq server) is mapped in /etc/hosts
3. use 'controller' as rabbitmq server in nova.conf

The eventlet greendns has been always enabled by monkey_patch since
0.20.0, and this will introduce some compatibility problems, e.g.

1. We create a connection to rabbitmq server using 'controller:5672'
2. patched socket.getaddrinfo('controller', 5672, 0) is called by amqp (0 for both ipv4 and ipv6)
3. greendns will use '127.0.0.1' as dns nameserver if there is no usable nameserver in /etc/resolv.conf
4. greendns will perform name resolving for 'controller', ipv6 dns lookup will be performed if there is no ipv6 mapping for 'controller' in /etc/hosts, so is ipv4. One of the dns lookup is leading to a timeout, and cause the problem mentioned above.

I think it's better to use an eventlet version < 0.20.0.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: amqp dns eventlet

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1717707

Title:
  nova-compute failed to communicate with nova-conductor on start

Status in OpenStack Compute (nova):
  New

Bug description:
  Related to bug #1696094.

  An 'Timed out waiting for nova-conductor.  Is it running? Or did this
  service start before nova-conductor?  Reattempting establishment of
  nova-conductor connection...' error occurs in nova-compute.log when:

  on compute node
  1. no usable nameserver in /etc/resolv.conf
  2. only ipv4 or only ipv6 address of 'controller' (as rabbitmq server) is mapped in /etc/hosts
  3. use 'controller' as rabbitmq server in nova.conf

  The eventlet greendns has been always enabled by monkey_patch since
  0.20.0, and this will introduce some compatibility problems, e.g.

  1. We create a connection to rabbitmq server using 'controller:5672'
  2. patched socket.getaddrinfo('controller', 5672, 0) is called by amqp (0 for both ipv4 and ipv6)
  3. greendns will use '127.0.0.1' as dns nameserver if there is no usable nameserver in /etc/resolv.conf
  4. greendns will perform name resolving for 'controller', ipv6 dns lookup will be performed if there is no ipv6 mapping for 'controller' in /etc/hosts, so is ipv4. One of the dns lookup is leading to a timeout, and cause the problem mentioned above.

  I think it's better to use an eventlet version < 0.20.0.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1717707/+subscriptions