yahoo-eng-team team mailing list archive

[Huang Cheng <1717707@xxxxxxxxxxxxxxxxxx>]

** Description changed:

  Related to bug #1696094.
  
  An 'Timed out waiting for nova-conductor.  Is it running? Or did this
  service start before nova-conductor?  Reattempting establishment of
  nova-conductor connection...' error occurs in nova-compute.log when:
  
  on compute node
  1. no usable nameserver in /etc/resolv.conf
  2. only ipv4 or only ipv6 address of 'controller' (as rabbitmq server) is mapped in /etc/hosts
  3. use 'controller' as rabbitmq server in nova.conf
  
  The eventlet greendns has been always enabled by monkey_patch since
  0.20.0, and this will introduce some compatibility problems, e.g.
  
  1. We create a connection to rabbitmq server using 'controller:5672'
  2. patched socket.getaddrinfo('controller', 5672, 0) is called by amqp (0 for both ipv4 and ipv6)
  3. greendns will use '127.0.0.1' as dns nameserver if there is no usable nameserver in /etc/resolv.conf
  4. greendns will perform name resolving for 'controller', ipv6 dns lookup will be performed if there is no ipv6 mapping for 'controller' in /etc/hosts, so is ipv4. One of the dns lookup is leading to a timeout, and cause the problem mentioned above.
  
- The original socket.getaddrinfo is ok with this situation, I think it's
- better not to use eventlet greendns patch for now.
+ Looks like 'EVENTLET_NO_GREENDNS' option is diabled only in eventlet 0.20.0~0.20.1, it's back in 0.21.0.
+ So temporarily there are two ways for nodes to work well under greened dns:
+ 1. put a usable nameserver in resolv.conf.
+ 2. map both ipv4 and ipv6 address for rabbitmq server in /etc/hosts.

** Description changed:

  Related to bug #1696094.
  
  An 'Timed out waiting for nova-conductor.  Is it running? Or did this
  service start before nova-conductor?  Reattempting establishment of
  nova-conductor connection...' error occurs in nova-compute.log when:
  
  on compute node
  1. no usable nameserver in /etc/resolv.conf
  2. only ipv4 or only ipv6 address of 'controller' (as rabbitmq server) is mapped in /etc/hosts
  3. use 'controller' as rabbitmq server in nova.conf
  
  The eventlet greendns has been always enabled by monkey_patch since
  0.20.0, and this will introduce some compatibility problems, e.g.
  
  1. We create a connection to rabbitmq server using 'controller:5672'
  2. patched socket.getaddrinfo('controller', 5672, 0) is called by amqp (0 for both ipv4 and ipv6)
  3. greendns will use '127.0.0.1' as dns nameserver if there is no usable nameserver in /etc/resolv.conf
  4. greendns will perform name resolving for 'controller', ipv6 dns lookup will be performed if there is no ipv6 mapping for 'controller' in /etc/hosts, so is ipv4. One of the dns lookup is leading to a timeout, and cause the problem mentioned above.
  
  Looks like 'EVENTLET_NO_GREENDNS' option is diabled only in eventlet 0.20.0~0.20.1, it's back in 0.21.0.
  So temporarily there are two ways for nodes to work well under greened dns:
  1. put a usable nameserver in resolv.conf.
- 2. map both ipv4 and ipv6 address for rabbitmq server in /etc/hosts.
+ 2. map both ipv4 and ipv6 address for rabbitmq server(e.g. controller) in /etc/hosts.

** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1717707

Title:
  nova-compute failed to communicate with nova-conductor on start

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Related to bug #1696094.

  An 'Timed out waiting for nova-conductor.  Is it running? Or did this
  service start before nova-conductor?  Reattempting establishment of
  nova-conductor connection...' error occurs in nova-compute.log when:

  on compute node
  1. no usable nameserver in /etc/resolv.conf
  2. only ipv4 or only ipv6 address of 'controller' (as rabbitmq server) is mapped in /etc/hosts
  3. use 'controller' as rabbitmq server in nova.conf

  The eventlet greendns has been always enabled by monkey_patch since
  0.20.0, and this will introduce some compatibility problems, e.g.

  1. We create a connection to rabbitmq server using 'controller:5672'
  2. patched socket.getaddrinfo('controller', 5672, 0) is called by amqp (0 for both ipv4 and ipv6)
  3. greendns will use '127.0.0.1' as dns nameserver if there is no usable nameserver in /etc/resolv.conf
  4. greendns will perform name resolving for 'controller', ipv6 dns lookup will be performed if there is no ipv6 mapping for 'controller' in /etc/hosts, so is ipv4. One of the dns lookup is leading to a timeout, and cause the problem mentioned above.

  Looks like 'EVENTLET_NO_GREENDNS' option is diabled only in eventlet 0.20.0~0.20.1, it's back in 0.21.0.
  So temporarily there are two ways for nodes to work well under greened dns:
  1. put a usable nameserver in resolv.conf.
  2. map both ipv4 and ipv6 address for rabbitmq server(e.g. controller) in /etc/hosts.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1717707/+subscriptions