yahoo-eng-team team mailing list archive

[Daniel Kuffner <daniel.kuffner@xxxxxxxxx>]

Public bug reported:

I was playing around with cloud-init. I wanted to use cloud init as substitute for the missing environment variables feature.
The basic idea is to define variables in the user data and inject the them before starting the service.

Following:
docker run -e "MY_Variable=MyValue" -d centos

would look like in openstack:

nova boot --image centos:latest --userdata "MY_Variable=MyValue"
myinstance

Unfortunately does the metadata service not work inside of a docker
container. After some testing I figured out that the reason for that is
that a container uses as default gateway the docker network (docker ip
address). The metadata service simple rejects the call since the IP
address of docker container is not associated with the nova instance.

I was able to work around the issue by simply changing the route inside
the container:

# Hack: In order to receive data from the metadata service we must make sure we resolve the data via our nova network.
#
# A docker container in openstack has two NICs.
# - eth0 has a IP address on the docker0 bridge which is usually an e.g. 172.0.0.0 IP address.
# - pvnetXXXX is a IP address assigned by nova.
#
# Extract the NIC name of the nova network.
#
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)

while [ "$NOVA_NIC" == "" ] ; do
   echo "Find nova NIC..."
   sleep 1
   NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
done

echo "Device $NOVA_NIC found. Wait until ready."
sleep 3

# Setup a network route to insure we use the nova network.
#
echo "[INFO] Create default route for $NOVA_NIC. Gateway 10.0.0.1"
ip r r default via 10.0.0.1 dev $NOVA_NIC

# Shutdown eth0 since icps will fetch enabled enterface for streaming.
ip l set down dev eth0

This approach is obviously a poor solution since it has certain
expectation of the network.

Another solution might be extend the docker driver to add a firewall
rule which will masquerade requests on 169.254.169.254 with the actual
nova network IP address

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: docker

** Description changed:

- 
- I was playing around with cloud-init. I wanted to use cloud init as substitute for the mission environment variables.
- The basic idea is to define variables in the user data and inject the them before starting the service. 
+ I was playing around with cloud-init. I wanted to use cloud init as substitute for the missing environment variables feature.
+ The basic idea is to define variables in the user data and inject the them before starting the service.
  
  Following:
- docker run -e "MY_Variable=MyValue" -d centos 
+ docker run -e "MY_Variable=MyValue" -d centos
  
  would look like in openstack:
  
  nova boot --image centos:latest --userdata "MY_Variable=MyValue"
  myinstance
  
- Unfortunately does the metadata service not work inside of a docker container. 
- After some testing I figured out that the reason for that is that a container 
+ Unfortunately does the metadata service not work inside of a docker container.
+ After some testing I figured out that the reason for that is that a container
  uses as default gateway the docker network (docker ip address). The metadata service
- simple rejects the call since the IP address of docker container is not associated with 
+ simple rejects the call since the IP address of docker container is not associated with
  the nova instance.
  
  I was able to work around the issue by simply changing the route inside
  the container:
  
- 
  # Hack: In order to receive data from the metadata service we must make sure we resolve the data via our nova network.
  #
- # A docker container in openstack has two NICs. 
+ # A docker container in openstack has two NICs.
  # - eth0 has a IP address on the docker0 bridge which is usually an e.g. 172.0.0.0 IP address.
  # - pvnetXXXX is a IP address assigned by nova.
  #
  # Extract the NIC name of the nova network.
  #
  NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
  
  while [ "$NOVA_NIC" == "" ] ; do
-    echo "Find nova NIC..."
-    sleep 1
-    NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
+    echo "Find nova NIC..."
+    sleep 1
+    NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
  done
  
  echo "Device $NOVA_NIC found. Wait until ready."
  sleep 3
  
  # Setup a network route to insure we use the nova network.
  #
  echo "[INFO] Create default route for $NOVA_NIC. Gateway 10.0.0.1"
  ip r r default via 10.0.0.1 dev $NOVA_NIC
  
  # Shutdown eth0 since icps will fetch enabled enterface for streaming.
  ip l set down dev eth0
  
  This approach is obviously a poor solution since it has certain
  expectation of the network.
  
  Another solution might be extend the docker driver to add a firewall
  rule which will masquerade requests on 169.254.169.254 with the actual
  nova network IP address

** Description changed:

  I was playing around with cloud-init. I wanted to use cloud init as substitute for the missing environment variables feature.
  The basic idea is to define variables in the user data and inject the them before starting the service.
  
  Following:
  docker run -e "MY_Variable=MyValue" -d centos
  
  would look like in openstack:
  
  nova boot --image centos:latest --userdata "MY_Variable=MyValue"
  myinstance
  
- Unfortunately does the metadata service not work inside of a docker container.
- After some testing I figured out that the reason for that is that a container
- uses as default gateway the docker network (docker ip address). The metadata service
- simple rejects the call since the IP address of docker container is not associated with
- the nova instance.
+ Unfortunately does the metadata service not work inside of a docker
+ container. After some testing I figured out that the reason for that is
+ that a container uses as default gateway the docker network (docker ip
+ address). The metadata service simple rejects the call since the IP
+ address of docker container is not associated with the nova instance.
  
  I was able to work around the issue by simply changing the route inside
  the container:
  
  # Hack: In order to receive data from the metadata service we must make sure we resolve the data via our nova network.
  #
  # A docker container in openstack has two NICs.
  # - eth0 has a IP address on the docker0 bridge which is usually an e.g. 172.0.0.0 IP address.
  # - pvnetXXXX is a IP address assigned by nova.
  #
  # Extract the NIC name of the nova network.
  #
  NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
  
  while [ "$NOVA_NIC" == "" ] ; do
     echo "Find nova NIC..."
     sleep 1
     NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
  done
  
  echo "Device $NOVA_NIC found. Wait until ready."
  sleep 3
  
  # Setup a network route to insure we use the nova network.
  #
  echo "[INFO] Create default route for $NOVA_NIC. Gateway 10.0.0.1"
  ip r r default via 10.0.0.1 dev $NOVA_NIC
  
  # Shutdown eth0 since icps will fetch enabled enterface for streaming.
  ip l set down dev eth0
  
  This approach is obviously a poor solution since it has certain
  expectation of the network.
  
  Another solution might be extend the docker driver to add a firewall
  rule which will masquerade requests on 169.254.169.254 with the actual
  nova network IP address

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1259267

Title:
  Docker: Metadata service doesn't work

Status in OpenStack Compute (Nova):
  New

Bug description:
  I was playing around with cloud-init. I wanted to use cloud init as substitute for the missing environment variables feature.
  The basic idea is to define variables in the user data and inject the them before starting the service.

  Following:
  docker run -e "MY_Variable=MyValue" -d centos

  would look like in openstack:

  nova boot --image centos:latest --userdata "MY_Variable=MyValue"
  myinstance

  Unfortunately does the metadata service not work inside of a docker
  container. After some testing I figured out that the reason for that
  is that a container uses as default gateway the docker network (docker
  ip address). The metadata service simple rejects the call since the IP
  address of docker container is not associated with the nova instance.

  I was able to work around the issue by simply changing the route
  inside the container:

  # Hack: In order to receive data from the metadata service we must make sure we resolve the data via our nova network.
  #
  # A docker container in openstack has two NICs.
  # - eth0 has a IP address on the docker0 bridge which is usually an e.g. 172.0.0.0 IP address.
  # - pvnetXXXX is a IP address assigned by nova.
  #
  # Extract the NIC name of the nova network.
  #
  NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)

  while [ "$NOVA_NIC" == "" ] ; do
     echo "Find nova NIC..."
     sleep 1
     NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
  done

  echo "Device $NOVA_NIC found. Wait until ready."
  sleep 3

  # Setup a network route to insure we use the nova network.
  #
  echo "[INFO] Create default route for $NOVA_NIC. Gateway 10.0.0.1"
  ip r r default via 10.0.0.1 dev $NOVA_NIC

  # Shutdown eth0 since icps will fetch enabled enterface for streaming.
  ip l set down dev eth0

  This approach is obviously a poor solution since it has certain
  expectation of the network.

  Another solution might be extend the docker driver to add a firewall
  rule which will masquerade requests on 169.254.169.254 with the actual
  nova network IP address

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1259267/+subscriptions