← Back to team overview

openstack team mailing list archive

bad nova-manage service list

 

Hi guys, 

I am trying to set up an openstack small cluster and I am facing a
problem.
I have two nodes, a controller node on which I have installed all the
nova-* services and a compute node on which I have installed
nova-compute and nova-network.

When I run 'nova-manage service list', I only see the nova-compute
service from the compute-node, not the other services from the
controller node. However, if I run 'ps aux | grep nova-' on the
controller node, all the nova-* are up and running. It seems that my
controller's nova-* services do not want to register to the mysql DB.

My guess is that services are not able to connect to the local mysql
db on the controller node may be due to iptables bad settings but I am
not able to figure it out.
Worth to mention it, my nova-controller node is running in a virtual
machine on virtual box, both nodes are ubuntu natty server amd64.

If someone have any idea on that problem, please let me know, I added
some more information about my configuration in attachment below.

Than you in advance.

Best Regards,
Ivan Frain

HERE IS MY NOVA.CONF FILE: IT IS THE SAME ON BOTH MACHINES: 

--dhcpbridge_flagfile=/etc/nova/nova.conf

--dhcpbridge=/usr/bin/nova-dhcpbridge

--logdir=/var/log/nova

--state_path=/var/lib/nova

--lock_path=/var/lock/nova

--verbose

--sql_connection=mysql://nova:notnova@nova-controller/nova

--s3_host=nova-controller

--rabbit_host=nova-controller

--ec2_api=nova-controller

--ec2_url=http://nova-controller:8773/services/Cloud

--fixed_range=192.168.0.0/16

--network_size=8

--routing_source_ip=nova-controller

THE OUTPUT OF PS AUX | GREP NOVA COMMAND:

root@nova-controller:~# ps aux | grep nova

rabbitmq  1118  0.9  5.3  53644 27224 ?        Ssl  21:06  
0:47 /usr/lib/erlang/erts-5.7.4/bin/beam -W w -K true -A30 -P 1048576
-- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq --
-noshell -noinput -sname rabbit@nova-controller -boot
/var/lib/rabbitmq/mnesia/rabbit@nova-controller-plugins-expand/rabbit
-kernel inet_default_listen_options [{nodelay,true}] -kernel
inet_default_connect_options [{nodelay,true}] -sasl errlog_type error
-kernel error_logger
{file,"/var/log/rabbitmq/rabbit@xxxxxxxxxxxxxxxxxxx"} -sasl
sasl_error_logger
{file,"/var/log/rabbitmq/rabbit@xxxxxxxxxxxxxxxxxxxxxxxx"} -os_mon
start_cpu_sup true -os_mon start_disksup false -os_mon start_memsup
false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@nova-controller"
-noshell -noinput

root     19856  0.1  3.2  61228 16232 ?        S    22:24
  0:00 /usr/bin/python /usr/bin/nova-objectstore --uid 108 --gid
65534 --pidfile /var/run/nova/nova-objectstore.pid
--flagfile=/etc/nova/nova.conf --nodaemon
--logfile=/var/log/nova/nova-objectstore.log

nova     23074  0.0  0.2  35708  1280 ?        Ss   22:26
  0:00 su -c nova-compute --flagfile=/etc/nova/nova.conf nova

nova     23075 30.3  8.2 145228 41740 ?        R    22:26  
0:00 /usr/bin/python /usr/bin/nova-compute
--flagfile=/etc/nova/nova.conf

nova     23132  0.0  0.2  35708  1276 ?        Ss   22:26
  0:00 su -c nova-api --flagfile=/etc/nova/nova.conf nova

nova     23133 35.0  5.3  89264 26884 ?        R    22:26  
0:00 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf

nova     23145  0.0  0.2  35708  1280 ?        Ss   22:26
  0:00 su -c nova-network --flagfile=/etc/nova/nova.conf nova

nova     23146 23.0  4.2  75476 21596 ?        R    22:26  
0:00 /usr/bin/python /usr/bin/nova-network
--flagfile=/etc/nova/nova.conf

nova     23152  0.0  0.2  35708  1280 ?        Ss   22:26
  0:00 su -c nova-scheduler --flagfile=/etc/nova/nova.conf nova

nova     23153  0.0  3.7  72924 19112 ?        R    22:26
  0:00 /usr/bin/python /usr/bin/nova-scheduler
--flagfile=/etc/nova/nova.conf

root     23167  0.0  0.2   9136  1060 pts/1    S+   22:26  
0:00 grep --color=auto nova

THE OUTPUT OF SERVICE-MANAGE COMMAND:

root@nova-controller:~# nova-manage service list
Obelix     nova-compute enabled  :-) 2011-08-08 20:27:26

NB: Obelix is the name of my compute node.

MYSQL> SELECT HOST,USER,PASSWORD FROM USER;
+-----------------+------------------+-------------------------------------------+
| Host            | User             | Password      
                           |
+-----------------+------------------+-------------------------------------------+
| localhost       | root             |
*0BE3B501084D35F4C66DD3AC4569EAE5EA738212 |
| nova-controller | root             |
*0BE3B501084D35F4C66DD3AC4569EAE5EA738212 |
| 127.0.0.1       | root             |
*0BE3B501084D35F4C66DD3AC4569EAE5EA738212 |
| localhost       | debian-sys-maint |
*B23864E8C293F7A6E977A8CF31AC4ED99570A161 |
| %               | nova             |
*8134721C1A2B651300B26AC135712CD373E4D471 |
+-----------------+------------------+-------------------------------------------+

A PROBLEM IN THE OUTPUT FROM NOVA-NETWORK LOG (IPTABLES PROBLEM):
root@nova-controller:~# tail -n 50  /var/log/nova/nova-network.log 
2011-08-08 19:34:11,852 DEBUG nova.utils [-] Attempting to grab file
lock "iptables" for method "apply"... from (pid=10073) inner
/usr/lib/pymodules/python2.7/nova/utils.py:599
2011-08-08 19:34:11,854 DEBUG nova.utils [-] Running cmd (subprocess):
sudo iptables-save -t filter from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:150
2011-08-08 19:34:11,870 DEBUG nova.utils [-] Running cmd (subprocess):
sudo iptables-restore from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:150
2011-08-08 19:34:11,911 DEBUG nova.utils [-] Running cmd (subprocess):
sudo iptables-save -t nat from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:150
2011-08-08 19:34:11,929 DEBUG nova.utils [-] Running cmd (subprocess):
sudo iptables-restore from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:150
2011-08-08 19:34:11,963 DEBUG nova.utils [-] Result was 2 from
(pid=10073) execute /usr/lib/pymodules/python2.7/nova/utils.py:166
2011-08-08 19:34:11,963 DEBUG nova.utils [-] ['sudo',
'iptables-restore'] failed. Retrying. from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:179
2011-08-08 19:34:12,776 DEBUG nova.utils [-] Running cmd (subprocess):
sudo iptables-restore from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:150
2011-08-08 19:34:12,802 DEBUG nova.utils [-] Result was 2 from
(pid=10073) execute /usr/lib/pymodules/python2.7/nova/utils.py:166
2011-08-08 19:34:12,803 DEBUG nova.utils [-] ['sudo',
'iptables-restore'] failed. Retrying. from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:179
2011-08-08 19:34:14,616 DEBUG nova.utils [-] Running cmd (subprocess):
sudo iptables-restore from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:150
2011-08-08 19:34:14,641 DEBUG nova.utils [-] Result was 2 from
(pid=10073) execute /usr/lib/pymodules/python2.7/nova/utils.py:166
2011-08-08 19:34:14,642 DEBUG nova.utils [-] ['sudo',
'iptables-restore'] failed. Retrying. from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:179
2011-08-08 19:34:15,723 DEBUG nova.utils [-] Running cmd (subprocess):
sudo iptables-restore from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:150
2011-08-08 19:34:15,742 DEBUG nova.utils [-] Result was 2 from
(pid=10073) execute /usr/lib/pymodules/python2.7/nova/utils.py:166
2011-08-08 19:34:15,742 DEBUG nova.utils [-] ['sudo',
'iptables-restore'] failed. Retrying. from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:179
2011-08-08 19:34:16,703 DEBUG nova.utils [-] Running cmd (subprocess):
sudo iptables-restore from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:150
2011-08-08 19:34:16,732 DEBUG nova.utils [-] Result was 2 from
(pid=10073) execute /usr/lib/pymodules/python2.7/nova/utils.py:166
2011-08-08 19:34:16,732 CRITICAL nova [-] Unexpected error while
running command.
Command: sudo iptables-restore
Exit code: 2
Stdout: ''
Stderr: 'iptables-restore v1.4.10: Bad IP address "nova"\n\nError
occurred at line: 18\nTry `iptables-restore -h\' or \'iptables-restore
--help\' for more information.\n'
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE:   File "/usr/bin/nova-network", line 48, in 
(nova): TRACE:     service.serve()
(nova): TRACE:   File "/usr/lib/pymodules/python2.7/nova/service.py",
line 290, in serve
(nova): TRACE:     x.start()
(nova): TRACE:   File "/usr/lib/pymodules/python2.7/nova/service.py",
line 84, in start
(nova): TRACE:     self.manager.init_host()
(nova): TRACE:   File
"/usr/lib/pymodules/python2.7/nova/network/manager.py", line 499, in
init_host
(nova): TRACE:     super(VlanManager, self).init_host()
(nova): TRACE:   File
"/usr/lib/pymodules/python2.7/nova/network/manager.py", line 128, in
init_host
(nova): TRACE:     self.driver.init_host()
(nova): TRACE:   File
"/usr/lib/pymodules/python2.7/nova/network/linux_net.py", line 378, in
init_host
(nova): TRACE:     iptables_manager.apply()
(nova): TRACE:   File "/usr/lib/pymodules/python2.7/nova/utils.py",
line 607, in inner
(nova): TRACE:     retval = f(*args, **kwargs)
(nova): TRACE:   File
"/usr/lib/pymodules/python2.7/nova/network/linux_net.py", line 292, in
apply
(nova): TRACE:     attempts=5)
(nova): TRACE:   File
"/usr/lib/pymodules/python2.7/nova/network/linux_net.py", line 666, in
_execute
(nova): TRACE:     return utils.execute(*cmd, **kwargs)
(nova): TRACE:   File "/usr/lib/pymodules/python2.7/nova/utils.py",
line 173, in execute
(nova): TRACE:     cmd=' '.join(cmd))
(nova): TRACE: ProcessExecutionError: Unexpected error while running
command.
(nova): TRACE: Command: sudo iptables-restore
(nova): TRACE: Exit code: 2
(nova): TRACE: Stdout: ''
(nova): TRACE: Stderr: 'iptables-restore v1.4.10: Bad IP address
"nova"\n\nError occurred at line: 18\nTry `iptables-restore -h\' or
\'iptables-restore --help\' for more information.\n'
(nova): TRACE: 

A PROBLEM OF DB CONNECTION IN NOVA COMPUTE LOG FROM THE CONTROLLER:
(nova): TRACE: ProgrammingError: (ProgrammingError) (1146, "Table
'nova.instances' doesn't exist") 'SELECT instances.created_at AS
instances_created_at, 

[......]

ting_ips_1.deleted = %s \nWHERE instances.host = %s AND
instances.deleted = %s' (False, False, False, False, False,
'nova-controller', False)
(nova): TRACE: 

IFCONFIG AND IPTABLES OF THE CONTROLLER NODE:
root@nova-controller:~# ifconfig
br100     Link encap:Ethernet  HWaddr 08:00:27:3d:e9:7f  
          inet addr:192.168.1.4  Bcast:192.168.1.255
 Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe3d:e97f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:352069 errors:0 dropped:0 overruns:0 frame:0
          TX packets:178694 errors:0 dropped:0 overruns:0
carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:28108787 (28.1 MB)  TX bytes:18241764 (18.2
MB)

eth0      Link encap:Ethernet  HWaddr 08:00:27:3d:e9:7f  
          inet6 addr: fe80::a00:27ff:fe3d:e97f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:352848 errors:0 dropped:0 overruns:0 frame:0
          TX packets:179157 errors:0 dropped:0 overruns:0
carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:33117855 (33.1 MB)  TX bytes:18271426 (18.2
MB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:498 errors:0 dropped:0 overruns:0 frame:0
          TX packets:498 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:108091 (108.0 KB)  TX bytes:108091 (108.0 KB)

virbr0    Link encap:Ethernet  HWaddr 22:0f:89:53:eb:82  
          inet addr:192.168.122.1  Bcast:192.168.122.255
 Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

root@nova-controller:~# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination      
  
ACCEPT     udp  --  anywhere             anywhere      
     udp dpt:domain 
ACCEPT     tcp  --  anywhere             anywhere      
     tcp dpt:domain 
ACCEPT     udp  --  anywhere             anywhere      
     udp dpt:bootps 
ACCEPT     tcp  --  anywhere             anywhere      
     tcp dpt:bootps 

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination      
  
ACCEPT     all  --  anywhere             192.168.122.0/24  
 state RELATED,ESTABLISHED 
ACCEPT     all  --  192.168.122.0/24     anywhere          
 
ACCEPT     all  --  anywhere             anywhere      
     
REJECT     all  --  anywhere             anywhere      
     reject-with icmp-port-unreachable 
REJECT     all  --  anywhere             anywhere      
     reject-with icmp-port-unreachable 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination