openstack team mailing list archive
-
openstack team
-
Mailing list archive
-
Message #03458
bad nova-manage service list
Hi guys,
I am trying to set up an openstack small cluster and I am facing a
problem.
I have two nodes, a controller node on which I have installed all the
nova-* services and a compute node on which I have installed
nova-compute and nova-network.
When I run 'nova-manage service list', I only see the nova-compute
service from the compute-node, not the other services from the
controller node. However, if I run 'ps aux | grep nova-' on the
controller node, all the nova-* are up and running. It seems that my
controller's nova-* services do not want to register to the mysql DB.
My guess is that services are not able to connect to the local mysql
db on the controller node may be due to iptables bad settings but I am
not able to figure it out.
Worth to mention it, my nova-controller node is running in a virtual
machine on virtual box, both nodes are ubuntu natty server amd64.
If someone have any idea on that problem, please let me know, I added
some more information about my configuration in attachment below.
Than you in advance.
Best Regards,
Ivan Frain
HERE IS MY NOVA.CONF FILE: IT IS THE SAME ON BOTH MACHINES:
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--verbose
--sql_connection=mysql://nova:notnova@nova-controller/nova
--s3_host=nova-controller
--rabbit_host=nova-controller
--ec2_api=nova-controller
--ec2_url=http://nova-controller:8773/services/Cloud
--fixed_range=192.168.0.0/16
--network_size=8
--routing_source_ip=nova-controller
THE OUTPUT OF PS AUX | GREP NOVA COMMAND:
root@nova-controller:~# ps aux | grep nova
rabbitmq 1118 0.9 5.3 53644 27224 ? Ssl 21:06
0:47 /usr/lib/erlang/erts-5.7.4/bin/beam -W w -K true -A30 -P 1048576
-- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq --
-noshell -noinput -sname rabbit@nova-controller -boot
/var/lib/rabbitmq/mnesia/rabbit@nova-controller-plugins-expand/rabbit
-kernel inet_default_listen_options [{nodelay,true}] -kernel
inet_default_connect_options [{nodelay,true}] -sasl errlog_type error
-kernel error_logger
{file,"/var/log/rabbitmq/rabbit@xxxxxxxxxxxxxxxxxxx"} -sasl
sasl_error_logger
{file,"/var/log/rabbitmq/rabbit@xxxxxxxxxxxxxxxxxxxxxxxx"} -os_mon
start_cpu_sup true -os_mon start_disksup false -os_mon start_memsup
false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@nova-controller"
-noshell -noinput
root 19856 0.1 3.2 61228 16232 ? S 22:24
0:00 /usr/bin/python /usr/bin/nova-objectstore --uid 108 --gid
65534 --pidfile /var/run/nova/nova-objectstore.pid
--flagfile=/etc/nova/nova.conf --nodaemon
--logfile=/var/log/nova/nova-objectstore.log
nova 23074 0.0 0.2 35708 1280 ? Ss 22:26
0:00 su -c nova-compute --flagfile=/etc/nova/nova.conf nova
nova 23075 30.3 8.2 145228 41740 ? R 22:26
0:00 /usr/bin/python /usr/bin/nova-compute
--flagfile=/etc/nova/nova.conf
nova 23132 0.0 0.2 35708 1276 ? Ss 22:26
0:00 su -c nova-api --flagfile=/etc/nova/nova.conf nova
nova 23133 35.0 5.3 89264 26884 ? R 22:26
0:00 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf
nova 23145 0.0 0.2 35708 1280 ? Ss 22:26
0:00 su -c nova-network --flagfile=/etc/nova/nova.conf nova
nova 23146 23.0 4.2 75476 21596 ? R 22:26
0:00 /usr/bin/python /usr/bin/nova-network
--flagfile=/etc/nova/nova.conf
nova 23152 0.0 0.2 35708 1280 ? Ss 22:26
0:00 su -c nova-scheduler --flagfile=/etc/nova/nova.conf nova
nova 23153 0.0 3.7 72924 19112 ? R 22:26
0:00 /usr/bin/python /usr/bin/nova-scheduler
--flagfile=/etc/nova/nova.conf
root 23167 0.0 0.2 9136 1060 pts/1 S+ 22:26
0:00 grep --color=auto nova
THE OUTPUT OF SERVICE-MANAGE COMMAND:
root@nova-controller:~# nova-manage service list
Obelix nova-compute enabled :-) 2011-08-08 20:27:26
NB: Obelix is the name of my compute node.
MYSQL> SELECT HOST,USER,PASSWORD FROM USER;
+-----------------+------------------+-------------------------------------------+
| Host | User | Password
|
+-----------------+------------------+-------------------------------------------+
| localhost | root |
*0BE3B501084D35F4C66DD3AC4569EAE5EA738212 |
| nova-controller | root |
*0BE3B501084D35F4C66DD3AC4569EAE5EA738212 |
| 127.0.0.1 | root |
*0BE3B501084D35F4C66DD3AC4569EAE5EA738212 |
| localhost | debian-sys-maint |
*B23864E8C293F7A6E977A8CF31AC4ED99570A161 |
| % | nova |
*8134721C1A2B651300B26AC135712CD373E4D471 |
+-----------------+------------------+-------------------------------------------+
A PROBLEM IN THE OUTPUT FROM NOVA-NETWORK LOG (IPTABLES PROBLEM):
root@nova-controller:~# tail -n 50 /var/log/nova/nova-network.log
2011-08-08 19:34:11,852 DEBUG nova.utils [-] Attempting to grab file
lock "iptables" for method "apply"... from (pid=10073) inner
/usr/lib/pymodules/python2.7/nova/utils.py:599
2011-08-08 19:34:11,854 DEBUG nova.utils [-] Running cmd (subprocess):
sudo iptables-save -t filter from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:150
2011-08-08 19:34:11,870 DEBUG nova.utils [-] Running cmd (subprocess):
sudo iptables-restore from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:150
2011-08-08 19:34:11,911 DEBUG nova.utils [-] Running cmd (subprocess):
sudo iptables-save -t nat from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:150
2011-08-08 19:34:11,929 DEBUG nova.utils [-] Running cmd (subprocess):
sudo iptables-restore from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:150
2011-08-08 19:34:11,963 DEBUG nova.utils [-] Result was 2 from
(pid=10073) execute /usr/lib/pymodules/python2.7/nova/utils.py:166
2011-08-08 19:34:11,963 DEBUG nova.utils [-] ['sudo',
'iptables-restore'] failed. Retrying. from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:179
2011-08-08 19:34:12,776 DEBUG nova.utils [-] Running cmd (subprocess):
sudo iptables-restore from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:150
2011-08-08 19:34:12,802 DEBUG nova.utils [-] Result was 2 from
(pid=10073) execute /usr/lib/pymodules/python2.7/nova/utils.py:166
2011-08-08 19:34:12,803 DEBUG nova.utils [-] ['sudo',
'iptables-restore'] failed. Retrying. from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:179
2011-08-08 19:34:14,616 DEBUG nova.utils [-] Running cmd (subprocess):
sudo iptables-restore from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:150
2011-08-08 19:34:14,641 DEBUG nova.utils [-] Result was 2 from
(pid=10073) execute /usr/lib/pymodules/python2.7/nova/utils.py:166
2011-08-08 19:34:14,642 DEBUG nova.utils [-] ['sudo',
'iptables-restore'] failed. Retrying. from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:179
2011-08-08 19:34:15,723 DEBUG nova.utils [-] Running cmd (subprocess):
sudo iptables-restore from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:150
2011-08-08 19:34:15,742 DEBUG nova.utils [-] Result was 2 from
(pid=10073) execute /usr/lib/pymodules/python2.7/nova/utils.py:166
2011-08-08 19:34:15,742 DEBUG nova.utils [-] ['sudo',
'iptables-restore'] failed. Retrying. from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:179
2011-08-08 19:34:16,703 DEBUG nova.utils [-] Running cmd (subprocess):
sudo iptables-restore from (pid=10073) execute
/usr/lib/pymodules/python2.7/nova/utils.py:150
2011-08-08 19:34:16,732 DEBUG nova.utils [-] Result was 2 from
(pid=10073) execute /usr/lib/pymodules/python2.7/nova/utils.py:166
2011-08-08 19:34:16,732 CRITICAL nova [-] Unexpected error while
running command.
Command: sudo iptables-restore
Exit code: 2
Stdout: ''
Stderr: 'iptables-restore v1.4.10: Bad IP address "nova"\n\nError
occurred at line: 18\nTry `iptables-restore -h\' or \'iptables-restore
--help\' for more information.\n'
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/usr/bin/nova-network", line 48, in
(nova): TRACE: service.serve()
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/service.py",
line 290, in serve
(nova): TRACE: x.start()
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/service.py",
line 84, in start
(nova): TRACE: self.manager.init_host()
(nova): TRACE: File
"/usr/lib/pymodules/python2.7/nova/network/manager.py", line 499, in
init_host
(nova): TRACE: super(VlanManager, self).init_host()
(nova): TRACE: File
"/usr/lib/pymodules/python2.7/nova/network/manager.py", line 128, in
init_host
(nova): TRACE: self.driver.init_host()
(nova): TRACE: File
"/usr/lib/pymodules/python2.7/nova/network/linux_net.py", line 378, in
init_host
(nova): TRACE: iptables_manager.apply()
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/utils.py",
line 607, in inner
(nova): TRACE: retval = f(*args, **kwargs)
(nova): TRACE: File
"/usr/lib/pymodules/python2.7/nova/network/linux_net.py", line 292, in
apply
(nova): TRACE: attempts=5)
(nova): TRACE: File
"/usr/lib/pymodules/python2.7/nova/network/linux_net.py", line 666, in
_execute
(nova): TRACE: return utils.execute(*cmd, **kwargs)
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/utils.py",
line 173, in execute
(nova): TRACE: cmd=' '.join(cmd))
(nova): TRACE: ProcessExecutionError: Unexpected error while running
command.
(nova): TRACE: Command: sudo iptables-restore
(nova): TRACE: Exit code: 2
(nova): TRACE: Stdout: ''
(nova): TRACE: Stderr: 'iptables-restore v1.4.10: Bad IP address
"nova"\n\nError occurred at line: 18\nTry `iptables-restore -h\' or
\'iptables-restore --help\' for more information.\n'
(nova): TRACE:
A PROBLEM OF DB CONNECTION IN NOVA COMPUTE LOG FROM THE CONTROLLER:
(nova): TRACE: ProgrammingError: (ProgrammingError) (1146, "Table
'nova.instances' doesn't exist") 'SELECT instances.created_at AS
instances_created_at,
[......]
ting_ips_1.deleted = %s \nWHERE instances.host = %s AND
instances.deleted = %s' (False, False, False, False, False,
'nova-controller', False)
(nova): TRACE:
IFCONFIG AND IPTABLES OF THE CONTROLLER NODE:
root@nova-controller:~# ifconfig
br100 Link encap:Ethernet HWaddr 08:00:27:3d:e9:7f
inet addr:192.168.1.4 Bcast:192.168.1.255
Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe3d:e97f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:352069 errors:0 dropped:0 overruns:0 frame:0
TX packets:178694 errors:0 dropped:0 overruns:0
carrier:0
collisions:0 txqueuelen:0
RX bytes:28108787 (28.1 MB) TX bytes:18241764 (18.2
MB)
eth0 Link encap:Ethernet HWaddr 08:00:27:3d:e9:7f
inet6 addr: fe80::a00:27ff:fe3d:e97f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:352848 errors:0 dropped:0 overruns:0 frame:0
TX packets:179157 errors:0 dropped:0 overruns:0
carrier:0
collisions:0 txqueuelen:1000
RX bytes:33117855 (33.1 MB) TX bytes:18271426 (18.2
MB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:498 errors:0 dropped:0 overruns:0 frame:0
TX packets:498 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:108091 (108.0 KB) TX bytes:108091 (108.0 KB)
virbr0 Link encap:Ethernet HWaddr 22:0f:89:53:eb:82
inet addr:192.168.122.1 Bcast:192.168.122.255
Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
root@nova-controller:~# iptables --list
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere
udp dpt:domain
ACCEPT tcp -- anywhere anywhere
tcp dpt:domain
ACCEPT udp -- anywhere anywhere
udp dpt:bootps
ACCEPT tcp -- anywhere anywhere
tcp dpt:bootps
Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere 192.168.122.0/24
state RELATED,ESTABLISHED
ACCEPT all -- 192.168.122.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere
reject-with icmp-port-unreachable
REJECT all -- anywhere anywhere
reject-with icmp-port-unreachable
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
