yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #16395
[Bug 1332036] [NEW] Instance status would never changed from BUILD after rabbitmq restarted
Public bug reported:
[Issue]
I'm checking the behavior when system failure cases.
Instance status never changed from BUILD after rabbitmq restarted.
[How to reproduce]
1) boot services that contained nova and rabbit.
2) kill rabbitmq service for simulating system failure.
$ ps aux|grep beam|grep -v grep
rabbitmq 15816 2.4 1.3 146496 27048 ? Sl 20:05 0:01 /usr/lib/erlang/erts-5.10.4/bin/beam -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@interdev -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@xxxxxxxxxxxx"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@xxxxxxxxxxxxxxxxx"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@interdev-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@interdev"
$ sudo kill -SIGKILL 15816
3) restart rabbitmq service.
$ sudo /etc/init.d/rabbitmq-server start
* Starting message broker rabbitmq-server [ OK ]
4) request to boot instance
$ nova boot --flavor 1 --image 41793321-51b1-413a-a69f-0c25d7a63b22 aaa
+--------------------------------------+-------------------------------------------------+
| Property | Value |
+--------------------------------------+-------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-00000005 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | 2gV49MZNHLQm |
| config_drive | |
| created | 2014-06-19T11:06:51Z |
| flavor | m1.tiny (1) |
| hostId | |
| id | 157f3d5d-5d26-4a55-9d7c-ab0dc3d9fdc9 |
| image | CorePlus (41793321-51b1-413a-a69f-0c25d7a63b22) |
| key_name | - |
| metadata | {} |
| name | aaa |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | 5924ee51bb6945f39dc7010687ef43af |
| updated | 2014-06-19T11:06:54Z |
| user_id | 35b1c3ef8b9d442cad696d049f7b2041 |
+--------------------------------------+-------------------------------------------------+
5) however, instance status never changed from BUILD/NOSTATE
$ nova list
+--------------------------------------+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+----------+
| 157f3d5d-5d26-4a55-9d7c-ab0dc3d9fdc9 | aaa | BUILD | - | NOSTATE | |
+--------------------------------------+------+--------+------------+-------------+----------+
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1332036
Title:
Instance status would never changed from BUILD after rabbitmq
restarted
Status in OpenStack Compute (Nova):
New
Bug description:
[Issue]
I'm checking the behavior when system failure cases.
Instance status never changed from BUILD after rabbitmq restarted.
[How to reproduce]
1) boot services that contained nova and rabbit.
2) kill rabbitmq service for simulating system failure.
$ ps aux|grep beam|grep -v grep
rabbitmq 15816 2.4 1.3 146496 27048 ? Sl 20:05 0:01 /usr/lib/erlang/erts-5.10.4/bin/beam -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@interdev -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@xxxxxxxxxxxx"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@xxxxxxxxxxxxxxxxx"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@interdev-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@interdev"
$ sudo kill -SIGKILL 15816
3) restart rabbitmq service.
$ sudo /etc/init.d/rabbitmq-server start
* Starting message broker rabbitmq-server [ OK ]
4) request to boot instance
$ nova boot --flavor 1 --image 41793321-51b1-413a-a69f-0c25d7a63b22 aaa
+--------------------------------------+-------------------------------------------------+
| Property | Value |
+--------------------------------------+-------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-00000005 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | 2gV49MZNHLQm |
| config_drive | |
| created | 2014-06-19T11:06:51Z |
| flavor | m1.tiny (1) |
| hostId | |
| id | 157f3d5d-5d26-4a55-9d7c-ab0dc3d9fdc9 |
| image | CorePlus (41793321-51b1-413a-a69f-0c25d7a63b22) |
| key_name | - |
| metadata | {} |
| name | aaa |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | 5924ee51bb6945f39dc7010687ef43af |
| updated | 2014-06-19T11:06:54Z |
| user_id | 35b1c3ef8b9d442cad696d049f7b2041 |
+--------------------------------------+-------------------------------------------------+
5) however, instance status never changed from BUILD/NOSTATE
$ nova list
+--------------------------------------+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+----------+
| 157f3d5d-5d26-4a55-9d7c-ab0dc3d9fdc9 | aaa | BUILD | - | NOSTATE | |
+--------------------------------------+------+--------+------------+-------------+----------+
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1332036/+subscriptions
Follow ups
References
