yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #30791
[Bug 1254677] Re: Stopping a Nova instance in ERROR state results in -- "Conflict": Instance is not ready
very old bug that reporter says we are free to close
** Changed in: nova
Status: Incomplete => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1254677
Title:
Stopping a Nova instance in ERROR state results in -- "Conflict":
Instance is not ready
Status in OpenStack Compute (Nova):
Invalid
Bug description:
Description
--------------
Attempt to stop a Nova instance (Cirros guest) in an ERROR state results in a stack trace:
The instance ended up in ERROR state as Compute node cannot connect to the AMQP server ( running on Controller node).
===
Conflict: Instance 19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246 is not ready (HTTP 409) (Request-ID: req-ecf547ce-cdd8-458b-bad8-32907527a506)
ERROR: Instance 19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246 is not ready (HTTP 409) (Request-ID: req-ecf547ce-cdd8-458b-bad8-32907527a506)
===
[I briefly discussed this with Nikola Dipanov, he mentioned it's
worth reporting. Also, Attitla Fazekas had this say -- if an instance
reaches ERROR state before a 'stop' action, 409 is expected.]
Setup/Version info
----------------------
It's a two-node Havana setup (hand-configured).
Havana:
$ rpm -q openstack-nova python-novaclient python-nova
python-nova-2013.2-2.fc20.noarch
openstack-nova-2013.2-2.fc20.noarch
python-novaclient-2.15.0-1.fc20.noarch
Test
-----
List the instance (in ERROR state).
$ nova list
+--------------------------------------+-------------+--------+------------+-------------+--------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------+--------+------------+-------------+--------------+
| 7a3eb749-62fb-481d-9f28-bb46494431f8 | cirr-guest5 | ERROR | None | NOSTATE | |
+--------------------------------------+-------------+--------+------------+-------------+--------------+
Stop the instance:
$ nova stop cirr-guest5
ERROR: Instance 7a3eb749-62fb-481d-9f28-bb46494431f8 is not ready (HTTP 409) (Request-ID: req-64e5d573-7a0a-48fb-bd98-bd783331b5b5)
Stop the instance (with --debug):
========================
$ nova --debug stop cirr-guest4
REQ: curl -i http://192.169.142.118:35357/v2.0/tokens -X POST -H
"Content-Type: application/json" -H "Accept: application/json" -H
"User-Agent: python-novaclient" -d '{"auth": {"tenantName":
"ostenant", "passwordCredentials": {"username": "kashyap", "password":
"fedora"}}}'
INFO (connectionpool:236) Starting new HTTP connection (1): 192.169.142.118
DEBUG (connectionpool:330) "POST /v2.0/tokens HTTP/1.1" 200 5475
RESP: [200] CaseInsensitiveDict({'date': 'Mon, 25 Nov 2013 10:01:10 GMT', 'vary': 'X-Auth-Token', 'content-length': '5475', 'content-type': 'application/json'})
RESP BODY: {"access": {"token": {"issued_at": "2013-11-25T10:01:10.091651", "expires": "2013-11-26T10:01:10Z", "id": "MIIJ6wYJKoZIhvcNAQcCoIIJ3DCCCdgCAQExCTAHBgUrDgMCGjCCCEEGCSqGSIb3DQEHAaCCCDIEgggueyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0xMS0yNVQxMDowMToxMC4wOTE2NTEiLCAiZXhwaXJlcyI6ICIyMDEzLTExLTI2VDEwOjAxOjEwWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNjE5ZGI3OWRkNGE5NDcwNDg5MmM4NTJiMDRiYzU1MzEiLCAibmFtZSI6ICJvc3RlbmFudCJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo4Nzc2L3YxLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo4Nzc2L3YxLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgImlkIjogIjAyNTBlOTgzMmU3ZTRjMDdhYmEyZWQzZDM2YjNiMWMxIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzYvdjEvNjE5ZGI3OWRkNGE5NDcwNDg5MmM4NTJiMDRiYzU1MzEifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAidm9sdW1lIiwgIm5hbWUiOiAiY2luZGVyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5MjkyIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5MjkyIiwgImlkIjogIjJhYTM4OWVlYjA2NTRiZjE5NzFmZDg5M2ZjNzU2ZWQ5IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4OjkyOTIifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiaW1hZ2UiLCAibmFtZSI6ICJnbGFuY2UifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzQvdjEuMS82MTlkYjc5ZGQ0YTk0NzA0ODkyYzg1MmIwNGJjNTUzMSIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6ODc3NC92MS4xLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgImlkIjogIjJhNGMzZDRiMGU4NjQ5YWNhNmJlOTA5YWEwMmMzYWY4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzQvdjEuMS82MTlkYjc5ZGQ0YTk0NzA0ODkyYzg1MmIwNGJjNTUzMSJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJjb21wdXRlIiwgIm5hbWUiOiAibm92YSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6OTY5NiIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6OTY5NiIsICJpZCI6ICJiNzkzMWU1N2ExNWI0MGRjODg5N2Q0MDJmOGNkYzJlYSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5Njk2In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm5ldHdvcmsiLCAibmFtZSI6ICJuZXV0cm9uIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo1MDAwL3YyLjAiLCAiaWQiOiAiM2I4OGY5MDc2NDEzNDkyMmI3MjFjMzhmZGFmZGEyNzQiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJrYXNoeWFwIiwgInJvbGVzX2xpbmtzIjogW10sICJpZCI6ICJkNGI0Zjg4MGY3M2M0NGEyOTBlMmU5Y2FkMTAwMjU0MCIsICJyb2xlcyI6IFt7Im5hbWUiOiAidXNlciJ9XSwgIm5hbWUiOiAia2FzaHlhcCJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgInJvbGVzIjogWyIxN2E3OTNjMmM3ZmQ0MGJhYTBlOGE0ZDYzNTU5OTJhMSJdfX19MYIBgTCCAX0CAQEwXDBXMQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVW5zZXQxDjAMBgNVBAcMBVVuc2V0MQ4wDAYDVQQKDAVVbnNldDEYMBYGA1UEAwwPd3d3LmV4YW1wbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqGSIb3DQEBAQUABIIBAHEVndw9iBsw9Mn6b9W2NvXDX0LxyG31PFeEB8hrXVoaWp5dhd7lSt9G8WyAzFIeIqdWmudiZuloDZSPlxLB9vLE-PKvJpYNd2-Lw53sv4R5XCcVBP779tyrjTaBpiYMk9pBiupSzyxhwAJozTSZKsow5HwrtSZmlEdW8ngkybqz9b2p2M11h7xIA6hRXlFBUlUmM9k+NhV-MZjSt356cIAIzRiMQya4w6rcmf+otV4yrDEVhjWP9pD7cjGSVeqxKVCshZOgRsFBmlqaG6z01RHk55eypzYknVxPK646+y3S48elB39YdZ+tBUGTucavpWlXedzZAiR2TIr6tmysMJY=", "tenant": {"description": null, "enabled": true, "id": "619db79dd4a94704892c852b04bc5531", "name": "ostenant"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://192.169.142.118:8776/v1/619db79dd4a94704892c852b04bc5531", "region": "regionOne", "internalURL": "http://192.169.142.118:8776/v1/619db79dd4a94704892c852b04bc5531", "id": "0250e9832e7e4c07aba2ed3d36b3b1c1", "publicURL": "http://192.169.142.118:8776/v1/619db79dd4a94704892c852b04bc5531"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://192.169.142.118:9292", "region": "regionOne", "internalURL": "http://192.169.142.118:9292", "id": "2aa389eeb0654bf1971fd893fc756ed9", "publicURL": "http://192.169.142.118:9292"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531", "region": "regionOne", "internalURL": "http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531", "id": "2a4c3d4b0e8649aca6be909aa02c3af8", "publicURL": "http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://192.169.142.118:9696", "region": "regionOne", "internalURL": "http://192.169.142.118:9696", "id": "b7931e57a15b40dc8897d402f8cdc2ea", "publicURL": "http://192.169.142.118:9696"}], "endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints": [{"adminURL": "http://192.169.142.118:35357/v2.0", "region": "regionOne", "internalURL": "http://192.169.142.118:5000/v2.0", "id": "3b88f90764134922b721c38fdafda274", "publicURL": "http://192.169.142.118:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "kashyap", "roles_links": [], "id": "d4b4f880f73c44a290e2e9cad1002540", "roles": [{"name": "user"}], "name": "kashyap"}, "metadata": {"is_admin": 0, "roles": ["17a793c2c7fd40baa0e8a4d6355992a1"]}}}
REQ: curl -i
http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531/servers
-X GET -H "X-Auth-Project-Id: ostenant" -H "User-Agent: python-
novaclient" -H "Accept: application/json" -H "X-Auth-Token:
MIIJ6wYJKoZIhvcNAQcCoIIJ3DCCCdgCAQExCTAHBgUrDgMCGjCCCEEGCSqGSIb3DQEHAaCCCDIEgggueyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0xMS0yNVQxMDowMToxMC4wOTE2NTEiLCAiZXhwaXJlcyI6ICIyMDEzLTExLTI2VDEwOjAxOjEwWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNjE5ZGI3OWRkNGE5NDcwNDg5MmM4NTJiMDRiYzU1MzEiLCAibmFtZSI6ICJvc3RlbmFudCJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo4Nzc2L3YxLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo4Nzc2L3YxLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgImlkIjogIjAyNTBlOTgzMmU3ZTRjMDdhYmEyZWQzZDM2YjNiMWMxIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzYvdjEvNjE5ZGI3OWRkNGE5NDcwNDg5MmM4NTJiMDRiYzU1MzEifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAidm9sdW1lIiwgIm5hbWUiOiAiY2luZGVyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5MjkyIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5MjkyIiwgImlkIjogIjJhYTM4OWVlYjA2NTRiZjE5NzFmZDg5M2ZjNzU2ZWQ5IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4OjkyOTIifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiaW1hZ2UiLCAibmFtZSI6ICJnbGFuY2UifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzQvdjEuMS82MTlkYjc5ZGQ0YTk0NzA0ODkyYzg1MmIwNGJjNTUzMSIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6ODc3NC92MS4xLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgImlkIjogIjJhNGMzZDRiMGU4NjQ5YWNhNmJlOTA5YWEwMmMzYWY4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzQvdjEuMS82MTlkYjc5ZGQ0YTk0NzA0ODkyYzg1MmIwNGJjNTUzMSJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJjb21wdXRlIiwgIm5hbWUiOiAibm92YSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6OTY5NiIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6OTY5NiIsICJpZCI6ICJiNzkzMWU1N2ExNWI0MGRjODg5N2Q0MDJmOGNkYzJlYSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5Njk2In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm5ldHdvcmsiLCAibmFtZSI6ICJuZXV0cm9uIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo1MDAwL3YyLjAiLCAiaWQiOiAiM2I4OGY5MDc2NDEzNDkyMmI3MjFjMzhmZGFmZGEyNzQiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJrYXNoeWFwIiwgInJvbGVzX2xpbmtzIjogW10sICJpZCI6ICJkNGI0Zjg4MGY3M2M0NGEyOTBlMmU5Y2FkMTAwMjU0MCIsICJyb2xlcyI6IFt7Im5hbWUiOiAidXNlciJ9XSwgIm5hbWUiOiAia2FzaHlhcCJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgInJvbGVzIjogWyIxN2E3OTNjMmM3ZmQ0MGJhYTBlOGE0ZDYzNTU5OTJhMSJdfX19MYIBgTCCAX0CAQEwXDBXMQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVW5zZXQxDjAMBgNVBAcMBVVuc2V0MQ4wDAYDVQQKDAVVbnNldDEYMBYGA1UEAwwPd3d3LmV4YW1wbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqGSIb3DQEBAQUABIIBAHEVndw9iBsw9Mn6b9W2NvXDX0LxyG31PFeEB8hrXVoaWp5dhd7lSt9G8WyAzFIeIqdWmudiZuloDZSPlxLB9vLE-
PKvJpYNd2-Lw53sv4R5XCcVBP779tyrjTaBpiYMk9pBiupSzyxhwAJozTSZKsow5HwrtSZmlEdW8ngkybqz9b2p2M11h7xIA6hRXlFBUlUmM9k
+NhV-
MZjSt356cIAIzRiMQya4w6rcmf+otV4yrDEVhjWP9pD7cjGSVeqxKVCshZOgRsFBmlqaG6z01RHk55eypzYknVxPK646+y3S48elB39YdZ+tBUGTucavpWlXedzZAiR2TIr6tmysMJY="
INFO (connectionpool:236) Starting new HTTP connection (1): 192.169.142.118
DEBUG (connectionpool:330) "GET /v1.1/619db79dd4a94704892c852b04bc5531/servers HTTP/1.1" 200 731
RESP: [200] CaseInsensitiveDict({'date': 'Mon, 25 Nov 2013 10:01:10 GMT', 'content-length': '731', 'content-type': 'application/json', 'x-compute-request-id': 'req-f3469b88-bd24-44d9-a3a5-67441b39a292'})
RESP BODY: {"servers": [{"id": "7a3eb749-62fb-481d-9f28-bb46494431f8", "links": [{"href": "http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531/servers/7a3eb749-62fb-481d-9f28-bb46494431f8", "rel": "self"}, {"href": "http://192.169.142.118:8774/619db79dd4a94704892c852b04bc5531/servers/7a3eb749-62fb-481d-9f28-bb46494431f8", "rel": "bookmark"}], "name": "cirr-guest5"}, {"id": "19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246", "links": [{"href": "http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531/servers/19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246", "rel": "self"}, {"href": "http://192.169.142.118:8774/619db79dd4a94704892c852b04bc5531/servers/19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246", "rel": "bookmark"}], "name": "cirr-guest4"}]}
REQ: curl -i
http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531/servers
/19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246 -X GET -H "X-Auth-Project-Id:
ostenant" -H "User-Agent: python-novaclient" -H "Accept:
application/json" -H "X-Auth-Token:
MIIJ6wYJKoZIhvcNAQcCoIIJ3DCCCdgCAQExCTAHBgUrDgMCGjCCCEEGCSqGSIb3DQEHAaCCCDIEgggueyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0xMS0yNVQxMDowMToxMC4wOTE2NTEiLCAiZXhwaXJlcyI6ICIyMDEzLTExLTI2VDEwOjAxOjEwWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNjE5ZGI3OWRkNGE5NDcwNDg5MmM4NTJiMDRiYzU1MzEiLCAibmFtZSI6ICJvc3RlbmFudCJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo4Nzc2L3YxLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo4Nzc2L3YxLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgImlkIjogIjAyNTBlOTgzMmU3ZTRjMDdhYmEyZWQzZDM2YjNiMWMxIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzYvdjEvNjE5ZGI3OWRkNGE5NDcwNDg5MmM4NTJiMDRiYzU1MzEifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAidm9sdW1lIiwgIm5hbWUiOiAiY2luZGVyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5MjkyIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5MjkyIiwgImlkIjogIjJhYTM4OWVlYjA2NTRiZjE5NzFmZDg5M2ZjNzU2ZWQ5IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4OjkyOTIifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiaW1hZ2UiLCAibmFtZSI6ICJnbGFuY2UifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzQvdjEuMS82MTlkYjc5ZGQ0YTk0NzA0ODkyYzg1MmIwNGJjNTUzMSIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6ODc3NC92MS4xLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgImlkIjogIjJhNGMzZDRiMGU4NjQ5YWNhNmJlOTA5YWEwMmMzYWY4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzQvdjEuMS82MTlkYjc5ZGQ0YTk0NzA0ODkyYzg1MmIwNGJjNTUzMSJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJjb21wdXRlIiwgIm5hbWUiOiAibm92YSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6OTY5NiIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6OTY5NiIsICJpZCI6ICJiNzkzMWU1N2ExNWI0MGRjODg5N2Q0MDJmOGNkYzJlYSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5Njk2In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm5ldHdvcmsiLCAibmFtZSI6ICJuZXV0cm9uIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo1MDAwL3YyLjAiLCAiaWQiOiAiM2I4OGY5MDc2NDEzNDkyMmI3MjFjMzhmZGFmZGEyNzQiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJrYXNoeWFwIiwgInJvbGVzX2xpbmtzIjogW10sICJpZCI6ICJkNGI0Zjg4MGY3M2M0NGEyOTBlMmU5Y2FkMTAwMjU0MCIsICJyb2xlcyI6IFt7Im5hbWUiOiAidXNlciJ9XSwgIm5hbWUiOiAia2FzaHlhcCJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgInJvbGVzIjogWyIxN2E3OTNjMmM3ZmQ0MGJhYTBlOGE0ZDYzNTU5OTJhMSJdfX19MYIBgTCCAX0CAQEwXDBXMQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVW5zZXQxDjAMBgNVBAcMBVVuc2V0MQ4wDAYDVQQKDAVVbnNldDEYMBYGA1UEAwwPd3d3LmV4YW1wbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqGSIb3DQEBAQUABIIBAHEVndw9iBsw9Mn6b9W2NvXDX0LxyG31PFeEB8hrXVoaWp5dhd7lSt9G8WyAzFIeIqdWmudiZuloDZSPlxLB9vLE-
PKvJpYNd2-Lw53sv4R5XCcVBP779tyrjTaBpiYMk9pBiupSzyxhwAJozTSZKsow5HwrtSZmlEdW8ngkybqz9b2p2M11h7xIA6hRXlFBUlUmM9k
+NhV-
MZjSt356cIAIzRiMQya4w6rcmf+otV4yrDEVhjWP9pD7cjGSVeqxKVCshZOgRsFBmlqaG6z01RHk55eypzYknVxPK646+y3S48elB39YdZ+tBUGTucavpWlXedzZAiR2TIr6tmysMJY="
DEBUG (connectionpool:330) "GET /v1.1/619db79dd4a94704892c852b04bc5531/servers/19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246 HTTP/1.1" 200 1392
RESP: [200] CaseInsensitiveDict({'date': 'Mon, 25 Nov 2013 10:01:10 GMT', 'content-length': '1392', 'content-type': 'application/json', 'x-compute-request-id': 'req-0732db3a-b866-4217-b341-44f35bed3b81'})
RESP BODY: {"server": {"status": "ERROR", "updated": "2013-11-25T09:33:00Z", "hostId": "", "addresses": {}, "links": [{"href": "http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531/servers/19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246", "rel": "self"}, {"href": "http://192.169.142.118:8774/619db79dd4a94704892c852b04bc5531/servers/19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246", "rel": "bookmark"}], "key_name": "oskey1", "image": {"id": "e1d7f58f-60cc-48f1-bb26-91f2f8b17bfa", "links": [{"href": "http://192.169.142.118:8774/619db79dd4a94704892c852b04bc5531/images/e1d7f58f-60cc-48f1-bb26-91f2f8b17bfa", "rel": "bookmark"}]}, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "error", "OS-SRV-USG:launched_at": null, "flavor": {"id": "2", "links": [{"href": "http://192.169.142.118:8774/619db79dd4a94704892c852b04bc5531/flavors/2", "rel": "bookmark"}]}, "id": "19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246", "OS-SRV-USG:terminated_at": null, "OS-EXT-AZ:availability_zone": "nova", "user_id": "d4b4f880f73c44a290e2e9cad1002540", "name": "cirr-guest4", "created": "2013-11-25T09:32:59Z", "tenant_id": "619db79dd4a94704892c852b04bc5531", "OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [], "accessIPv4": "", "accessIPv6": "", "fault": {"message": "No valid host was found. ", "code": 500, "created": "2013-11-25T09:33:00Z"}, "OS-EXT-STS:power_state": 0, "config_drive": "", "metadata": {}}}
REQ: curl -i
http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531/servers
/19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246/action -X POST -H "X-Auth-
Project-Id: ostenant" -H "User-Agent: python-novaclient" -H "Content-
Type: application/json" -H "Accept: application/json" -H "X-Auth-
Token:
MIIJ6wYJKoZIhvcNAQcCoIIJ3DCCCdgCAQExCTAHBgUrDgMCGjCCCEEGCSqGSIb3DQEHAaCCCDIEgggueyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0xMS0yNVQxMDowMToxMC4wOTE2NTEiLCAiZXhwaXJlcyI6ICIyMDEzLTExLTI2VDEwOjAxOjEwWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNjE5ZGI3OWRkNGE5NDcwNDg5MmM4NTJiMDRiYzU1MzEiLCAibmFtZSI6ICJvc3RlbmFudCJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo4Nzc2L3YxLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo4Nzc2L3YxLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgImlkIjogIjAyNTBlOTgzMmU3ZTRjMDdhYmEyZWQzZDM2YjNiMWMxIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzYvdjEvNjE5ZGI3OWRkNGE5NDcwNDg5MmM4NTJiMDRiYzU1MzEifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAidm9sdW1lIiwgIm5hbWUiOiAiY2luZGVyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5MjkyIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5MjkyIiwgImlkIjogIjJhYTM4OWVlYjA2NTRiZjE5NzFmZDg5M2ZjNzU2ZWQ5IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4OjkyOTIifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiaW1hZ2UiLCAibmFtZSI6ICJnbGFuY2UifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzQvdjEuMS82MTlkYjc5ZGQ0YTk0NzA0ODkyYzg1MmIwNGJjNTUzMSIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6ODc3NC92MS4xLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgImlkIjogIjJhNGMzZDRiMGU4NjQ5YWNhNmJlOTA5YWEwMmMzYWY4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzQvdjEuMS82MTlkYjc5ZGQ0YTk0NzA0ODkyYzg1MmIwNGJjNTUzMSJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJjb21wdXRlIiwgIm5hbWUiOiAibm92YSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6OTY5NiIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6OTY5NiIsICJpZCI6ICJiNzkzMWU1N2ExNWI0MGRjODg5N2Q0MDJmOGNkYzJlYSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5Njk2In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm5ldHdvcmsiLCAibmFtZSI6ICJuZXV0cm9uIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo1MDAwL3YyLjAiLCAiaWQiOiAiM2I4OGY5MDc2NDEzNDkyMmI3MjFjMzhmZGFmZGEyNzQiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJrYXNoeWFwIiwgInJvbGVzX2xpbmtzIjogW10sICJpZCI6ICJkNGI0Zjg4MGY3M2M0NGEyOTBlMmU5Y2FkMTAwMjU0MCIsICJyb2xlcyI6IFt7Im5hbWUiOiAidXNlciJ9XSwgIm5hbWUiOiAia2FzaHlhcCJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgInJvbGVzIjogWyIxN2E3OTNjMmM3ZmQ0MGJhYTBlOGE0ZDYzNTU5OTJhMSJdfX19MYIBgTCCAX0CAQEwXDBXMQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVW5zZXQxDjAMBgNVBAcMBVVuc2V0MQ4wDAYDVQQKDAVVbnNldDEYMBYGA1UEAwwPd3d3LmV4YW1wbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqGSIb3DQEBAQUABIIBAHEVndw9iBsw9Mn6b9W2NvXDX0LxyG31PFeEB8hrXVoaWp5dhd7lSt9G8WyAzFIeIqdWmudiZuloDZSPlxLB9vLE-
PKvJpYNd2-Lw53sv4R5XCcVBP779tyrjTaBpiYMk9pBiupSzyxhwAJozTSZKsow5HwrtSZmlEdW8ngkybqz9b2p2M11h7xIA6hRXlFBUlUmM9k
+NhV-
MZjSt356cIAIzRiMQya4w6rcmf+otV4yrDEVhjWP9pD7cjGSVeqxKVCshZOgRsFBmlqaG6z01RHk55eypzYknVxPK646+y3S48elB39YdZ+tBUGTucavpWlXedzZAiR2TIr6tmysMJY="
-d '{"os-stop": null}'
DEBUG (connectionpool:330) "POST /v1.1/619db79dd4a94704892c852b04bc5531/servers/19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246/action HTTP/1.1" 409 110
RESP: [409] CaseInsensitiveDict({'date': 'Mon, 25 Nov 2013 10:01:10 GMT', 'content-length': '110', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-ecf547ce-cdd8-458b-bad8-32907527a506'})
RESP BODY: {"conflictingRequest": {"message": "Instance 19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246 is not ready", "code": 409}}
DEBUG (shell:740) Instance 19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246 is not ready (HTTP 409) (Request-ID: req-ecf547ce-cdd8-458b-bad8-32907527a506)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 737, in main
OpenStackComputeShell().main(map(strutils.safe_decode, sys.argv[1:]))
File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 673, in main
args.func(self.cs, args)
File "/usr/lib/python2.7/site-packages/novaclient/v1_1/shell.py", line 1281, in do_stop
_find_server(cs, args.server).stop()
File "/usr/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 122, in stop
self.manager.stop(self)
File "/usr/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 509, in stop
return self._action('os-stop', server, None)
File "/usr/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 951, in _action
return self.api.client.post(url, body=body)
File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 232, in post
return self._cs_request(url, 'POST', **kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 213, in _cs_request
**kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 195, in _time_request
resp, body = self.request(url, method, **kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 189, in request
raise exceptions.from_response(resp, body, url, method)
Conflict: Instance 19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246 is not ready (HTTP 409) (Request-ID: req-ecf547ce-cdd8-458b-bad8-32907527a506)
ERROR: Instance 19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246 is not ready (HTTP 409) (Request-ID: req-ecf547ce-cdd8-458b-bad8-32907527a506)
$
========================
Other info:
------------
I'm trying to figure, *why* Controller node (where AMQP is running)
can't connect to AMQP (or drops connection)
$ tail scheduler.log
2013-11-25 06:11:29.880 10206 ERROR nova.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: connection aborted. Sleeping 1 seconds
Despite (on Controller node):
$ netstat -lnptu | grep -i qpid
tcp 0 0 0.0.0.0:5672 0.0.0.0:* LISTEN 28783/qpidd
tcp6 0 0 :::5672 :::* LISTEN 28783/qpidd
$ iptables -nL | grep qpid
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 5672 /* 001 qpid incoming */
$ qpid-config -a 192.169.142.118:5672 ( <-- Same
Total Exchanges: 48
topic: 6
headers: 1
fanout: 11
direct: 30
Total Queues: 30
durable: 0
non-durable: 30
$ systemctl status qpidd
qpidd.service - An AMQP message broker daemon.
Loaded: loaded (/usr/lib/systemd/system/qpidd.service; enabled)
Active: active (running) since Mon 2013-11-25 06:11:29 EST; 7min ago
Docs: man:qpidd(1)
http://qpid.apache.org/
Main PID: 28783 (qpidd)
CGroup: /system.slice/qpidd.service
└─28783 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1254677/+subscriptions