yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1254677] Re: Stopping a Nova instance in ERROR state results in -- "Conflict": Instance is not ready

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Wed, 01 Apr 2015 12:37:30 -0000
Reply-to: Bug 1254677 <1254677@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
very old bug that reporter says we are free to close

** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1254677

Title:
  Stopping a Nova instance in ERROR state results in -- "Conflict":
  Instance is not ready

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  Description
  --------------
  Attempt to stop a Nova instance (Cirros guest) in an ERROR state results in a stack trace:

  The instance ended up in ERROR state as Compute node cannot connect to the AMQP server ( running on Controller node).
  ===
  Conflict: Instance 19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246 is not ready (HTTP 409) (Request-ID: req-ecf547ce-cdd8-458b-bad8-32907527a506)
  ERROR: Instance 19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246 is not ready (HTTP 409) (Request-ID: req-ecf547ce-cdd8-458b-bad8-32907527a506)
  ===

  [I  briefly discussed this with Nikola Dipanov, he mentioned it's
  worth reporting.  Also, Attitla Fazekas had this say -- if an instance
  reaches ERROR state before a 'stop' action,  409 is expected.]

  Setup/Version info
  ----------------------

  It's a two-node Havana setup (hand-configured).

  Havana:

      $ rpm -q openstack-nova python-novaclient python-nova
      python-nova-2013.2-2.fc20.noarch
      openstack-nova-2013.2-2.fc20.noarch
      python-novaclient-2.15.0-1.fc20.noarch

  Test
  -----

  List the instance (in ERROR state).

      $ nova list
  +--------------------------------------+-------------+--------+------------+-------------+--------------+
  | ID                                   | Name        | Status | Task State | Power State | Networks     |
  +--------------------------------------+-------------+--------+------------+-------------+--------------+
  | 7a3eb749-62fb-481d-9f28-bb46494431f8 | cirr-guest5 | ERROR  | None       | NOSTATE     |              |
  +--------------------------------------+-------------+--------+------------+-------------+--------------+

  Stop the instance:

      $ nova stop cirr-guest5
  ERROR: Instance 7a3eb749-62fb-481d-9f28-bb46494431f8 is not ready (HTTP 409) (Request-ID: req-64e5d573-7a0a-48fb-bd98-bd783331b5b5)

  Stop the instance (with --debug):
  ========================
      $ nova --debug stop cirr-guest4

  REQ: curl -i http://192.169.142.118:35357/v2.0/tokens -X POST -H
  "Content-Type: application/json" -H "Accept: application/json" -H
  "User-Agent: python-novaclient" -d '{"auth": {"tenantName":
  "ostenant", "passwordCredentials": {"username": "kashyap", "password":
  "fedora"}}}'

  INFO (connectionpool:236) Starting new HTTP connection (1): 192.169.142.118
  DEBUG (connectionpool:330) "POST /v2.0/tokens HTTP/1.1" 200 5475
  RESP: [200] CaseInsensitiveDict({'date': 'Mon, 25 Nov 2013 10:01:10 GMT', 'vary': 'X-Auth-Token', 'content-length': '5475', 'content-type': 'application/json'})
  RESP BODY: {"access": {"token": {"issued_at": "2013-11-25T10:01:10.091651", "expires": "2013-11-26T10:01:10Z", "id": "MIIJ6wYJKoZIhvcNAQcCoIIJ3DCCCdgCAQExCTAHBgUrDgMCGjCCCEEGCSqGSIb3DQEHAaCCCDIEgggueyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0xMS0yNVQxMDowMToxMC4wOTE2NTEiLCAiZXhwaXJlcyI6ICIyMDEzLTExLTI2VDEwOjAxOjEwWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNjE5ZGI3OWRkNGE5NDcwNDg5MmM4NTJiMDRiYzU1MzEiLCAibmFtZSI6ICJvc3RlbmFudCJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo4Nzc2L3YxLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo4Nzc2L3YxLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgImlkIjogIjAyNTBlOTgzMmU3ZTRjMDdhYmEyZWQzZDM2YjNiMWMxIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzYvdjEvNjE5ZGI3OWRkNGE5NDcwNDg5MmM4NTJiMDRiYzU1MzEifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAidm9sdW1lIiwgIm5hbWUiOiAiY2luZGVyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5MjkyIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5MjkyIiwgImlkIjogIjJhYTM4OWVlYjA2NTRiZjE5NzFmZDg5M2ZjNzU2ZWQ5IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4OjkyOTIifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiaW1hZ2UiLCAibmFtZSI6ICJnbGFuY2UifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzQvdjEuMS82MTlkYjc5ZGQ0YTk0NzA0ODkyYzg1MmIwNGJjNTUzMSIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6ODc3NC92MS4xLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgImlkIjogIjJhNGMzZDRiMGU4NjQ5YWNhNmJlOTA5YWEwMmMzYWY4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzQvdjEuMS82MTlkYjc5ZGQ0YTk0NzA0ODkyYzg1MmIwNGJjNTUzMSJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJjb21wdXRlIiwgIm5hbWUiOiAibm92YSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6OTY5NiIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6OTY5NiIsICJpZCI6ICJiNzkzMWU1N2ExNWI0MGRjODg5N2Q0MDJmOGNkYzJlYSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5Njk2In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm5ldHdvcmsiLCAibmFtZSI6ICJuZXV0cm9uIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo1MDAwL3YyLjAiLCAiaWQiOiAiM2I4OGY5MDc2NDEzNDkyMmI3MjFjMzhmZGFmZGEyNzQiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJrYXNoeWFwIiwgInJvbGVzX2xpbmtzIjogW10sICJpZCI6ICJkNGI0Zjg4MGY3M2M0NGEyOTBlMmU5Y2FkMTAwMjU0MCIsICJyb2xlcyI6IFt7Im5hbWUiOiAidXNlciJ9XSwgIm5hbWUiOiAia2FzaHlhcCJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgInJvbGVzIjogWyIxN2E3OTNjMmM3ZmQ0MGJhYTBlOGE0ZDYzNTU5OTJhMSJdfX19MYIBgTCCAX0CAQEwXDBXMQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVW5zZXQxDjAMBgNVBAcMBVVuc2V0MQ4wDAYDVQQKDAVVbnNldDEYMBYGA1UEAwwPd3d3LmV4YW1wbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqGSIb3DQEBAQUABIIBAHEVndw9iBsw9Mn6b9W2NvXDX0LxyG31PFeEB8hrXVoaWp5dhd7lSt9G8WyAzFIeIqdWmudiZuloDZSPlxLB9vLE-PKvJpYNd2-Lw53sv4R5XCcVBP779tyrjTaBpiYMk9pBiupSzyxhwAJozTSZKsow5HwrtSZmlEdW8ngkybqz9b2p2M11h7xIA6hRXlFBUlUmM9k+NhV-MZjSt356cIAIzRiMQya4w6rcmf+otV4yrDEVhjWP9pD7cjGSVeqxKVCshZOgRsFBmlqaG6z01RHk55eypzYknVxPK646+y3S48elB39YdZ+tBUGTucavpWlXedzZAiR2TIr6tmysMJY=", "tenant": {"description": null, "enabled": true, "id": "619db79dd4a94704892c852b04bc5531", "name": "ostenant"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://192.169.142.118:8776/v1/619db79dd4a94704892c852b04bc5531";, "region": "regionOne", "internalURL": "http://192.169.142.118:8776/v1/619db79dd4a94704892c852b04bc5531";, "id": "0250e9832e7e4c07aba2ed3d36b3b1c1", "publicURL": "http://192.169.142.118:8776/v1/619db79dd4a94704892c852b04bc5531"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://192.169.142.118:9292";, "region": "regionOne", "internalURL": "http://192.169.142.118:9292";, "id": "2aa389eeb0654bf1971fd893fc756ed9", "publicURL": "http://192.169.142.118:9292"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531";, "region": "regionOne", "internalURL": "http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531";, "id": "2a4c3d4b0e8649aca6be909aa02c3af8", "publicURL": "http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://192.169.142.118:9696";, "region": "regionOne", "internalURL": "http://192.169.142.118:9696";, "id": "b7931e57a15b40dc8897d402f8cdc2ea", "publicURL": "http://192.169.142.118:9696"}], "endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints": [{"adminURL": "http://192.169.142.118:35357/v2.0";, "region": "regionOne", "internalURL": "http://192.169.142.118:5000/v2.0";, "id": "3b88f90764134922b721c38fdafda274", "publicURL": "http://192.169.142.118:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "kashyap", "roles_links": [], "id": "d4b4f880f73c44a290e2e9cad1002540", "roles": [{"name": "user"}], "name": "kashyap"}, "metadata": {"is_admin": 0, "roles": ["17a793c2c7fd40baa0e8a4d6355992a1"]}}}

  REQ: curl -i
  http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531/servers
  -X GET -H "X-Auth-Project-Id: ostenant" -H "User-Agent: python-
  novaclient" -H "Accept: application/json" -H "X-Auth-Token:
  MIIJ6wYJKoZIhvcNAQcCoIIJ3DCCCdgCAQExCTAHBgUrDgMCGjCCCEEGCSqGSIb3DQEHAaCCCDIEgggueyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0xMS0yNVQxMDowMToxMC4wOTE2NTEiLCAiZXhwaXJlcyI6ICIyMDEzLTExLTI2VDEwOjAxOjEwWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNjE5ZGI3OWRkNGE5NDcwNDg5MmM4NTJiMDRiYzU1MzEiLCAibmFtZSI6ICJvc3RlbmFudCJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo4Nzc2L3YxLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo4Nzc2L3YxLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgImlkIjogIjAyNTBlOTgzMmU3ZTRjMDdhYmEyZWQzZDM2YjNiMWMxIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzYvdjEvNjE5ZGI3OWRkNGE5NDcwNDg5MmM4NTJiMDRiYzU1MzEifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAidm9sdW1lIiwgIm5hbWUiOiAiY2luZGVyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5MjkyIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5MjkyIiwgImlkIjogIjJhYTM4OWVlYjA2NTRiZjE5NzFmZDg5M2ZjNzU2ZWQ5IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4OjkyOTIifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiaW1hZ2UiLCAibmFtZSI6ICJnbGFuY2UifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzQvdjEuMS82MTlkYjc5ZGQ0YTk0NzA0ODkyYzg1MmIwNGJjNTUzMSIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6ODc3NC92MS4xLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgImlkIjogIjJhNGMzZDRiMGU4NjQ5YWNhNmJlOTA5YWEwMmMzYWY4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzQvdjEuMS82MTlkYjc5ZGQ0YTk0NzA0ODkyYzg1MmIwNGJjNTUzMSJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJjb21wdXRlIiwgIm5hbWUiOiAibm92YSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6OTY5NiIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6OTY5NiIsICJpZCI6ICJiNzkzMWU1N2ExNWI0MGRjODg5N2Q0MDJmOGNkYzJlYSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5Njk2In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm5ldHdvcmsiLCAibmFtZSI6ICJuZXV0cm9uIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo1MDAwL3YyLjAiLCAiaWQiOiAiM2I4OGY5MDc2NDEzNDkyMmI3MjFjMzhmZGFmZGEyNzQiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJrYXNoeWFwIiwgInJvbGVzX2xpbmtzIjogW10sICJpZCI6ICJkNGI0Zjg4MGY3M2M0NGEyOTBlMmU5Y2FkMTAwMjU0MCIsICJyb2xlcyI6IFt7Im5hbWUiOiAidXNlciJ9XSwgIm5hbWUiOiAia2FzaHlhcCJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgInJvbGVzIjogWyIxN2E3OTNjMmM3ZmQ0MGJhYTBlOGE0ZDYzNTU5OTJhMSJdfX19MYIBgTCCAX0CAQEwXDBXMQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVW5zZXQxDjAMBgNVBAcMBVVuc2V0MQ4wDAYDVQQKDAVVbnNldDEYMBYGA1UEAwwPd3d3LmV4YW1wbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqGSIb3DQEBAQUABIIBAHEVndw9iBsw9Mn6b9W2NvXDX0LxyG31PFeEB8hrXVoaWp5dhd7lSt9G8WyAzFIeIqdWmudiZuloDZSPlxLB9vLE-
  PKvJpYNd2-Lw53sv4R5XCcVBP779tyrjTaBpiYMk9pBiupSzyxhwAJozTSZKsow5HwrtSZmlEdW8ngkybqz9b2p2M11h7xIA6hRXlFBUlUmM9k
  +NhV-
  MZjSt356cIAIzRiMQya4w6rcmf+otV4yrDEVhjWP9pD7cjGSVeqxKVCshZOgRsFBmlqaG6z01RHk55eypzYknVxPK646+y3S48elB39YdZ+tBUGTucavpWlXedzZAiR2TIr6tmysMJY="

  INFO (connectionpool:236) Starting new HTTP connection (1): 192.169.142.118
  DEBUG (connectionpool:330) "GET /v1.1/619db79dd4a94704892c852b04bc5531/servers HTTP/1.1" 200 731
  RESP: [200] CaseInsensitiveDict({'date': 'Mon, 25 Nov 2013 10:01:10 GMT', 'content-length': '731', 'content-type': 'application/json', 'x-compute-request-id': 'req-f3469b88-bd24-44d9-a3a5-67441b39a292'})
  RESP BODY: {"servers": [{"id": "7a3eb749-62fb-481d-9f28-bb46494431f8", "links": [{"href": "http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531/servers/7a3eb749-62fb-481d-9f28-bb46494431f8";, "rel": "self"}, {"href": "http://192.169.142.118:8774/619db79dd4a94704892c852b04bc5531/servers/7a3eb749-62fb-481d-9f28-bb46494431f8";, "rel": "bookmark"}], "name": "cirr-guest5"}, {"id": "19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246", "links": [{"href": "http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531/servers/19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246";, "rel": "self"}, {"href": "http://192.169.142.118:8774/619db79dd4a94704892c852b04bc5531/servers/19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246";, "rel": "bookmark"}], "name": "cirr-guest4"}]}

  REQ: curl -i
  http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531/servers
  /19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246 -X GET -H "X-Auth-Project-Id:
  ostenant" -H "User-Agent: python-novaclient" -H "Accept:
  application/json" -H "X-Auth-Token:
  MIIJ6wYJKoZIhvcNAQcCoIIJ3DCCCdgCAQExCTAHBgUrDgMCGjCCCEEGCSqGSIb3DQEHAaCCCDIEgggueyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0xMS0yNVQxMDowMToxMC4wOTE2NTEiLCAiZXhwaXJlcyI6ICIyMDEzLTExLTI2VDEwOjAxOjEwWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNjE5ZGI3OWRkNGE5NDcwNDg5MmM4NTJiMDRiYzU1MzEiLCAibmFtZSI6ICJvc3RlbmFudCJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo4Nzc2L3YxLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo4Nzc2L3YxLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgImlkIjogIjAyNTBlOTgzMmU3ZTRjMDdhYmEyZWQzZDM2YjNiMWMxIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzYvdjEvNjE5ZGI3OWRkNGE5NDcwNDg5MmM4NTJiMDRiYzU1MzEifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAidm9sdW1lIiwgIm5hbWUiOiAiY2luZGVyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5MjkyIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5MjkyIiwgImlkIjogIjJhYTM4OWVlYjA2NTRiZjE5NzFmZDg5M2ZjNzU2ZWQ5IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4OjkyOTIifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiaW1hZ2UiLCAibmFtZSI6ICJnbGFuY2UifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzQvdjEuMS82MTlkYjc5ZGQ0YTk0NzA0ODkyYzg1MmIwNGJjNTUzMSIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6ODc3NC92MS4xLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgImlkIjogIjJhNGMzZDRiMGU4NjQ5YWNhNmJlOTA5YWEwMmMzYWY4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzQvdjEuMS82MTlkYjc5ZGQ0YTk0NzA0ODkyYzg1MmIwNGJjNTUzMSJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJjb21wdXRlIiwgIm5hbWUiOiAibm92YSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6OTY5NiIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6OTY5NiIsICJpZCI6ICJiNzkzMWU1N2ExNWI0MGRjODg5N2Q0MDJmOGNkYzJlYSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5Njk2In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm5ldHdvcmsiLCAibmFtZSI6ICJuZXV0cm9uIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo1MDAwL3YyLjAiLCAiaWQiOiAiM2I4OGY5MDc2NDEzNDkyMmI3MjFjMzhmZGFmZGEyNzQiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJrYXNoeWFwIiwgInJvbGVzX2xpbmtzIjogW10sICJpZCI6ICJkNGI0Zjg4MGY3M2M0NGEyOTBlMmU5Y2FkMTAwMjU0MCIsICJyb2xlcyI6IFt7Im5hbWUiOiAidXNlciJ9XSwgIm5hbWUiOiAia2FzaHlhcCJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgInJvbGVzIjogWyIxN2E3OTNjMmM3ZmQ0MGJhYTBlOGE0ZDYzNTU5OTJhMSJdfX19MYIBgTCCAX0CAQEwXDBXMQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVW5zZXQxDjAMBgNVBAcMBVVuc2V0MQ4wDAYDVQQKDAVVbnNldDEYMBYGA1UEAwwPd3d3LmV4YW1wbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqGSIb3DQEBAQUABIIBAHEVndw9iBsw9Mn6b9W2NvXDX0LxyG31PFeEB8hrXVoaWp5dhd7lSt9G8WyAzFIeIqdWmudiZuloDZSPlxLB9vLE-
  PKvJpYNd2-Lw53sv4R5XCcVBP779tyrjTaBpiYMk9pBiupSzyxhwAJozTSZKsow5HwrtSZmlEdW8ngkybqz9b2p2M11h7xIA6hRXlFBUlUmM9k
  +NhV-
  MZjSt356cIAIzRiMQya4w6rcmf+otV4yrDEVhjWP9pD7cjGSVeqxKVCshZOgRsFBmlqaG6z01RHk55eypzYknVxPK646+y3S48elB39YdZ+tBUGTucavpWlXedzZAiR2TIr6tmysMJY="

  DEBUG (connectionpool:330) "GET /v1.1/619db79dd4a94704892c852b04bc5531/servers/19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246 HTTP/1.1" 200 1392
  RESP: [200] CaseInsensitiveDict({'date': 'Mon, 25 Nov 2013 10:01:10 GMT', 'content-length': '1392', 'content-type': 'application/json', 'x-compute-request-id': 'req-0732db3a-b866-4217-b341-44f35bed3b81'})
  RESP BODY: {"server": {"status": "ERROR", "updated": "2013-11-25T09:33:00Z", "hostId": "", "addresses": {}, "links": [{"href": "http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531/servers/19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246";, "rel": "self"}, {"href": "http://192.169.142.118:8774/619db79dd4a94704892c852b04bc5531/servers/19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246";, "rel": "bookmark"}], "key_name": "oskey1", "image": {"id": "e1d7f58f-60cc-48f1-bb26-91f2f8b17bfa", "links": [{"href": "http://192.169.142.118:8774/619db79dd4a94704892c852b04bc5531/images/e1d7f58f-60cc-48f1-bb26-91f2f8b17bfa";, "rel": "bookmark"}]}, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "error", "OS-SRV-USG:launched_at": null, "flavor": {"id": "2", "links": [{"href": "http://192.169.142.118:8774/619db79dd4a94704892c852b04bc5531/flavors/2";, "rel": "bookmark"}]}, "id": "19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246", "OS-SRV-USG:terminated_at": null, "OS-EXT-AZ:availability_zone": "nova", "user_id": "d4b4f880f73c44a290e2e9cad1002540", "name": "cirr-guest4", "created": "2013-11-25T09:32:59Z", "tenant_id": "619db79dd4a94704892c852b04bc5531", "OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [], "accessIPv4": "", "accessIPv6": "", "fault": {"message": "No valid host was found. ", "code": 500, "created": "2013-11-25T09:33:00Z"}, "OS-EXT-STS:power_state": 0, "config_drive": "", "metadata": {}}}

  REQ: curl -i
  http://192.169.142.118:8774/v1.1/619db79dd4a94704892c852b04bc5531/servers
  /19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246/action -X POST -H "X-Auth-
  Project-Id: ostenant" -H "User-Agent: python-novaclient" -H "Content-
  Type: application/json" -H "Accept: application/json" -H "X-Auth-
  Token:
  MIIJ6wYJKoZIhvcNAQcCoIIJ3DCCCdgCAQExCTAHBgUrDgMCGjCCCEEGCSqGSIb3DQEHAaCCCDIEgggueyJhY2Nlc3MiOiB7InRva2VuIjogeyJpc3N1ZWRfYXQiOiAiMjAxMy0xMS0yNVQxMDowMToxMC4wOTE2NTEiLCAiZXhwaXJlcyI6ICIyMDEzLTExLTI2VDEwOjAxOjEwWiIsICJpZCI6ICJwbGFjZWhvbGRlciIsICJ0ZW5hbnQiOiB7ImRlc2NyaXB0aW9uIjogbnVsbCwgImVuYWJsZWQiOiB0cnVlLCAiaWQiOiAiNjE5ZGI3OWRkNGE5NDcwNDg5MmM4NTJiMDRiYzU1MzEiLCAibmFtZSI6ICJvc3RlbmFudCJ9fSwgInNlcnZpY2VDYXRhbG9nIjogW3siZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo4Nzc2L3YxLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo4Nzc2L3YxLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgImlkIjogIjAyNTBlOTgzMmU3ZTRjMDdhYmEyZWQzZDM2YjNiMWMxIiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzYvdjEvNjE5ZGI3OWRkNGE5NDcwNDg5MmM4NTJiMDRiYzU1MzEifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAidm9sdW1lIiwgIm5hbWUiOiAiY2luZGVyIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5MjkyIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5MjkyIiwgImlkIjogIjJhYTM4OWVlYjA2NTRiZjE5NzFmZDg5M2ZjNzU2ZWQ5IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4OjkyOTIifV0sICJlbmRwb2ludHNfbGlua3MiOiBbXSwgInR5cGUiOiAiaW1hZ2UiLCAibmFtZSI6ICJnbGFuY2UifSwgeyJlbmRwb2ludHMiOiBbeyJhZG1pblVSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzQvdjEuMS82MTlkYjc5ZGQ0YTk0NzA0ODkyYzg1MmIwNGJjNTUzMSIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6ODc3NC92MS4xLzYxOWRiNzlkZDRhOTQ3MDQ4OTJjODUyYjA0YmM1NTMxIiwgImlkIjogIjJhNGMzZDRiMGU4NjQ5YWNhNmJlOTA5YWEwMmMzYWY4IiwgInB1YmxpY1VSTCI6ICJodHRwOi8vMTkyLjE2OS4xNDIuMTE4Ojg3NzQvdjEuMS82MTlkYjc5ZGQ0YTk0NzA0ODkyYzg1MmIwNGJjNTUzMSJ9XSwgImVuZHBvaW50c19saW5rcyI6IFtdLCAidHlwZSI6ICJjb21wdXRlIiwgIm5hbWUiOiAibm92YSJ9LCB7ImVuZHBvaW50cyI6IFt7ImFkbWluVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6OTY5NiIsICJyZWdpb24iOiAicmVnaW9uT25lIiwgImludGVybmFsVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6OTY5NiIsICJpZCI6ICJiNzkzMWU1N2ExNWI0MGRjODg5N2Q0MDJmOGNkYzJlYSIsICJwdWJsaWNVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo5Njk2In1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogIm5ldHdvcmsiLCAibmFtZSI6ICJuZXV0cm9uIn0sIHsiZW5kcG9pbnRzIjogW3siYWRtaW5VUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODozNTM1Ny92Mi4wIiwgInJlZ2lvbiI6ICJyZWdpb25PbmUiLCAiaW50ZXJuYWxVUkwiOiAiaHR0cDovLzE5Mi4xNjkuMTQyLjExODo1MDAwL3YyLjAiLCAiaWQiOiAiM2I4OGY5MDc2NDEzNDkyMmI3MjFjMzhmZGFmZGEyNzQiLCAicHVibGljVVJMIjogImh0dHA6Ly8xOTIuMTY5LjE0Mi4xMTg6NTAwMC92Mi4wIn1dLCAiZW5kcG9pbnRzX2xpbmtzIjogW10sICJ0eXBlIjogImlkZW50aXR5IiwgIm5hbWUiOiAia2V5c3RvbmUifV0sICJ1c2VyIjogeyJ1c2VybmFtZSI6ICJrYXNoeWFwIiwgInJvbGVzX2xpbmtzIjogW10sICJpZCI6ICJkNGI0Zjg4MGY3M2M0NGEyOTBlMmU5Y2FkMTAwMjU0MCIsICJyb2xlcyI6IFt7Im5hbWUiOiAidXNlciJ9XSwgIm5hbWUiOiAia2FzaHlhcCJ9LCAibWV0YWRhdGEiOiB7ImlzX2FkbWluIjogMCwgInJvbGVzIjogWyIxN2E3OTNjMmM3ZmQ0MGJhYTBlOGE0ZDYzNTU5OTJhMSJdfX19MYIBgTCCAX0CAQEwXDBXMQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVW5zZXQxDjAMBgNVBAcMBVVuc2V0MQ4wDAYDVQQKDAVVbnNldDEYMBYGA1UEAwwPd3d3LmV4YW1wbGUuY29tAgEBMAcGBSsOAwIaMA0GCSqGSIb3DQEBAQUABIIBAHEVndw9iBsw9Mn6b9W2NvXDX0LxyG31PFeEB8hrXVoaWp5dhd7lSt9G8WyAzFIeIqdWmudiZuloDZSPlxLB9vLE-
  PKvJpYNd2-Lw53sv4R5XCcVBP779tyrjTaBpiYMk9pBiupSzyxhwAJozTSZKsow5HwrtSZmlEdW8ngkybqz9b2p2M11h7xIA6hRXlFBUlUmM9k
  +NhV-
  MZjSt356cIAIzRiMQya4w6rcmf+otV4yrDEVhjWP9pD7cjGSVeqxKVCshZOgRsFBmlqaG6z01RHk55eypzYknVxPK646+y3S48elB39YdZ+tBUGTucavpWlXedzZAiR2TIr6tmysMJY="
  -d '{"os-stop": null}'

  DEBUG (connectionpool:330) "POST /v1.1/619db79dd4a94704892c852b04bc5531/servers/19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246/action HTTP/1.1" 409 110
  RESP: [409] CaseInsensitiveDict({'date': 'Mon, 25 Nov 2013 10:01:10 GMT', 'content-length': '110', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-ecf547ce-cdd8-458b-bad8-32907527a506'})
  RESP BODY: {"conflictingRequest": {"message": "Instance 19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246 is not ready", "code": 409}}

  DEBUG (shell:740) Instance 19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246 is not ready (HTTP 409) (Request-ID: req-ecf547ce-cdd8-458b-bad8-32907527a506)
  Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 737, in main
      OpenStackComputeShell().main(map(strutils.safe_decode, sys.argv[1:]))
    File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 673, in main
      args.func(self.cs, args)
    File "/usr/lib/python2.7/site-packages/novaclient/v1_1/shell.py", line 1281, in do_stop
      _find_server(cs, args.server).stop()
    File "/usr/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 122, in stop
      self.manager.stop(self)
    File "/usr/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 509, in stop
      return self._action('os-stop', server, None)
    File "/usr/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 951, in _action
      return self.api.client.post(url, body=body)
    File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 232, in post
      return self._cs_request(url, 'POST', **kwargs)
    File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 213, in _cs_request
      **kwargs)
    File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 195, in _time_request
      resp, body = self.request(url, method, **kwargs)
    File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 189, in request
      raise exceptions.from_response(resp, body, url, method)
  Conflict: Instance 19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246 is not ready (HTTP 409) (Request-ID: req-ecf547ce-cdd8-458b-bad8-32907527a506)
  ERROR: Instance 19b8a59b-5cd3-4ebe-a1fc-0cea2f41d246 is not ready (HTTP 409) (Request-ID: req-ecf547ce-cdd8-458b-bad8-32907527a506)
  $
  ========================

  Other info:
  ------------

  I'm trying to figure, *why* Controller node (where AMQP is running)
  can't connect to AMQP (or drops connection)

  $ tail scheduler.log
  2013-11-25 06:11:29.880 10206 ERROR nova.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: connection aborted. Sleeping 1 seconds

  Despite (on Controller node):

      $ netstat -lnptu | grep -i qpid
  tcp        0      0 0.0.0.0:5672            0.0.0.0:*               LISTEN      28783/qpidd
  tcp6       0      0 :::5672                 :::*                    LISTEN      28783/qpidd

      $ iptables -nL | grep qpid
  ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            multiport dports 5672 /* 001 qpid incoming */

      $ qpid-config -a 192.169.142.118:5672   ( <-- Same
  Total Exchanges: 48
            topic: 6
          headers: 1
           fanout: 11
           direct: 30

     Total Queues: 30
          durable: 0
      non-durable: 30

      $ systemctl status qpidd
  qpidd.service - An AMQP message broker daemon.
     Loaded: loaded (/usr/lib/systemd/system/qpidd.service; enabled)
     Active: active (running) since Mon 2013-11-25 06:11:29 EST; 7min ago
       Docs: man:qpidd(1)
             http://qpid.apache.org/
   Main PID: 28783 (qpidd)
     CGroup: /system.slice/qpidd.service
             └─28783 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1254677/+subscriptions