openstack team mailing list archive

Thread
Date
Re: Instance termination is not stable

To: openstack@xxxxxxxxxxxxxxxxxxx
From: Sajith Kariyawasam <sajhak@xxxxxxxxx>
Date: Tue, 19 Jun 2012 22:51:34 +0530
In-reply-to: <CANGgF+8QsTsM4cdMiq4nU7DgyHMv2Rtbw28C447dJWW-VrsWFw@mail.gmail.com>
Any clue on this guys?

On Mon, Jun 18, 2012 at 7:08 PM, Sajith Kariyawasam <sajhak@xxxxxxxxx>wrote:

> Hi all,
>
> I have Openstack Essex version installed and I have created several
> instances based on an Ubuntu-12.04 UEC image in Openstack and those are up
> and running.
>
> When I'm trying to terminate an instance I'm getting an exception (log is
> mentioned below) and, in console its status is shown as "Shutoff" and the
> task is "Deleting". Even though i tried terminating the instance again and
> again nothing happens. But after I restart machine (nova) those instances
> can be terminated.
>
> This issue is not occurred everytime, but occassionally, as I noted this
> occurs when there are more than 2 instances up and running at the same
> time.. If I create one instance, terminate that, again create one,
> terminate that one, if goes like that, there wont be an issue in
> terminating.
>
> What could be the problem here? any suggestions are highly appreciated.
>
> Thanks
>
>
> *ERROR LOG ( /var/log/nova/nova-compute.log )
> ==========*
>
> 2012-06-18 18:43:55 DEBUG nova.manager [-] Skipping
> ComputeManager._run_image_cache_manager_pass, 17 ticks left until next run
> from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:147
> 2012-06-18 18:43:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._reclaim_queued_deletes from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:43:55 DEBUG nova.compute.manager [-]
> FLAGS.reclaim_instance_interval <= 0, skipping... from (pid=24151)
> _reclaim_queued_deletes
> /usr/lib/python2.7/dist-packages/nova/compute/manager.py:2380
> 2012-06-18 18:43:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._report_driver_status from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:43:55 INFO nova.compute.manager [-] Updating host status
> 2012-06-18 18:43:55 DEBUG nova.virt.libvirt.connection [-] Updating host
> stats from (pid=24151) update_status
> /usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py:2467
> 2012-06-18 18:43:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._poll_unconfirmed_resizes from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:44:17 DEBUG nova.rpc.amqp [-] received {u'_context_roles':
> [u'swiftoperator', u'Member', u'admin'], u'_context_request_id':
> u'req-01ca70c8-2240-407b-92d1-5a59ee497291', u'_context_read_deleted':
> u'no', u'args': {u'instance_uuid':
> u'9999d250-1d8b-4973-8320-e6058a2058b9'}, u'_context_auth_token':
> '<SANITIZED>', u'_context_is_admin': True, u'_context_project_id':
> u'194d6e24ec1843fb8fbd94c3fb519deb', u'_context_timestamp':
> u'2012-06-18T13:14:17.013212', u'_context_user_id':
> u'f8a75778c36241479693ff61a754f67b', u'method': u'terminate_instance',
> u'_context_remote_address': u'172.16.0.254'} from (pid=24151) _safe_log
> /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160
> 2012-06-18 18:44:17 DEBUG nova.rpc.amqp
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] unpacked context: {'user_id':
> u'f8a75778c36241479693ff61a754f67b', 'roles': [u'swiftoperator', u'Member',
> u'admin'], 'timestamp': '2012-06-18T13:14:17.013212', 'auth_token':
> '<SANITIZED>', 'remote_address': u'172.16.0.254', 'is_admin': True,
> 'request_id': u'req-01ca70c8-2240-407b-92d1-5a59ee497291', 'project_id':
> u'194d6e24ec1843fb8fbd94c3fb519deb', 'read_deleted': u'no'} from
> (pid=24151) _safe_log
> /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160
> 2012-06-18 18:44:17 INFO nova.compute.manager
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] check_instance_lock: decorating:
> |<function terminate_instance at 0x2bd3050>|
> 2012-06-18 18:44:17 INFO nova.compute.manager
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] check_instance_lock: arguments:
> |<nova.compute.manager.ComputeManager object at 0x20ffb90>|
> |<nova.rpc.amqp.RpcContext object at 0x4d2a450>|
> |9999d250-1d8b-4973-8320-e6058a2058b9|
> 2012-06-18 18:44:17 DEBUG nova.compute.manager
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] instance
> 9999d250-1d8b-4973-8320-e6058a2058b9: getting locked state from (pid=24151)
> get_lock /usr/lib/python2.7/dist-packages/nova/compute/manager.py:1597
> 2012-06-18 18:44:17 INFO nova.compute.manager
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] check_instance_lock: locked: |False|
> 2012-06-18 18:44:17 INFO nova.compute.manager
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] check_instance_lock: admin: |True|
> 2012-06-18 18:44:17 INFO nova.compute.manager
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] check_instance_lock: executing:
> |<function terminate_instance at 0x2bd3050>|
> 2012-06-18 18:44:17 DEBUG nova.utils
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] Attempting to grab semaphore
> "9999d250-1d8b-4973-8320-e6058a2058b9" for method
> "do_terminate_instance"... from (pid=24151) inner
> /usr/lib/python2.7/dist-packages/nova/utils.py:927
> 2012-06-18 18:44:17 DEBUG nova.utils
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] Got semaphore
> "9999d250-1d8b-4973-8320-e6058a2058b9" for method
> "do_terminate_instance"... from (pid=24151) inner
> /usr/lib/python2.7/dist-packages/nova/utils.py:931
> 2012-06-18 18:44:17 AUDIT nova.compute.manager
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] [instance:
> 9999d250-1d8b-4973-8320-e6058a2058b9] Terminating instance
> 2012-06-18 18:44:17 DEBUG nova.rpc.amqp
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] Making asynchronous call on network ...
> from (pid=24151) multicall
> /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:321
> 2012-06-18 18:44:17 DEBUG nova.rpc.amqp
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] MSG_ID is
> 2fba7314616d480fa39d5d4d1d942c46 from (pid=24151) multicall
> /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:324
> 2012-06-18 18:44:17 DEBUG nova.compute.manager
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] [instance:
> 9999d250-1d8b-4973-8320-e6058a2058b9] Deallocating network for instance
> from (pid=24151) _deallocate_network
> /usr/lib/python2.7/dist-packages/nova/compute/manager.py:616
> 2012-06-18 18:44:17 DEBUG nova.rpc.amqp
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] Making asynchronous cast on network...
> from (pid=24151) cast /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:346
> 2012-06-18 18:44:20 WARNING nova.virt.libvirt.connection
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] [instance:
> 9999d250-1d8b-4973-8320-e6058a2058b9] Error from libvirt during saved
> instance removal. Code=3 Error=this function is not supported by the
> connection driver: virDomainHasManagedSaveImage
> 2012-06-18 18:44:20 DEBUG nova.utils
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] Attempting to grab semaphore "iptables"
> for method "apply"... from (pid=24151) inner
> /usr/lib/python2.7/dist-packages/nova/utils.py:927
> 2012-06-18 18:44:20 DEBUG nova.utils
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] Got semaphore "iptables" for method
> "apply"... from (pid=24151) inner
> /usr/lib/python2.7/dist-packages/nova/utils.py:931
> 2012-06-18 18:44:20 DEBUG nova.utils
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] Attempting to grab file lock "iptables"
> for method "apply"... from (pid=24151) inner
> /usr/lib/python2.7/dist-packages/nova/utils.py:935
> 2012-06-18 18:44:20 DEBUG nova.utils
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] Got file lock "iptables" for method
> "apply"... from (pid=24151) inner
> /usr/lib/python2.7/dist-packages/nova/utils.py:942
> 2012-06-18 18:44:20 DEBUG nova.utils
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] Running cmd (subprocess): sudo
> nova-rootwrap iptables-save -t filter from (pid=24151) execute
> /usr/lib/python2.7/dist-packages/nova/utils.py:219
> 2012-06-18 18:44:20 INFO nova.virt.libvirt.connection [-] [instance:
> 9999d250-1d8b-4973-8320-e6058a2058b9] Instance destroyed successfully.
> 2012-06-18 18:44:20 DEBUG nova.utils
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] Running cmd (subprocess): sudo
> nova-rootwrap iptables-restore from (pid=24151) execute
> /usr/lib/python2.7/dist-packages/nova/utils.py:219
> 2012-06-18 18:44:20 DEBUG nova.utils
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] Running cmd (subprocess): sudo
> nova-rootwrap iptables-save -t nat from (pid=24151) execute
> /usr/lib/python2.7/dist-packages/nova/utils.py:219
> 2012-06-18 18:44:21 DEBUG nova.utils
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] Running cmd (subprocess): sudo
> nova-rootwrap iptables-restore from (pid=24151) execute
> /usr/lib/python2.7/dist-packages/nova/utils.py:219
> 2012-06-18 18:44:21 DEBUG nova.network.linux_net
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] IPTablesManager.apply completed with
> success from (pid=24151) apply
> /usr/lib/python2.7/dist-packages/nova/network/linux_net.py:335
> 2012-06-18 18:44:21 INFO nova.virt.libvirt.connection
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] [instance:
> 9999d250-1d8b-4973-8320-e6058a2058b9] Deleting instance files
> /var/lib/nova/instances/instance-00000037
> 2012-06-18 18:44:21 DEBUG nova.utils
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] Running cmd (subprocess): sudo
> nova-rootwrap umount /dev/nbd11 from (pid=24151) execute
> /usr/lib/python2.7/dist-packages/nova/utils.py:219
> 2012-06-18 18:44:21 DEBUG nova.utils
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] Running cmd (subprocess): sudo
> nova-rootwrap qemu-nbd -d /dev/nbd11 from (pid=24151) execute
> /usr/lib/python2.7/dist-packages/nova/utils.py:219
> 2012-06-18 18:44:21 ERROR nova.rpc.amqp
> [req-01ca70c8-2240-407b-92d1-5a59ee497291 f8a75778c36241479693ff61a754f67b
> 194d6e24ec1843fb8fbd94c3fb519deb] Exception during message handling
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp Traceback (most recent call last):
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 252, in
> _process_data
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp     rval = node_func(context=ctxt,
> **node_args)
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp     return f(*args, **kw)
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 153, in
> decorated_function
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp     function(self, context,
> instance_uuid, *args, **kwargs)
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 177, in
> decorated_function
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp     sys.exc_info())
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp     self.gen.next()
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 171, in
> decorated_function
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp     return function(self, context,
> instance_uuid, *args, **kwargs)
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 747, in
> terminate_instance
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp     do_terminate_instance()
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/utils.py", line 945, in inner
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp     retval = f(*args, **kwargs)
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 740, in
> do_terminate_instance
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp     self._delete_instance(context,
> instance)
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 718, in
> _delete_instance
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp
> self._shutdown_instance(context, instance, 'Terminating')
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 687, in
> _shutdown_instance
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp     block_device_info)
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line
> 484, in destroy
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp     cleanup=True)
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line
> 478, in _destroy
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp     self._cleanup(instance)
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line
> 493, in _cleanup
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp     shutil.rmtree(target)
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/shutil.py", line 245, in rmtree
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp     rmtree(fullname,
> ignore_errors, onerror)
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/shutil.py", line 245, in rmtree
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp     rmtree(fullname,
> ignore_errors, onerror)
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/shutil.py", line 250, in rmtree
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp     onerror(os.remove, fullname,
> sys.exc_info())
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp   File
> "/usr/lib/python2.7/shutil.py", line 248, in rmtree
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp     os.remove(fullname)
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp OSError: [Errno 13] Permission
> denied:
> '/var/lib/nova/instances/instance-00000037/rootfs/boot/memtest86+.bin'
> 2012-06-18 18:44:21 TRACE nova.rpc.amqp
> 2012-06-18 18:44:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._publish_service_capabilities from (pid=24151)
> periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:44:55 DEBUG nova.manager [-] Notifying Schedulers of
> capabilities ... from (pid=24151) _publish_service_capabilities
> /usr/lib/python2.7/dist-packages/nova/manager.py:203
> 2012-06-18 18:44:55 DEBUG nova.rpc.amqp [-] Making asynchronous fanout
> cast... from (pid=24151) fanout_cast
> /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:354
> 2012-06-18 18:44:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._poll_rescued_instances from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:44:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._sync_power_states from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:44:55 WARNING nova.compute.manager [-] Found 5 in the
> database and 4 on the hypervisor.
> 2012-06-18 18:44:55 WARNING nova.compute.manager [-] [instance:
> 9999d250-1d8b-4973-8320-e6058a2058b9] Instance found in database but not
> known by hypervisor. Setting power state to NOSTATE
> 2012-06-18 18:44:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._poll_bandwidth_usage from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:44:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager.update_available_resource from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:44:55 INFO nova.virt.libvirt.connection [-] Compute_service
> record updated for sajithvb2
> 2012-06-18 18:44:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._poll_rebooting_instances from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:44:55 DEBUG nova.manager [-] Skipping
> ComputeManager._cleanup_running_deleted_instances, 27 ticks left until next
> run from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:147
> 2012-06-18 18:44:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._heal_instance_info_cache from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:44:55 DEBUG nova.rpc.amqp [-] Making asynchronous call on
> network ... from (pid=24151) multicall
> /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:321
> 2012-06-18 18:44:55 DEBUG nova.rpc.amqp [-] MSG_ID is
> e220048302744b3180c048d0e410aa29 from (pid=24151) multicall
> /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:324
> 2012-06-18 18:44:55 DEBUG nova.compute.manager [-] Updated the info_cache
> for instance fa181c09-f78e-441f-afda-8c8eb84f24bd from (pid=24151)
> _heal_instance_info_cache
> /usr/lib/python2.7/dist-packages/nova/compute/manager.py:2227
> 2012-06-18 18:44:55 DEBUG nova.manager [-] Skipping
> ComputeManager._run_image_cache_manager_pass, 16 ticks left until next run
> from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:147
> 2012-06-18 18:44:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._reclaim_queued_deletes from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:44:55 DEBUG nova.compute.manager [-]
> FLAGS.reclaim_instance_interval <= 0, skipping... from (pid=24151)
> _reclaim_queued_deletes
> /usr/lib/python2.7/dist-packages/nova/compute/manager.py:2380
> 2012-06-18 18:44:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._report_driver_status from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:44:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._poll_unconfirmed_resizes from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:45:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._publish_service_capabilities from (pid=24151)
> periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:45:55 DEBUG nova.manager [-] Notifying Schedulers of
> capabilities ... from (pid=24151) _publish_service_capabilities
> /usr/lib/python2.7/dist-packages/nova/manager.py:203
> 2012-06-18 18:45:55 DEBUG nova.rpc.amqp [-] Making asynchronous fanout
> cast... from (pid=24151) fanout_cast
> /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:354
> 2012-06-18 18:45:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._poll_rescued_instances from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:45:55 DEBUG nova.manager [-] Skipping
> ComputeManager._sync_power_states, 10 ticks left until next run from
> (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:147
> 2012-06-18 18:45:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._poll_bandwidth_usage from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:45:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager.update_available_resource from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:45:55 INFO nova.virt.libvirt.connection [-] Compute_service
> record updated for sajithvb2
> 2012-06-18 18:45:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._poll_rebooting_instances from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:45:55 DEBUG nova.manager [-] Skipping
> ComputeManager._cleanup_running_deleted_instances, 26 ticks left until next
> run from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:147
> 2012-06-18 18:45:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._heal_instance_info_cache from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:45:55 DEBUG nova.rpc.amqp [-] Making asynchronous call on
> network ... from (pid=24151) multicall
> /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:321
> 2012-06-18 18:45:55 DEBUG nova.rpc.amqp [-] MSG_ID is
> 4dfad20f5f9f498997cf6ee7141e563d from (pid=24151) multicall
> /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:324
> 2012-06-18 18:45:55 DEBUG nova.compute.manager [-] Updated the info_cache
> for instance 8ff80284-ff59-4eb7-b5e7-5e3a36fc4144 from (pid=24151)
> _heal_instance_info_cache
> /usr/lib/python2.7/dist-packages/nova/compute/manager.py:2227
> 2012-06-18 18:45:55 DEBUG nova.manager [-] Skipping
> ComputeManager._run_image_cache_manager_pass, 15 ticks left until next run
> from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:147
> 2012-06-18 18:45:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._reclaim_queued_deletes from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:45:55 DEBUG nova.compute.manager [-]
> FLAGS.reclaim_instance_interval <= 0, skipping... from (pid=24151)
> _reclaim_queued_deletes
> /usr/lib/python2.7/dist-packages/nova/compute/manager.py:2380
> 2012-06-18 18:45:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._report_driver_status from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:45:55 INFO nova.compute.manager [-] Updating host status
> 2012-06-18 18:45:55 DEBUG nova.virt.libvirt.connection [-] Updating host
> stats from (pid=24151) update_status
> /usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py:2467
> 2012-06-18 18:45:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._poll_unconfirmed_resizes from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:46:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._publish_service_capabilities from (pid=24151)
> periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:46:55 DEBUG nova.manager [-] Notifying Schedulers of
> capabilities ... from (pid=24151) _publish_service_capabilities
> /usr/lib/python2.7/dist-packages/nova/manager.py:203
> 2012-06-18 18:46:55 DEBUG nova.rpc.amqp [-] Making asynchronous fanout
> cast... from (pid=24151) fanout_cast
> /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:354
> 2012-06-18 18:46:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._poll_rescued_instances from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:46:55 DEBUG nova.manager [-] Skipping
> ComputeManager._sync_power_states, 9 ticks left until next run from
> (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:147
> 2012-06-18 18:46:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._poll_bandwidth_usage from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:46:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager.update_available_resource from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:46:55 INFO nova.virt.libvirt.connection [-] Compute_service
> record updated for sajithvb2
> 2012-06-18 18:46:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._poll_rebooting_instances from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:46:55 DEBUG nova.manager [-] Skipping
> ComputeManager._cleanup_running_deleted_instances, 25 ticks left until next
> run from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:147
> 2012-06-18 18:46:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._heal_instance_info_cache from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:46:55 DEBUG nova.rpc.amqp [-] Making asynchronous call on
> network ... from (pid=24151) multicall
> /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:321
> 2012-06-18 18:46:55 DEBUG nova.rpc.amqp [-] MSG_ID is
> fcebfbce0666469dbefdd4b1a2c4df38 from (pid=24151) multicall
> /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:324
> 2012-06-18 18:46:55 DEBUG nova.compute.manager [-] Updated the info_cache
> for instance dda6d890-72bf-4538-816d-5e19702902a4 from (pid=24151)
> _heal_instance_info_cache
> /usr/lib/python2.7/dist-packages/nova/compute/manager.py:2227
> 2012-06-18 18:46:55 DEBUG nova.manager [-] Skipping
> ComputeManager._run_image_cache_manager_pass, 14 ticks left until next run
> from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:147
> 2012-06-18 18:46:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._reclaim_queued_deletes from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:46:55 DEBUG nova.compute.manager [-]
> FLAGS.reclaim_instance_interval <= 0, skipping... from (pid=24151)
> _reclaim_queued_deletes
> /usr/lib/python2.7/dist-packages/nova/compute/manager.py:2380
> 2012-06-18 18:46:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._report_driver_status from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
> 2012-06-18 18:46:55 DEBUG nova.manager [-] Running periodic task
> ComputeManager._poll_unconfirmed_resizes from (pid=24151) periodic_tasks
> /usr/lib/python2.7/dist-packages/nova/manager.py:152
>
>
> --
> Best Regards
> Sajith
>
>


-- 
Best Regards
Sajith
References

Instance termination is not stable
From: Sajith Kariyawasam, 2012-06-18