openstack team mailing list archive

["Deepak A.P" <swift007.deepak@xxxxxxxxx>]

trace of error log when nova-compute service was tried to re-started


sudo cat /var/log/nova/nova-compute.log


2013-04-16 06:36:46 CRITICAL nova [-] Unexpected error while running
command.
Command: sudo nova-rootwrap ip link add link eth2 name vlan100 type vlan id
100
Exit code: 1
Stdout: ''
Stderr: 'Cannot find device "eth2"\n'
2013-04-16 06:36:46 TRACE nova Traceback (most recent call last):
2013-04-16 06:36:46 TRACE nova   File "/usr/bin/nova-network", line 49, in
<module>
2013-04-16 06:36:46 TRACE nova     service.wait()
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/nova/service.py", line 413, in wait
2013-04-16 06:36:46 TRACE nova     _launcher.wait()
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/nova/service.py", line 131, in wait
2013-04-16 06:36:46 TRACE nova     service.wait()
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in
wait
2013-04-16 06:36:46 TRACE nova     return self._exit_event.wait()
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2013-04-16 06:36:46 TRACE nova     return hubs.get_hub().switch()
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
2013-04-16 06:36:46 TRACE nova     return self.greenlet.switch()
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in
main
2013-04-16 06:36:46 TRACE nova     result = function(*args, **kwargs)
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/nova/service.py", line 101, in run_server
2013-04-16 06:36:46 TRACE nova     server.start()
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/nova/service.py", line 162, in start
2013-04-16 06:36:46 TRACE nova     self.manager.init_host()
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1766, in
init_host
2013-04-16 06:36:46 TRACE nova     NetworkManager.init_host(self)
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 758, in
init_host
2013-04-16 06:36:46 TRACE nova     self._setup_network_on_host(ctxt,
network)
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1845, in
_setup_network_on_host
2013-04-16 06:36:46 TRACE nova     self.l3driver.initialize_gateway(network)
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/nova/network/l3.py", line 98, in
initialize_gateway
2013-04-16 06:36:46 TRACE nova     gateway=(network_ref['gateway'] is not
None))
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 896, in
plug
2013-04-16 06:36:46 TRACE nova     return
_get_interface_driver().plug(network, mac_address, gateway)
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 935, in
plug
2013-04-16 06:36:46 TRACE nova     mac_address)
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 958, in
ensure_vlan_bridge
2013-04-16 06:36:46 TRACE nova     bridge_interface, mac_address)
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/nova/utils.py", line 987, in inner
2013-04-16 06:36:46 TRACE nova     retval = f(*args, **kwargs)
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 971, in
ensure_vlan
2013-04-16 06:36:46 TRACE nova     'id', vlan_num, run_as_root=True)
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 813, in
_execute
2013-04-16 06:36:46 TRACE nova     return utils.execute(*cmd, **kwargs)
2013-04-16 06:36:46 TRACE nova   File
"/usr/lib/python2.7/dist-packages/nova/utils.py", line 246, in execute
2013-04-16 06:36:46 TRACE nova     cmd=' '.join(cmd))
2013-04-16 06:36:46 TRACE nova ProcessExecutionError: Unexpected error
while running command.
2013-04-16 06:36:46 TRACE nova Command: sudo nova-rootwrap ip link add link
eth2 name vlan100 type vlan id 100
2013-04-16 06:36:46 TRACE nova Exit code: 1
2013-04-16 06:36:46 TRACE nova Stdout: ''
2013-04-16 06:36:46 TRACE nova* Stderr: 'Cannot find device "eth2"\n'*
2013-04-16 06:36:46 TRACE nova



On Tue, Apr 16, 2013 at 3:43 PM, Wangpan <hzwangpan@xxxxxxxxxxxxxxxx> wrote:

> **
> you may need to check the log of nova-network, may be exist at
> /var/log/nova/nova-network.log, look into it and check if any exception
> there.
> if nothing there, you may run 'nova-network' as nova user, and check the
> output on shell.
> 2013-04-16
>  ------------------------------
>  Wangpan
>  ------------------------------
>  *发件人：*Deepak A.P
> *发送时间：*2013-04-16 18:01
> *主题：*Re: Re: Re: Re: [Openstack] Fwd: Re: [openStack] instance status
> *收件人：*"Wangpan"<hzwangpan@xxxxxxxxxxxxxxxx>
> *抄送：*"openstack-operators@xxxxxxxxxxxxxxxxxxx"<
> openstack-operators@xxxxxxxxxxxxxxxxxxx>,"OpenStack Mailing List"<
> openstack@xxxxxxxxxxxxxxxxxxx>
>
>   ok will go ahead and delete the existing stuck instance and re-create a
> new instance with tiny flavor.
>
>  Am getting another issue , however how many times i restart the nova
> services , the 'nova-network' service is not starting as below
>
>  openstack@openStack:~$ ls /etc/init/nova-* | cut -d '/' -f4 | cut -d '.'
> -f1 | while read S; do sudo status $S; done
> nova-api start/running, process 923
> nova-cert start/running, process 32736
> nova-compute start/running, process 32752
> nova-network *stop/waiting*
> nova-objectstore start/running, process 29539
> nova-scheduler start/running, process 316
>
>
> i tried stopping and re-starting the 'nova-newtwork' individually , but
> still the status shows  '*stop/waiting' *for 'nova-compute'   what could
> be the problem ?
> *
> *
>
>
>
>
> On Tue, Apr 16, 2013 at 3:23 PM, Wangpan <hzwangpan@xxxxxxxxxxxxxxxx>wrote:
>
>> **
>> delete the building state instances and re-create them may be better.
>>
>> 2013-04-16
>>  ------------------------------
>>  Wangpan
>>  ------------------------------
>>  *发件人：*Deepak A.P
>> *发送时间：*2013-04-16 17:47
>> *主题：*Re: Re: Re: [Openstack] Fwd: Re: [openStack] instance status
>>  *收件人：*"Wangpan"<hzwangpan@xxxxxxxxxxxxxxxx>
>> *抄送：*"Aaron Rosen"<arosen@xxxxxxxxxx>,"
>> openstack-operators@xxxxxxxxxxxxxxxxxxx"<
>> openstack-operators@xxxxxxxxxxxxxxxxxxx>,"OpenStack Mailing List"<
>> openstack@xxxxxxxxxxxxxxxxxxx>
>>
>>  yes i re-started all the services
>>
>> openstack@openStack:~$ ls /etc/init/nova-* | cut -d '/' -f4 | cut -d '.'
>> -f1 | while read S; do sudo start $S; done
>> start: Job is *already running:* nova-api
>> nova-cert *start/running*, process 32736
>> nova-compute *start/running*, process 32752
>> nova-network *start/running,* process 32767
>> start: *Job is already running*: nova-objectstore
>> nova-scheduler* start/running*, process 316
>>
>>
>> Should i create a new instance or reload , reboot the existing instance ?
>>
>>
>>
>>
>>
>>
>> On Tue, Apr 16, 2013 at 2:41 PM, Wangpan <hzwangpan@xxxxxxxxxxxxxxxx>wrote:
>>
>>> **
>>>  nova-cert nova-compute nova-network nova-scheduler are all stopped,
>>>
>>> nova-api start/running, process 923             ---this is running
>>> nova-cert stop/waiting                  ---this is stop
>>> nova-compute stop/waiting               ---this is stop
>>> nova-network stop/waiting               ---this is stop
>>> nova-objectstore start/running, process 29539   ---this is running
>>> nova-scheduler stop/waiting             ---this is stop
>>>
>>> 2013-04-16
>>>  ------------------------------
>>>  Wangpan
>>>  ------------------------------
>>>  *发件人：*Deepak A.P
>>> *发送时间：*2013-04-16 16:58
>>> *主题：*Re: Re: [Openstack] Fwd: Re: [openStack] instance status
>>> *收件人：*"Wangpan"<hzwangpan@xxxxxxxxxxxxxxxx>
>>> *抄送：*"Aaron Rosen"<arosen@xxxxxxxxxx>,"
>>> openstack-operators@xxxxxxxxxxxxxxxxxxx"<
>>> openstack-operators@xxxxxxxxxxxxxxxxxxx>,"OpenStack Mailing List"<
>>> openstack@xxxxxxxxxxxxxxxxxxx>
>>>
>>>  All the nova service are running as u can see it below
>>>
>>>  openstack@openStack:~$ ls /etc/init/nova-* | cut -d '/' -f4 | cut -d
>>> '.' -f1 | while read S; do sudo status $S; done
>>> [sudo] password for openstack:
>>> nova-api start/running, process 923
>>> nova-cert stop/waiting
>>> nova-compute stop/waiting
>>> nova-network stop/waiting
>>> nova-objectstore start/running, process 29539
>>> nova-scheduler stop/waiting
>>>
>>>
>>> and the glance service
>>>
>>>  openstack@openStack:~$ ls /etc/init/glance-* | cut -d '/' -f4 | cut -d
>>> '.' -f1 | while read S; do sudo status $S; done
>>> glance-api start/running, process 882
>>> glance-registry start/running, process 879
>>>
>>>
>>> On Tue, Apr 16, 2013 at 2:22 PM, Wangpan <hzwangpan@xxxxxxxxxxxxxxxx>wrote:
>>>
>>>> **
>>>> Looks like all of your nova sevices are stopped runing, and what the
>>>> result of 'service nova-compute status'?
>>>>
>>>> 2013-04-16
>>>>  ------------------------------
>>>>  Wangpan
>>>>  ------------------------------
>>>>  *发件人：*Deepak A.P
>>>> *发送时间：*2013-04-16 14:36
>>>> *主题：*Re: [Openstack] Fwd: Re: [openStack] instance status
>>>> *收件人：*"Aaron Rosen"<arosen@xxxxxxxxxx>
>>>> *抄送：*"openstack-operators@xxxxxxxxxxxxxxxxxxx"<
>>>> openstack-operators@xxxxxxxxxxxxxxxxxxx>,"OpenStack Mailing List"<
>>>> openstack@xxxxxxxxxxxxxxxxxxx>
>>>>
>>>>  openstack@openStack:~$ sudo service ntp status
>>>>  * NTP server is running
>>>>
>>>>
>>>> On Fri, Apr 12, 2013 at 11:22 AM, Aaron Rosen <arosen@xxxxxxxxxx>wrote:
>>>>
>>>>> Do you have NTP configured? If the nodes running nova-compute have
>>>>> clocks that differ from each other the status shows XXX . (Not sure why
>>>>> it's done this way though).
>>>>>
>>>>> Aaron
>>>>>
>>>>>
>>>>> On Thu, Apr 11, 2013 at 10:44 PM, Deepak A.P <
>>>>> swift007.deepak@xxxxxxxxx> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> ---------- Forwarded message ----------
>>>>>> From: Deepak A.P <swift007.deepak@xxxxxxxxx>
>>>>>> Date: Tue, Apr 9, 2013 at 2:25 PM
>>>>>> Subject: Re: Re: [Openstack] [openStack] instance status
>>>>>> To: Wangpan <hzwangpan@xxxxxxxxxxxxxxxx>
>>>>>>
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>>         The output of 'nova-manage service list'    command is
>>>>>>
>>>>>>
>>>>>>
>>>>>>         * Binary                   Host                  Zone
>>>>>>                 Status                    State
>>>>>>  Updated_At*
>>>>>>
>>>>>>         nova-compute        openStack           nova
>>>>>>        Enabled                   XXX                       2012-04-03
>>>>>>
>>>>>>         nova-network         openStack           nova
>>>>>>        Enabled                   XXX                      2012-04-03
>>>>>>
>>>>>>         nova-scheduler       openStack          nova
>>>>>>        Enabled                  XXX                        2012-04-05
>>>>>>
>>>>>>          nova-cert              openStack           nova
>>>>>>            Enabled                   XXX                       2012-04-05
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 9, 2013 at 12:21 PM, Wangpan <hzwangpan@xxxxxxxxxxxxxxxx>wrote:
>>>>>>
>>>>>>> **
>>>>>>> What the result of running 'nova-manage service list'， 'sudo' may be
>>>>>>> needed by non-root user.
>>>>>>>
>>>>>>> 2013-04-09
>>>>>>>  ------------------------------
>>>>>>>  Wangpan
>>>>>>>  ------------------------------
>>>>>>>  *发件人：*Deepak A.P
>>>>>>> *发送时间：*2013-04-09 13:54
>>>>>>> *主题：*Re: [Openstack] [openStack] instance status
>>>>>>> *收件人：*"sachin tripathi"<sachinkuber@xxxxxxxxx>
>>>>>>> *抄送：*"OpenStack Mailing List"<openstack@xxxxxxxxxxxxxxxxxxx>
>>>>>>>
>>>>>>>  Below is output which i got on running '*nova console-log*'
>>>>>>>
>>>>>>> nova console-log *d46530e2-.....*
>>>>>>>
>>>>>>> Error: The server has either erred or either incapable of performing
>>>>>>> the requested operation. HTTP(500) (Request-ID: req-be41c539.......)
>>>>>>>
>>>>>>> i tried the other nova commands like 'nova reboot' , ' nova reload'
>>>>>>> but got the similar error,
>>>>>>>
>>>>>>> and logs are not created under '/var/log/nova/'
>>>>>>>
>>>>>>> stuck with the above issue
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Apr 9, 2013 at 10:32 AM, sachin tripathi <
>>>>>>> sachinkuber@xxxxxxxxx> wrote:
>>>>>>>
>>>>>>>>    Hello,
>>>>>>>> You can check the console logs
>>>>>>>> nova console-log <instance_id>
>>>>>>>>
>>>>>>>> On the api node, you can find /var/log/nova/nova-*.log,
>>>>>>>>
>>>>>>>> And from "nova  show <instance_id> " get the hypervisor hostname
>>>>>>>> and check the hypervisor compute-log too.
>>>>>>>>
>>>>>>>> Hope, this will give some info to start troubleshoot.
>>>>>>>>
>>>>>>>> +Sachin
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>  On Tue, Apr 9, 2013 at 10:03 AM, Deepak A.P <
>>>>>>>> swift007.deepak@xxxxxxxxx> wrote:
>>>>>>>>
>>>>>>>>>  Hello,
>>>>>>>>>
>>>>>>>>>         After launching the instance i waited for long time , but
>>>>>>>>> still the status of the instance  shows 'BUILD' , the flavor of the image
>>>>>>>>> is 'm1.small' . Is there any work around to know what's happening with the
>>>>>>>>>
>>>>>>>>>         instance ?. Are there any logs where i can check the
>>>>>>>>> instance launch status ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Apr 6, 2013 at 12:01 AM, Lloyd Dewolf <
>>>>>>>>> lloydostack@xxxxxxxxx> wrote:
>>>>>>>>>
>>>>>>>>>>  On Fri, Apr 5, 2013 at 4:18 AM, Deepak A.P <
>>>>>>>>>> swift007.deepak@xxxxxxxxx> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi ,
>>>>>>>>>>>
>>>>>>>>>>>      i have a list of instances created using the below command
>>>>>>>>>>>
>>>>>>>>>>>      nova boot myInstance --image
>>>>>>>>>>> 0e2f43a8-e614-48ff-92bd-be0c68da19f4
>>>>>>>>>>>
>>>>>>>>>>>        --flavor 2 --key_name openstack
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>        i ran the below command to check the status of instances
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>     nova list
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> all the instances show status as  "*BUILD*" , how to se the status of the image to
>>>>>>>>>>>
>>>>>>>>>>> "ACTIVE " , i tried rebooting the instance am getting the below error
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Once the instance finishes building then it will be in the
>>>>>>>>>> "active" state. Depending on the image, flavor and configuration starting
>>>>>>>>>> an instance can take a long time. I would suggest first trying with a small
>>>>>>>>>> image like Cirrus and using a "tiny" flavor.
>>>>>>>>>>
>>>>>>>>>> http://docs.openstack.org/trunk/openstack-compute/admin/content/starting-images.html
>>>>>>>>>>
>>>>>>>>>> Hope that helps,
>>>>>>>>>> --
>>>>>>>>>> @lloyddewolf
>>>>>>>>>> http://www.pistoncloud.com/
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Cheers,
>>>>>>>>> Deepak
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Cheers,
>>>>>>> Deepak
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Cheers,
>>>>>> Deepak
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Cheers,
>>>>>> Deepak
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Deepak
>>>>
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Deepak
>>>
>>
>>
>>
>> --
>> Cheers,
>> Deepak
>>
>
>
>
> --
> Cheers,
> Deepak
>



-- 
Cheers,
Deepak