openstack team mailing list archive

[Martinx - ジェームズ <thiagocmartinsc@xxxxxxxxx>]

Hi John! Thank you for your support!

My answers are at the bottom.

On 2 February 2013 22:51, John Griffith <john.griffith@xxxxxxxxxxxxx> wrote:

>
>
> On Sat, Feb 2, 2013 at 1:34 PM, Martinx - ジェームズ <thiagocmartinsc@xxxxxxxxx
> > wrote:
>
>> No one can help me with this?
>> My cinder-volume process is completely down.
>> I don't know what I can do to fix this issue...
>>
>> Thanks!
>> Thiago
>>
>>
>> On 1 February 2013 19:49, Martinx - ジェームズ <thiagocmartinsc@xxxxxxxxx>wrote:
>>
>>> Guys,
>>>
>>>  At first, tgt was running after one "controller.teste.com" reboot but,
>>> I know I'm using ietd, so, I stop tgt and remove the init script of if and
>>> started ietd manually...
>>>
>>>  But cinder doesn't start up anymore...   :(
>>>
>>>  I really appreciate any kind of help here...
>>>
>>>  Someone on IRC channel told me that Cinder is trying to use
>>> "_get_target" function in the TargetAdmin class but, how can Cinder use tgt
>>> if "iscsi_helper = ietadm" ?
>>>
>>> Best,
>>> Thiago
>>>
>>> On 1 February 2013 19:20, Martinx - ジェームズ <thiagocmartinsc@xxxxxxxxx>wrote:
>>>
>>>> Guys,
>>>>
>>>>  At my Cinder logs, I'm seeing this when I run
>>>> "/etc/init.d/cinder-volume start":
>>>>
>>>> ----
>>>> 2013-02-01 18:16:20 1075 AUDIT cinder.service [-] Starting
>>>> cinder-volume node (version 2012.2.1-LOCALBRANCH:LOCALREVISION)
>>>> 2013-02-01 18:16:21 DEBUG cinder.utils
>>>> [req-1ebb9638-2300-456a-995e-382c96f6632d None None] Running cmd
>>>> (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf vgs
>>>> --noheadings -o name execute
>>>> /usr/lib/python2.7/dist-packages/cinder/utils.py:163
>>>> 2013-02-01 18:16:22 DEBUG cinder.utils
>>>> [req-1ebb9638-2300-456a-995e-382c96f6632d None None] backend <module
>>>> 'cinder.db.sqlalchemy.api' from
>>>> '/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.pyc'>
>>>> __get_backend /usr/lib/python2.7/dist-packages/cinder/utils.py:477
>>>> 2013-02-01 18:16:23 DEBUG cinder.volume.manager
>>>> [req-1ebb9638-2300-456a-995e-382c96f6632d None None] Re-exporting 1 volumes
>>>> init_host /usr/lib/python2.7/dist-packages/cinder/volume/manager.py:98
>>>> 2013-02-01 18:16:23 1075 CRITICAL cinder [-] Got unknown keyword args
>>>> to utils.execute: {'old_name': None}
>>>> 2013-02-01 18:16:23 1075 TRACE cinder Traceback (most recent call last):
>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File "/usr/bin/cinder-volume",
>>>> line 48, in <module>
>>>> 2013-02-01 18:16:23 1075 TRACE cinder     service.wait()
>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>> "/usr/lib/python2.7/dist-packages/cinder/service.py", line 422, in wait
>>>> 2013-02-01 18:16:23 1075 TRACE cinder     _launcher.wait()
>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>> "/usr/lib/python2.7/dist-packages/cinder/service.py", line 127, in wait
>>>> 2013-02-01 18:16:23 1075 TRACE cinder     service.wait()
>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>> "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in
>>>> wait
>>>> 2013-02-01 18:16:23 1075 TRACE cinder     return self._exit_event.wait()
>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>> "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
>>>> 2013-02-01 18:16:23 1075 TRACE cinder     return hubs.get_hub().switch()
>>>>  2013-02-01 18:16:23 1075 TRACE cinder   File
>>>> "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
>>>> 2013-02-01 18:16:23 1075 TRACE cinder     return self.greenlet.switch()
>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>> "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in
>>>> main
>>>> 2013-02-01 18:16:23 1075 TRACE cinder     result = function(*args,
>>>> **kwargs)
>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>> "/usr/lib/python2.7/dist-packages/cinder/service.py", line 88, in run_server
>>>> 2013-02-01 18:16:23 1075 TRACE cinder     server.start()
>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>> "/usr/lib/python2.7/dist-packages/cinder/service.py", line 159, in start
>>>> 2013-02-01 18:16:23 1075 TRACE cinder     self.manager.init_host()
>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>> "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 101, in
>>>> init_host
>>>> 2013-02-01 18:16:23 1075 TRACE cinder
>>>> self.driver.ensure_export(ctxt, volume)
>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>> "/usr/lib/python2.7/dist-packages/cinder/volume/driver.py", line 352, in
>>>> ensure_export
>>>> 2013-02-01 18:16:23 1075 TRACE cinder     old_name=old_name)
>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>> "/usr/lib/python2.7/dist-packages/cinder/volume/iscsi.py", line 198, in
>>>> create_iscsi_target
>>>> 2013-02-01 18:16:23 1075 TRACE cinder     self._new_target(name, tid,
>>>> **kwargs)
>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>> "/usr/lib/python2.7/dist-packages/cinder/volume/iscsi.py", line 211, in
>>>> _new_target
>>>> 2013-02-01 18:16:23 1075 TRACE cinder     **kwargs)
>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>> "/usr/lib/python2.7/dist-packages/cinder/volume/iscsi.py", line 60, in _run
>>>> 2013-02-01 18:16:23 1075 TRACE cinder     self._execute(self._cmd,
>>>> *args, run_as_root=True, **kwargs)
>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>> "/usr/lib/python2.7/dist-packages/cinder/utils.py", line 142, in execute
>>>> 2013-02-01 18:16:23 1075 TRACE cinder     'to utils.execute: %r') %
>>>> kwargs)
>>>> 2013-02-01 18:16:23 1075 TRACE cinder Error: Got unknown keyword args
>>>> to utils.execute: {'old_name': None}
>>>> 2013-02-01 18:16:23 1075 TRACE cinder
>>>> ----
>>>>
>>>>  My cinder.conf contains:
>>>>
>>>> ----
>>>> [DEFAULT]
>>>> sql_connection = mysql://
>>>> cinderUser:cinderPass@xxxxxxxxxxxxxxxxxxxx/cinder
>>>> rootwrap_config = /etc/cinder/rootwrap.conf
>>>> api_paste_confg = /etc/cinder/api-paste.ini
>>>> iscsi_helper = ietadm
>>>> volume_name_template = volume-%s
>>>> volume_group = cinder-volumes
>>>> verbose = True
>>>> auth_strategy = keystone
>>>> #osapi_volume_listen_port=5900
>>>> ---
>>>>
>>>>  My /etc/cinder/api-paste.ini relevant parts:
>>>>
>>>> ----
>>>> [filter:authtoken]
>>>> paste.filter_factory = keystone.middleware.auth_token:filter_factory
>>>> service_protocol = http
>>>> service_host = controller.teste.com
>>>> service_port = 5000
>>>> auth_host = controller.teste.com
>>>> auth_port = 35357
>>>> auth_protocol = http
>>>> admin_tenant_name = service
>>>> admin_user = cinder
>>>> admin_password = service_pass
>>>> ----
>>>>
>>>>  The "controller.teste.com" host resolution is okay (IP 10.32.14.232).
>>>> It was working yesterday, volumes creation, attaching, deleting, etc...
>>>>  =(
>>>>
>>>>  I'm using Openstack Folsom on top of Ubuntu 12.04.
>>>>
>>>> Thanks,
>>>> Thiago
>>>>
>>>
>>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
> First let's make sure I have this correct:
>
> 1. You setup and installed using tgt iscsi-helper
>
No. I'm using ietd since the beginning.

My "ISCSITARGET_ENABLE=true" at /etc/default/iscsitarget (to start ietd).

The tgt come up after a server reboot... I didn't remove the tgt from the
initscripts, my bad...

But, have this triggered this problem?

2. Created a volume succesfully
>
Yes.


> 3. Something went wrong, had some problem so changed to iet
>
No.


> 4. Did NOT explicitly delete the volume that you had created (via 'cinder
> delete xxxx')
>
> If this is accurate then here's what I would try depending on your needs:
>
> 1. Revert back to using the tgt iscsi-helper
>
The tgt was never used.


>
> If you don't want to do that, and/or if you still have the same issue:
>
> 1. Delete the entry for the volume that you created earlier from the
> database
>
        I suspect what's happening here is that on startup it's trying to
> reset for the volume you created earlier under tgt
>         Either the volume and persistence path is no longer there, or
> since you've switched to iet it's messed up and doesn't know
>         how to reinit the iscsi connection.
>
>
> If these things don't work we can try some more detailed attempts at
> cleaning up, but if this isn't production data I'd be inclined to just
> reset and use the tgt iscsi helper.
>
> John
>

I'll start another Cinder from scratch using the tgt but, I think we need
to know what's going on here with this server.

I mean, maybe this is a BUG and no one knows.

I think that this is important, because if this was a production
environment, well, I'd be in trouble now.

Thanks!
Thiago