openstack team mailing list archive

[Martinx - ジェームズ <thiagocmartinsc@xxxxxxxxx>]

John,

 I reinstalled it from scratch, using TGT now. Everything seems to be
running fine now.

 Do you think that it worth try to identify this problem or just forget it?
I can try to reproduce it.

Thank you!
Thiago

On 3 February 2013 03:13, Martinx - ジェームズ <thiagocmartinsc@xxxxxxxxx> wrote:

> Hi John! Thank you for your support!
>
> My answers are at the bottom.
>
> On 2 February 2013 22:51, John Griffith <john.griffith@xxxxxxxxxxxxx>wrote:
>
>>
>>
>> On Sat, Feb 2, 2013 at 1:34 PM, Martinx - ジェームズ <
>> thiagocmartinsc@xxxxxxxxx> wrote:
>>
>>> No one can help me with this?
>>> My cinder-volume process is completely down.
>>> I don't know what I can do to fix this issue...
>>>
>>> Thanks!
>>> Thiago
>>>
>>>
>>> On 1 February 2013 19:49, Martinx - ジェームズ <thiagocmartinsc@xxxxxxxxx>wrote:
>>>
>>>> Guys,
>>>>
>>>>  At first, tgt was running after one "controller.teste.com" reboot
>>>> but, I know I'm using ietd, so, I stop tgt and remove the init script of if
>>>> and started ietd manually...
>>>>
>>>>  But cinder doesn't start up anymore...   :(
>>>>
>>>>  I really appreciate any kind of help here...
>>>>
>>>>  Someone on IRC channel told me that Cinder is trying to use
>>>> "_get_target" function in the TargetAdmin class but, how can Cinder use tgt
>>>> if "iscsi_helper = ietadm" ?
>>>>
>>>> Best,
>>>> Thiago
>>>>
>>>> On 1 February 2013 19:20, Martinx - ジェームズ <thiagocmartinsc@xxxxxxxxx>wrote:
>>>>
>>>>> Guys,
>>>>>
>>>>>  At my Cinder logs, I'm seeing this when I run
>>>>> "/etc/init.d/cinder-volume start":
>>>>>
>>>>> ----
>>>>> 2013-02-01 18:16:20 1075 AUDIT cinder.service [-] Starting
>>>>> cinder-volume node (version 2012.2.1-LOCALBRANCH:LOCALREVISION)
>>>>> 2013-02-01 18:16:21 DEBUG cinder.utils
>>>>> [req-1ebb9638-2300-456a-995e-382c96f6632d None None] Running cmd
>>>>> (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf vgs
>>>>> --noheadings -o name execute
>>>>> /usr/lib/python2.7/dist-packages/cinder/utils.py:163
>>>>> 2013-02-01 18:16:22 DEBUG cinder.utils
>>>>> [req-1ebb9638-2300-456a-995e-382c96f6632d None None] backend <module
>>>>> 'cinder.db.sqlalchemy.api' from
>>>>> '/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.pyc'>
>>>>> __get_backend /usr/lib/python2.7/dist-packages/cinder/utils.py:477
>>>>> 2013-02-01 18:16:23 DEBUG cinder.volume.manager
>>>>> [req-1ebb9638-2300-456a-995e-382c96f6632d None None] Re-exporting 1 volumes
>>>>> init_host /usr/lib/python2.7/dist-packages/cinder/volume/manager.py:98
>>>>> 2013-02-01 18:16:23 1075 CRITICAL cinder [-] Got unknown keyword args
>>>>> to utils.execute: {'old_name': None}
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder Traceback (most recent call
>>>>> last):
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File "/usr/bin/cinder-volume",
>>>>> line 48, in <module>
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder     service.wait()
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>>> "/usr/lib/python2.7/dist-packages/cinder/service.py", line 422, in wait
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder     _launcher.wait()
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>>> "/usr/lib/python2.7/dist-packages/cinder/service.py", line 127, in wait
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder     service.wait()
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>>> "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in
>>>>> wait
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder     return
>>>>> self._exit_event.wait()
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>>> "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder     return
>>>>> hubs.get_hub().switch()
>>>>>  2013-02-01 18:16:23 1075 TRACE cinder   File
>>>>> "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder     return self.greenlet.switch()
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>>> "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in
>>>>> main
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder     result = function(*args,
>>>>> **kwargs)
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>>> "/usr/lib/python2.7/dist-packages/cinder/service.py", line 88, in run_server
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder     server.start()
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>>> "/usr/lib/python2.7/dist-packages/cinder/service.py", line 159, in start
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder     self.manager.init_host()
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>>> "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 101, in
>>>>> init_host
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder
>>>>> self.driver.ensure_export(ctxt, volume)
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>>> "/usr/lib/python2.7/dist-packages/cinder/volume/driver.py", line 352, in
>>>>> ensure_export
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder     old_name=old_name)
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>>> "/usr/lib/python2.7/dist-packages/cinder/volume/iscsi.py", line 198, in
>>>>> create_iscsi_target
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder     self._new_target(name, tid,
>>>>> **kwargs)
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>>> "/usr/lib/python2.7/dist-packages/cinder/volume/iscsi.py", line 211, in
>>>>> _new_target
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder     **kwargs)
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>>> "/usr/lib/python2.7/dist-packages/cinder/volume/iscsi.py", line 60, in _run
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder     self._execute(self._cmd,
>>>>> *args, run_as_root=True, **kwargs)
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder   File
>>>>> "/usr/lib/python2.7/dist-packages/cinder/utils.py", line 142, in execute
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder     'to utils.execute: %r') %
>>>>> kwargs)
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder Error: Got unknown keyword args
>>>>> to utils.execute: {'old_name': None}
>>>>> 2013-02-01 18:16:23 1075 TRACE cinder
>>>>> ----
>>>>>
>>>>>  My cinder.conf contains:
>>>>>
>>>>> ----
>>>>> [DEFAULT]
>>>>> sql_connection = mysql://
>>>>> cinderUser:cinderPass@xxxxxxxxxxxxxxxxxxxx/cinder
>>>>> rootwrap_config = /etc/cinder/rootwrap.conf
>>>>> api_paste_confg = /etc/cinder/api-paste.ini
>>>>> iscsi_helper = ietadm
>>>>> volume_name_template = volume-%s
>>>>> volume_group = cinder-volumes
>>>>> verbose = True
>>>>> auth_strategy = keystone
>>>>> #osapi_volume_listen_port=5900
>>>>> ---
>>>>>
>>>>>  My /etc/cinder/api-paste.ini relevant parts:
>>>>>
>>>>> ----
>>>>> [filter:authtoken]
>>>>> paste.filter_factory = keystone.middleware.auth_token:filter_factory
>>>>> service_protocol = http
>>>>> service_host = controller.teste.com
>>>>> service_port = 5000
>>>>> auth_host = controller.teste.com
>>>>> auth_port = 35357
>>>>> auth_protocol = http
>>>>> admin_tenant_name = service
>>>>> admin_user = cinder
>>>>> admin_password = service_pass
>>>>> ----
>>>>>
>>>>>  The "controller.teste.com" host resolution is okay (IP
>>>>> 10.32.14.232). It was working yesterday, volumes creation, attaching,
>>>>> deleting, etc...    =(
>>>>>
>>>>>  I'm using Openstack Folsom on top of Ubuntu 12.04.
>>>>>
>>>>> Thanks,
>>>>> Thiago
>>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>> First let's make sure I have this correct:
>>
>> 1. You setup and installed using tgt iscsi-helper
>>
> No. I'm using ietd since the beginning.
>
> My "ISCSITARGET_ENABLE=true" at /etc/default/iscsitarget (to start ietd).
>
> The tgt come up after a server reboot... I didn't remove the tgt from the
> initscripts, my bad...
>
> But, have this triggered this problem?
>
> 2. Created a volume succesfully
>>
> Yes.
>
>
>> 3. Something went wrong, had some problem so changed to iet
>>
> No.
>
>
>> 4. Did NOT explicitly delete the volume that you had created (via 'cinder
>> delete xxxx')
>>
>> If this is accurate then here's what I would try depending on your needs:
>>
>> 1. Revert back to using the tgt iscsi-helper
>>
> The tgt was never used.
>
>
>>
>> If you don't want to do that, and/or if you still have the same issue:
>>
>> 1. Delete the entry for the volume that you created earlier from the
>> database
>>
>         I suspect what's happening here is that on startup it's trying to
>> reset for the volume you created earlier under tgt
>>         Either the volume and persistence path is no longer there, or
>> since you've switched to iet it's messed up and doesn't know
>>         how to reinit the iscsi connection.
>>
>>
>> If these things don't work we can try some more detailed attempts at
>> cleaning up, but if this isn't production data I'd be inclined to just
>> reset and use the tgt iscsi helper.
>>
>> John
>>
>
> I'll start another Cinder from scratch using the tgt but, I think we need
> to know what's going on here with this server.
>
> I mean, maybe this is a BUG and no one knows.
>
> I think that this is important, because if this was a production
> environment, well, I'd be in trouble now.
>
> Thanks!
> Thiago
>