yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #72951
[Bug 1764125] Re: Re-attaching an encrypted(Barbican) Cinder (RBD) volume to an instance fails
Thanks to Tzach I was able to get access to an env downstream and
confirm whats going on here.
c-vol appears to be creating a fresh secret for the new volume that
isn't capable of unlocking the volume. IMHO c-vol should just copy the
associated secret UUID during the creation process from an image with
one already associated to it.
Additionally, the create flow here is really weird, I can see that we
download the image twice and try to import into rbd twice. The first
import appears to be a fresh LUKS encrypted image, the second a raw to
raw conversion that does nothing to the original LUKS encryption of the
image.
Anyway, I'm removing nova from this bug and adding cinder. More detailed
notes can be found below.
[ notes ]
I can see multiple keys used by n-cpu :
2018-05-17 11:47:47.382 1 DEBUG barbicanclient.v1.secrets [req-
6c45d622-ecf1-4cbb-a038-b8eaaf776818 ea26e0f59cf44f909a0dbe86f1f21078
3d16a4daf99042d5adbc4f0d55dbf322 - default default] Getting secret -
Secret href: http://172.17.1.12:9311/v1/secrets/a3c400ce-
8b94-4ee5-90e9-564bab6c823b get /usr/lib/python2.7/site-
packages/barbicanclient/v1/secrets.py:457
2018-05-17 11:52:26.413 1 DEBUG barbicanclient.v1.secrets [req-dfe882de-
0b11-4a70-b527-78b47a7faf2e ea26e0f59cf44f909a0dbe86f1f21078
3d16a4daf99042d5adbc4f0d55dbf322 - default default] Getting secret -
Secret href: http://172.17.1.12:9311/v1/secrets/3b88eedc-813e-
4e01-bec7-d8d2b7d2ef42 get /usr/lib/python2.7/site-
packages/barbicanclient/v1/secrets.py:457
Fetching these we can see that they are not the same :
$ curl -vv -H "X-Auth-Token: $TOKEN" -H 'Accept: application/octet-stream' -o a3c400ce-8b94-4ee5-90e9-564bab6c823b http://10.0.0.106:9311/v1/secrets/a3c400ce-8b94-4ee5-90e9-564bab6c823b
* About to connect() to 10.0.0.106 port 9311 (#0)
* Trying 10.0.0.106...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 10.0.0.106 (10.0.0.106) port 9311 (#0)
> GET /v1/secrets/a3c400ce-8b94-4ee5-90e9-564bab6c823b HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.0.0.106:9311
> X-Auth-Token: gAAAAABa_W8gLDSMIfr7hzDC385Qpjewpy2awYIrqyO0O8U5VceB4YX_xyDlH7zBBPyR68L5krAEvCzkJq-b335TbGGeqQ_EDFNa9pclZo7Qm3m0_E8ofv0W9Ny8XWwhKERNK-3BxuUUMf1N7CgexHnkIgFye23EpzZF8lcxAKWmNCIiY_p2h9g
> Accept: application/octet-stream
>
< HTTP/1.1 200 OK
< Date: Thu, 17 May 2018 12:12:16 GMT
< Server: Apache
< x-openstack-request-id: req-e32e0e58-8234-4fd3-90d8-50f9f72d617c
< Content-Length: 32
< Content-Type: application/octet-stream
<
{ [data not shown]
100 32 100 32 0 0 115 0 --:--:-- --:--:-- --:--:-- 115
* Connection #0 to host 10.0.0.106 left intact
$ curl -vv -H "X-Auth-Token: $TOKEN" -H 'Accept: application/octet-stream' -o 3b88eedc-813e-4e01-bec7-d8d2b7d2ef42 http://10.0.0.106:9311/v1/secrets/3b88eedc-813e-4e01-bec7-d8d2b7d2ef42
* About to connect() to 10.0.0.106 port 9311 (#0)
* Trying 10.0.0.106...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 10.0.0.106 (10.0.0.106) port 9311 (#0)
> GET /v1/secrets/3b88eedc-813e-4e01-bec7-d8d2b7d2ef42 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.0.0.106:9311
> X-Auth-Token: gAAAAABa_W8gLDSMIfr7hzDC385Qpjewpy2awYIrqyO0O8U5VceB4YX_xyDlH7zBBPyR68L5krAEvCzkJq-b335TbGGeqQ_EDFNa9pclZo7Qm3m0_E8ofv0W9Ny8XWwhKERNK-3BxuUUMf1N7CgexHnkIgFye23EpzZF8lcxAKWmNCIiY_p2h9g
> Accept: application/octet-stream
>
< HTTP/1.1 200 OK
< Date: Thu, 17 May 2018 12:12:33 GMT
< Server: Apache
< x-openstack-request-id: req-cd6964b5-eaac-4d97-b3a5-ae5dc2d2e474
< Content-Length: 32
< Content-Type: application/octet-stream
<
{ [data not shown]
100 32 100 32 0 0 112 0 --:--:-- --:--:-- --:--:-- 112
* Connection #0 to host 10.0.0.106 left intact
Decoding as n-cpu does to set the passphrase (urgh!):
$ python
[..]
>>> for key in ['3b88eedc-813e-4e01-bec7-d8d2b7d2ef42', 'a3c400ce-8b94-4ee5-90e9-564bab6c823b']:
... with open(key) as f:
... binascii.hexlify(f.read()).decode('utf-8')
...
u'08293dc9fc1dbbb34fa3b464a851db40b3bd4f819b96ababff9dbc13eb9ebe05'
u'a01c2b1721ce9e893ee7856aa7b9c6252d3d6ff77c9ab20c4e83e7355958352b'
$ sudo rbd -k /etc/ceph/ceph.client.openstack.keyring --user openstack export volumes/volume-17276f8b-c21d-479e-a58a-522319d01ff8 17276f8b-c21d-479e-a58a-522319d01ff8.img
Exporting image: 100% complete...done.
$ sudo rbd -k /etc/ceph/ceph.client.openstack.keyring --user openstack export volumes/volume-50fb8cc6-fd69-4a7e-844e-e0629d81a3b7 50fb8cc6-fd69-4a7e-844e-e0629d81a3b7.img
Exporting image: 100% complete...done.
We can see the same LUKS header in both volumes:
$ qemu-img info 17276f8b-c21d-479e-a58a-522319d01ff8.img
image: 17276f8b-c21d-479e-a58a-522319d01ff8.img
file format: luks
virtual size: 1.0G (1073741824 bytes)
disk size: 134M
encrypted: yes
Format specific information:
ivgen alg: plain64
hash alg: sha256
cipher alg: aes-256
uuid: 8d41fc3e-d134-480c-a35e-a42bf8a5763b
cipher mode: xts
slots:
[0]:
active: true
iters: 1038604
key offset: 4096
stripes: 4000
[1]:
active: false
key offset: 262144
[2]:
active: false
key offset: 520192
[3]:
active: false
key offset: 778240
[4]:
active: false
key offset: 1036288
[5]:
active: false
key offset: 1294336
[6]:
active: false
key offset: 1552384
[7]:
active: false
key offset: 1810432
payload offset: 2068480
master key iters: 259425
$ qemu-img info 50fb8cc6-fd69-4a7e-844e-e0629d81a3b7.img
image: 50fb8cc6-fd69-4a7e-844e-e0629d81a3b7.img
file format: luks
virtual size: 2.0G (2145415168 bytes)
disk size: 112M
encrypted: yes
Format specific information:
ivgen alg: plain64
hash alg: sha256
cipher alg: aes-256
uuid: 8d41fc3e-d134-480c-a35e-a42bf8a5763b
cipher mode: xts
slots:
[0]:
active: true
iters: 1038604
key offset: 4096
stripes: 4000
[1]:
active: false
key offset: 262144
[2]:
active: false
key offset: 520192
[3]:
active: false
key offset: 778240
[4]:
active: false
key offset: 1036288
[5]:
active: false
key offset: 1294336
[6]:
active: false
key offset: 1552384
[7]:
active: false
key offset: 1810432
payload offset: 2068480
master key iters: 259425
However only one key works here:
$ sudo losetup -f 17276f8b-c21d-479e-a58a-522319d01ff8.img
$ sudo losetup -f 50fb8cc6-fd69-4a7e-844e-e0629d81a3b7.img
$ sudo cryptsetup luksOpen /dev/loop0 17276f8b-c21d-479e-a58a-522319d01ff8
Enter passphrase for /home/heat-admin/17276f8b-c21d-479e-a58a-522319d01ff8.img: a01c2b1721ce9e893ee7856aa7b9c6252d3d6ff77c9ab20c4e83e7355958352b
$ sudo cryptsetup luksOpen /dev/loop1 50fb8cc6-fd69-4a7e-844e-e0629d81a3b7
Enter passphrase for /home/heat-admin/50fb8cc6-fd69-4a7e-844e-e0629d81a3b7.img:
a01c2b1721ce9e893ee7856aa7b9c6252d3d6ff77c9ab20c4e83e7355958352b
$ sudo cryptsetup close 17276f8b-c21d-479e-a58a-522319d01ff8
$ sudo cryptsetup close 50fb8cc6-fd69-4a7e-844e-e0629d81a3b7
$ sudo cryptsetup luksOpen /dev/loop0 17276f8b-c21d-479e-a58a-522319d01ff8
Enter passphrase for /home/heat-admin/17276f8b-c21d-479e-a58a-522319d01ff8.img: 08293dc9fc1dbbb34fa3b464a851db40b3bd4f819b96ababff9dbc13eb9ebe05
No key available with this passphrase.
Enter passphrase for /home/heat-admin/17276f8b-c21d-479e-a58a-522319d01ff8.img: Error reading passphrase from terminal.
$ sudo cryptsetup luksOpen /dev/loop1 50fb8cc6-fd69-4a7e-844e-e0629d81a3b7
Enter passphrase for /home/heat-admin/50fb8cc6-fd69-4a7e-844e-e0629d81a3b7.img: 08293dc9fc1dbbb34fa3b464a851db40b3bd4f819b96ababff9dbc13eb9ebe05
No key available with this passphrase.
Enter passphrase for /home/heat-admin/50fb8cc6-fd69-4a7e-844e-e0629d81a3b7.img: Error reading passphrase from terminal.
** Project changed: nova => cinder
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1764125
Title:
Re-attaching an encrypted(Barbican) Cinder (RBD) volume to an instance
fails
Status in Cinder:
New
Bug description:
Description of problem:
An attached encrypted (Barbican) RBD Cinder volume was attached to instance, write data to it.
Then volume was detached, when trying to reattach the volume to same instance volume fails to attach. Odd errors on attached nova-compute.log
2018-04-15 13:14:06.274 1 ERROR nova.compute.manager [instance: 923c5318-8502-4f85-a215-78afc4fd641b] uuid=managed_object_id)
2018-04-15 13:14:06.274 1 ERROR nova.compute.manager [instance: 923c5318-8502-4f85-a215-78afc4fd641b] ManagedObjectNotFoundError: Key not found, uuid: 7912eac8-2652-4c92-b53f-3db4ecca7bc7
2018-04-15 13:14:06.523 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 177, in request
2018-04-15 13:14:06.523 1 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body)
2018-04-15 13:14:06.523 1 ERROR oslo_messaging.rpc.server VolumeAttachmentNotFound: Volume attachment c17e2b89-5a36-4e7e-8c71-b975f2f5ccb3 could not be found.
2018-04-15 13:14:06.523 1 ERROR oslo_messaging.rpc.server
How reproducible:
Unsure looks like every time I try to re-attach.
Steps to Reproduce:
1. Boot an instance
2. Create an encrypted(Barbican) backed Cinder(RBD) volume, attach to instance write data.
3. Detach volume from instance
4. Try to reattach same volume to same instance.
$nova volume-attach 923c5318-8502-4f85-a215-78afc4fd641b
16584072-ef78-4a80-91ab-cbd47e9bc70d auto
5. Volume fails to attach
No error volume remains unattached
cinder list
+--------------------------------------+-----------+-------------+------+----------------------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-------------+------+----------------------------+----------+--------------------------------------+
| 16584072-ef78-4a80-91ab-cbd47e9bc70d | available | 2-Encrypted | 1 | LuksEncryptor-Template-256 | false | |
Actual results:
Volume fails to attach.
Expected results:
Volume should successfully reattach.
Environment / Version-Release number of selected component (if applicable):
rhel7.5
openstack-nova-conductor-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
python-nova-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
python-novaclient-9.1.1-1.el7ost.noarch
openstack-cinder-12.0.1-0.20180326201852.46c4ec1.el7ost.noarch
openstack-nova-scheduler-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-nova-console-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
puppet-cinder-12.3.1-0.20180222074326.18152ac.el7ost.noarch
openstack-nova-compute-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
python2-cinderclient-3.5.0-1.el7ost.noarch
python-cinder-12.0.1-0.20180326201852.46c4ec1.el7ost.noarch
openstack-nova-api-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-nova-novncproxy-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
puppet-nova-12.3.1-0.20180319062741.9db79a6.el7ost.noarch
openstack-nova-common-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-nova-migration-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
openstack-nova-placement-api-17.0.2-0.20180323024604.0390d5f.el7ost.noarch
Libvirt + KVM
Neutron networking
Cinder volume is RBD backed and encrypted via Barbican.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1764125/+subscriptions
References
