yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #85563
[Bug 1919357] [NEW] "Secure live migration with QEMU-native TLS in nova"-guide misses essential config option
*** This bug is a security vulnerability ***
Private security bug reported:
- [x] This doc is inaccurate in this way: ______
I followed the guide to setup qemu native tls for live migration. After
checking, that libvirt is able to use tls using tcpdump to listen on the
port for tls, I also wanted to check that it works when I live migrate
an instance. Apparently it didn't. But it used the port for unencrypted
TCP [1].
After digging through documentation and code afterwards I found that in
this code part:
https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L1120
@staticmethod
def _live_migration_uri(dest):
uris = {
'kvm': 'qemu+%(scheme)s://%(dest)s/system',
'qemu': 'qemu+%(scheme)s://%(dest)s/system',
'xen': 'xenmigr://%(dest)s/system',
'parallels': 'parallels+tcp://%(dest)s/system',
}
dest = oslo_netutils.escape_ipv6(dest)
virt_type = CONF.libvirt.virt_type
# TODO(pkoniszewski): Remove fetching live_migration_uri in Pike
uri = CONF.libvirt.live_migration_uri
if uri:
return uri % dest
uri = uris.get(virt_type)
if uri is None:
raise exception.LiveMigrationURINotAvailable(virt_type=virt_type)
str_format = {
'dest': dest,
'scheme': CONF.libvirt.live_migration_scheme or 'tcp',
}
return uri % str_format
the uri is calculated using the config parameter 'live_migration_scheme'
or using the hard coded tcp parameter. Coming from the guide for qemu
native tls, there was no hint that this config option needs to be set.
In fact without setting this 'live_migration_scheme' config option to
tls, there is no way to see, that the live migration still uses the
unencrypted tcp connection - one has to use tcpdump and listen for tcp
or tls to recognize it. Neither in the logs nor in any debug output
there is any hint that it is still unencrypted!
Thus I conclude there might be OpenStack deployments which are
configured as the guide say but these config changes have no effect!
- [x] This is a doc addition request.
To fix this the config parameter 'live_migration_scheme' should be set
to tls and maybe there should be a warning in the documentation, that
without doing this, the traffic is still unencrypted.
- [ ] I have a fix to the document that I can paste below including
example: input and output.
[1] without setting 'live_migration_scheme' in the nova.conf
$ tcpdump -i INTERFACE -n -X port 16509 and '(tcp[((tcp[12] & 0xf0) >> 2)] < 0x14 || tcp[((tcp[12] & 0xf0) >> 2)] > 0x17)'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes
17:10:56.387407 IP 192.168.70.101.50900 > 192.168.70.100.16509: Flags [P.], seq 304:6488, ack 285, win 502, options [nop,nop,TS val 424149655 ecr 1875309961], length 6184
0x0000: 4500 185c ad05 4000 4006 677c c0a8 4665 E..\..@.@.g|..Fe
0x0010: c0a8 4664 c6d4 407d a407 70a6 15ad 0a5a ..Fd..@}..p....Z
0x0020: 8018 01f6 2669 0000 0101 080a 1948 0297 ....&i.......H..
0x0030: 6fc6 f589 0000 1828 2000 8086 0000 0001 o......(........
0x0040: 0000 012f 0000 0000 0000 0009 0000 0000 .../............
0x0050: 0000 0001 0000 000f 6465 7374 696e 6174 ........destinat
0x0060: 696f 6e5f 786d 6c00 0000 0007 0000 129b ion_xml.........
0x0070: 3c64 6f6d 6169 6e20 7479 7065 3d27 6b76 <domain.type='kv
0x0080: 6d27 3e0a 2020 3c6e 616d 653e 696e 7374 m'>...<name>inst
0x0090: 616e 6365 2d30 3030 3032 6539 393c 2f6e ance-00002e99</n
0x00a0: 616d 653e 0a20 203c 7575 6964 3e37 6532 ame>...<uuid>7e2
0x00b0: 6364 3839 352d 6263 3765 2d34 6634 352d cd895-bc7e-4f45-
0x00c0: 6166 6264 2d33 3732 3166 3735 6134 3064 afbd-3721f75a40d
0x00d0: 383c 2f75 7569 643e 0a20 203c 6d65 7461 8</uuid>...<meta
[2] after setting 'live_migration_scheme = tls'
$ tcpdump -i INTERFACE -n -X port 16514 and '(tcp[((tcp[12] & 0xf0) >> 2)] > 0x13 && tcp[((tcp[12] & 0xf0) >> 2)] < 0x18)'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes
16:55:47.746851 IP 192.168.70.100.35620 > 192.168.70.101.16514: Flags [P.], seq 1849334708:1849334914, ack 3121294199, win 502, options [nop,nop,TS val 1874401351 ecr 423241020], length 206
0x0000: 4500 0102 a605 4000 4006 85d6 c0a8 4664 E.....@.@.....Fd
0x0010: c0a8 4665 8b24 4082 6e3a 9bb4 ba0b 2b77 ..Fe.$@.n:....+w
0x0020: 8018 01f6 2f43 0000 0101 080a 6fb9 1847 ..../C......o..G
0x0030: 193a 253c 1603 0100 c901 0000 c503 0360 .:%<...........`
0x0040: 50e3 23f7 2ab9 57b2 47f5 e0e8 b550 014b P.#.*.W.G....P.K
0x0050: 35c6 fe95 3475 cc66 9f81 fc0d 1646 ab00 5...4u.f.....F..
0x0060: 0052 c02c c087 cca9 c0ad c00a c02b c086 .R.,.........+..
0x0070: c0ac c009 c008 c030 c08b cca8 c014 c02f .......0......./
All in all: I can switch back and forth using TLS or unencrypted TCP via
the different ports with just changing the config parameter
'live_migration_scheme'. All tests were done on an OpenStack Train
deployment.
-----------------------------------
Release: 22.1.0.dev604 on 2019-03-11 17:38:39
SHA: 6e6d78d204dbf23d1ffaf232456fb2917b4f8372
Source: https://opendev.org/openstack/nova/src/doc/source/admin/secure-live-migration-with-qemu-native-tls.rst
URL: https://docs.openstack.org/nova/latest/admin/secure-live-migration-with-qemu-native-tls.html
** Affects: nova
Importance: Undecided
Status: New
** Tags: doc
** Information type changed from Public to Private Security
** Description changed:
-
- [x] This doc is inaccurate in this way: ______
I followed the guide to setup qemu native tls for live migration. After
checking, that libvirt is able to use tls using tcpdump to listen on the
port for tls, I also wanted to check that it works when I live migrate
an instance. Apparently it didn't. But it used the port for unencrypted
TCP [1].
After digging through documentation and code afterwards I found that in
this code part:
https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L1120
@staticmethod
- def _live_migration_uri(dest):
- uris = {
- 'kvm': 'qemu+%(scheme)s://%(dest)s/system',
- 'qemu': 'qemu+%(scheme)s://%(dest)s/system',
- 'xen': 'xenmigr://%(dest)s/system',
- 'parallels': 'parallels+tcp://%(dest)s/system',
- }
- dest = oslo_netutils.escape_ipv6(dest)
+ def _live_migration_uri(dest):
+ uris = {
+ 'kvm': 'qemu+%(scheme)s://%(dest)s/system',
+ 'qemu': 'qemu+%(scheme)s://%(dest)s/system',
+ 'xen': 'xenmigr://%(dest)s/system',
+ 'parallels': 'parallels+tcp://%(dest)s/system',
+ }
+ dest = oslo_netutils.escape_ipv6(dest)
- virt_type = CONF.libvirt.virt_type
- # TODO(pkoniszewski): Remove fetching live_migration_uri in Pike
- uri = CONF.libvirt.live_migration_uri
- if uri:
- return uri % dest
+ virt_type = CONF.libvirt.virt_type
+ # TODO(pkoniszewski): Remove fetching live_migration_uri in Pike
+ uri = CONF.libvirt.live_migration_uri
+ if uri:
+ return uri % dest
- uri = uris.get(virt_type)
- if uri is None:
- raise exception.LiveMigrationURINotAvailable(virt_type=virt_type)
+ uri = uris.get(virt_type)
+ if uri is None:
+ raise exception.LiveMigrationURINotAvailable(virt_type=virt_type)
- str_format = {
- 'dest': dest,
- 'scheme': CONF.libvirt.live_migration_scheme or 'tcp',
- }
- return uri % str_format
+ str_format = {
+ 'dest': dest,
+ 'scheme': CONF.libvirt.live_migration_scheme or 'tcp',
+ }
+ return uri % str_format
the uri is calculated using the config parameter 'live_migration_scheme'
or using the hard coded tcp parameter. Coming from the guide for qemu
native tls, there was no hint that this config option needs to be set.
In fact without setting this 'live_migration_scheme' config option to
tls, there is no way to see, that the live migration still uses the
unencrypted tcp connection - one has to use tcpdump and listen for tcp
or tls to recognize it. Neither in the logs nor in any debug output
there is any hint that it is still unencrypted!
Thus I conclude there might be OpenStack deployments which are
configured as the guide say but these config changes have no effect!
- [x] This is a doc addition request.
To fix this the config parameter 'live_migration_scheme' should be set
to tls and maybe there should be a warning in the documentation, that
without doing this, the traffic is still unencrypted.
- [ ] I have a fix to the document that I can paste below including
example: input and output.
-
[1] without setting 'live_migration_scheme' in the nova.conf
- tcpdump -i eno1 -n -X port 16509 and '(tcp[((tcp[12] & 0xf0) >> 2)] < 0x14 || tcp[((tcp[12] & 0xf0) >> 2)] > 0x17)'
+ $ tcpdump -i INTERFACE -n -X port 16509 and '(tcp[((tcp[12] & 0xf0) >> 2)] < 0x14 || tcp[((tcp[12] & 0xf0) >> 2)] > 0x17)'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
- listening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes
+ listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes
17:10:56.387407 IP 192.168.70.101.50900 > 192.168.70.100.16509: Flags [P.], seq 304:6488, ack 285, win 502, options [nop,nop,TS val 424149655 ecr 1875309961], length 6184
- 0x0000: 4500 185c ad05 4000 4006 677c c0a8 4665 E..\..@.@.g|..Fe
- 0x0010: c0a8 4664 c6d4 407d a407 70a6 15ad 0a5a ..Fd..@}..p....Z
- 0x0020: 8018 01f6 2669 0000 0101 080a 1948 0297 ....&i.......H..
- 0x0030: 6fc6 f589 0000 1828 2000 8086 0000 0001 o......(........
- 0x0040: 0000 012f 0000 0000 0000 0009 0000 0000 .../............
- 0x0050: 0000 0001 0000 000f 6465 7374 696e 6174 ........destinat
- 0x0060: 696f 6e5f 786d 6c00 0000 0007 0000 129b ion_xml.........
- 0x0070: 3c64 6f6d 6169 6e20 7479 7065 3d27 6b76 <domain.type='kv
- 0x0080: 6d27 3e0a 2020 3c6e 616d 653e 696e 7374 m'>...<name>inst
- 0x0090: 616e 6365 2d30 3030 3032 6539 393c 2f6e ance-00002e99</n
- 0x00a0: 616d 653e 0a20 203c 7575 6964 3e37 6532 ame>...<uuid>7e2
- 0x00b0: 6364 3839 352d 6263 3765 2d34 6634 352d cd895-bc7e-4f45-
- 0x00c0: 6166 6264 2d33 3732 3166 3735 6134 3064 afbd-3721f75a40d
- 0x00d0: 383c 2f75 7569 643e 0a20 203c 6d65 7461 8</uuid>...<meta
-
+ 0x0000: 4500 185c ad05 4000 4006 677c c0a8 4665 E..\..@.@.g|..Fe
+ 0x0010: c0a8 4664 c6d4 407d a407 70a6 15ad 0a5a ..Fd..@}..p....Z
+ 0x0020: 8018 01f6 2669 0000 0101 080a 1948 0297 ....&i.......H..
+ 0x0030: 6fc6 f589 0000 1828 2000 8086 0000 0001 o......(........
+ 0x0040: 0000 012f 0000 0000 0000 0009 0000 0000 .../............
+ 0x0050: 0000 0001 0000 000f 6465 7374 696e 6174 ........destinat
+ 0x0060: 696f 6e5f 786d 6c00 0000 0007 0000 129b ion_xml.........
+ 0x0070: 3c64 6f6d 6169 6e20 7479 7065 3d27 6b76 <domain.type='kv
+ 0x0080: 6d27 3e0a 2020 3c6e 616d 653e 696e 7374 m'>...<name>inst
+ 0x0090: 616e 6365 2d30 3030 3032 6539 393c 2f6e ance-00002e99</n
+ 0x00a0: 616d 653e 0a20 203c 7575 6964 3e37 6532 ame>...<uuid>7e2
+ 0x00b0: 6364 3839 352d 6263 3765 2d34 6634 352d cd895-bc7e-4f45-
+ 0x00c0: 6166 6264 2d33 3732 3166 3735 6134 3064 afbd-3721f75a40d
+ 0x00d0: 383c 2f75 7569 643e 0a20 203c 6d65 7461 8</uuid>...<meta
[2] after setting 'live_migration_scheme = tls'
$ tcpdump -i INTERFACE -n -X port 16514 and '(tcp[((tcp[12] & 0xf0) >> 2)] > 0x13 && tcp[((tcp[12] & 0xf0) >> 2)] < 0x18)'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes
16:55:47.746851 IP 192.168.70.100.35620 > 192.168.70.101.16514: Flags [P.], seq 1849334708:1849334914, ack 3121294199, win 502, options [nop,nop,TS val 1874401351 ecr 423241020], length 206
- 0x0000: 4500 0102 a605 4000 4006 85d6 c0a8 4664 E.....@.@.....Fd
- 0x0010: c0a8 4665 8b24 4082 6e3a 9bb4 ba0b 2b77 ..Fe.$@.n:....+w
- 0x0020: 8018 01f6 2f43 0000 0101 080a 6fb9 1847 ..../C......o..G
- 0x0030: 193a 253c 1603 0100 c901 0000 c503 0360 .:%<...........`
- 0x0040: 50e3 23f7 2ab9 57b2 47f5 e0e8 b550 014b P.#.*.W.G....P.K
- 0x0050: 35c6 fe95 3475 cc66 9f81 fc0d 1646 ab00 5...4u.f.....F..
- 0x0060: 0052 c02c c087 cca9 c0ad c00a c02b c086 .R.,.........+..
- 0x0070: c0ac c009 c008 c030 c08b cca8 c014 c02f .......0......./
+ 0x0000: 4500 0102 a605 4000 4006 85d6 c0a8 4664 E.....@.@.....Fd
+ 0x0010: c0a8 4665 8b24 4082 6e3a 9bb4 ba0b 2b77 ..Fe.$@.n:....+w
+ 0x0020: 8018 01f6 2f43 0000 0101 080a 6fb9 1847 ..../C......o..G
+ 0x0030: 193a 253c 1603 0100 c901 0000 c503 0360 .:%<...........`
+ 0x0040: 50e3 23f7 2ab9 57b2 47f5 e0e8 b550 014b P.#.*.W.G....P.K
+ 0x0050: 35c6 fe95 3475 cc66 9f81 fc0d 1646 ab00 5...4u.f.....F..
+ 0x0060: 0052 c02c c087 cca9 c0ad c00a c02b c086 .R.,.........+..
+ 0x0070: c0ac c009 c008 c030 c08b cca8 c014 c02f .......0......./
-
- All in all: I can switch back and forth using TLS or unencrypted TCP via the different ports with just changing the config parameter 'live_migration_scheme'. All tests were done on an OpenStack Train deployment.
+ All in all: I can switch back and forth using TLS or unencrypted TCP via
+ the different ports with just changing the config parameter
+ 'live_migration_scheme'. All tests were done on an OpenStack Train
+ deployment.
-----------------------------------
Release: 22.1.0.dev604 on 2019-03-11 17:38:39
SHA: 6e6d78d204dbf23d1ffaf232456fb2917b4f8372
Source: https://opendev.org/openstack/nova/src/doc/source/admin/secure-live-migration-with-qemu-native-tls.rst
URL: https://docs.openstack.org/nova/latest/admin/secure-live-migration-with-qemu-native-tls.html
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1919357
Title:
"Secure live migration with QEMU-native TLS in nova"-guide misses
essential config option
Status in OpenStack Compute (nova):
New
Bug description:
- [x] This doc is inaccurate in this way: ______
I followed the guide to setup qemu native tls for live migration.
After checking, that libvirt is able to use tls using tcpdump to
listen on the port for tls, I also wanted to check that it works when
I live migrate an instance. Apparently it didn't. But it used the port
for unencrypted TCP [1].
After digging through documentation and code afterwards I found that
in this code part:
https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L1120
@staticmethod
def _live_migration_uri(dest):
uris = {
'kvm': 'qemu+%(scheme)s://%(dest)s/system',
'qemu': 'qemu+%(scheme)s://%(dest)s/system',
'xen': 'xenmigr://%(dest)s/system',
'parallels': 'parallels+tcp://%(dest)s/system',
}
dest = oslo_netutils.escape_ipv6(dest)
virt_type = CONF.libvirt.virt_type
# TODO(pkoniszewski): Remove fetching live_migration_uri in Pike
uri = CONF.libvirt.live_migration_uri
if uri:
return uri % dest
uri = uris.get(virt_type)
if uri is None:
raise exception.LiveMigrationURINotAvailable(virt_type=virt_type)
str_format = {
'dest': dest,
'scheme': CONF.libvirt.live_migration_scheme or 'tcp',
}
return uri % str_format
the uri is calculated using the config parameter
'live_migration_scheme' or using the hard coded tcp parameter. Coming
from the guide for qemu native tls, there was no hint that this config
option needs to be set.
In fact without setting this 'live_migration_scheme' config option to
tls, there is no way to see, that the live migration still uses the
unencrypted tcp connection - one has to use tcpdump and listen for tcp
or tls to recognize it. Neither in the logs nor in any debug output
there is any hint that it is still unencrypted!
Thus I conclude there might be OpenStack deployments which are
configured as the guide say but these config changes have no effect!
- [x] This is a doc addition request.
To fix this the config parameter 'live_migration_scheme' should be set
to tls and maybe there should be a warning in the documentation, that
without doing this, the traffic is still unencrypted.
- [ ] I have a fix to the document that I can paste below including
example: input and output.
[1] without setting 'live_migration_scheme' in the nova.conf
$ tcpdump -i INTERFACE -n -X port 16509 and '(tcp[((tcp[12] & 0xf0) >> 2)] < 0x14 || tcp[((tcp[12] & 0xf0) >> 2)] > 0x17)'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes
17:10:56.387407 IP 192.168.70.101.50900 > 192.168.70.100.16509: Flags [P.], seq 304:6488, ack 285, win 502, options [nop,nop,TS val 424149655 ecr 1875309961], length 6184
0x0000: 4500 185c ad05 4000 4006 677c c0a8 4665 E..\..@.@.g|..Fe
0x0010: c0a8 4664 c6d4 407d a407 70a6 15ad 0a5a ..Fd..@}..p....Z
0x0020: 8018 01f6 2669 0000 0101 080a 1948 0297 ....&i.......H..
0x0030: 6fc6 f589 0000 1828 2000 8086 0000 0001 o......(........
0x0040: 0000 012f 0000 0000 0000 0009 0000 0000 .../............
0x0050: 0000 0001 0000 000f 6465 7374 696e 6174 ........destinat
0x0060: 696f 6e5f 786d 6c00 0000 0007 0000 129b ion_xml.........
0x0070: 3c64 6f6d 6169 6e20 7479 7065 3d27 6b76 <domain.type='kv
0x0080: 6d27 3e0a 2020 3c6e 616d 653e 696e 7374 m'>...<name>inst
0x0090: 616e 6365 2d30 3030 3032 6539 393c 2f6e ance-00002e99</n
0x00a0: 616d 653e 0a20 203c 7575 6964 3e37 6532 ame>...<uuid>7e2
0x00b0: 6364 3839 352d 6263 3765 2d34 6634 352d cd895-bc7e-4f45-
0x00c0: 6166 6264 2d33 3732 3166 3735 6134 3064 afbd-3721f75a40d
0x00d0: 383c 2f75 7569 643e 0a20 203c 6d65 7461 8</uuid>...<meta
[2] after setting 'live_migration_scheme = tls'
$ tcpdump -i INTERFACE -n -X port 16514 and '(tcp[((tcp[12] & 0xf0) >> 2)] > 0x13 && tcp[((tcp[12] & 0xf0) >> 2)] < 0x18)'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes
16:55:47.746851 IP 192.168.70.100.35620 > 192.168.70.101.16514: Flags [P.], seq 1849334708:1849334914, ack 3121294199, win 502, options [nop,nop,TS val 1874401351 ecr 423241020], length 206
0x0000: 4500 0102 a605 4000 4006 85d6 c0a8 4664 E.....@.@.....Fd
0x0010: c0a8 4665 8b24 4082 6e3a 9bb4 ba0b 2b77 ..Fe.$@.n:....+w
0x0020: 8018 01f6 2f43 0000 0101 080a 6fb9 1847 ..../C......o..G
0x0030: 193a 253c 1603 0100 c901 0000 c503 0360 .:%<...........`
0x0040: 50e3 23f7 2ab9 57b2 47f5 e0e8 b550 014b P.#.*.W.G....P.K
0x0050: 35c6 fe95 3475 cc66 9f81 fc0d 1646 ab00 5...4u.f.....F..
0x0060: 0052 c02c c087 cca9 c0ad c00a c02b c086 .R.,.........+..
0x0070: c0ac c009 c008 c030 c08b cca8 c014 c02f .......0......./
All in all: I can switch back and forth using TLS or unencrypted TCP
via the different ports with just changing the config parameter
'live_migration_scheme'. All tests were done on an OpenStack Train
deployment.
-----------------------------------
Release: 22.1.0.dev604 on 2019-03-11 17:38:39
SHA: 6e6d78d204dbf23d1ffaf232456fb2917b4f8372
Source: https://opendev.org/openstack/nova/src/doc/source/admin/secure-live-migration-with-qemu-native-tls.rst
URL: https://docs.openstack.org/nova/latest/admin/secure-live-migration-with-qemu-native-tls.html
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1919357/+subscriptions
Follow ups
