yahoo-eng-team team mailing list archive

[Lee Yarwood <1919357@xxxxxxxxxxxxxxxxxx>]

** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1919357

Title:
  "Secure live migration with QEMU-native TLS in nova"-guide misses
  essential config option

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) stein series:
  New
Status in OpenStack Compute (nova) train series:
  Fix Released
Status in OpenStack Compute (nova) ussuri series:
  Fix Released
Status in OpenStack Compute (nova) victoria series:
  Fix Released
Status in OpenStack Security Advisory:
  Won't Fix
Status in OpenStack Security Notes:
  In Progress

Bug description:
  - [x] This doc is inaccurate in this way: ______

  I followed the guide to setup qemu native tls for live migration.
  After checking, that libvirt is able to use tls using tcpdump to
  listen on the port for tls,  I also wanted to check that it works when
  I live migrate an instance. Apparently it didn't. But it used the port
  for unencrypted TCP [1].

  After digging through documentation and code afterwards I found that
  in this code part:
  https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L1120

  @staticmethod
      def _live_migration_uri(dest):
          uris = {
              'kvm': 'qemu+%(scheme)s://%(dest)s/system',
              'qemu': 'qemu+%(scheme)s://%(dest)s/system',
              'xen': 'xenmigr://%(dest)s/system',
              'parallels': 'parallels+tcp://%(dest)s/system',
          }
          dest = oslo_netutils.escape_ipv6(dest)

          virt_type = CONF.libvirt.virt_type
          # TODO(pkoniszewski): Remove fetching live_migration_uri in Pike
          uri = CONF.libvirt.live_migration_uri
          if uri:
              return uri % dest

          uri = uris.get(virt_type)
          if uri is None:
              raise exception.LiveMigrationURINotAvailable(virt_type=virt_type)

          str_format = {
              'dest': dest,
              'scheme': CONF.libvirt.live_migration_scheme or 'tcp',
          }
          return uri % str_format

  the uri is calculated using the config parameter
  'live_migration_scheme' or using the hard coded tcp parameter. Coming
  from the guide for qemu native tls, there was no hint that this config
  option needs to be set.

  In fact without setting this 'live_migration_scheme' config option to
  tls, there is no way to see, that the live migration still uses the
  unencrypted tcp connection - one has to use tcpdump and listen for tcp
  or tls to recognize it. Neither in the logs nor in any debug output
  there is any hint that it is still unencrypted!

  Thus I conclude there might be OpenStack deployments which are
  configured as the guide say but these config changes have no effect!

  - [x] This is a doc addition request.

  To fix this the config parameter 'live_migration_scheme' should be set
  to tls and maybe there should be a warning in the documentation, that
  without doing this, the traffic is still unencrypted.

  - [ ] I have a fix to the document that I can paste below including
  example: input and output.

  [1] without setting 'live_migration_scheme' in the nova.conf
  $ tcpdump -i INTERFACE -n -X port 16509 and '(tcp[((tcp[12] & 0xf0) >> 2)] < 0x14 || tcp[((tcp[12] & 0xf0) >> 2)] > 0x17)'
  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes
  17:10:56.387407 IP 192.168.70.101.50900 > 192.168.70.100.16509: Flags [P.], seq 304:6488, ack 285, win 502, options [nop,nop,TS val 424149655 ecr 1875309961], length 6184
   0x0000:  4500 185c ad05 4000 4006 677c c0a8 4665  E..\..@.@.g|..Fe
   0x0010:  c0a8 4664 c6d4 407d a407 70a6 15ad 0a5a  ..Fd..@}..p....Z
   0x0020:  8018 01f6 2669 0000 0101 080a 1948 0297  ....&i.......H..
   0x0030:  6fc6 f589 0000 1828 2000 8086 0000 0001  o......(........
   0x0040:  0000 012f 0000 0000 0000 0009 0000 0000  .../............
   0x0050:  0000 0001 0000 000f 6465 7374 696e 6174  ........destinat
   0x0060:  696f 6e5f 786d 6c00 0000 0007 0000 129b  ion_xml.........
   0x0070:  3c64 6f6d 6169 6e20 7479 7065 3d27 6b76  <domain.type='kv
   0x0080:  6d27 3e0a 2020 3c6e 616d 653e 696e 7374  m'>...<name>inst
   0x0090:  616e 6365 2d30 3030 3032 6539 393c 2f6e  ance-00002e99</n
   0x00a0:  616d 653e 0a20 203c 7575 6964 3e37 6532  ame>...<uuid>7e2
   0x00b0:  6364 3839 352d 6263 3765 2d34 6634 352d  cd895-bc7e-4f45-
   0x00c0:  6166 6264 2d33 3732 3166 3735 6134 3064  afbd-3721f75a40d
   0x00d0:  383c 2f75 7569 643e 0a20 203c 6d65 7461  8</uuid>...<meta

  [2] after setting 'live_migration_scheme = tls'
  $ tcpdump -i INTERFACE -n -X port 16514 and '(tcp[((tcp[12] & 0xf0) >> 2)] > 0x13 && tcp[((tcp[12] & 0xf0) >> 2)] < 0x18)'
  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes
  16:55:47.746851 IP 192.168.70.100.35620 > 192.168.70.101.16514: Flags [P.], seq 1849334708:1849334914, ack 3121294199, win 502, options [nop,nop,TS val 1874401351 ecr 423241020], length 206
   0x0000:  4500 0102 a605 4000 4006 85d6 c0a8 4664  E.....@.@.....Fd
   0x0010:  c0a8 4665 8b24 4082 6e3a 9bb4 ba0b 2b77  ..Fe.$@.n:....+w
   0x0020:  8018 01f6 2f43 0000 0101 080a 6fb9 1847  ..../C......o..G
   0x0030:  193a 253c 1603 0100 c901 0000 c503 0360  .:%<...........`
   0x0040:  50e3 23f7 2ab9 57b2 47f5 e0e8 b550 014b  P.#.*.W.G....P.K
   0x0050:  35c6 fe95 3475 cc66 9f81 fc0d 1646 ab00  5...4u.f.....F..
   0x0060:  0052 c02c c087 cca9 c0ad c00a c02b c086  .R.,.........+..
   0x0070:  c0ac c009 c008 c030 c08b cca8 c014 c02f  .......0......./

  All in all: I can switch back and forth using TLS or unencrypted TCP
  via the different ports with just changing the config parameter
  'live_migration_scheme'. All tests were done on an OpenStack Train
  deployment.

  -----------------------------------
  Release: 22.1.0.dev604 on 2019-03-11 17:38:39
  SHA: 6e6d78d204dbf23d1ffaf232456fb2917b4f8372
  Source: https://opendev.org/openstack/nova/src/doc/source/admin/secure-live-migration-with-qemu-native-tls.rst
  URL: https://docs.openstack.org/nova/latest/admin/secure-live-migration-with-qemu-native-tls.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1919357/+subscriptions