yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #95537
[Bug 2103513] [NEW] Spice console TLS doesnt work as expected
Public bug reported:
Description
===========
I attempted to configure TLS between nova-spicehtml5proxy and Openstack nova KVM hypervisor.
after enabling require_secure option and giving the paths to SSL related information on nova.conf,
nova-spice5htmlproxy still connects to non-ssl port.
Steps to reproduce
==================
on nova [spice] section set
require_secure = True
on compute set according your environment the following variables:
server_proxyclient_address =
html5proxy_base_url =
html5proxy_host =
html5proxy_port =
server_listen =
on nova-server set with your environment specific values following:
server_listen =
server_proxyclient_address =
html5proxy_host =
html5proxy_port =
on [DEFAULT] section set values for
key = <ssl_cert_key>
cert = <ssl_cert>
ssl_only = true
both on compute hypervisor and nova-api server.
configure /etc/libvirt/qemu.conf and set values:
spice_tls = 1
spice_tls_x509_cert_dir = "/etc/pki/libvirt-spice"
restart nova-api, nova-spice5html proxy on api server
restart libvirtd and nova-compute on the hypervisor
create a server on the specific hypervisor:
openstack server create --image cirros --flavor m1.mini --network myipv4 --host node14.openstack.host test-spice-tls --os-compute-api-version 2.74
Login to your horizon instance, and go to instances -> console.
We get just a gray spice window instead of console.
Expected result
===============
Spice console to the VM
Actual result
=============
A grey window without spice console
Environment
===========
Server environment:
ii nova-api 2:30.0.0-5~bpo12+1 all OpenStack Compute - compute API frontend
ii nova-common 2:30.0.0-5~bpo12+1 all OpenStack Compute - common files
ii nova-conductor 2:30.0.0-5~bpo12+1 all OpenStack Compute - conductor service
ii nova-consoleproxy 2:30.0.0-5~bpo12+1 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 2:30.0.0-5~bpo12+1 all OpenStack Compute - virtual machine scheduler
ii python3-nova 2:30.0.0-5~bpo12+1 all OpenStack Compute - libraries
ii python3-novaclient 2:18.7.0-3~bpo12+1 all client library for OpenStack Compute API - 3.x
hypervisor:
ii nova-common 2:30.0.0-5~bpo12+1 all OpenStack Compute - common files
ii nova-compute 2:30.0.0-5~bpo12+1 all OpenStack Compute - compute node
ii nova-compute-kvm 2:30.0.0-5~bpo12+1 all OpenStack Compute - compute node (KVM)
ii python3-nova 2:30.0.0-5~bpo12+1 all OpenStack Compute - libraries
ii python3-novaclient 2:18.7.0-3~bpo12+1 all client library for OpenStack Compute API - 3.x
2. Which hypervisor did you use?
Libvirt + KVM
dpkg -l|grep qemu
ii ipxe-qemu 1.0.0+git-20190125.36a4c85-5.1 all PXE boot firmware - ROM images for qemu
ii libvirt-daemon-driver-qemu 9.0.0-4+deb12u2 amd64 Virtualization daemon QEMU connection driver
ii qemu-block-extra 1:7.2+dfsg-7+deb12u7 amd64 extra block backend modules for qemu-system and qemu-utils
ii qemu-system-common 1:7.2+dfsg-7+deb12u7 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-data 1:7.2+dfsg-7+deb12u7 all QEMU full system emulation (data files)
ii qemu-system-x86 1:7.2+dfsg-7+deb12u7 amd64 QEMU full system emulation binaries (x86)
ii qemu-utils 1:7.2+dfsg-7+deb12u7 amd64 QEMU utilities
dpkg -l|grep libvirt
ii libvirt-clients 9.0.0-4+deb12u2 amd64 Programs for the libvirt library
ii libvirt-daemon 9.0.0-4+deb12u2 amd64 Virtualization daemon
ii libvirt-daemon-config-network 9.0.0-4+deb12u2 all Libvirt daemon configuration files (default network)
ii libvirt-daemon-config-nwfilter 9.0.0-4+deb12u2 all Libvirt daemon configuration files (default network filters)
ii libvirt-daemon-driver-qemu 9.0.0-4+deb12u2 amd64 Virtualization daemon QEMU connection driver
ii libvirt-daemon-driver-storage-rbd 9.0.0-4+deb12u2 amd64 Virtualization daemon RBD storage driver
ii libvirt-daemon-system 9.0.0-4+deb12u2 amd64 Libvirt daemon configuration files
ii libvirt-daemon-system-systemd 9.0.0-4+deb12u2 all Libvirt daemon configuration files (systemd)
ii libvirt0:amd64 9.0.0-4+deb12u2 amd64 library for interfacing with different virtualization systems
ii python3-libvirt 10.0.0-1~bpo12+1 amd64 libvirt Python 3 bindings
2. Which storage type did you use?
Ceph
version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
3. Which networking type did you use?
Neutron with OVN
Logs & Configs
==============
nova-api/spice5html-proxy
[DEFAULT] ...
key = /srv/cert/privkey.pem
cert = /srv/cert/cert.pem
ssl_only = true
...
[spice]
enabled = true
require_secure = true
agent_enabled = False
source_is_ipv6 = True
server_listen = <public ip>
server_proxyclient_address = <public ip>
html5proxy_host = <public ip>
html5proxy_port = 6082
...
Open ports when running spice5html
ss -tulpn
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 127.0.0.54:53 0.0.0.0:* users:(("systemd-resolve",pid=84,fd=16))
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=84,fd=14))
tcp LISTEN 0 100 0.0.0.0:25 0.0.0.0:* users:(("master",pid=465,fd=13))
tcp LISTEN 0 4096 127.0.0.54:53 0.0.0.0:* users:(("systemd-resolve",pid=84,fd=17))
tcp LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=84,fd=15))
tcp LISTEN 0 511 *:8775 *:* users:(("apache2",pid=66907,fd=6),("apache2",pid=66906,fd=6),("apache2",pid=66890,fd=6))
tcp LISTEN 0 511 *:8774 *:* users:(("apache2",pid=66907,fd=5),("apache2",pid=66906,fd=5),("apache2",pid=66890,fd=5))
tcp LISTEN 0 511 *:80 *:* users:(("apache2",pid=66907,fd=3),("apache2",pid=66906,fd=3),("apache2",pid=66890,fd=3))
tcp LISTEN 0 4096 *:22 *:* users:(("sshd",pid=97,fd=3),("systemd",pid=1,fd=78))
tcp LISTEN 0 100 [::]:25 [::]:* users:(("master",pid=465,fd=14))
tcp LISTEN 0 511 *:443 *:* users:(("apache2",pid=66907,fd=4),("apache2",pid=66906,fd=4),("apache2",pid=66890,fd=4))
tcp LISTEN 0 100 [public_ipv6]:6082 [::]:* users:(("nova-spicehtml5",pid=66877,fd=10))
Compute node configuration
[DEFAULT]
debug = true
my_ip = <my_ipv6_internal>
pybasedir = /usr/lib/python3/dist-packages
state_path = /var/lib/nova
transport_url = rabbit://openstack:rabbitpass@rabbitmq.internal.cloud:5671//
key = /srv/hostname/certs/privkey.pem
cert = /srv/hostname/certs/cert.pem
ssl_only = True
...
[spice]
source_is_ipv6 = true
enabled = true
agent_enabled = False
require_secure = True
server_proxyclient_address = $my_ip
html5proxy_base_url = https:/<proxy public ip>:6082/spice_auto.html
html5proxy_host = "<proxy public ip>"
html5proxy_port = 6082
server_listen = $my_ip
...
/etc/libvirt/qemu.conf
spice_tls = 1
spice_tls_x509_cert_dir = "/etc/pki/libvirt-spice"
virtual machine XML console part after creation:
<graphics type='spice' port='5900' tlsPort='5901' autoport='yes' listen='$my_ip'>
<listen type='address' address='$my_ip'/>
<channel name='main' mode='secure'/>
<channel name='display' mode='secure'/>
<channel name='inputs' mode='secure'/>
<channel name='cursor' mode='secure'/>
<channel name='playback' mode='secure'/>
<channel name='record' mode='secure'/>
<channel name='smartcard' mode='secure'/>
<channel name='usbredir' mode='secure'/>
</graphics>
tcpdump on server when trying to access spice console on horizon:
tcpdump -i eno1 dst node14.internal.openstack.cloud
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eno1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:25:13.095419 IP6 nova.internal.openstack.cloud.50286 > node14.internal.openstack.cloud.5900: Flags [S], seq 3792735556, win 65535, options [mss 1440,sackOK,TS val 3475825695 ecr 0,nop,wscale 8], length 0
12:25:13.095700 IP6 nova.internal.openstack.cloud.50286 > node14.internal.openstack.cloud.5900: Flags [.], ack 1127036222, win 19530, options [nop,nop,TS val 3475825696 ecr 835323250], length 0
12:25:13.097215 IP6 nova.internal.openstack.cloud.50286 > node14.internal.openstack.cloud.5900: Flags [P.], seq 0:42, ack 1, win 19530, options [nop,nop,TS val 3475825697 ecr 835323250], length 42
12:25:13.097582 IP6 nova.internal.openstack.cloud.50286 > node14.internal.openstack.cloud.5900: Flags [.], ack 195, win 19530, options [nop,nop,TS val 3475825698 ecr 835323252], length 0
12:25:13.098171 IP6 nova.internal.openstack.cloud.50286 > node14.internal.openstack.cloud.5900: Flags [F.], seq 42, ack 196, win 19530, options [nop,nop,TS val 3475825698 ecr 835323252], length 0
12:25:54.221794 IP6 nova.internal.openstack.cloud.39964 > node14.internal.openstack.cloud.5900: Flags [S], seq 2638634823, win 65535, options [mss 1440,sackOK,TS val 3475866822 ecr 0,nop,wscale 8], length 0
12:25:54.222118 IP6 nova.internal.openstack.cloud.39964 > node14.internal.openstack.cloud.5900: Flags [.], ack 2255119891, win 19530, options [nop,nop,TS val 3475866822 ecr 835364376], length 0
12:25:54.223687 IP6 nova.internal.openstack.cloud.39964 > node14.internal.openstack.cloud.5900: Flags [P.], seq 0:42, ack 1, win 19530, options [nop,nop,TS val 3475866824 ecr 835364376], length 42
12:25:54.223997 IP6 nova.internal.openstack.cloud.39964 > node14.internal.openstack.cloud.5900: Flags [.], ack 195, win 19530, options [nop,nop,TS val 3475866824 ecr 835364378], length 0
12:25:54.224505 IP6 nova.internal.openstack.cloud.39964 > node14.internal.openstack.cloud.5900: Flags [F.], seq 42, ack 196, win 19530, options [nop,nop,TS val 3475866824 ecr 835364378], length 0
on nova hypervisor:
tcpdump -i cloud-int src nova.internal.openstack.cloud
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on cloud-int, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:25:54.221632 IP6 nova.internal.openstack..cloud.39964 > node14.internal.openstack.cloud.5900: Flags [S], seq 2638634823, win 65535, options [mss 1440,sackOK,TS val 3475866822 ecr 0,nop,wscale 8], length 0
12:25:54.221888 IP6 nova.internal.openstack..cloud.39964 > node14.internal.openstack.cloud.5900: Flags [.], ack 2255119891, win 19530, options [nop,nop,TS val 3475866822 ecr 835364376], length 0
12:25:54.223502 IP6 nova.internal.openstack..cloud.39964 > node14.internal.openstack.cloud.5900: Flags [P.], seq 0:42, ack 1, win 19530, options [nop,nop,TS val 3475866824 ecr 835364376], length 42
12:25:54.223760 IP6 nova.internal.openstack..cloud.39964 > node14.internal.openstack.cloud.5900: Flags [.], ack 195, win 19530, options [nop,nop,TS val 3475866824 ecr 835364378], length 0
12:25:54.224261 IP6 nova.internal.openstack..cloud.39964 > node14.internal.openstack.cloud.5900: Flags [F.], seq 42, ack 196, win 19530, options [nop,nop,TS val 3475866824 ecr 835364378], length 0
Traffic should go to 5901, not 5900!
We run nova-api with apache2 wsgi, config here:
Listen 8774
Listen 8775
<VirtualHost *:8774>
SSLEngine on
SSLHonorCipherOrder on
SSLCertificateFile /srv/nova.openstack.cloud/dehydrated/certs/nova.openstack.cloud/fullchain.pem
SSLCertificateKeyFile /srv/nova.openstack.cloud/dehydrated/certs/nova.openstack.cloud/privkey.pem
WSGIScriptAlias / /usr/bin/nova-api-wsgi
WSGIDaemonProcess nova-api processes=5 threads=1 user=nova group=nova display-name=%{GROUP}
WSGIProcessGroup nova-api
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On
LimitRequestBody 114688
<IfVersion >= 2.4>
ErrorLogFormat "%{cu}t %M"
</IfVersion>
ErrorLog /var/log/apache2/nova-api-error.log
CustomLog /var/log/apache2/nova-api-access.log combined
<Directory /usr/bin>
Require all granted
</Directory>
</VirtualHost>
Alias /compute /usr/bin/nova-api-wsgi
<Location /compute>
SetHandler wsgi-script
Options +ExecCGI
WSGIProcessGroup nova-api
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On
</Location>
<VirtualHost *:8775>
SSLEngine on
SSLHonorCipherOrder on
SSLCertificateFile /srv/nova.openstack.cloud/dehydrated/certs/nova.openstack.cloud/fullchain.pem
SSLCertificateKeyFile /srv/nova.openstack.cloud/dehydrated/certs/nova.openstack.cloud/privkey.pem
WSGIScriptAlias / /usr/bin/nova-metadata-wsgi
WSGIDaemonProcess nova-api-metadata processes=5 threads=1 user=nova group=nova display-name=%{GROUP}
WSGIProcessGroup nova-api-metadata
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On
LimitRequestBody 114688
<IfVersion >= 2.4>
ErrorLogFormat "%{cu}t %M"
</IfVersion>
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2103513
Title:
Spice console TLS doesnt work as expected
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
I attempted to configure TLS between nova-spicehtml5proxy and Openstack nova KVM hypervisor.
after enabling require_secure option and giving the paths to SSL related information on nova.conf,
nova-spice5htmlproxy still connects to non-ssl port.
Steps to reproduce
==================
on nova [spice] section set
require_secure = True
on compute set according your environment the following variables:
server_proxyclient_address =
html5proxy_base_url =
html5proxy_host =
html5proxy_port =
server_listen =
on nova-server set with your environment specific values following:
server_listen =
server_proxyclient_address =
html5proxy_host =
html5proxy_port =
on [DEFAULT] section set values for
key = <ssl_cert_key>
cert = <ssl_cert>
ssl_only = true
both on compute hypervisor and nova-api server.
configure /etc/libvirt/qemu.conf and set values:
spice_tls = 1
spice_tls_x509_cert_dir = "/etc/pki/libvirt-spice"
restart nova-api, nova-spice5html proxy on api server
restart libvirtd and nova-compute on the hypervisor
create a server on the specific hypervisor:
openstack server create --image cirros --flavor m1.mini --network myipv4 --host node14.openstack.host test-spice-tls --os-compute-api-version 2.74
Login to your horizon instance, and go to instances -> console.
We get just a gray spice window instead of console.
Expected result
===============
Spice console to the VM
Actual result
=============
A grey window without spice console
Environment
===========
Server environment:
ii nova-api 2:30.0.0-5~bpo12+1 all OpenStack Compute - compute API frontend
ii nova-common 2:30.0.0-5~bpo12+1 all OpenStack Compute - common files
ii nova-conductor 2:30.0.0-5~bpo12+1 all OpenStack Compute - conductor service
ii nova-consoleproxy 2:30.0.0-5~bpo12+1 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 2:30.0.0-5~bpo12+1 all OpenStack Compute - virtual machine scheduler
ii python3-nova 2:30.0.0-5~bpo12+1 all OpenStack Compute - libraries
ii python3-novaclient 2:18.7.0-3~bpo12+1 all client library for OpenStack Compute API - 3.x
hypervisor:
ii nova-common 2:30.0.0-5~bpo12+1 all OpenStack Compute - common files
ii nova-compute 2:30.0.0-5~bpo12+1 all OpenStack Compute - compute node
ii nova-compute-kvm 2:30.0.0-5~bpo12+1 all OpenStack Compute - compute node (KVM)
ii python3-nova 2:30.0.0-5~bpo12+1 all OpenStack Compute - libraries
ii python3-novaclient 2:18.7.0-3~bpo12+1 all client library for OpenStack Compute API - 3.x
2. Which hypervisor did you use?
Libvirt + KVM
dpkg -l|grep qemu
ii ipxe-qemu 1.0.0+git-20190125.36a4c85-5.1 all PXE boot firmware - ROM images for qemu
ii libvirt-daemon-driver-qemu 9.0.0-4+deb12u2 amd64 Virtualization daemon QEMU connection driver
ii qemu-block-extra 1:7.2+dfsg-7+deb12u7 amd64 extra block backend modules for qemu-system and qemu-utils
ii qemu-system-common 1:7.2+dfsg-7+deb12u7 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-data 1:7.2+dfsg-7+deb12u7 all QEMU full system emulation (data files)
ii qemu-system-x86 1:7.2+dfsg-7+deb12u7 amd64 QEMU full system emulation binaries (x86)
ii qemu-utils 1:7.2+dfsg-7+deb12u7 amd64 QEMU utilities
dpkg -l|grep libvirt
ii libvirt-clients 9.0.0-4+deb12u2 amd64 Programs for the libvirt library
ii libvirt-daemon 9.0.0-4+deb12u2 amd64 Virtualization daemon
ii libvirt-daemon-config-network 9.0.0-4+deb12u2 all Libvirt daemon configuration files (default network)
ii libvirt-daemon-config-nwfilter 9.0.0-4+deb12u2 all Libvirt daemon configuration files (default network filters)
ii libvirt-daemon-driver-qemu 9.0.0-4+deb12u2 amd64 Virtualization daemon QEMU connection driver
ii libvirt-daemon-driver-storage-rbd 9.0.0-4+deb12u2 amd64 Virtualization daemon RBD storage driver
ii libvirt-daemon-system 9.0.0-4+deb12u2 amd64 Libvirt daemon configuration files
ii libvirt-daemon-system-systemd 9.0.0-4+deb12u2 all Libvirt daemon configuration files (systemd)
ii libvirt0:amd64 9.0.0-4+deb12u2 amd64 library for interfacing with different virtualization systems
ii python3-libvirt 10.0.0-1~bpo12+1 amd64 libvirt Python 3 bindings
2. Which storage type did you use?
Ceph
version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
3. Which networking type did you use?
Neutron with OVN
Logs & Configs
==============
nova-api/spice5html-proxy
[DEFAULT] ...
key = /srv/cert/privkey.pem
cert = /srv/cert/cert.pem
ssl_only = true
...
[spice]
enabled = true
require_secure = true
agent_enabled = False
source_is_ipv6 = True
server_listen = <public ip>
server_proxyclient_address = <public ip>
html5proxy_host = <public ip>
html5proxy_port = 6082
...
Open ports when running spice5html
ss -tulpn
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 127.0.0.54:53 0.0.0.0:* users:(("systemd-resolve",pid=84,fd=16))
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=84,fd=14))
tcp LISTEN 0 100 0.0.0.0:25 0.0.0.0:* users:(("master",pid=465,fd=13))
tcp LISTEN 0 4096 127.0.0.54:53 0.0.0.0:* users:(("systemd-resolve",pid=84,fd=17))
tcp LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=84,fd=15))
tcp LISTEN 0 511 *:8775 *:* users:(("apache2",pid=66907,fd=6),("apache2",pid=66906,fd=6),("apache2",pid=66890,fd=6))
tcp LISTEN 0 511 *:8774 *:* users:(("apache2",pid=66907,fd=5),("apache2",pid=66906,fd=5),("apache2",pid=66890,fd=5))
tcp LISTEN 0 511 *:80 *:* users:(("apache2",pid=66907,fd=3),("apache2",pid=66906,fd=3),("apache2",pid=66890,fd=3))
tcp LISTEN 0 4096 *:22 *:* users:(("sshd",pid=97,fd=3),("systemd",pid=1,fd=78))
tcp LISTEN 0 100 [::]:25 [::]:* users:(("master",pid=465,fd=14))
tcp LISTEN 0 511 *:443 *:* users:(("apache2",pid=66907,fd=4),("apache2",pid=66906,fd=4),("apache2",pid=66890,fd=4))
tcp LISTEN 0 100 [public_ipv6]:6082 [::]:* users:(("nova-spicehtml5",pid=66877,fd=10))
Compute node configuration
[DEFAULT]
debug = true
my_ip = <my_ipv6_internal>
pybasedir = /usr/lib/python3/dist-packages
state_path = /var/lib/nova
transport_url = rabbit://openstack:rabbitpass@rabbitmq.internal.cloud:5671//
key = /srv/hostname/certs/privkey.pem
cert = /srv/hostname/certs/cert.pem
ssl_only = True
...
[spice]
source_is_ipv6 = true
enabled = true
agent_enabled = False
require_secure = True
server_proxyclient_address = $my_ip
html5proxy_base_url = https:/<proxy public ip>:6082/spice_auto.html
html5proxy_host = "<proxy public ip>"
html5proxy_port = 6082
server_listen = $my_ip
...
/etc/libvirt/qemu.conf
spice_tls = 1
spice_tls_x509_cert_dir = "/etc/pki/libvirt-spice"
virtual machine XML console part after creation:
<graphics type='spice' port='5900' tlsPort='5901' autoport='yes' listen='$my_ip'>
<listen type='address' address='$my_ip'/>
<channel name='main' mode='secure'/>
<channel name='display' mode='secure'/>
<channel name='inputs' mode='secure'/>
<channel name='cursor' mode='secure'/>
<channel name='playback' mode='secure'/>
<channel name='record' mode='secure'/>
<channel name='smartcard' mode='secure'/>
<channel name='usbredir' mode='secure'/>
</graphics>
tcpdump on server when trying to access spice console on horizon:
tcpdump -i eno1 dst node14.internal.openstack.cloud
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eno1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:25:13.095419 IP6 nova.internal.openstack.cloud.50286 > node14.internal.openstack.cloud.5900: Flags [S], seq 3792735556, win 65535, options [mss 1440,sackOK,TS val 3475825695 ecr 0,nop,wscale 8], length 0
12:25:13.095700 IP6 nova.internal.openstack.cloud.50286 > node14.internal.openstack.cloud.5900: Flags [.], ack 1127036222, win 19530, options [nop,nop,TS val 3475825696 ecr 835323250], length 0
12:25:13.097215 IP6 nova.internal.openstack.cloud.50286 > node14.internal.openstack.cloud.5900: Flags [P.], seq 0:42, ack 1, win 19530, options [nop,nop,TS val 3475825697 ecr 835323250], length 42
12:25:13.097582 IP6 nova.internal.openstack.cloud.50286 > node14.internal.openstack.cloud.5900: Flags [.], ack 195, win 19530, options [nop,nop,TS val 3475825698 ecr 835323252], length 0
12:25:13.098171 IP6 nova.internal.openstack.cloud.50286 > node14.internal.openstack.cloud.5900: Flags [F.], seq 42, ack 196, win 19530, options [nop,nop,TS val 3475825698 ecr 835323252], length 0
12:25:54.221794 IP6 nova.internal.openstack.cloud.39964 > node14.internal.openstack.cloud.5900: Flags [S], seq 2638634823, win 65535, options [mss 1440,sackOK,TS val 3475866822 ecr 0,nop,wscale 8], length 0
12:25:54.222118 IP6 nova.internal.openstack.cloud.39964 > node14.internal.openstack.cloud.5900: Flags [.], ack 2255119891, win 19530, options [nop,nop,TS val 3475866822 ecr 835364376], length 0
12:25:54.223687 IP6 nova.internal.openstack.cloud.39964 > node14.internal.openstack.cloud.5900: Flags [P.], seq 0:42, ack 1, win 19530, options [nop,nop,TS val 3475866824 ecr 835364376], length 42
12:25:54.223997 IP6 nova.internal.openstack.cloud.39964 > node14.internal.openstack.cloud.5900: Flags [.], ack 195, win 19530, options [nop,nop,TS val 3475866824 ecr 835364378], length 0
12:25:54.224505 IP6 nova.internal.openstack.cloud.39964 > node14.internal.openstack.cloud.5900: Flags [F.], seq 42, ack 196, win 19530, options [nop,nop,TS val 3475866824 ecr 835364378], length 0
on nova hypervisor:
tcpdump -i cloud-int src nova.internal.openstack.cloud
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on cloud-int, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:25:54.221632 IP6 nova.internal.openstack..cloud.39964 > node14.internal.openstack.cloud.5900: Flags [S], seq 2638634823, win 65535, options [mss 1440,sackOK,TS val 3475866822 ecr 0,nop,wscale 8], length 0
12:25:54.221888 IP6 nova.internal.openstack..cloud.39964 > node14.internal.openstack.cloud.5900: Flags [.], ack 2255119891, win 19530, options [nop,nop,TS val 3475866822 ecr 835364376], length 0
12:25:54.223502 IP6 nova.internal.openstack..cloud.39964 > node14.internal.openstack.cloud.5900: Flags [P.], seq 0:42, ack 1, win 19530, options [nop,nop,TS val 3475866824 ecr 835364376], length 42
12:25:54.223760 IP6 nova.internal.openstack..cloud.39964 > node14.internal.openstack.cloud.5900: Flags [.], ack 195, win 19530, options [nop,nop,TS val 3475866824 ecr 835364378], length 0
12:25:54.224261 IP6 nova.internal.openstack..cloud.39964 > node14.internal.openstack.cloud.5900: Flags [F.], seq 42, ack 196, win 19530, options [nop,nop,TS val 3475866824 ecr 835364378], length 0
Traffic should go to 5901, not 5900!
We run nova-api with apache2 wsgi, config here:
Listen 8774
Listen 8775
<VirtualHost *:8774>
SSLEngine on
SSLHonorCipherOrder on
SSLCertificateFile /srv/nova.openstack.cloud/dehydrated/certs/nova.openstack.cloud/fullchain.pem
SSLCertificateKeyFile /srv/nova.openstack.cloud/dehydrated/certs/nova.openstack.cloud/privkey.pem
WSGIScriptAlias / /usr/bin/nova-api-wsgi
WSGIDaemonProcess nova-api processes=5 threads=1 user=nova group=nova display-name=%{GROUP}
WSGIProcessGroup nova-api
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On
LimitRequestBody 114688
<IfVersion >= 2.4>
ErrorLogFormat "%{cu}t %M"
</IfVersion>
ErrorLog /var/log/apache2/nova-api-error.log
CustomLog /var/log/apache2/nova-api-access.log combined
<Directory /usr/bin>
Require all granted
</Directory>
</VirtualHost>
Alias /compute /usr/bin/nova-api-wsgi
<Location /compute>
SetHandler wsgi-script
Options +ExecCGI
WSGIProcessGroup nova-api
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On
</Location>
<VirtualHost *:8775>
SSLEngine on
SSLHonorCipherOrder on
SSLCertificateFile /srv/nova.openstack.cloud/dehydrated/certs/nova.openstack.cloud/fullchain.pem
SSLCertificateKeyFile /srv/nova.openstack.cloud/dehydrated/certs/nova.openstack.cloud/privkey.pem
WSGIScriptAlias / /usr/bin/nova-metadata-wsgi
WSGIDaemonProcess nova-api-metadata processes=5 threads=1 user=nova group=nova display-name=%{GROUP}
WSGIProcessGroup nova-api-metadata
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On
LimitRequestBody 114688
<IfVersion >= 2.4>
ErrorLogFormat "%{cu}t %M"
</IfVersion>
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2103513/+subscriptions
