yahoo-eng-team team mailing list archive

[Carlo Baijens <1711547@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Description
===========
Virtuozzo storage mounts invalid state after Openstack Nova Compute service restart.

==================
Running Openstack Ocata release with Virtuozzo KVM hypervisor on Vstorage backed by Cinder. A restart of the openstack-nova-compute service causes vzstorage mounts to be unreachable. The restart keeps VMs running using the mount but the filesystem is unreachable.

Before restart:
[user@node ~]$ sudo systemctl status  -l openstack-nova-compute.service
● openstack-nova-compute.service - OpenStack Nova Compute Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2017-08-14 13:47:00 CEST; 1 day 23h ago
 Main PID: 3932 (nova-compute)
   CGroup: /system.slice/openstack-nova-compute.service
           ├─3932 /usr/bin/python2 /usr/bin/nova-compute
           ├─4251 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_linux_bridge.privsep.vif_plug --privsep_sock_path /tmp/tmpe2LS_d/privsep.sock
           ├─4454 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmpOXZxyO/privsep.sock
           └─4498 pstorage-mount -c openstackpoc -u nova -g root -m 0770 -l /var/log/vstorage/nova-openstackpoc.log.gz /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee

We restart the service :
sudo systemctl restart openstack-nova-compute.service

After this we see:

[user@node ~]$ sudo systemctl status  -l openstack-nova-compute.service
● openstack-nova-compute.service - OpenStack Nova Compute Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2017-08-16 13:15:26 CEST; 50s ago
 Main PID: 199753 (nova-compute)
   CGroup: /system.slice/openstack-nova-compute.service
           ├─199753 /usr/bin/python2 /usr/bin/nova-compute
           └─199798 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_linux_bridge.privsep.vif_plug --privsep_sock_path /tmp/tmpjDaH7y/privsep.sock

The vstorage-mount is not reinitialised by nova-compute.

Expected result
===============
The vstorage-mount is available after nova-compute restart.

Actual result
=============
The vstorage-mount is not reinitialised by nova-compute.

Nova mounts do not disappear after restart but are unreachable:

[user@node ~]$ sudo lsof | grep /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee
lsof: WARNING: can't stat() fuse.vstorage file system /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee
      Output information may be incomplete.
qemu-kvm    7366                   root   18u  unknown                                          /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee/volume-7d555095-f36e-4673-ad10-30786d105270 (stat: Transport endpoint is not connected)
qemu-kvm    7366   7370            root   18u  unknown                                          /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee/volume-7d555095-f36e-4673-ad10-30786d105270 (stat: Transport endpoint is not connected)

We currently implemented a workaround by adding

KillMode=process

to the openstack nova compute unit file override. This leave open the
vzstorage mounts and keeps the VMs running.

Environment
===========
1. Openstack Ocata release 15.0.3-2.el7

2. Which hypervisor did you use?

qemu-kvm-vz 2.6.0-28.3.9.vz7.56.1

2. Which storage type did you use?

Vstorage, 7.4.106-1

3. Which networking type did you use?
Neutron with Linuxbridge

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  Description
  ===========
- Virtuozzo storage mounts invalid state after Openstack Nova Compute service restart. 
+ Virtuozzo storage mounts invalid state after Openstack Nova Compute service restart.
  
  ==================
  Running Openstack Ocata release with Virtuozzo KVM hypervisor on Vstorage backed by Cinder. A restart of the openstack-nova-compute service causes vzstorage mounts to be unreachable. The restart keeps VMs running using the mount but the filesystem is unreachable.
  
  Before restart:
  [user@node ~]$ sudo systemctl status  -l openstack-nova-compute.service
  ● openstack-nova-compute.service - OpenStack Nova Compute Server
-    Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
-    Active: active (running) since Mon 2017-08-14 13:47:00 CEST; 1 day 23h ago
-  Main PID: 3932 (nova-compute)
-    CGroup: /system.slice/openstack-nova-compute.service
-            ├─3932 /usr/bin/python2 /usr/bin/nova-compute
-            ├─4251 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_linux_bridge.privsep.vif_plug --privsep_sock_path /tmp/tmpe2LS_d/privsep.sock
-            ├─4454 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmpOXZxyO/privsep.sock
-            └─4498 pstorage-mount -c openstackpoc -u nova -g root -m 0770 -l /var/log/vstorage/nova-openstackpoc.log.gz /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee
+    Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
+    Active: active (running) since Mon 2017-08-14 13:47:00 CEST; 1 day 23h ago
+  Main PID: 3932 (nova-compute)
+    CGroup: /system.slice/openstack-nova-compute.service
+            ├─3932 /usr/bin/python2 /usr/bin/nova-compute
+            ├─4251 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_linux_bridge.privsep.vif_plug --privsep_sock_path /tmp/tmpe2LS_d/privsep.sock
+            ├─4454 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmpOXZxyO/privsep.sock
+            └─4498 pstorage-mount -c openstackpoc -u nova -g root -m 0770 -l /var/log/vstorage/nova-openstackpoc.log.gz /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee
  
  We restart the service :
  sudo systemctl restart openstack-nova-compute.service
  
  After this we see:
  
  [user@node ~]$ sudo systemctl status  -l openstack-nova-compute.service
  ● openstack-nova-compute.service - OpenStack Nova Compute Server
-    Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
-    Active: active (running) since Wed 2017-08-16 13:15:26 CEST; 50s ago
-  Main PID: 199753 (nova-compute)
-    CGroup: /system.slice/openstack-nova-compute.service
-            ├─199753 /usr/bin/python2 /usr/bin/nova-compute
-            └─199798 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_linux_bridge.privsep.vif_plug --privsep_sock_path /tmp/tmpjDaH7y/privsep.sock
+    Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
+    Active: active (running) since Wed 2017-08-16 13:15:26 CEST; 50s ago
+  Main PID: 199753 (nova-compute)
+    CGroup: /system.slice/openstack-nova-compute.service
+            ├─199753 /usr/bin/python2 /usr/bin/nova-compute
+            └─199798 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_linux_bridge.privsep.vif_plug --privsep_sock_path /tmp/tmpjDaH7y/privsep.sock
  
- The pstorage-mount is not reinitialised by nova-compute.
- 
+ The vstorage-mount is not reinitialised by nova-compute.
  
  Expected result
  ===============
- The pstorage-mount is available after nova-compute restart.
+ The vstorage-mount is available after nova-compute restart.
  
  Actual result
  =============
- The pstorage-mount is not reinitialised by nova-compute.
+ The vstorage-mount is not reinitialised by nova-compute.
  
  Nova mounts do not disappear after restart but are unreachable:
  
  [user@node ~]$ sudo lsof | grep /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee
  lsof: WARNING: can't stat() fuse.vstorage file system /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee
-       Output information may be incomplete.
+       Output information may be incomplete.
  qemu-kvm    7366                   root   18u  unknown                                          /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee/volume-7d555095-f36e-4673-ad10-30786d105270 (stat: Transport endpoint is not connected)
  qemu-kvm    7366   7370            root   18u  unknown                                          /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee/volume-7d555095-f36e-4673-ad10-30786d105270 (stat: Transport endpoint is not connected)
  
  We currently implemented a workaround by adding
  
  KillMode=process
  
  to the openstack nova compute unit file override. This leave open the
  vzstorage mounts and keeps the VMs running.
- 
  
  Environment
  ===========
  1. Openstack Ocata release 15.0.3-2.el7
  
  2. Which hypervisor did you use?
  
  qemu-kvm-vz 2.6.0-28.3.9.vz7.56.1
  
  2. Which storage type did you use?
  
  Vstorage, 7.4.106-1
  
  3. Which networking type did you use?
  Neutron with Linuxbridge

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1711547

Title:
  Nova service restart disconnects vzstorage volumes

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  Virtuozzo storage mounts invalid state after Openstack Nova Compute service restart.

  ==================
  Running Openstack Ocata release with Virtuozzo KVM hypervisor on Vstorage backed by Cinder. A restart of the openstack-nova-compute service causes vzstorage mounts to be unreachable. The restart keeps VMs running using the mount but the filesystem is unreachable.

  Before restart:
  [user@node ~]$ sudo systemctl status  -l openstack-nova-compute.service
  ● openstack-nova-compute.service - OpenStack Nova Compute Server
     Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
     Active: active (running) since Mon 2017-08-14 13:47:00 CEST; 1 day 23h ago
   Main PID: 3932 (nova-compute)
     CGroup: /system.slice/openstack-nova-compute.service
             ├─3932 /usr/bin/python2 /usr/bin/nova-compute
             ├─4251 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_linux_bridge.privsep.vif_plug --privsep_sock_path /tmp/tmpe2LS_d/privsep.sock
             ├─4454 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmpOXZxyO/privsep.sock
             └─4498 pstorage-mount -c openstackpoc -u nova -g root -m 0770 -l /var/log/vstorage/nova-openstackpoc.log.gz /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee

  We restart the service :
  sudo systemctl restart openstack-nova-compute.service

  After this we see:

  [user@node ~]$ sudo systemctl status  -l openstack-nova-compute.service
  ● openstack-nova-compute.service - OpenStack Nova Compute Server
     Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
     Active: active (running) since Wed 2017-08-16 13:15:26 CEST; 50s ago
   Main PID: 199753 (nova-compute)
     CGroup: /system.slice/openstack-nova-compute.service
             ├─199753 /usr/bin/python2 /usr/bin/nova-compute
             └─199798 /usr/bin/python2 /bin/privsep-helper --config-file /usr/share/nova/nova-dist.conf --config-file /etc/nova/nova.conf --privsep_context vif_plug_linux_bridge.privsep.vif_plug --privsep_sock_path /tmp/tmpjDaH7y/privsep.sock

  The vstorage-mount is not reinitialised by nova-compute.

  Expected result
  ===============
  The vstorage-mount is available after nova-compute restart.

  Actual result
  =============
  The vstorage-mount is not reinitialised by nova-compute.

  Nova mounts do not disappear after restart but are unreachable:

  [user@node ~]$ sudo lsof | grep /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee
  lsof: WARNING: can't stat() fuse.vstorage file system /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee
        Output information may be incomplete.
  qemu-kvm    7366                   root   18u  unknown                                          /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee/volume-7d555095-f36e-4673-ad10-30786d105270 (stat: Transport endpoint is not connected)
  qemu-kvm    7366   7370            root   18u  unknown                                          /var/lib/nova/mnt/b2b894afb0fb3e47348888c87ad01eee/volume-7d555095-f36e-4673-ad10-30786d105270 (stat: Transport endpoint is not connected)

  We currently implemented a workaround by adding

  KillMode=process

  to the openstack nova compute unit file override. This leave open the
  vzstorage mounts and keeps the VMs running.

  Environment
  ===========
  1. Openstack Ocata release 15.0.3-2.el7

  2. Which hypervisor did you use?

  qemu-kvm-vz 2.6.0-28.3.9.vz7.56.1

  2. Which storage type did you use?

  Vstorage, 7.4.106-1

  3. Which networking type did you use?
  Neutron with Linuxbridge

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1711547/+subscriptions