yahoo-eng-team team mailing list archive

Thread
Date

[Bug 1528756] [NEW] nova instance was read-only mode

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Kyoungduk Yoo <tog09@xxxxxxxxx>
Date: Wed, 23 Dec 2015 06:00:27 -0000
Reply-to: Bug 1528756 <1528756@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx

Public bug reported:

Our Openstack brief Information & Issue

1. Version : Juno

2. Management(Nova, CInder, LB etc) & Compute Node OS ver : Ubuntu 14.04

3. H/W Model
- Management Server : Dell R720
- Compute Node : HP C7000 (Blade 460C *6ea)
- Storage : Netapp FAS 8020
- Switch : Nexus 4568

4. Issues
- nova instance was read-only mode.
- then we did reboot all instances except for Bootable Volume Instances.
- I have thought that the NAS volume what have nova instance images temporarily disconnected.

there are two Instance type we have
1) First type instance(OS) is launched from nfs (glance images)
2) Second is installed by cinder volume (Bootable Volume OS)

5. cause analysis
- After Rebooting instances, all instances were normal status.
- But there were no logs about NFS N/W disconnection in Compute Node, Storage, Switch.
- And no Performance Issues (CPU, MEM, Disk IO, N/W traffic)

- VM kernel I/O Error log
kernel: end_request: I/O error, dev vda, sector 12523664
kernel: Buffer I/O error on device dm-1, logical block 388370
kernel: lost page write due to I/O error on dm-1
kernel: end_request: I/O error, dev vda, sector 36124456
kernel: Buffer I/O error on device dm-1, logical block 3338469
kernel: lost page write due to I/O error on dm-1

6. etc
- nfs client version
# dpkg -l |grep nfs
ii libnfsidmap2:amd64 0.25-5 amd64 NFS idmapping library
ii nfs-common 1:1.2.8-6ubuntu1.1 amd64 NFS support files common to client and server

What is causing NFS temporary disconnection?

** Affects: nova
Importance: Undecided
Status: New

** Description changed:

Our Openstack brief Information & Issue

1. Version : Juno

2. Management(Nova, CInder, LB etc) & Compute Node OS ver : Ubuntu 14.04

3. H/W Model
- - Management Server : Dell R720
- - Compute Node : HP C7000 (Blade 460C *6ea)
- - Storage : Netapp FAS 8020
- - Switch : Nexus 4568
+ - Management Server : Dell R720
+ - Compute Node : HP C7000 (Blade 460C *6ea)
+ - Storage : Netapp FAS 8020
+ - Switch : Nexus 4568

- 4. Issues
- - nova instance was read-only mode.
- - then we did reboot all instances except for Bootable Volume Instances.
- - I have thought that the NAS volume what have nova instance images temporarily disconnected.
+ 4. Issues
+ - nova instance was read-only mode.
+ - then we did reboot all instances except for Bootable Volume Instances.
+ - I have thought that the NAS volume what have nova instance images temporarily disconnected.

there are two Instance type we have
- 1) First type instance(OS) is launched nfs (glance images)
- 2) Second is installed by cinder volume (Bootable Volume OS)
+ 1) First type instance(OS) is launched nfs (glance images)
+ 2) Second is installed by cinder volume (Bootable Volume OS)

5. cause analysis
- - After Rebooting instances, all instances were normal status.
- - But there were no logs about NFS N/W disconnection in Compute Node, Storage, Switch.
- - And no Performance Issues (CPU, MEM, Disk IO, N/W traffic)
+ - After Rebooting instances, all instances were normal status.
+ - But there were no logs about NFS N/W disconnection in Compute Node, Storage, Switch.
+ - And no Performance Issues (CPU, MEM, Disk IO, N/W traffic)

- - VM kernel I/O Error log
+ - VM kernel I/O Error log
kernel: end_request: I/O error, dev vda, sector 12523664
kernel: Buffer I/O error on device dm-1, logical block 388370
kernel: lost page write due to I/O error on dm-1
kernel: end_request: I/O error, dev vda, sector 36124456
kernel: Buffer I/O error on device dm-1, logical block 3338469
kernel: lost page write due to I/O error on dm-1

6. etc
- - nfs client version
+ - nfs client version
# dpkg -l |grep nfs
ii libnfsidmap2:amd64 0.25-5 amd64 NFS idmapping library
ii nfs-common 1:1.2.8-6ubuntu1.1 amd64 NFS support files common to client and server

-
-
- What is causing NFS temporarily disconnection?
+ What is causing NFS temporary disconnection?

** Description changed:

Our Openstack brief Information & Issue

1. Version : Juno

2. Management(Nova, CInder, LB etc) & Compute Node OS ver : Ubuntu 14.04

3. H/W Model
- Management Server : Dell R720
- Compute Node : HP C7000 (Blade 460C *6ea)
- Storage : Netapp FAS 8020
- Switch : Nexus 4568

there are two Instance type we have
- 1) First type instance(OS) is launched nfs (glance images)
+ 1) First type instance(OS) is launched from nfs (glance images)
2) Second is installed by cinder volume (Bootable Volume OS)

6. etc
- nfs client version
# dpkg -l |grep nfs
ii libnfsidmap2:amd64 0.25-5 amd64 NFS idmapping library
ii nfs-common 1:1.2.8-6ubuntu1.1 amd64 NFS support files common to client and server

What is causing NFS temporary disconnection?

--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1528756

Title:
nova instance was read-only mode

Status in OpenStack Compute (nova):
New

Bug description:
Our Openstack brief Information & Issue

1. Version : Juno

2. Management(Nova, CInder, LB etc) & Compute Node OS ver : Ubuntu
14.04

3. H/W Model
- Management Server : Dell R720
- Compute Node : HP C7000 (Blade 460C *6ea)
- Storage : Netapp FAS 8020
- Switch : Nexus 4568

there are two Instance type we have
1) First type instance(OS) is launched from nfs (glance images)
2) Second is installed by cinder volume (Bootable Volume OS)

6. etc
- nfs client version
# dpkg -l |grep nfs
ii libnfsidmap2:amd64 0.25-5 amd64 NFS idmapping library
ii nfs-common 1:1.2.8-6ubuntu1.1 amd64 NFS support files common to client and server

What is causing NFS temporary disconnection?

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1528756/+subscriptions

Follow ups

[Bug 1528756] Re: nova instance was read-only mode
From: Augustina Ragwitz, 2016-01-27