kernel-packages team mailing list archive

[Frederik Bosch <f.bosch@xxxxxxxxx>]

@decui @jsalisbury That is indeed good news. Since we did not make any
changes to our platform yet, we can help to test possible fixes. There
were also some questions.

1. Occurrence of the issue: with the fixes of bug #1445195 the amount of
read-only errors reduced drastically. But it is still there. We are
making backups two times a week on three Ubuntu machines, and read-only
occurs I believe once a month. And when a machine goes read-only, it is
always just one machine, not all three. This situation requires us to
create backups during working days because then we are able to restart
machines immediately, hence causing little downtime.

2. Specificness of the issue: it is indeed a Ubuntu only problem, at
least for us. In our Hyper-V cluster we have four virtual machines,
three Ubuntu, one CentOS. The latter never suffered from the read-only
problem.

3. IO-spike issue: you are correct that we have a I/O performance
downgrade. And I now for sure it is related to backups. During a week we
never have any I/O problems, only those exactly two times a week when we
create backups. This downgrade came up recently, at least we are feeling
the downgrade now. The cause might be that more and more disk space is
getting occupied. Or that was a (kernel) update that is causing this
issue. But my guess (and it is really a guess) is that is caused by the
disk space and therefore the amount of data to backup.

Regarding our system. Since the fixes for bug #1445195 were released, we
switched back to the latest 3.13 kernel in our production environments.
But if I need to switch kernel to help fixing the bug, I think that
would be no problem. Luckily we can now, with the work of @jsalisbury,
do this without problems on a testing machine. Thank you for the work so
far already!

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1470250

Title:
  [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based
  Backups

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Trusty:
  In Progress
Status in linux source package in Utopic:
  In Progress
Status in linux source package in Vivid:
  In Progress
Status in linux source package in Wily:
  In Progress

Bug description:
  Customers have reported running various versions of Ubuntu 14.04.2 LTS
  on Generation 2 Hyper-V Hosts.    On a random Basis, the file system
  will be mounted Read-Only due to a "disk error" (which really isn't
  the case here).    As a result, they must reboot the Ubuntu guest to
  get the file system to mount RW again.

  The Error seen are the following:
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] 
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition
  Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters.
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt ffff88006eb6c700 autosense data valid - len 18
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] 
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition
  Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed.  The Linux SCSI layer does not automatically adjust these parameters.

  This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later.   
  Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have.   We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both.
  As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this.   We can stop the backup process and all is good, but we need nightly backups to image all of our VM's.   All the windows guests have no issues of course.   We also have some CentOS based guests running without issues from what we've seen.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions