yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

Because of this: https://review.openstack.org/#/c/57774/

** Changed in: nova
       Status: Triaged => Won't Fix

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1241619

Title:
  Turn off image compression in the powervm driver by default

Status in OpenStack Compute (Nova):
  Won't Fix

Bug description:
  There are a few image-intensive compute tests in tempest that take way
  too long to run when you're using the powervm driver, but I use
  tempest/tempest/api/compute/images/test_images_oneserver.py:ImagesOneServerTestJSON.test_create_delete_image
  as my benchmark.  That takes the powervm driver about 26 minutes on a
  dedicated VM and backing IVM.

  The other tests are at least
  test_create_second_image_when_first_image_is_being_saved and
  test_create_image_from_stopped_server.

  We had our performance guy do some profiling while the tests were
  running and found the majority of the time is spent when the powervm
  driver is doing the gzip/gunzip operations before/after ftp to/from
  the backing IVM.  I've attached his findings.

  So here is the image create path in the driver :
  https://github.com/openstack/nova/blob/master/nova/virt/powervm/blockdev.py#L208

  It appears the gzip process in _copy_image_file_from_host (with
  compress=True) is taking the longest amount of time. We should add a
  config flag to optionally do compression at this step (and also to not
  decompress when we are deploying, since we can't assume images in
  glance are compressed anymore)

  ===========

  Note that we're testing powervm with a RHEL image that is about ~462MB
  since we don't have a small Cirros image that works on ppc64, but
  we're working on getting one.

  Also note a conversation with sdague and afazekas in IRC about how
  glance was changed in havana to no longer do image compression by
  default, which further justifies why the powervm driver shouldn't be
  doing it by default:

  (8:59:17 AM) sdague: mriedem: it's not possible to take a straight glance pull?
  (8:59:33 AM) sdague: or it's because it has to be pushed over to the ivm
  (8:59:37 AM) mriedem: sdague: correct
  (8:59:49 AM) sdague: right, the joys of type 1 hypervisors
  (9:00:04 AM) mriedem: sdague: so what mrodden and i were talking about doing was changing the powervm driver to not gzip/gunzip and see if network was faster than i/o, in our case it should be
  (9:00:13 AM) sdague: wel, the cirros size should still help a lot
  (9:00:24 AM) mriedem: problem was when i hacked the powervm driver to do that, it puked...so it's a WIP
  (9:00:35 AM) mriedem: sdague: that will definitely be a blocker for powervm CI for nova
  (9:01:13 AM) sdague: yeh, actually, when people looked at the glance numbers for gzipping before sending, it turns out that only helped if you were on < Gb network
  (9:01:22 AM) sdague: so we defaulted that off in havana
  (9:01:36 AM) sdague: I think the gzip before the wire is all premature optimization
  (9:01:54 AM) afazekas: what about lzop ?
  (9:02:28 AM) afazekas: The lzop compression is much faster, and relatively good
  (9:02:30 AM) sdague: afazekas: honestly, at this point I'd just assume you have the bits, and only worry about compression if you really need it later
  (9:02:52 AM) krtaylor: mriedem, I believe that we have a very small image for power, I am tracking that down today, hopefully
  (9:03:40 AM) krtaylor: mriedem, we'll need it for our kvm on power CI anyway, I'll let you know what I find
  (9:03:48 AM) sdague: heh
  (9:04:01 AM) mriedem: krtaylor: awesome, thanks
  (9:04:25 AM) sdague: yeh, the HP folks did a ton of analysis on it for the glance case, and NSA and CERN jumped in on that thread and said "yes, please change this, we hacked around it"
  (9:04:41 AM) sdague: which was convincing enough for me
  (9:04:49 AM) mriedem: sdague: good, glad to know what i was working on what the right thing
  (9:04:55 AM) mriedem: *was
  (9:05:04 AM) mriedem: now i just need to get the damn driver to work with it
  (9:05:16 AM) sdague: yeh, each case might be different, but it's definitely naive to believe adding compression is always a win

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1241619/+subscriptions