yahoo-eng-team team mailing list archive

[hossein zabolzadeh <zabolzadeh@xxxxxxxxx>]

Hi Sekiyama,
After tracing the error, I found that the hang state(in the case of image delete operation) is related to the delete function at /glance_source/glance/glance/api/v1/images.py file. See the following piece of code:
        else:
            #LOG.debug("&&&&&&&&&&&&&&&&&&& BEFORE NOTIFIER &&&&&&&&&&&&&&&&")
            self.notifier.info('image.delete', redact_loc(image))
            #LOG.debug("&&&&&&&&&&&&&&&&&&&THE END OF DELETE&&&&&&&&&&&&&&&&")
            return Response(body='', status=200)

The glance service go into hang when execute the self.notifier.info line of code. This line, tries to put delete notification message at INFO queue created on rabbitmq. But rabbitmq did not allow glance to put the message on INFO queue. The reason behind this restriction, is turned back to the default configuration value of rabbitmq service in general and "disk_free_limit" configuration parameter in particular. This parameter restrict any put operation on the rabbitmq, when disk space is lower than the value specified in this parameter.
To resove the issue, I changed the disk_free_limit value to 5MB.  Then the delete operation did not go into hang, and glance service works properly. You can create rabbitmq config file at /etc/rabbitmq/rabbitmq.config and change the disk_free_limit parameter to e.g. 5MB. Put the following lines on your rabbitmq.config:

[
 {rabbit,
  [
%% Set disk free limit (in bytes). Once free disk space reaches this
   %% lower bound, a disk alarm will be set - see the documentation
   %% listed above for more details.
   %%
    {disk_free_limit, 5000000}
 ]}
].

Then restart the rabbitmq service.

service rabbitmq-server restart

And then check the delete operation when the available space is smaller
than e.g. 600MB. I checked it, on my two devstack machines(At home, and
at my office) and both works properly after this change, and the glance
did not go into hang state.

** Changed in: glance
       Status: In Progress => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1376512

Title:
  hang when disk has little free space

Status in OpenStack Image Registry and Delivery Service (Glance):
  Invalid

Bug description:
  When the disk has little free space ( e.g. ~600MB ), glance operations
  (even glance image-delete) hang up and don't finish forever.

  
  This issue is seen in the one node setup with devstack.

  % glance image-list
  +--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+
  | ID                                   | Name                            | Disk Format | Container Format | Size      | Status |
  +--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+
  | 535f59b9-118b-4960-bfbe-17ac9a7f17c5 | cirros-0.3.2-x86_64-uec         | ami         | ami              | 25165824  | active |
  | e8e25ec5-853e-4492-9a21-d6d413dc49ae | cirros-0.3.2-x86_64-uec-kernel  | aki         | aki              | 4969360   | active |
  | ef56e39a-6c59-4503-9662-dbe2e6f9e794 | cirros-0.3.2-x86_64-uec-ramdisk | ari         | ari              | 3723817   | active |
  | dcc47833-1fc5-4c9a-8131-4560ca94f2de | f20-qga                         | qcow2       | bare             | 645332992 | active |
  | e82ca789-47cb-4dcd-9fec-737c695d7cb7 | Fedora-x86_64-20-20140618-sda   | qcow2       | bare             | 209649664 | active |
  +--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+

  
  (.... do something like "nova boot ..." and disk space will go below 600MB)

  
  % glance image-delete f20-qga

  ( =>  will hang up. Any other operations, such as  creating new image,
  are also hang.)

  
  % rm -rf /tmp/...  (free some disk spaces)

  (=> glance will recover from hang up)

  
  Expected behavior is that the operation fails with IO error (No disk space).

  
  (If the filesystem free space is really low, e.g. 50MB, it fails with IO error, as expected.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1376512/+subscriptions