yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1801828] [NEW] get_hash_str() not working in FIPS mode

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Joshua Cornutt <1801828@xxxxxxxxxxxxxxxxxx>
Date: Tue, 06 Nov 2018 02:01:12 -0000
Reply-to: Bug 1801828 <1801828@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
I run a Red Hat OpenStack 13 lab cloud in "forced" FIPS mode (fips=1 kernel flag) and this is one of the few issues I encountered along the way that required patching. Basically, MD5 is disabled in OpenSSL / Python's hashlib / NSS and anything else that honors FIPS mode, regardless of MD5's intended usage (for data security or not). 

When creating an instance, MD5 is being used by privsep for get 7 random
characters to append to the image cache filename.

Proposed patch - https://review.openstack.org/#/c/615704/


Steps to reproduce
==================
Launch an instance using CLI/API or Horizon. 
(bootstrap-venv) [root@compute-1 bootstrap]# openstack server create \
    --image '54568928-d6bd-47b0-8985-faa34b8aefd2' \
    --flavor '63c23b65-6e9c-4910-887a-188a7c8b61da' \
    --key-name 'default' \
    --security-group '59c728b5-4856-4714-a4ae-21fcb239f8e1' \
    --network '8017a74e-de17-4abd-8261-55abad7fbf35' \
    'test-fips'


Expected result
===============
Instance gets created and enters the "running" state.


Actual result
=============
Instance sometimes gets stuck in "scheduling" state but other times fails with a Horizon error popup of "Error: Failed to perform requested operation on instance "test-fips", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 585f5f53-aa31-4e88-9a13-bc1d517e9d0f.]". 

/var/log/nova/nova-compute.log shows a FIPS violation, stack trace -

2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [req-65312099-b42c-464e-a5d7-e46e0b07edc9 31a8c18286284697a9714cfe327f535f 3c3458e78f864f9385171fa621e5c757 - default default] [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] Instance failed to spawn: ValueError: error:060800A
3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] Traceback (most recent call last):
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2236, in _build_resources
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]     yield resources
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2016, in _build_and_run_instance
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]     block_device_info=block_device_info)
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3084, in spawn
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]     block_device_info=block_device_info)
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3470, in _create_image
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]     os_type_with_default)
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]   File "/usr/lib/python2.7/site-packages/nova/virt/disk/api.py", line 97, in get_file_extension_for_os_type
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]     return utils.get_hash_str(extension)[:7]
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 1064, in get_hash_str
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]     return hashlib.md5(base_str).hexdigest()
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] 
2018-11-06 01:42:24.204 35447 INFO nova.compute.manager [req-65312099-b42c-464e-a5d7-e46e0b07edc9 31a8c18286284697a9714cfe327f535f 3c3458e78f864f9385171fa621e5c757 - default default] [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] Terminating instance


/var/log/nova/nova-conductor.log also shows a FIPS error - 

2018-11-06 01:42:25.728 35493 ERROR nova.scheduler.utils [req-
73ae3a5e-9936-4f1d-a45e-4dc9e145cd77 5e361d86f0fa4cfc82a1a0d481538d2c
0088c9e9ed484e41be132623e4a9e780 - a7052d336b4d45b196ab457dde9b34c6
a7052d336b4d45b196ab457dde9b34c6] [instance:
585f5f53-aa31-4e88-9a13-bc1d517e9d0f] Error from last host: compute-1
.us-east-1.nonplus.io (node compute-1.us-east-1.nonplus.io):
[u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7
/site-packages/nova/compute/manager.py", line 1825, in
_do_build_and_run_instance\n    filter_properties, request_spec)\n', u'
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
2105, in _build_and_run_instance\n    instance_uuid=instance.uuid,
reason=six.text_type(e))\n', u'RescheduledException: Build of instance
585f5f53-aa31-4e88-9a13-bc1d517e9d0f was re-scheduled:
error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for
fips\n']


Environment
===========

(bootstrap-venv) [root@compute-1 bootstrap]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.5 (Maipo)

(bootstrap-venv) [root@compute-1 bootstrap]# sysctl crypto.fips_enabled && cat /proc/sys/crypto/fips_enabled
crypto.fips_enabled = 1
1

(bootstrap-venv) [root@compute-1 bootstrap]# rpm -qa | grep nova
openstack-nova-api-17.0.5-3.d7864fbgit.el7ost.noarch
openstack-nova-compute-17.0.5-3.d7864fbgit.el7ost.noarch
openstack-nova-novncproxy-17.0.5-3.d7864fbgit.el7ost.noarch
openstack-nova-placement-api-17.0.5-3.d7864fbgit.el7ost.noarch
python2-novaclient-10.1.0-1.el7ost.noarch
openstack-nova-common-17.0.5-3.d7864fbgit.el7ost.noarch
openstack-nova-conductor-17.0.5-3.d7864fbgit.el7ost.noarch
python-nova-17.0.5-3.d7864fbgit.el7ost.noarch
openstack-nova-console-17.0.5-3.d7864fbgit.el7ost.noarch
openstack-nova-scheduler-17.0.5-3.d7864fbgit.el7ost.noarch

** Affects: nova
     Importance: Undecided
     Assignee: Joshua Cornutt (oioooioi)
         Status: In Progress


** Tags: fips

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1801828

Title:
  get_hash_str() not working in FIPS mode

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Description
  ===========
  I run a Red Hat OpenStack 13 lab cloud in "forced" FIPS mode (fips=1 kernel flag) and this is one of the few issues I encountered along the way that required patching. Basically, MD5 is disabled in OpenSSL / Python's hashlib / NSS and anything else that honors FIPS mode, regardless of MD5's intended usage (for data security or not). 

  When creating an instance, MD5 is being used by privsep for get 7
  random characters to append to the image cache filename.

  Proposed patch - https://review.openstack.org/#/c/615704/

  
  Steps to reproduce
  ==================
  Launch an instance using CLI/API or Horizon. 
  (bootstrap-venv) [root@compute-1 bootstrap]# openstack server create \
      --image '54568928-d6bd-47b0-8985-faa34b8aefd2' \
      --flavor '63c23b65-6e9c-4910-887a-188a7c8b61da' \
      --key-name 'default' \
      --security-group '59c728b5-4856-4714-a4ae-21fcb239f8e1' \
      --network '8017a74e-de17-4abd-8261-55abad7fbf35' \
      'test-fips'

  
  Expected result
  ===============
  Instance gets created and enters the "running" state.

  
  Actual result
  =============
  Instance sometimes gets stuck in "scheduling" state but other times fails with a Horizon error popup of "Error: Failed to perform requested operation on instance "test-fips", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 585f5f53-aa31-4e88-9a13-bc1d517e9d0f.]". 

  /var/log/nova/nova-compute.log shows a FIPS violation, stack trace -

  2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [req-65312099-b42c-464e-a5d7-e46e0b07edc9 31a8c18286284697a9714cfe327f535f 3c3458e78f864f9385171fa621e5c757 - default default] [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] Instance failed to spawn: ValueError: error:060800A
  3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
  2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] Traceback (most recent call last):
  2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2236, in _build_resources
  2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]     yield resources
  2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2016, in _build_and_run_instance
  2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]     block_device_info=block_device_info)
  2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3084, in spawn
  2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]     block_device_info=block_device_info)
  2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3470, in _create_image
  2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]     os_type_with_default)
  2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]   File "/usr/lib/python2.7/site-packages/nova/virt/disk/api.py", line 97, in get_file_extension_for_os_type
  2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]     return utils.get_hash_str(extension)[:7]
  2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 1064, in get_hash_str
  2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f]     return hashlib.md5(base_str).hexdigest()
  2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
  2018-11-06 01:42:24.202 35447 ERROR nova.compute.manager [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] 
  2018-11-06 01:42:24.204 35447 INFO nova.compute.manager [req-65312099-b42c-464e-a5d7-e46e0b07edc9 31a8c18286284697a9714cfe327f535f 3c3458e78f864f9385171fa621e5c757 - default default] [instance: 585f5f53-aa31-4e88-9a13-bc1d517e9d0f] Terminating instance

  
  /var/log/nova/nova-conductor.log also shows a FIPS error - 

  2018-11-06 01:42:25.728 35493 ERROR nova.scheduler.utils [req-
  73ae3a5e-9936-4f1d-a45e-4dc9e145cd77 5e361d86f0fa4cfc82a1a0d481538d2c
  0088c9e9ed484e41be132623e4a9e780 - a7052d336b4d45b196ab457dde9b34c6
  a7052d336b4d45b196ab457dde9b34c6] [instance:
  585f5f53-aa31-4e88-9a13-bc1d517e9d0f] Error from last host: compute-1
  .us-east-1.nonplus.io (node compute-1.us-east-1.nonplus.io):
  [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7
  /site-packages/nova/compute/manager.py", line 1825, in
  _do_build_and_run_instance\n    filter_properties, request_spec)\n',
  u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py",
  line 2105, in _build_and_run_instance\n
  instance_uuid=instance.uuid, reason=six.text_type(e))\n',
  u'RescheduledException: Build of instance
  585f5f53-aa31-4e88-9a13-bc1d517e9d0f was re-scheduled:
  error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled
  for fips\n']

  
  Environment
  ===========

  (bootstrap-venv) [root@compute-1 bootstrap]# cat /etc/redhat-release 
  Red Hat Enterprise Linux Server release 7.5 (Maipo)

  (bootstrap-venv) [root@compute-1 bootstrap]# sysctl crypto.fips_enabled && cat /proc/sys/crypto/fips_enabled
  crypto.fips_enabled = 1
  1

  (bootstrap-venv) [root@compute-1 bootstrap]# rpm -qa | grep nova
  openstack-nova-api-17.0.5-3.d7864fbgit.el7ost.noarch
  openstack-nova-compute-17.0.5-3.d7864fbgit.el7ost.noarch
  openstack-nova-novncproxy-17.0.5-3.d7864fbgit.el7ost.noarch
  openstack-nova-placement-api-17.0.5-3.d7864fbgit.el7ost.noarch
  python2-novaclient-10.1.0-1.el7ost.noarch
  openstack-nova-common-17.0.5-3.d7864fbgit.el7ost.noarch
  openstack-nova-conductor-17.0.5-3.d7864fbgit.el7ost.noarch
  python-nova-17.0.5-3.d7864fbgit.el7ost.noarch
  openstack-nova-console-17.0.5-3.d7864fbgit.el7ost.noarch
  openstack-nova-scheduler-17.0.5-3.d7864fbgit.el7ost.noarch

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1801828/+subscriptions