yahoo-eng-team team mailing list archive

[Masahito Muroi <2075100@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Description:

Instance boot operation is stuck in BUILDING status if user has some
roles which total name length is over 255 chars.


Details:

The nova-compute manager tries to store all roles list of the request
context in InstanceSystemMetadata in the __build_and_run_instance
method[1], but the metadata value is limited to 255 chars as DB
model[2]. So the total role name string length including ","-joined is
over 255 chars, the instance boot operation gets fail due to the DB
operation failure.


1. https://opendev.org/openstack/nova/src/commit/7a7427691e0bd4818bb7a2c5f5371e0244addbbb/nova/compute/manager.py#L2584-L2587
2. https://opendev.org/openstack/nova/src/commit/7a7427691e0bd4818bb7a2c5f5371e0244addbbb/nova/db/main/models.py#L980


Step to reproduce:


Create some roles which name length is over 255 characters.
In this sample, use three 100 characters roles.

$ openstack role create aaaaaa...<omit 90 a chars>..aaaa
$ openstack role create bbbbbb...<omit 90 b chars>..bbbb
$ openstack role create cccccc...<omit 90 b chars>..cccc

$ openstack role add --project admin --user admin <aaaa role id> 
$ openstack role add --project admin --user admin <bbbb role id> 
$ openstack role add --project admin --user admin <cccc role id> 

$ openstack server create --image <image-id> --nic net-id=<net-id>
--flavor <flavor-id> stuck-vm


The server, stuck-vm, gets stuck in status is "BUILD" and vm_state is "building" status.

Error log by nova-compute:

DBDataError (pymysql.err.DataError) (1406, "Data too long for column
'value' at row 1")

[SQL: INSERT INTO instance_system_metadata (created_at, updated_at, deleted_at, deleted, `key`, value, instance_uuid) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(key)s, %(value)s, %(instance_uuid)s)]
[parameters: {'created_at': datetime.datetime(2024, 7, 12, 18, 57, 35, 750579), 'updated_at': None, 'deleted_at': None, 'deleted': 0, 'key': 'boot_roles', 'value': 'ccccccccccbbbbbbbbbbccccccccccbbbbbbbbbbccccccccccbbbbbbbbbbccccccccccbbbbbbbbbbccccccccccbbbbbbbbbb,ddddddddddbbbbbbbbbbddddddddddbbbbbbbbbbdddddddd ... (32 characters truncated) ... ddddddddddbbbbbbbbbb,manager,admin,reader,member,aaaaaaaaaabbbbbbbbbbaaaaaaaaaabbbbbbbbbbaaaaaaaaaabbbbbbbbbbaaaaaaaaaabbbbbbbbbbaaaaaaaaaabbbbbbbbbb', 'instance_uuid': '1bbd12b0-62d1-4b03-88e2-0edc77517261'}]
(Background on this error at: https://sqlalche.me/e/14/9h9h)


Environment:

Caracal release (2024.1) and master branch

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2075100

Title:
  Instance stuck on building status if user has many roles

Status in OpenStack Compute (nova):
  New

Bug description:
  Description:

  Instance boot operation is stuck in BUILDING status if user has some
  roles which total name length is over 255 chars.

  
  Details:

  The nova-compute manager tries to store all roles list of the request
  context in InstanceSystemMetadata in the __build_and_run_instance
  method[1], but the metadata value is limited to 255 chars as DB
  model[2]. So the total role name string length including ","-joined is
  over 255 chars, the instance boot operation gets fail due to the DB
  operation failure.

  
  1. https://opendev.org/openstack/nova/src/commit/7a7427691e0bd4818bb7a2c5f5371e0244addbbb/nova/compute/manager.py#L2584-L2587
  2. https://opendev.org/openstack/nova/src/commit/7a7427691e0bd4818bb7a2c5f5371e0244addbbb/nova/db/main/models.py#L980

  
  Step to reproduce:

  
  Create some roles which name length is over 255 characters.
  In this sample, use three 100 characters roles.

  $ openstack role create aaaaaa...<omit 90 a chars>..aaaa
  $ openstack role create bbbbbb...<omit 90 b chars>..bbbb
  $ openstack role create cccccc...<omit 90 b chars>..cccc

  $ openstack role add --project admin --user admin <aaaa role id> 
  $ openstack role add --project admin --user admin <bbbb role id> 
  $ openstack role add --project admin --user admin <cccc role id> 

  $ openstack server create --image <image-id> --nic net-id=<net-id>
  --flavor <flavor-id> stuck-vm

  
  The server, stuck-vm, gets stuck in status is "BUILD" and vm_state is "building" status.

  Error log by nova-compute:

  DBDataError (pymysql.err.DataError) (1406, "Data too long for column
  'value' at row 1")

  [SQL: INSERT INTO instance_system_metadata (created_at, updated_at, deleted_at, deleted, `key`, value, instance_uuid) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(key)s, %(value)s, %(instance_uuid)s)]
  [parameters: {'created_at': datetime.datetime(2024, 7, 12, 18, 57, 35, 750579), 'updated_at': None, 'deleted_at': None, 'deleted': 0, 'key': 'boot_roles', 'value': 'ccccccccccbbbbbbbbbbccccccccccbbbbbbbbbbccccccccccbbbbbbbbbbccccccccccbbbbbbbbbbccccccccccbbbbbbbbbb,ddddddddddbbbbbbbbbbddddddddddbbbbbbbbbbdddddddd ... (32 characters truncated) ... ddddddddddbbbbbbbbbb,manager,admin,reader,member,aaaaaaaaaabbbbbbbbbbaaaaaaaaaabbbbbbbbbbaaaaaaaaaabbbbbbbbbbaaaaaaaaaabbbbbbbbbbaaaaaaaaaabbbbbbbbbb', 'instance_uuid': '1bbd12b0-62d1-4b03-88e2-0edc77517261'}]
  (Background on this error at: https://sqlalche.me/e/14/9h9h)

  
  Environment:

  Caracal release (2024.1) and master branch

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2075100/+subscriptions