yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2102180] [NEW] DBDataError :Data too long for column 'pci_stats' at row 1

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: gaolc <2102180@xxxxxxxxxxxxxxxxxx>
Date: Thu, 13 Mar 2025 04:20:46 -0000
Reply-to: Bug 2102180 <2102180@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

 Description
   ===========
   When I use OpenStack 2024.1 version to start a virtual machine with PCI devices, after a period of time, there may be a situation where the virtual machine cannot be deleted. Our usage will frequently delete and create virtual machines, and the log error is as follows:

   DBDataError (pymysql.err.DataError) (1406, "Data too long for column
'pci_stats' at row 1")

   Steps to reproduce
   ==================
   * Create VM instance
   * End of use, delete VM instance
   * After repeating the above steps for a period of time, there will be a failure to delete the instance
   * After restarting nova_compute, it can be deleted again
   
   Expected result
   ===============
   pci_stats["nova_object.data"]["objects"] is a list type.
   The quantity of this list should be consistent with the number of PCI devices on the computing node
   
   Actual result
   =============
   The list will continue to accumulate, with a lot of duplicate data, until it exceeds the database field limit
   
   Environment
   Openstack: 2024.1
   nova: 29.2.1 
   
   Logs & Configs
   ==============
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager Traceback (most recent call last):
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10574, in _update_available_resource_for_node
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     self.rt.update_available_resource(context, nodename,
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 935, in update_available_resource
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 412, in inner
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     return f(*args, **kwargs)
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1066, in _update_available_resource
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1400, in _update
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     with excutils.save_and_reraise_exception(logger=LOG):
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     self.force_reraise()
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     raise self.value
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1395, in _update
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     compute_node.save()
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 209, in wrapper
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     updates, result = self.indirection_api.object_action(
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/conductor/rpcapi.py", line 247, in object_action
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     return cctxt.call(context, 'object_action', objinst=objinst,
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     result = self.transport._send(
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     return self._driver.send(target, ctxt, message,
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 793, in send
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     return self._send(target, ctxt, message, wait_for_reply, timeout,
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 785, in _send
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     raise result
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager oslo_messaging.rpc.client.RemoteError: Remote error: DBDataError (pymysql.err.DataError) (1406, "Data too long for column 'pci_stats' at row 1")
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager [SQL: UPDATE compute_nodes SET updated_at=%(updated_at)s, disk_available_least=%(disk_available_least)s, pci_stats=%(pci_stats)s, stats=%(stats)s WHERE compute_nodes.id = %(compute_nodes_id)s]
2025-03-07 03:13:18.353 7 ERROR nova.compute.manager [parameters: {'updated_at': datetime.datetime(2025, 3, 6, 19, 13, 18, 318788), 'disk_available_least': 3875, 'pci_stats': '{"nova_object.name": "PciDevicePoolList", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"objects": [{"nova_obje ... (65447 characters truncated) ... a23c67025681"}, "count": 1}, "nova_object.changes": ["numa_node", "count", "product_id", "vendor_id", "tags"]}]}, "nova_object.changes": ["objects"]}', 'stats': '{"failed_builds": "0", "num_instances": "3", "num_vm_active": "2", "num_task_None": "3", "num_os_type_linux": "3", "num_proj_2694810932f0494f903be14ef7247221": "3", "io_workload": "0", "num_vm_error": "1"}', 'compute_nodes_id': 60}]

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2102180

Title:
  DBDataError :Data too long for column 'pci_stats' at row 1

Status in OpenStack Compute (nova):
  New

Bug description:
   Description
     ===========
     When I use OpenStack 2024.1 version to start a virtual machine with PCI devices, after a period of time, there may be a situation where the virtual machine cannot be deleted. Our usage will frequently delete and create virtual machines, and the log error is as follows:

     DBDataError (pymysql.err.DataError) (1406, "Data too long for
  column 'pci_stats' at row 1")

     Steps to reproduce
     ==================
     * Create VM instance
     * End of use, delete VM instance
     * After repeating the above steps for a period of time, there will be a failure to delete the instance
     * After restarting nova_compute, it can be deleted again
     
     Expected result
     ===============
     pci_stats["nova_object.data"]["objects"] is a list type.
     The quantity of this list should be consistent with the number of PCI devices on the computing node
     
     Actual result
     =============
     The list will continue to accumulate, with a lot of duplicate data, until it exceeds the database field limit
     
     Environment
     Openstack: 2024.1
     nova: 29.2.1 
     
     Logs & Configs
     ==============
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager Traceback (most recent call last):
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10574, in _update_available_resource_for_node
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     self.rt.update_available_resource(context, nodename,
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 935, in update_available_resource
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     self._update_available_resource(context, resources, startup=startup)
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 412, in inner
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     return f(*args, **kwargs)
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1066, in _update_available_resource
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     self._update(context, cn, startup=startup)
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1400, in _update
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     with excutils.save_and_reraise_exception(logger=LOG):
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     self.force_reraise()
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     raise self.value
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1395, in _update
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     compute_node.save()
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_versionedobjects/base.py", line 209, in wrapper
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     updates, result = self.indirection_api.object_action(
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/conductor/rpcapi.py", line 247, in object_action
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     return cctxt.call(context, 'object_action', objinst=objinst,
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     result = self.transport._send(
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     return self._driver.send(target, ctxt, message,
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 793, in send
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     return self._send(target, ctxt, message, wait_for_reply, timeout,
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 785, in _send
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager     raise result
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager oslo_messaging.rpc.client.RemoteError: Remote error: DBDataError (pymysql.err.DataError) (1406, "Data too long for column 'pci_stats' at row 1")
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager [SQL: UPDATE compute_nodes SET updated_at=%(updated_at)s, disk_available_least=%(disk_available_least)s, pci_stats=%(pci_stats)s, stats=%(stats)s WHERE compute_nodes.id = %(compute_nodes_id)s]
  2025-03-07 03:13:18.353 7 ERROR nova.compute.manager [parameters: {'updated_at': datetime.datetime(2025, 3, 6, 19, 13, 18, 318788), 'disk_available_least': 3875, 'pci_stats': '{"nova_object.name": "PciDevicePoolList", "nova_object.namespace": "nova", "nova_object.version": "1.1", "nova_object.data": {"objects": [{"nova_obje ... (65447 characters truncated) ... a23c67025681"}, "count": 1}, "nova_object.changes": ["numa_node", "count", "product_id", "vendor_id", "tags"]}]}, "nova_object.changes": ["objects"]}', 'stats': '{"failed_builds": "0", "num_instances": "3", "num_vm_active": "2", "num_task_None": "3", "num_os_type_linux": "3", "num_proj_2694810932f0494f903be14ef7247221": "3", "io_workload": "0", "num_vm_error": "1"}', 'compute_nodes_id': 60}]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2102180/+subscriptions