yahoo-eng-team team mailing list archive

[Takashi NATSUME <natsume.takashi@xxxxxxxxxxxxx>]

My guess is as follows:

In nova,
if the record of the marker VM instance exists in the cell
but the 'cell_mapping' in the InstanceMapping of the marker VM instance is null (None),
the issue occurs.

https://github.com/openstack/nova/blob/f902e0d5d87fb05207e4a7aca73d185775d43df2/nova/compute/instance_list.py#L56-L73

--------------------------------------------------------------------------
    def get_marker_record(self, ctx, marker):
        try:
            im = objects.InstanceMapping.get_by_instance_uuid(ctx, marker)
        except exception.InstanceMappingNotFound:
            raise exception.MarkerNotFound(marker=marker)

        elevated = ctx.elevated(read_deleted='yes')
        with context.target_cell(elevated, im.cell_mapping) as cctx:
            try:
                # NOTE(danms): We query this with no columns_to_join()
                # as we're just getting values for the sort keys from
                # it and none of the valid sort keys are on joined
                # columns.
                db_inst = db.instance_get_by_uuid(cctx, marker,
                                                  columns_to_join=[])
            except exception.InstanceNotFound:
                raise exception.MarkerNotFound(marker=marker) <------ Here
    return db_inst
--------------------------------------------------------------------------

In nova-conductor, the record of the VM instance exists in the cell
but the 'cell_mapping' in the InstanceMapping of the VM instance is null (None)
between *1 and *2.

*1: https://github.com/openstack/nova/blob/f902e0d5d87fb05207e4a7aca73d185775d43df2/nova/conductor/manager.py#L1172-L1175
*2: https://github.com/openstack/nova/blob/f902e0d5d87fb05207e4a7aca73d185775d43df2/nova/conductor/manager.py#L1236-L1239


** Tags added: cells

** Project changed: python-novaclient => nova

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1773945

Title:
  nova client servers.list crashes with bad marker

Status in OpenStack Compute (nova):
  New

Bug description:
  We have a python script that called servers.list() on an instance of
  novaclient.v2.client.Client . Sometimes that raises a "BadRequest
  marker not found" exception:

  Our call:

    client = nova_client.Client("2", session=some_session)
    client.servers.list()

  Observed Stacktrace:

    File "/usr/lib/python2.7/site-packages/XXXX/XXXX.py", line 630, in XXXX:
      all_servers = self.nova.servers.list()
    File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 854, in list
      "servers")
    File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 257, in _list
      resp, body = self.api.client.get(url)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get
      return self.request(url, 'GET', **kwargs)
    File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
      raise exceptions.from_response(resp, body, url, method)
  BadRequest: marker [6a91d602-ab6e-42e0-929e-5ec33df2ddef] not found (HTTP 400) (Request-ID: req-78827725-801d-4514-8cc8-e4b94f15c191)

  Discussion:

  We have a lot of stacks and we sometimes create multiple stacks at the same time. We've noticed that that the stacks with the mentioned UUIDs were created just before these errors occur. It seems that when a newly-created stack appears at a certain location in the server list, its UUID is used as a marker, but the code that validates the marker does
  not recognize such stacks.

  Relevant versions:

  - python-novaclient (9.1.0)
  - nova (16.0.0)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1773945/+subscriptions