yahoo-eng-team team mailing list archive

[sean mooney <1896226@xxxxxxxxxxxxxxxxxx>]

** Also affects: neutron
   Importance: Undecided
       Status: New

** Tags added: libvirt neutron ovs

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1896226

Title:
  The vnics are disappearing in the vm

Status in neutron:
  New
Status in OpenStack Compute (nova):
  New

Bug description:
  Hi,

  We have a rocky OSA setup of branch 18.1.9.

  When we create a vm from a particular image, the vm comes with two
  missing vnics inside of it , out of four, which we have provisioned to
  it from four dhcp tenant networks. The plugin is OVS and the firewall
  driver is conntrack. It was working earlier until we have noticed this
  issue.

  So, if we reboot the VM, then, the vnics appear for a short time and
  again after a few seconds, they are disappearing. The disappearing
  vnics inside the vm have a floating ip associated with one of it and
  hence the vm is becoming unpingable. Again if we reboot the vm , it is
  comping up for a short time and again vanishing. At the moment when
  the pinging of the vm stops, we are not noticing any messages in the
  neutron server logs for that port, except when we try to manually do a
  vm reboot:


  LOGS
  -------------
  2020-09-18 08:41:50.197 17808 INFO neutron.wsgi [req-ff4ad70c-568d-4625-9bea-2a472351d00a dae4d1b704943b11cb10287e984f9367070915c18f7cadd48f915af92d4b4d03 35391b98793b4c09bf87c91006d123c2 - f7834cb0083b4f8f81184b6595b46b34 f7834cb0083b4f8f81184b6595b46b34] 172.29.236.183,172.29.236.21 "GET /v2.0/floatingips?tenant_id=35391b98793b4c09bf87c91006d123c2&port_id=095edf83-2d8d-494d-b820-ef7540aefa7c&port_id=0da3066c-a0f5-49df-a10a-20919595a5b8&port_id=298916ee-91a2-428f-86aa-c5ed5f034563&port_id=3e6d29fe-7bee-47d4-bc98-00d934fc5764&port_id=6d456cbe-425a-4d17-86f6-2b77ab88a42f&port_id=853adf00-92ac-4444-8487-32a77e3efb66&port_id=86ec85de-609a-403d-b027-3097ac597e0c&port_id=86f1034d-837d-4e67-ad5e-63d9642a0b2a&port_id=947c5610-b0f4-439a-abf8-51ba3dc8d212&port_id=b00f8eae-18fa-44c4-92e6-9ee75c7c599c&port_id=b8ba1a0c-ebd9-4278-bc83-9b60e1036f63&port_id=c6ce1a3c-b8b3-493d-b052-2810efacbf5e&port_id=c7facefa-565f-46be-8048-a333505ee177&port_id=d5e1fc5f-1a84-4366-bf36-548f2bdc0366 HTTP/1.1" status: 200  len: 4363 time: 0.0851929
  2020-09-18 08:41:50.585 17814 INFO neutron.wsgi [req-349c0d49-9671-43a4-a541-f04facff2ee7 c42abde21dee4c848dc653df8ec429aa e02428b1700247b98ad1d563133f6174 - default default] 172.29.236.57,172.29.236.21 "GET /v2.0/floatingips?fixed_ip_address=172.16.1.19&port_id=86f1034d-837d-4e67-ad5e-63d9642a0b2a HTTP/1.1" status: 200  len: 1042 time: 0.0781569
  2020-09-18 08:41:59.951 17808 INFO neutron.wsgi [req-4bc06b83-cb44-469a-9ef1-0ce9a4fa0753 dae4d1b704943b11cb10287e984f9367070915c18f7cadd48f915af92d4b4d03 35391b98793b4c09bf87c91006d123c2 - f7834cb0083b4f8f81184b6595b46b34 f7834cb0083b4f8f81184b6595b46b34] 172.29.236.183,172.29.236.21 "GET /v2.0/floatingips?tenant_id=35391b98793b4c09bf87c91006d123c2&port_id=298916ee-91a2-428f-86aa-c5ed5f034563&port_id=6d456cbe-425a-4d17-86f6-2b77ab88a42f&port_id=86f1034d-837d-4e67-ad5e-63d9642a0b2a&port_id=947c5610-b0f4-439a-abf8-51ba3dc8d212 HTTP/1.1" status: 200  len: 1042 time: 0.0917962
  2020-09-18 08:42:06.094 17817 DEBUG neutron.plugins.ml2.rpc [req-9cc5b8b2-a616-466d-9ed8-eae2b9b6056b - - - - -] Device 86f1034d-837d-4e67-ad5e-63d9642a0b2a up at agent ovs-agent-b7w update_device_up /openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:256
  2020-09-18 08:42:06.151 17817 DEBUG neutron.db.provisioning_blocks [req-9cc5b8b2-a616-466d-9ed8-eae2b9b6056b - - - - -] Provisioning complete for port 86f1034d-837d-4e67-ad5e-63d9642a0b2a triggered by entity L2. provisioning_complete /openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/neutron/db/provisioning_blocks.py:138
  -----------------

  The port we are talking about is "86f1034d-837d-4e67-ad5e-
  63d9642a0b2a" in the above logs.

  
  ------------
  1.9/lib/python2.7/site-packages/neutron/notifiers/nova.py:242
  2020-09-18 08:41:56.744 16567 DEBUG novaclient.v2.client [-] REQ: curl -g -i -X POST http://wtl-int.sandvine.cloud:8774/v2.1/os-server-external-events -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}c8de3bb0dae419214d99c02879f89bd3a6a4dd78" -H "X-OpenStack-Nova-API-Version: 2.1" -d '{"events": [{"status": "completed", "tag": "6d456cbe-425a-4d17-86f6-2b77ab88a42f", "name": "network-vif-unplugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412"}, {"status": "completed", "tag": "298916ee-91a2-428f-86aa-c5ed5f034563", "name": "network-vif-unplugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412"}, {"status": "completed", "tag": "86f1034d-837d-4e67-ad5e-63d9642a0b2a", "name": "network-vif-unplugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412"}]}' _http_log_request /openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/keystoneauth1/session.py:448
  2020-09-18 08:41:56.801 16567 DEBUG novaclient.v2.client [-] RESP BODY: {"events": [{"status": "completed", "tag": "6d456cbe-425a-4d17-86f6-2b77ab88a42f", "name": "network-vif-unplugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412", "code": 200}, {"status": "completed", "tag": "298916ee-91a2-428f-86aa-c5ed5f034563", "name": "network-vif-unplugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412", "code": 200}, {"status": "completed", "tag": "86f1034d-837d-4e67-ad5e-63d9642a0b2a", "name": "network-vif-unplugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412", "code": 200}]} _http_log_response /openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/keystoneauth1/session.py:511
  2020-09-18 08:41:56.803 16567 INFO neutron.notifiers.nova [-] Nova event response: {u'status': u'completed', u'tag': u'86f1034d-837d-4e67-ad5e-63d9642a0b2a', u'name': u'network-vif-unplugged', u'server_uuid': u'713376a9-c354-4fb7-946c-e926c1cd9412', u'code': 200}
  --------------------- 

  
  -------------

  2020-09-18 08:42:01.553 16363 DEBUG novaclient.v2.client [-] REQ: curl -g -i -X POST http://wtl-int.sandvine.cloud:8774/v2.1/os-server-external-events -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}e28d38f7ee67e8258671b28e839dbdf049f13b48" -H "X-OpenStack-Nova-API-Version: 2.1" -d '{"events": [{"status": "completed", "tag": "6d456cbe-425a-4d17-86f6-2b77ab88a42f", "name": "network-vif-plugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412"}, {"status": "completed", "tag": "298916ee-91a2-428f-86aa-c5ed5f034563", "name": "network-vif-plugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412"}, {"status": "completed", "tag": "86f1034d-837d-4e67-ad5e-63d9642a0b2a", "name": "network-vif-plugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412"}]}' _http_log_request /openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/keystoneauth1/session.py:448
  2020-09-18 08:42:01.608 16363 DEBUG novaclient.v2.client [-] RESP BODY: {"events": [{"status": "completed", "tag": "6d456cbe-425a-4d17-86f6-2b77ab88a42f", "name": "network-vif-plugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412", "code": 200}, {"status": "completed", "tag": "298916ee-91a2-428f-86aa-c5ed5f034563", "name": "network-vif-plugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412", "code": 200}, {"status": "completed", "tag": "86f1034d-837d-4e67-ad5e-63d9642a0b2a", "name": "network-vif-plugged", "server_uuid": "713376a9-c354-4fb7-946c-e926c1cd9412", "code": 200}]} _http_log_response /openstack/venvs/neutron-18.1.9/lib/python2.7/site-packages/keystoneauth1/session.py:511
  2020-09-18 08:42:01.610 16363 INFO neutron.notifiers.nova [-] Nova event response: {u'status': u'completed', u'tag': u'86f1034d-837d-4e67-ad5e-63d9642a0b2a', u'name': u'network-vif-plugged', u'server_uuid': u'713376a9-c354-4fb7-946c-e926c1cd9412', u'code': 200}
  ------------

  
  Please suggest a workaround if it is a known bug.

  
  Thanks
  Kumar

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1896226/+subscriptions