← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #14090

[Bug 1315467] [NEW] Neutron deletes the router interface instead of adding a floatingip

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

After parsing a lot of log files related to check failure, looks like the q-vpn at the time when I would expect to add
floating ip , it destroys the router's qg- and qr-  interfaces.

However after the floating ip deletion request the q-vpn service
restores the qg-, qr- interfaces.


tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario[compute,image,network,volume]
failed in http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/console.html.

admin user: admin/9e02f14321454af6bb27587770f27d9b
admin tenant id: admin/413bb1232bca45069f3a3256839effa1

test user: TestMinimumBasicScenario-1306090821/c8dd95056c0b407e8dd168dbf410a66a
test Tenant:  TestMinimumBasicScenario-993819377/2527b8222e3343bca9f70343e608880c
External Net : public/c29040d3-7e73-4a87-9f73-bb5cbe602afb
External subnet: public-subnet/ee754eb6-6194-4a25-a4cc-f9233d366c1e
Network: TestMinimumBasicScenario-1375749858-network/f029d7a8-54e0-484c-a215-cc34066ae830
Subnet: TestMinimumBasicScenario-1375749858-subnet/7edc72be-1207-4571-95d4-911223885ae7  10.100.0.0/28
Router id:TestMinimumBasicScenario-1375749858-router/08216822-5ee2-4313-be7e-dad2d84147db
Expected interfaces in the qrouter-08216822-5ee2-4313-be7e-dad2d84147db:
* lo 127.0.0.1
* qr-529eddd4-2c 10.100.0.1/28 iface_id: 529eddd4-2ca8-43ec-9cab-29c3a6632604, attached-mac: fa:16:3e:2a:f8:ba, ofport 166
* qg-9be8f502-93 172.24.4.85/24 iface-id: 9be8f502-9360-47dc-9eff-33c8743e7c2b  attached-mac: fa:16:3e:be:a1:54, ofport 37

Floating IP: 172.24.4.87 (Never appears in the q-vpn log)
port: (net/subnet/port)(c29040d3-7e73-4a87-9f73-bb5cbe602afb/ee754eb6-6194-4a25-a4cc-f9233d366c1e/013b0b2d-80ed-403d-b380-6b6895ce34f5)
mac?: fa:16:3e:15:f2:57
floating ip uuid: cd84111e-af6a-4c26-af73-3167419c664a

Instance:
Ipv4: 10.100.0.2 mac: FA:16:3E:18:F1:69 
Instance uuid: 8a552eda-2fbd-4972-bfcf-cee7e6472871
iface_id/port_id: 4188c532-3265-4294-8b4e-9bbfe5a482e8
ovsdb_interface_uuid: 9d7b858b-745e-482b-b91a-1e9ae34fc545
intbr tag: 49

dhcp server dev: tap4c6c6e06-e4
ns: qdhcp-f029d7a8-54e0-484c-a215-cc34066ae830
ip: 10.100.0.3 mac: fa:16:3e:a2:f1:ea
intbr tag: 49

Host:
eth0: 10.7.16.229/15 mac: 02:16:3e:52:5d:ff


Router + router interface creation in the logs:
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/tempest.txt.gz#_2014-05-01_19_49_46_924
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-q-svc.txt.gz#_2014-05-01_19_49_46_724

Floating IP create:
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-n-api.txt.gz#_2014-05-01_19_50_18_447
Floating IP associate:
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-n-api.txt.gz#_2014-05-01_19_50_18_814

q-svc starts destroying the router:
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-q-vpn.txt.gz#_2014-05-01_19_50_20_277
Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-c0cd93e7-5cb0-403b-a509-8e07b352b89d', 'ipsec', 'whack', '--ctlbase', '/opt/stack/data/neutron/ipsec/c0cd93e7-5cb0-403b-a509-8e07b352b89d/var/run/pluto', '--status']
Exit code: 1
Stdout: ''
Stderr: 'whack: Pluto is not running (no "/opt/stack/data/neutron/ipsec/c0cd93e7-5cb0-403b-a509-8e07b352b89d/var/run/pluto.ctl")\n' execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:74
2014-05-01 19:50:20.277 19279 DEBUG neutron.openstack.common.lockutils [-] Semaphore / lock released "sync" inner /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:252
2014-05-01 19:50:20.277 19279 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-08216822-5ee2-4313-be7e-dad2d84147db', 'ip', '-o', 'link', 'show', 'qr-529eddd4-2c'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:48
2014-05-01 19:50:20.516 19279 DEBUG neutron.agent.linux.utils [-] 
Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-08216822-5ee2-4313-be7e-dad2d84147db', 'ip', '-o', 'link', 'show', 'qr-529eddd4-2c']
Exit code: 0
Stdout: '605: qr-529eddd4-2c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \\    link/ether fa:16:3e:2a:f8:ba brd ff:ff:ff:ff:ff:ff\n'
Stderr: '' execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:74
2014-05-01 19:50:20.517 19279 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', '-o', 'link', 'show', 'br-int'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:48
2014-05-01 19:50:20.533 19279 DEBUG neutron.agent.linux.utils [-] 
Command: ['ip', '-o', 'link', 'show', 'br-int']
Exit code: 0
Stdout: '6: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN \\    link/ether c6:49:9f:72:d2:4a brd ff:ff:ff:ff:ff:ff\n'
Stderr: '' execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:74
2014-05-01 19:50:20.534 19279 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', '--', '--if-exists', 'del-port', 'br-int', 'qr-529eddd4-2c'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:48
2014-05-01 19:50:21.061 19279 DEBUG neutron.agent.linux.utils [-] 
Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', '--', '--if-exists', 'del-port', 'br-int', 'qr-529eddd4-2c']
Exit code: 0

Tempest tries to connect to the VM:
2014-05-01 19:50:19,463 .. 2014-05-01 19:53:48,494 ~ 209 sec

Delete request for the floatingip:
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/tempest.txt.gz#_2014-05-01_19_54_24_191
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-n-api.txt.gz#_2014-05-01_19_54_25_047

Neutron touches the qr-529eddd4-2c interface + port, 
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-q-vpn.txt.gz#_2014-05-01_19_54_37_886

Delete request for router and router interface:
http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/tempest.txt.gz#_2014-05-01_19_54_57_423

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1315467

Title:
  Neutron deletes the router interface instead of adding a floatingip

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  After parsing a lot of log files related to check failure, looks like the q-vpn at the time when I would expect to add
  floating ip , it destroys the router's qg- and qr-  interfaces.

  However after the floating ip deletion request the q-vpn service
  restores the qg-, qr- interfaces.

  
  tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario[compute,image,network,volume]
  failed in http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/console.html.

  admin user: admin/9e02f14321454af6bb27587770f27d9b
  admin tenant id: admin/413bb1232bca45069f3a3256839effa1

  test user: TestMinimumBasicScenario-1306090821/c8dd95056c0b407e8dd168dbf410a66a
  test Tenant:  TestMinimumBasicScenario-993819377/2527b8222e3343bca9f70343e608880c
  External Net : public/c29040d3-7e73-4a87-9f73-bb5cbe602afb
  External subnet: public-subnet/ee754eb6-6194-4a25-a4cc-f9233d366c1e
  Network: TestMinimumBasicScenario-1375749858-network/f029d7a8-54e0-484c-a215-cc34066ae830
  Subnet: TestMinimumBasicScenario-1375749858-subnet/7edc72be-1207-4571-95d4-911223885ae7  10.100.0.0/28
  Router id:TestMinimumBasicScenario-1375749858-router/08216822-5ee2-4313-be7e-dad2d84147db
  Expected interfaces in the qrouter-08216822-5ee2-4313-be7e-dad2d84147db:
  * lo 127.0.0.1
  * qr-529eddd4-2c 10.100.0.1/28 iface_id: 529eddd4-2ca8-43ec-9cab-29c3a6632604, attached-mac: fa:16:3e:2a:f8:ba, ofport 166
  * qg-9be8f502-93 172.24.4.85/24 iface-id: 9be8f502-9360-47dc-9eff-33c8743e7c2b  attached-mac: fa:16:3e:be:a1:54, ofport 37

  Floating IP: 172.24.4.87 (Never appears in the q-vpn log)
  port: (net/subnet/port)(c29040d3-7e73-4a87-9f73-bb5cbe602afb/ee754eb6-6194-4a25-a4cc-f9233d366c1e/013b0b2d-80ed-403d-b380-6b6895ce34f5)
  mac?: fa:16:3e:15:f2:57
  floating ip uuid: cd84111e-af6a-4c26-af73-3167419c664a

  Instance:
  Ipv4: 10.100.0.2 mac: FA:16:3E:18:F1:69 
  Instance uuid: 8a552eda-2fbd-4972-bfcf-cee7e6472871
  iface_id/port_id: 4188c532-3265-4294-8b4e-9bbfe5a482e8
  ovsdb_interface_uuid: 9d7b858b-745e-482b-b91a-1e9ae34fc545
  intbr tag: 49

  dhcp server dev: tap4c6c6e06-e4
  ns: qdhcp-f029d7a8-54e0-484c-a215-cc34066ae830
  ip: 10.100.0.3 mac: fa:16:3e:a2:f1:ea
  intbr tag: 49

  Host:
  eth0: 10.7.16.229/15 mac: 02:16:3e:52:5d:ff

  
  Router + router interface creation in the logs:
  http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/tempest.txt.gz#_2014-05-01_19_49_46_924
  http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-q-svc.txt.gz#_2014-05-01_19_49_46_724

  Floating IP create:
  http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-n-api.txt.gz#_2014-05-01_19_50_18_447
  Floating IP associate:
  http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-n-api.txt.gz#_2014-05-01_19_50_18_814

  q-svc starts destroying the router:
  http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-q-vpn.txt.gz#_2014-05-01_19_50_20_277
  Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-c0cd93e7-5cb0-403b-a509-8e07b352b89d', 'ipsec', 'whack', '--ctlbase', '/opt/stack/data/neutron/ipsec/c0cd93e7-5cb0-403b-a509-8e07b352b89d/var/run/pluto', '--status']
  Exit code: 1
  Stdout: ''
  Stderr: 'whack: Pluto is not running (no "/opt/stack/data/neutron/ipsec/c0cd93e7-5cb0-403b-a509-8e07b352b89d/var/run/pluto.ctl")\n' execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:74
  2014-05-01 19:50:20.277 19279 DEBUG neutron.openstack.common.lockutils [-] Semaphore / lock released "sync" inner /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:252
  2014-05-01 19:50:20.277 19279 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-08216822-5ee2-4313-be7e-dad2d84147db', 'ip', '-o', 'link', 'show', 'qr-529eddd4-2c'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:48
  2014-05-01 19:50:20.516 19279 DEBUG neutron.agent.linux.utils [-] 
  Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-08216822-5ee2-4313-be7e-dad2d84147db', 'ip', '-o', 'link', 'show', 'qr-529eddd4-2c']
  Exit code: 0
  Stdout: '605: qr-529eddd4-2c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \\    link/ether fa:16:3e:2a:f8:ba brd ff:ff:ff:ff:ff:ff\n'
  Stderr: '' execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:74
  2014-05-01 19:50:20.517 19279 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', '-o', 'link', 'show', 'br-int'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:48
  2014-05-01 19:50:20.533 19279 DEBUG neutron.agent.linux.utils [-] 
  Command: ['ip', '-o', 'link', 'show', 'br-int']
  Exit code: 0
  Stdout: '6: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN \\    link/ether c6:49:9f:72:d2:4a brd ff:ff:ff:ff:ff:ff\n'
  Stderr: '' execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:74
  2014-05-01 19:50:20.534 19279 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', '--', '--if-exists', 'del-port', 'br-int', 'qr-529eddd4-2c'] create_process /opt/stack/new/neutron/neutron/agent/linux/utils.py:48
  2014-05-01 19:50:21.061 19279 DEBUG neutron.agent.linux.utils [-] 
  Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', '--', '--if-exists', 'del-port', 'br-int', 'qr-529eddd4-2c']
  Exit code: 0

  Tempest tries to connect to the VM:
  2014-05-01 19:50:19,463 .. 2014-05-01 19:53:48,494 ~ 209 sec

  Delete request for the floatingip:
  http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/tempest.txt.gz#_2014-05-01_19_54_24_191
  http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-n-api.txt.gz#_2014-05-01_19_54_25_047

  Neutron touches the qr-529eddd4-2c interface + port, 
  http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/screen-q-vpn.txt.gz#_2014-05-01_19_54_37_886

  Delete request for router and router interface:
  http://logs.openstack.org/79/88579/2/check/check-tempest-dsvm-neutron-full/f76ee0e/logs/tempest.txt.gz#_2014-05-01_19_54_57_423

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1315467/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next