yahoo-eng-team team mailing list archive

[Anton Kurbatov <1929438@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

I ran into a problem when the network inside the newly created VM is not
working.


* Pre-conditions:
- the neutron ovs agent has not yet seen any ports from the VM network;
- any other bridge (except for the network in which the VM is created) is recreated on the node.


* Step-by-step reproduction steps:

The bridge mapping from ml2_conf.ini looks like:
[ovs]
bridge_mappings = Public:br-eth0,Test:br-test


The 'Test:br-test' mapping is a test bridge to demonstrate the problem.
I've created it using ovs-vsctl tool like: ovs-vsctl add-br br-test.

1) Recreate this test bridge that triggers
_reconfigure_physical_bridges:

[root@sqvm2-2009 ~]# ovs-vsctl del-br br-test; ovs-vsctl add-br br-test
[root@sqvm2-2009 ~]#


2) Create the first VM from the 'public' network that is mapped to the 'Public' bridge and try to ping it:
---
[root@sqvm2-2009 ~]# openstack server create test-vm --image cirros  --flavor 100 --network public --boot-from-volume 1
[root@sqvm2-2009 ~]# openstack server list
+--------------------------------------+---------+--------+---------------------+--------------------------+--------+
| ID                                   | Name    | Status | Networks            | Image                    | Flavor |
+--------------------------------------+---------+--------+---------------------+--------------------------+--------+
| 68c32b4d-8f90-4ced-8ca4-67a9e4ff255b | test-vm | ACTIVE | public=10.34.111.12 | N/A (booted from volume) | tiny   |
+--------------------------------------+---------+--------+---------------------+--------------------------+--------+
[root@sqvm2-2009 ~]# virsh console 68c32b4d-8f90-4ced-8ca4-67a9e4ff255b
Connected to domain instance-00000005
Escape character is ^]

login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root.
cirros login: cirros
Password:
$ sudo ip addr add 10.34.111.12/18 dev eth0
$ ip a s eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether fa:16:3e:67:f8:4e brd ff:ff:ff:ff:ff:ff
    inet 10.34.111.12/18 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe67:f84e/64 scope link
       valid_lft forever preferred_lft forever
$
[root@sqvm2-2009 ~]# ping 10.34.111.12
PING 10.34.111.12 (10.34.111.12) 56(84) bytes of data.
>From 10.34.66.138 icmp_seq=1 Destination Host Unreachable
>From 10.34.66.138 icmp_seq=2 Destination Host Unreachable
>From 10.34.66.138 icmp_seq=3 Destination Host Unreachable
>From 10.34.66.138 icmp_seq=4 Destination Host Unreachable
^C
--- 10.34.111.12 ping statistics ---
5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4000ms
pipe 4
[root@sqvm2-2009 ~]#
---


* Actual result:
The VM is not pingable, but should.
During port processing in neutron-openvswitch-agent rpc_loop, one can see the logs:

2021-05-24 17:28:29.776 13744 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-7f3667e0-56a5-4830-8376-10577a2ee167 - - - - -] Port c010955a-4782-418d-a612-0bfbd66b3c09 updated. Details: {'device': 'c010955a-4782-418d-a612-0bfbd66b3c09', 'device_id': '68c32b4d-8f90-4ced-8ca4-67a9e4ff255b', 'network_id': '568fb8ce-8f1b-456e-8a31-330ef19f2f5c', 'port_id': 'c010955a-4782-418d-a612-0bfbd66b3c09', 'mac_address': 'fa:16:3e:67:f8:4e', 'admin_state_up': True, 'network_type': 'flat', 'segmentation_id': None, 'physical_network': 'Public', 'fixed_ips': [{'subnet_id': 'b6f963e3-ad77-4bde-8431-049f87871422', 'ip_address': '10.34.111.12'}], 'device_owner': 'compute:nova', 'allowed_address_pairs': [], 'port_security_enabled': True, 'qos_policy_id': None, 'network_qos_policy_id': None, 'profile': {}, 'vif_type': 'ovs', 'vnic_type': 'normal', 'security_groups': ['09948793-2e11-4d89-ad1f-0c0d0eef80f0'], 'migrating_to': None}
2021-05-24 17:28:29.776 13744 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-7f3667e0-56a5-4830-8376-10577a2ee167 - - - - -] Assigning 2 as local vlan for net-id=568fb8ce-8f1b-456e-8a31-330ef19f2f5c
2021-05-24 17:28:29.777 13744 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-7f3667e0-56a5-4830-8376-10577a2ee167 - - - - -] Cannot provision flat network for net-id=568fb8ce-8f1b-456e-8a31-330ef19f2f5c - no bridge for physical_network Public


* Version:
Stein release.
The issue is also reproducible on the master branch.


* Attachments:
Full neutron-openvswitch-agent service logs attached

** Affects: neutron
     Importance: Undecided
         Status: New

** Attachment added: "neutron-openvswitch-agent.log"
   https://bugs.launchpad.net/bugs/1929438/+attachment/5499913/+files/neutron-openvswitch-agent.log

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1929438

Title:
  Cannot provision flat network after reconfiguring physical bridges

Status in neutron:
  New

Bug description:
  I ran into a problem when the network inside the newly created VM is
  not working.

  
  * Pre-conditions:
  - the neutron ovs agent has not yet seen any ports from the VM network;
  - any other bridge (except for the network in which the VM is created) is recreated on the node.

  
  * Step-by-step reproduction steps:

  The bridge mapping from ml2_conf.ini looks like:
  [ovs]
  bridge_mappings = Public:br-eth0,Test:br-test

  
  The 'Test:br-test' mapping is a test bridge to demonstrate the problem.
  I've created it using ovs-vsctl tool like: ovs-vsctl add-br br-test.

  1) Recreate this test bridge that triggers
  _reconfigure_physical_bridges:

  [root@sqvm2-2009 ~]# ovs-vsctl del-br br-test; ovs-vsctl add-br br-test
  [root@sqvm2-2009 ~]#

  
  2) Create the first VM from the 'public' network that is mapped to the 'Public' bridge and try to ping it:
  ---
  [root@sqvm2-2009 ~]# openstack server create test-vm --image cirros  --flavor 100 --network public --boot-from-volume 1
  [root@sqvm2-2009 ~]# openstack server list
  +--------------------------------------+---------+--------+---------------------+--------------------------+--------+
  | ID                                   | Name    | Status | Networks            | Image                    | Flavor |
  +--------------------------------------+---------+--------+---------------------+--------------------------+--------+
  | 68c32b4d-8f90-4ced-8ca4-67a9e4ff255b | test-vm | ACTIVE | public=10.34.111.12 | N/A (booted from volume) | tiny   |
  +--------------------------------------+---------+--------+---------------------+--------------------------+--------+
  [root@sqvm2-2009 ~]# virsh console 68c32b4d-8f90-4ced-8ca4-67a9e4ff255b
  Connected to domain instance-00000005
  Escape character is ^]

  login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root.
  cirros login: cirros
  Password:
  $ sudo ip addr add 10.34.111.12/18 dev eth0
  $ ip a s eth0
  2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
      link/ether fa:16:3e:67:f8:4e brd ff:ff:ff:ff:ff:ff
      inet 10.34.111.12/18 scope global eth0
         valid_lft forever preferred_lft forever
      inet6 fe80::f816:3eff:fe67:f84e/64 scope link
         valid_lft forever preferred_lft forever
  $
  [root@sqvm2-2009 ~]# ping 10.34.111.12
  PING 10.34.111.12 (10.34.111.12) 56(84) bytes of data.
  From 10.34.66.138 icmp_seq=1 Destination Host Unreachable
  From 10.34.66.138 icmp_seq=2 Destination Host Unreachable
  From 10.34.66.138 icmp_seq=3 Destination Host Unreachable
  From 10.34.66.138 icmp_seq=4 Destination Host Unreachable
  ^C
  --- 10.34.111.12 ping statistics ---
  5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4000ms
  pipe 4
  [root@sqvm2-2009 ~]#
  ---

  
  * Actual result:
  The VM is not pingable, but should.
  During port processing in neutron-openvswitch-agent rpc_loop, one can see the logs:

  2021-05-24 17:28:29.776 13744 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-7f3667e0-56a5-4830-8376-10577a2ee167 - - - - -] Port c010955a-4782-418d-a612-0bfbd66b3c09 updated. Details: {'device': 'c010955a-4782-418d-a612-0bfbd66b3c09', 'device_id': '68c32b4d-8f90-4ced-8ca4-67a9e4ff255b', 'network_id': '568fb8ce-8f1b-456e-8a31-330ef19f2f5c', 'port_id': 'c010955a-4782-418d-a612-0bfbd66b3c09', 'mac_address': 'fa:16:3e:67:f8:4e', 'admin_state_up': True, 'network_type': 'flat', 'segmentation_id': None, 'physical_network': 'Public', 'fixed_ips': [{'subnet_id': 'b6f963e3-ad77-4bde-8431-049f87871422', 'ip_address': '10.34.111.12'}], 'device_owner': 'compute:nova', 'allowed_address_pairs': [], 'port_security_enabled': True, 'qos_policy_id': None, 'network_qos_policy_id': None, 'profile': {}, 'vif_type': 'ovs', 'vnic_type': 'normal', 'security_groups': ['09948793-2e11-4d89-ad1f-0c0d0eef80f0'], 'migrating_to': None}
  2021-05-24 17:28:29.776 13744 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-7f3667e0-56a5-4830-8376-10577a2ee167 - - - - -] Assigning 2 as local vlan for net-id=568fb8ce-8f1b-456e-8a31-330ef19f2f5c
  2021-05-24 17:28:29.777 13744 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-7f3667e0-56a5-4830-8376-10577a2ee167 - - - - -] Cannot provision flat network for net-id=568fb8ce-8f1b-456e-8a31-330ef19f2f5c - no bridge for physical_network Public

  
  * Version:
  Stein release.
  The issue is also reproducible on the master branch.

  
  * Attachments:
  Full neutron-openvswitch-agent service logs attached

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1929438/+subscriptions