yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #60753
[Bug 1658802] [NEW] Issue booting instance with normal port and macvtap agent
Public bug reported:
OpenStack Release: Newton
Operating System: Ubuntu 16.04 LTS 4.4.0-45-generic
OpenStack Distro: OpenStack-Ansible 14.0.2
While working to test/implement macvtap functionality, I found it was
not possible to boot an instance when using the macvtap mech driver and
macvtap agent. My goal is/was to boot an instance with a 'normal' (non-
PCI) port, and have a macvtap interface created on the compute node.
What I found was that that booting the instance resulted in an ERROR
state, with errors reported in neutron-server.log[1] and nova-
compute.log[2]:
[1] Neutron Server: http://paste.openstack.org/show/596128/
[2] Nova Compute: http://paste.openstack.org/show/596129/
The Nova boot syntax can be seen here:
nova boot --image 'Cirros Test Image' --flavor 'm1.test' --nic port-
id=5f08dcec-6689-4c1c-9f95-ffb69548c606 --availability-zone=ZONE-B
TEST-1
An availability zone consisting of a single node was specified.
The interesting thing was observed (multiple times) in neutron-
server.log during the boot process:
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers [req-f531783a-8f29-423e-9764-026f4cd4693f 3e202337454b4813b0ce6ecff74d43d6 b009912cbdcd45a0b4ef34fb3d22e7e1 - - -] Mechanism driver macvtap failed in bind_port
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 787, in _bind_port_level
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers driver.obj.bind_port(context)
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/mech_agent.py", line 109, in bind_port
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers agent):
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/macvtap/mech_driver/mech_macvtap.py", line 106, in try_to_bind_segment_for_agent
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers if self._is_live_migration(context):
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/macvtap/mech_driver/mech_macvtap.py", line 68, in _is_live_migration
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers port_profile = context.original.get(portbindings.PROFILE)
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers AttributeError: 'NoneType' object has no attribute 'get'
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers
The _is_live_migration function was implemented in [3]:
[3]
https://review.openstack.org/#/c/361301/5/neutron/plugins/ml2/drivers/macvtap/mech_driver/mech_macvtap.py
context.original is set to None, which is resulting in the Traceback
observed here. I have logged the object data at the time it hits the
function in [4]:
[4] Object: http://paste.openstack.org/show/596130/
To work around this issue for the sake of booting the instance, I set
'port_profile' to None in mech_macvtap.py and restarted Neutron server.
I was then able to boot the instance, and the agent created the VLAN and
macvtap interfaces accordingly:
+--------------------------------------+------------------------------------+--------+------------+-------------+------------------------------------------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------------------------------+--------+------------+-------------+------------------------------------------------------------------------------+
| 0022df55-6500-44c0-8ff5-4433c37157c1 | TEST-1 | ERROR | - | NOSTATE | |
| 69eda7de-00ad-4927-955f-9e5e1d9d6909 | TEST-1 | ACTIVE | - | Running | TEST=192.168.7.8 |
113: em50.62@em50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 5c:b9:01:88:fd:a5 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5eb9:1ff:fe88:fda5/64 scope link
valid_lft forever preferred_lft forever
115: macvtap0@em50.62: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/ether fa:16:3e:67:da:b8 brd ff:ff:ff:ff:ff:ff
inet6 fe80::f816:3eff:fe67:dab8/64 scope link
valid_lft forever preferred_lft forever
root@817769-i812846:~# tcpdump -i macvtap0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on macvtap0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:30:52.524485 IP6 :: > ff02::1:ff67:dab8: ICMP6, neighbor solicitation, who has fe80::f816:3eff:fe67:dab8, length 24
15:30:52.994199 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:67:da:b8 (oui Unknown), length 290
15:30:52.998185 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
15:30:53.390218 IP6 :: > ff02::1:ff67:dab8: ICMP6, neighbor solicitation, who has fe80::f816:3eff:fe67:dab8, length 24
15:30:53.524385 IP6 fe80::f816:3eff:fe67:dab8 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
15:30:53.524420 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16
15:30:54.008276 IP6 fe80::f816:3eff:fe67:dab8 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
15:30:54.390184 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16
15:30:57.528300 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16
15:30:58.394100 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16
15:31:01.536286 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16
15:31:02.042017 IP6 fe80::f816:3eff:fe67:dab8 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
15:31:02.402014 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16
Port details can be seen here:
+-----------------------+-------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | 817769-i812846.NewtonTest.com |
| binding:profile | {} |
| binding:vif_details | {"port_filter": false, "macvtap_mode": "bridge", "vlan": 62, "physical_interface": "em50", "macvtap_source": "em50.62"} |
| binding:vif_type | macvtap |
| binding:vnic_type | normal |
| created_at | 2017-01-23T20:18:35Z |
| description | |
| device_id | 69eda7de-00ad-4927-955f-9e5e1d9d6909 |
| device_owner | compute:ZONE-B |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "898171fa-903e-4549-bb82-fbb5f16c9bb1", "ip_address": "192.168.7.8"} |
| id | 5f08dcec-6689-4c1c-9f95-ffb69548c606 |
| mac_address | fa:16:3e:67:da:b8 |
| name | TEST |
| network_id | 508e4e86-0137-4774-8df5-bd6919080bea |
| port_security_enabled | True |
| project_id | a2bc030466504dea81191913f1071a10 |
| revision_number | 23 |
| security_groups | |
| status | ACTIVE |
| tenant_id | a2bc030466504dea81191913f1071a10 |
| updated_at | 2017-01-23T20:29:07Z |
+-----------------------+-------------------------------------------------------------------------------------------------------------------------+
Please let me know what additional information you need, including
further troubleshooting or debug output.
James
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1658802
Title:
Issue booting instance with normal port and macvtap agent
Status in neutron:
New
Bug description:
OpenStack Release: Newton
Operating System: Ubuntu 16.04 LTS 4.4.0-45-generic
OpenStack Distro: OpenStack-Ansible 14.0.2
While working to test/implement macvtap functionality, I found it was
not possible to boot an instance when using the macvtap mech driver
and macvtap agent. My goal is/was to boot an instance with a 'normal'
(non-PCI) port, and have a macvtap interface created on the compute
node. What I found was that that booting the instance resulted in an
ERROR state, with errors reported in neutron-server.log[1] and nova-
compute.log[2]:
[1] Neutron Server: http://paste.openstack.org/show/596128/
[2] Nova Compute: http://paste.openstack.org/show/596129/
The Nova boot syntax can be seen here:
nova boot --image 'Cirros Test Image' --flavor 'm1.test' --nic port-
id=5f08dcec-6689-4c1c-9f95-ffb69548c606 --availability-zone=ZONE-B
TEST-1
An availability zone consisting of a single node was specified.
The interesting thing was observed (multiple times) in neutron-
server.log during the boot process:
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers [req-f531783a-8f29-423e-9764-026f4cd4693f 3e202337454b4813b0ce6ecff74d43d6 b009912cbdcd45a0b4ef34fb3d22e7e1 - - -] Mechanism driver macvtap failed in bind_port
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 787, in _bind_port_level
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers driver.obj.bind_port(context)
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/mech_agent.py", line 109, in bind_port
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers agent):
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/macvtap/mech_driver/mech_macvtap.py", line 106, in try_to_bind_segment_for_agent
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers if self._is_live_migration(context):
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/macvtap/mech_driver/mech_macvtap.py", line 68, in _is_live_migration
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers port_profile = context.original.get(portbindings.PROFILE)
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers AttributeError: 'NoneType' object has no attribute 'get'
2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers
The _is_live_migration function was implemented in [3]:
[3]
https://review.openstack.org/#/c/361301/5/neutron/plugins/ml2/drivers/macvtap/mech_driver/mech_macvtap.py
context.original is set to None, which is resulting in the Traceback
observed here. I have logged the object data at the time it hits the
function in [4]:
[4] Object: http://paste.openstack.org/show/596130/
To work around this issue for the sake of booting the instance, I set
'port_profile' to None in mech_macvtap.py and restarted Neutron
server. I was then able to boot the instance, and the agent created
the VLAN and macvtap interfaces accordingly:
+--------------------------------------+------------------------------------+--------+------------+-------------+------------------------------------------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------------------------------+--------+------------+-------------+------------------------------------------------------------------------------+
| 0022df55-6500-44c0-8ff5-4433c37157c1 | TEST-1 | ERROR | - | NOSTATE | |
| 69eda7de-00ad-4927-955f-9e5e1d9d6909 | TEST-1 | ACTIVE | - | Running | TEST=192.168.7.8 |
113: em50.62@em50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 5c:b9:01:88:fd:a5 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5eb9:1ff:fe88:fda5/64 scope link
valid_lft forever preferred_lft forever
115: macvtap0@em50.62: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/ether fa:16:3e:67:da:b8 brd ff:ff:ff:ff:ff:ff
inet6 fe80::f816:3eff:fe67:dab8/64 scope link
valid_lft forever preferred_lft forever
root@817769-i812846:~# tcpdump -i macvtap0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on macvtap0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:30:52.524485 IP6 :: > ff02::1:ff67:dab8: ICMP6, neighbor solicitation, who has fe80::f816:3eff:fe67:dab8, length 24
15:30:52.994199 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:67:da:b8 (oui Unknown), length 290
15:30:52.998185 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
15:30:53.390218 IP6 :: > ff02::1:ff67:dab8: ICMP6, neighbor solicitation, who has fe80::f816:3eff:fe67:dab8, length 24
15:30:53.524385 IP6 fe80::f816:3eff:fe67:dab8 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
15:30:53.524420 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16
15:30:54.008276 IP6 fe80::f816:3eff:fe67:dab8 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
15:30:54.390184 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16
15:30:57.528300 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16
15:30:58.394100 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16
15:31:01.536286 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16
15:31:02.042017 IP6 fe80::f816:3eff:fe67:dab8 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
15:31:02.402014 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16
Port details can be seen here:
+-----------------------+-------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | 817769-i812846.NewtonTest.com |
| binding:profile | {} |
| binding:vif_details | {"port_filter": false, "macvtap_mode": "bridge", "vlan": 62, "physical_interface": "em50", "macvtap_source": "em50.62"} |
| binding:vif_type | macvtap |
| binding:vnic_type | normal |
| created_at | 2017-01-23T20:18:35Z |
| description | |
| device_id | 69eda7de-00ad-4927-955f-9e5e1d9d6909 |
| device_owner | compute:ZONE-B |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "898171fa-903e-4549-bb82-fbb5f16c9bb1", "ip_address": "192.168.7.8"} |
| id | 5f08dcec-6689-4c1c-9f95-ffb69548c606 |
| mac_address | fa:16:3e:67:da:b8 |
| name | TEST |
| network_id | 508e4e86-0137-4774-8df5-bd6919080bea |
| port_security_enabled | True |
| project_id | a2bc030466504dea81191913f1071a10 |
| revision_number | 23 |
| security_groups | |
| status | ACTIVE |
| tenant_id | a2bc030466504dea81191913f1071a10 |
| updated_at | 2017-01-23T20:29:07Z |
+-----------------------+-------------------------------------------------------------------------------------------------------------------------+
Please let me know what additional information you need, including
further troubleshooting or debug output.
James
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1658802/+subscriptions
Follow ups
