yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1519537] [NEW] [RFE] - Diagnostics Extension for Neutron

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Ramu Ramamurthy <ramu.ramamurthy@xxxxxxxxxx>
Date: Tue, 24 Nov 2015 23:32:31 -0000
Reply-to: Bug 1519537 <1519537@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Problem
----------

Debugging common networking/neutron problems (1. cannot ping VM, 2. cannot ping FIP),
tends to be manual, and requires root access to look into the state of the agents or the datapath
on different hosts.

Neutron needs to provide a "diagnostics" extension api which can be used for debugging networking problems.
Each agent/driver exposes its own state in a structured (json) format via the diagnostics extension. The structured
content can be parsed by automation to pin-point problems or at least help get to the next step of the debugging much faster than done manually.
In addition, there should be diagnostics/operational support to ping a neutron port from the dhcp or l3 agents.

Nova "diagnostics" serves as an example here.
https://wiki.openstack.org/wiki/Nova_VM_Diagnostics

Fix
----

A "diagnostics" extension is added to neutron

Each agent and corresponding drivers  supports a get_diagnostics()  API invoked from neutron-server upon
the following GET APIs (limited by policy to admin-only). The outputs are structured so, they can be processed by other tools.

GET: /agent/:id/diagnostics
               example output from neutron-ovs agent:   OVS bridges, ports and flows

GET: /agent/:id/diagnostics/network/:id
               example output from dhcp-agent (dnsmasq driver): contents of host,lease files

GET: /agent/:id/diagnostics/port/:id
                example output from dhcp-agent: dhcp transactions for that port (from dnsmasq logs)
                example output from ovs-agent: stats on qvo,qbr,tap interfaces

GET: /agent/:id/diagnostics/port/:id/security-groups
                 example output from l2-agent (iptables-firewalldriver):  iptables rules programmed (ingress/egress/spoofing) for that port

GET: /agent/:id/diagnostics/port/:id/ping
                   This is an "operational" command - ping the port from the agent (dhcp/l3) network/router namespace

Neutron Command-line Client supports the following new commands
----------------------------------------------------------------

neutron l2-diagnostics --network-id <> --port-id <> agent

neutron dhcp-diagnostics --network-id <> --port-id <> --ping agent

neutron l3-diagnostics --network-id <> --port-id <> --ping agent

Sample Diagnostics Extension Code
------------------------------------------

See Attached Code Diff

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: rfe

** Patch added: "sample code diff for diagnostics extension"
   https://bugs.launchpad.net/bugs/1519537/+attachment/4524935/+files/diagnostics-extension-code-diff.txt

** Description changed:

- 
  Problem
  ----------
  
- Debugging common networking/neutron problems (1. cannot ping VM, 2. cannot ping FIP),  
+ Debugging common networking/neutron problems (1. cannot ping VM, 2. cannot ping FIP),
  tends to be manual, and requires root-shell access to look into the state of the agents or the datapath
  on different hosts.
  
  Neutron needs to provide a "diagnostics" extension api which can be used for debugging networking problems.
  Each agent/driver exposes its own state in a structured (json) format via the diagnostics extension. The structured
- content can be parsed by automation to pin-point problems or at least help get to 
- the next step of the debugging much faster than done manually.
+ content can be parsed by automation to pin-point problems or at least help get to the next step of the debugging much faster than done manually.
  In addition, there should be diagnostics/operational support to ping a neutron port from the dhcp or l3 agents.
  
  Nova "diagnostics" serves as an example here.
  https://wiki.openstack.org/wiki/Nova_VM_Diagnostics
  
  Fix
  ----
  
  A "diagnostics" extension is added to neutron
  
  Each agent and corresponding drivers  supports a get_diagnostics()  API invoked from neutron-server upon
- the following GET APIs (limited by policy to admin-only). The outputs are structured so, they can be
- processed by other tools.
+ the following GET APIs (limited by policy to admin-only). The outputs are structured so, they can be processed by other tools.
  
  GET: /agent/:id/diagnostics
-                example output from neutron-ovs agent:   OVS bridges, ports and flows
+                example output from neutron-ovs agent:   OVS bridges, ports and flows
  
  GET: /agent/:id/diagnostics/network/:id
-                example output from dhcp-agent (dnsmasq driver): contents of host,lease files
+                example output from dhcp-agent (dnsmasq driver): contents of host,lease files
  
  GET: /agent/:id/diagnostics/port/:id
-                 example output from dhcp-agent: dhcp transactions for that port (from dnsmasq logs)
-        
+                 example output from dhcp-agent: dhcp transactions for that port (from dnsmasq logs)
+                 example output from ovs-agent: stats on qvo,qbr,tap interfaces
+ 
  GET: /agent/:id/diagnostics/port/:id/security-groups
-                  example output from l2-agent (iptables-firewalldriver):  iptables rules programmed (ingress/egress/spoofing) for that port
+                  example output from l2-agent (iptables-firewalldriver):  iptables rules programmed (ingress/egress/spoofing) for that port
  
  GET: /agent/:id/diagnostics/port/:id/ping
-                    This is an "operational" command - ping the port from the agent (dhcp/l3) network/router namespace
+                    This is an "operational" command - ping the port from the agent (dhcp/l3) network/router namespace
  
  Sample Diagnostics Extension Code
  ------------------------------------------
  
  See Attached Code Diff

** Description changed:

  Problem
  ----------
  
  Debugging common networking/neutron problems (1. cannot ping VM, 2. cannot ping FIP),
  tends to be manual, and requires root-shell access to look into the state of the agents or the datapath
  on different hosts.
  
  Neutron needs to provide a "diagnostics" extension api which can be used for debugging networking problems.
  Each agent/driver exposes its own state in a structured (json) format via the diagnostics extension. The structured
  content can be parsed by automation to pin-point problems or at least help get to the next step of the debugging much faster than done manually.
  In addition, there should be diagnostics/operational support to ping a neutron port from the dhcp or l3 agents.
  
  Nova "diagnostics" serves as an example here.
  https://wiki.openstack.org/wiki/Nova_VM_Diagnostics
  
  Fix
  ----
  
  A "diagnostics" extension is added to neutron
  
  Each agent and corresponding drivers  supports a get_diagnostics()  API invoked from neutron-server upon
  the following GET APIs (limited by policy to admin-only). The outputs are structured so, they can be processed by other tools.
  
  GET: /agent/:id/diagnostics
                 example output from neutron-ovs agent:   OVS bridges, ports and flows
  
  GET: /agent/:id/diagnostics/network/:id
                 example output from dhcp-agent (dnsmasq driver): contents of host,lease files
  
  GET: /agent/:id/diagnostics/port/:id
                  example output from dhcp-agent: dhcp transactions for that port (from dnsmasq logs)
-                 example output from ovs-agent: stats on qvo,qbr,tap interfaces
+                 example output from ovs-agent: stats on qvo,qbr,tap interfaces
  
  GET: /agent/:id/diagnostics/port/:id/security-groups
                   example output from l2-agent (iptables-firewalldriver):  iptables rules programmed (ingress/egress/spoofing) for that port
  
  GET: /agent/:id/diagnostics/port/:id/ping
                     This is an "operational" command - ping the port from the agent (dhcp/l3) network/router namespace
  
+ 
+ Neutron Command-line Client supports the following new commands
+ ----------------------------------------------------------------
+ 
+ neutron l2-diagnostics --network-id <> --port-id <> agent
+ 
+ neutron dhcp-diagnostics --network-id <> --port-id <> --ping agent
+ 
+ neutron l3-diagnostics --network-id <> --port-id <> --ping agent
+ 
+ 
  Sample Diagnostics Extension Code
  ------------------------------------------
  
  See Attached Code Diff

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1519537

Title:
  [RFE] - Diagnostics Extension for Neutron

Status in neutron:
  New

Bug description:
  Problem
  ----------

  Debugging common networking/neutron problems (1. cannot ping VM, 2. cannot ping FIP),
  tends to be manual, and requires root access to look into the state of the agents or the datapath
  on different hosts.

  Neutron needs to provide a "diagnostics" extension api which can be used for debugging networking problems.
  Each agent/driver exposes its own state in a structured (json) format via the diagnostics extension. The structured
  content can be parsed by automation to pin-point problems or at least help get to the next step of the debugging much faster than done manually.
  In addition, there should be diagnostics/operational support to ping a neutron port from the dhcp or l3 agents.

  Nova "diagnostics" serves as an example here.
  https://wiki.openstack.org/wiki/Nova_VM_Diagnostics

  Fix
  ----

  A "diagnostics" extension is added to neutron

  Each agent and corresponding drivers  supports a get_diagnostics()  API invoked from neutron-server upon
  the following GET APIs (limited by policy to admin-only). The outputs are structured so, they can be processed by other tools.

  GET: /agent/:id/diagnostics
                 example output from neutron-ovs agent:   OVS bridges, ports and flows

  GET: /agent/:id/diagnostics/network/:id
                 example output from dhcp-agent (dnsmasq driver): contents of host,lease files

  GET: /agent/:id/diagnostics/port/:id
                  example output from dhcp-agent: dhcp transactions for that port (from dnsmasq logs)
                  example output from ovs-agent: stats on qvo,qbr,tap interfaces

  GET: /agent/:id/diagnostics/port/:id/security-groups
                   example output from l2-agent (iptables-firewalldriver):  iptables rules programmed (ingress/egress/spoofing) for that port

  GET: /agent/:id/diagnostics/port/:id/ping
                     This is an "operational" command - ping the port from the agent (dhcp/l3) network/router namespace

  Neutron Command-line Client supports the following new commands
  ----------------------------------------------------------------

  neutron l2-diagnostics --network-id <> --port-id <> agent

  neutron dhcp-diagnostics --network-id <> --port-id <> --ping agent

  neutron l3-diagnostics --network-id <> --port-id <> --ping agent

  Sample Diagnostics Extension Code
  ------------------------------------------

  See Attached Code Diff

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1519537/+subscriptions