yahoo-eng-team team mailing list archive

[Rafael Weingartner <1886949@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Problem Description
=================
Currently, when creating shared labels in Neutron metering, we only receive a total amount of accounting data for a label. This makes it hard for operators to identify what tenants and routers, where the labels were applied, are generating the traffic. Moreover, for shared labels, as they apply in all tenants, Neutron metering is using a random tenant_id, which can create confusion when used in production.

Proposed Change
===============
We propose to extend the neutron metering agent to enable/use a granular
message format. To maintain backward compatibility, this feature is enabled/disabled by the following neutron-metering-agent parameter:

* ``granular_traffic_data``: Defines if the metering agent driver should
present traffic data in a granular fashion, instead of grouping all of the
traffic data for all tenants and routers where the labels were assigned to. The default value is ``False`` for backward compatibility.

When the ``granular_traffic_data`` config is set to ``True``, we have the
following granularities:
* label -- all of the traffic counters for a given label. One must bear in
mind that a label (shared) can be assigned to multiple routers.
* router -- all of the traffic counter for all labels that are assigned to
the router.
* tenant -- all of the traffic counters for all labels of all routers that a tenant has.
* router-label -- all of the traffic counters for a router and the given label.
* tenant-label -- all of the traffic counters for all routers of a tenant that have a given label.

Each granularity presented here is sent to the message bus with different
events types that vary according to the granularity. The mapping between
granularity and event type is presented as follows.

* ``label`` -- event type ``l3.meter.label``.
* ``router`` -- event type ``l3.meter.router``.
* ``tenant`` -- event type ``l3.meter.tenant``..
* ``router-label`` -- event type ``l3.meter.label_router``.
* ``tenant-label`` -- event type ``l3.meter.label_tenant``.

Therefore, we will change the non-granular (``granular_traffic_data =
False``) traffic messages (here also called as legacy), which have the
following format.

     {"pkts": "<the number of packets that matched the rules of the labels>",
      "bytes": "<the number of bytes that matched the rules of the labels>",
      "time": "<seconds between the first data collection and the last one>",
      "first_update": "timeutils.utcnow_ts() of the first collection",
      "last_update": "timeutils.utcnow_ts() of the last collection",
      "host": "<neutron metering agent host name>",
      "label_id": "<the label id>",
      "tenant_id": "<the tenant id>"
      }

The following will be the new event message format, which will also contain some attributes that can be found in the legacy mode such as ``bytes``, ``pkts``, ``time``, ``first_update``, ``last_update``, and ``host``. As follows we present an example of JSON message with all of the
possible attributes.

     {"resource_id": "router-f0f745d9a59c47fdbbdd187d718f9e41-label-00c714f1-49c8-462c-8f5d-f05f21e035c7",
      "tenant_id": "f0f745d9a59c47fdbbdd187d718f9e41",
      "first_update": 1591058790,
      "bytes": 0,
      "label_id": "00c714f1-49c8-462c-8f5d-f05f21e035c7",
      "label_name": "test1",
      "last_update": 1591059037,
      "host": "<hostname>",
      "time": 247,
      "pkts": 0,
      "label_shared": true
      }

The ``resource_id`` is a unique identifier for the "resource" being monitored. Here we consider a resource to be any of the granularities that we handle. The following is the standard for resource ID when the legacy
mode is disabled (for each granularity).

* labels -- label-<label_id>
* routers -- router-<router_id>
* tenant -- tenant-<tenant_id>
* router-label -- router-<router_id>-label-<label_id>
* tenant-label -- tenant-<tenant_id>-label-<label_id>

** Affects: neutron
     Importance: Undecided
     Assignee: Rafael Weingartner (rafaelweingartner)
         Status: In Progress

** Changed in: neutron
     Assignee: (unassigned) => Rafael Weingartner (rafaelweingartner)

** Changed in: neutron
       Status: New => In Progress

** Description changed:

- 
  Problem Description
  =================
- Currently, when creating shared labels in Neutron metering, we only receive a
- total amount of accounting data for a label. This makes it hard for operators
- to identify what tenants and routers, where the labels were applied, are
- generating the traffic. Moreover, for shared labels, as they apply in all
- tenants, Neutron metering is using a random tenant_id, which can create
- confusion when used in production.
+ Currently, when creating shared labels in Neutron metering, we only receive a total amount of accounting data for a label. This makes it hard for operators to identify what tenants and routers, where the labels were applied, are generating the traffic. Moreover, for shared labels, as they apply in all tenants, Neutron metering is using a random tenant_id, which can create confusion when used in production.
  
  Proposed Change
  ===============
  We propose to extend the neutron metering agent to enable/use a granular
- message format. To maintain backward compatibility, this feature is enabled/disabled
- by the following neutron-metering-agent parameter:
+ message format. To maintain backward compatibility, this feature is enabled/disabled by the following neutron-metering-agent parameter:
  
  * ``granular_traffic_data``: Defines if the metering agent driver should
  present traffic data in a granular fashion, instead of grouping all of the
- traffic data for all tenants and routers where the labels were assigned to.
- The default value is ``False`` for backward compatibility.
+ traffic data for all tenants and routers where the labels were assigned to. The default value is ``False`` for backward compatibility.
  
  When the ``granular_traffic_data`` config is set to ``True``, we have the
  following granularities:
  * label -- all of the traffic counters for a given label. One must bear in
  mind that a label (shared) can be assigned to multiple routers.
  * router -- all of the traffic counter for all labels that are assigned to
  the router.
- * tenant -- all of the traffic counters for all labels of all routers that a
- tenant has.
+ * tenant -- all of the traffic counters for all labels of all routers that a tenant has.
  * router-label -- all of the traffic counters for a router and the given label.
- * tenant-label -- all of the traffic counters for all routers of a tenant that
- have a given label.
+ * tenant-label -- all of the traffic counters for all routers of a tenant that have a given label.
  
  Each granularity presented here is sent to the message bus with different
  events types that vary according to the granularity. The mapping between
  granularity and event type is presented as follows.
  
  * ``label`` -- event type ``l3.meter.label``.
  * ``router`` -- event type ``l3.meter.router``.
  * ``tenant`` -- event type ``l3.meter.tenant``..
  * ``router-label`` -- event type ``l3.meter.label_router``.
  * ``tenant-label`` -- event type ``l3.meter.label_tenant``.
  
- Therefore, we will change the non-granular (``granular_traffic_data = False``)
- traffic messages (here also called as legacy), which have the following format.
+ Therefore, we will change the non-granular (``granular_traffic_data =
+ False``) traffic messages (here also called as legacy), which have the
+ following format.
  
-      {"pkts": "<the number of packets that matched the rules of the labels>",
-       "bytes": "<the number of bytes that matched the rules of the labels>",
-       "time": "<seconds between the first data collection and the last one>",
-       "first_update": "timeutils.utcnow_ts() of the first collection",
-       "last_update": "timeutils.utcnow_ts() of the last collection",
-       "host": "<neutron metering agent host name>",
-       "label_id": "<the label id>",
-       "tenant_id": "<the tenant id>"
-       }
+      {"pkts": "<the number of packets that matched the rules of the labels>",
+       "bytes": "<the number of bytes that matched the rules of the labels>",
+       "time": "<seconds between the first data collection and the last one>",
+       "first_update": "timeutils.utcnow_ts() of the first collection",
+       "last_update": "timeutils.utcnow_ts() of the last collection",
+       "host": "<neutron metering agent host name>",
+       "label_id": "<the label id>",
+       "tenant_id": "<the tenant id>"
+       }
  
- The following will be the new event message format, which will also contain
- some attributes that can be found in the legacy mode such as ``bytes``,
- ``pkts``, ``time``, ``first_update``, ``last_update``, and ``host``.
- As follows we present an example of JSON message with all of the
+ The following will be the new event message format, which will also contain some attributes that can be found in the legacy mode such as ``bytes``, ``pkts``, ``time``, ``first_update``, ``last_update``, and ``host``. As follows we present an example of JSON message with all of the
  possible attributes.
  
-      {"resource_id": "router-f0f745d9a59c47fdbbdd187d718f9e41-label-00c714f1-49c8-462c-8f5d-f05f21e035c7",
-       "tenant_id": "f0f745d9a59c47fdbbdd187d718f9e41",
-       "first_update": 1591058790,
-       "bytes": 0,
-       "label_id": "00c714f1-49c8-462c-8f5d-f05f21e035c7",
-       "label_name": "test1",
-       "last_update": 1591059037,
-       "host": "<hostname>",
-       "time": 247,
-       "pkts": 0,
-       "label_shared": true
-       }
+      {"resource_id": "router-f0f745d9a59c47fdbbdd187d718f9e41-label-00c714f1-49c8-462c-8f5d-f05f21e035c7",
+       "tenant_id": "f0f745d9a59c47fdbbdd187d718f9e41",
+       "first_update": 1591058790,
+       "bytes": 0,
+       "label_id": "00c714f1-49c8-462c-8f5d-f05f21e035c7",
+       "label_name": "test1",
+       "last_update": 1591059037,
+       "host": "<hostname>",
+       "time": 247,
+       "pkts": 0,
+       "label_shared": true
+       }
  
- The ``resource_id`` is a unique identifier for the "resource" being monitored. Here we consider a resource
- to be any of the granularities that we handle. The following is the standard for resource ID when the legacy
+ The ``resource_id`` is a unique identifier for the "resource" being monitored. Here we consider a resource to be any of the granularities that we handle. The following is the standard for resource ID when the legacy
  mode is disabled (for each granularity).
  
  * labels -- label-<label_id>
  * routers -- router-<router_id>
  * tenant -- tenant-<tenant_id>
  * router-label -- router-<router_id>-label-<label_id>
  * tenant-label -- tenant-<tenant_id>-label-<label_id>

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1886949

Title:
  [RFE] Granular metering data in neutron-metering-agent

Status in neutron:
  In Progress

Bug description:
  Problem Description
  =================
  Currently, when creating shared labels in Neutron metering, we only receive a total amount of accounting data for a label. This makes it hard for operators to identify what tenants and routers, where the labels were applied, are generating the traffic. Moreover, for shared labels, as they apply in all tenants, Neutron metering is using a random tenant_id, which can create confusion when used in production.

  Proposed Change
  ===============
  We propose to extend the neutron metering agent to enable/use a granular
  message format. To maintain backward compatibility, this feature is enabled/disabled by the following neutron-metering-agent parameter:

  * ``granular_traffic_data``: Defines if the metering agent driver should
  present traffic data in a granular fashion, instead of grouping all of the
  traffic data for all tenants and routers where the labels were assigned to. The default value is ``False`` for backward compatibility.

  When the ``granular_traffic_data`` config is set to ``True``, we have the
  following granularities:
  * label -- all of the traffic counters for a given label. One must bear in
  mind that a label (shared) can be assigned to multiple routers.
  * router -- all of the traffic counter for all labels that are assigned to
  the router.
  * tenant -- all of the traffic counters for all labels of all routers that a tenant has.
  * router-label -- all of the traffic counters for a router and the given label.
  * tenant-label -- all of the traffic counters for all routers of a tenant that have a given label.

  Each granularity presented here is sent to the message bus with different
  events types that vary according to the granularity. The mapping between
  granularity and event type is presented as follows.

  * ``label`` -- event type ``l3.meter.label``.
  * ``router`` -- event type ``l3.meter.router``.
  * ``tenant`` -- event type ``l3.meter.tenant``..
  * ``router-label`` -- event type ``l3.meter.label_router``.
  * ``tenant-label`` -- event type ``l3.meter.label_tenant``.

  Therefore, we will change the non-granular (``granular_traffic_data =
  False``) traffic messages (here also called as legacy), which have the
  following format.

       {"pkts": "<the number of packets that matched the rules of the labels>",
        "bytes": "<the number of bytes that matched the rules of the labels>",
        "time": "<seconds between the first data collection and the last one>",
        "first_update": "timeutils.utcnow_ts() of the first collection",
        "last_update": "timeutils.utcnow_ts() of the last collection",
        "host": "<neutron metering agent host name>",
        "label_id": "<the label id>",
        "tenant_id": "<the tenant id>"
        }

  The following will be the new event message format, which will also contain some attributes that can be found in the legacy mode such as ``bytes``, ``pkts``, ``time``, ``first_update``, ``last_update``, and ``host``. As follows we present an example of JSON message with all of the
  possible attributes.

       {"resource_id": "router-f0f745d9a59c47fdbbdd187d718f9e41-label-00c714f1-49c8-462c-8f5d-f05f21e035c7",
        "tenant_id": "f0f745d9a59c47fdbbdd187d718f9e41",
        "first_update": 1591058790,
        "bytes": 0,
        "label_id": "00c714f1-49c8-462c-8f5d-f05f21e035c7",
        "label_name": "test1",
        "last_update": 1591059037,
        "host": "<hostname>",
        "time": 247,
        "pkts": 0,
        "label_shared": true
        }

  The ``resource_id`` is a unique identifier for the "resource" being monitored. Here we consider a resource to be any of the granularities that we handle. The following is the standard for resource ID when the legacy
  mode is disabled (for each granularity).

  * labels -- label-<label_id>
  * routers -- router-<router_id>
  * tenant -- tenant-<tenant_id>
  * router-label -- router-<router_id>-label-<label_id>
  * tenant-label -- tenant-<tenant_id>-label-<label_id>

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1886949/+subscriptions