yahoo-eng-team team mailing list archive

[Jeremy Stanley <2084081@xxxxxxxxxxxxxxxxxx>]

Seems like there's some consensus for this being a security hardening
opportunity which we can safely work through in public, and possibly
accompany with some written guidance to administrators if anyone wants
to draft that.

** Description changed:

- This issue is being treated as a potential security risk under
- embargo. Please do not make any public mention of embargoed
- (private) security vulnerabilities before their coordinated
- publication by the OpenStack Vulnerability Management Team in the
- form of an official OpenStack Security Advisory. This includes
- discussion of the bug or associated fixes in public forums such as
- mailing lists, code review systems and bug trackers. Please also
- avoid private disclosure to other individuals not already approved
- for access to this information, and provide this same reminder to
- those who are made aware of the issue prior to publication. All
- discussion should remain confined to this private bug report, and
- any proposed fixes should be added to the bug as attachments. This
- embargo shall not extend past 2025-01-07 and will be made
- public by or on that date even if no fix is identified.
- 
  This was reported to me by a G-Research engineer, Andrew Ellard, as
  tested against Nova+Ironic unmaintained/yoga branch. I am providing
  information from him here, but will validate it in devstack and against
  other branches in the next couple of days.
  
  Add the following to nova.conf on a nova-compute instance running Ironic
  driver (I'm unsure if this also impacts other drivers). Topic names,
  urls, and paths redacted (I have not validated if this happens with
  other oslo.notification drivers):
  
  [DEFAULT]
  notification_level = info
  versioned_notifications_topics = oslo.notifications
  
  [oslo_messaging_notifications]
  driver = messagingv2
  transport_url = $kafka_url
  topics = oslo.notifications.$env.$region
  
  [oslo_messaging_kafka]
  security_protocol = SASL_SSL
  sasl_mechanism = GSSAPI
  ssl_ca_file = /path/to/tls-ca-bundle.pem
  
  Rebuild a machine, and make that rebuild fail.
  Observe that the payload in the rebuild_instance notification contains an auth_token:
  
  "priority"=>"ERROR",
  "publisher_id"=>"compute.hostname-ironic",
  "timestamp"=>"2024-10-08 14:51:23.891320",
  "payload"=>{
   "exception"=>"{'kwargs': {'code': 400}, 'message': 'Failed to provision instance 91aa8063-cf88-4dbb-8644-5787ee7d0581: None', 'class': 'InstanceDeployFailure'}",
   "args"=>{...
   }
  },
  "message_id"=>"cdaaf8b9-1bc3-4d31-be58-315dd8acb9cd",
  "_context"=>{...
  },
  "event_type"=>"rebuild_instance"
  
  Auth tokens are found in:
  - payload.args.instance.ec2_ids_context.auth_token
  - payload.args.instance_info__cache_context.auth_token
  - payload.args.request_spec_context.auth_token
  - payload.args.request.spec_obj_image_context.auth_token
  - payload.args.request_spec_obj_limits_context.auth_token
  - payload.args.request_spec_obj_security_groups_context.auth_token
  
  All of the auth_tokens appear to be valid.
  
  It's notable that this notification appears different than most
  notifications from nova (e.g. compute.instance.XXX).
  
  I know VMT does not manage unmaintained branches, but I'd request we
  keep this treated as an embargoed bug until we determine what the scope
  is.

** Changed in: ossa
       Status: Incomplete => Won't Fix

** Information type changed from Private Security to Public Security

** Information type changed from Public Security to Public

** Tags added: security

** Also affects: ossn
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2084081

Title:
  Payload of "rebuild_instance" notification contains an auth_token

Status in OpenStack Compute (nova):
  New
Status in OpenStack Security Advisory:
  Won't Fix
Status in OpenStack Security Notes:
  New

Bug description:
  This was reported to me by a G-Research engineer, Andrew Ellard, as
  tested against Nova+Ironic unmaintained/yoga branch. I am providing
  information from him here, but will validate it in devstack and
  against other branches in the next couple of days.

  Add the following to nova.conf on a nova-compute instance running
  Ironic driver (I'm unsure if this also impacts other drivers). Topic
  names, urls, and paths redacted (I have not validated if this happens
  with other oslo.notification drivers):

  [DEFAULT]
  notification_level = info
  versioned_notifications_topics = oslo.notifications

  [oslo_messaging_notifications]
  driver = messagingv2
  transport_url = $kafka_url
  topics = oslo.notifications.$env.$region

  [oslo_messaging_kafka]
  security_protocol = SASL_SSL
  sasl_mechanism = GSSAPI
  ssl_ca_file = /path/to/tls-ca-bundle.pem

  Rebuild a machine, and make that rebuild fail.
  Observe that the payload in the rebuild_instance notification contains an auth_token:

  "priority"=>"ERROR",
  "publisher_id"=>"compute.hostname-ironic",
  "timestamp"=>"2024-10-08 14:51:23.891320",
  "payload"=>{
   "exception"=>"{'kwargs': {'code': 400}, 'message': 'Failed to provision instance 91aa8063-cf88-4dbb-8644-5787ee7d0581: None', 'class': 'InstanceDeployFailure'}",
   "args"=>{...
   }
  },
  "message_id"=>"cdaaf8b9-1bc3-4d31-be58-315dd8acb9cd",
  "_context"=>{...
  },
  "event_type"=>"rebuild_instance"

  Auth tokens are found in:
  - payload.args.instance.ec2_ids_context.auth_token
  - payload.args.instance_info__cache_context.auth_token
  - payload.args.request_spec_context.auth_token
  - payload.args.request.spec_obj_image_context.auth_token
  - payload.args.request_spec_obj_limits_context.auth_token
  - payload.args.request_spec_obj_security_groups_context.auth_token

  All of the auth_tokens appear to be valid.

  It's notable that this notification appears different than most
  notifications from nova (e.g. compute.instance.XXX).

  I know VMT does not manage unmaintained branches, but I'd request we
  keep this treated as an embargoed bug until we determine what the
  scope is.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2084081/+subscriptions