← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #91605

[Bug 2012993] [NEW] Unredacted X-Auth-Token logged at level DEBUG in nova-api when HTTP status code != 2xx

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Noticed this while working on something else, if the API is going to
return a non 2xx HTTP success status code, a lot of request details are
logged including the user's unsanitized auth token. In the past,
operators considered this to be a security issue despite logging only at
level DEBUG. For this reason I am opening a bug for review.

This particular logging code was added in the Zed release:

https://review.opendev.org/c/openstack/nova/+/806683

These are logged a lot when using OSC + server names because OSC always
tries to lookup a name as a UUID (which will fail with 404) before it
falls back on trying it as an ID. So commands such as 'openstack server
show MyServer' will produce debug logs like the following.

Example log for GET /servers HTTP 404:

Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: INFO nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] HTTP exception thrown: Instance test could not be found.
Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: DEBUG nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] Request method failure captured:
Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]:   request: GET /compute/v2.1/servers/test HTTP/1.1
Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Accept: application/json
Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Accept-Encoding: gzip, deflate
Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Connection: keep-alive
Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Content-Length: 0
Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Host: 192.168.44.11
Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Openstack-System-Scope: None
Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: User-Agent: python-novaclient
Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: X-Auth-Token: gAAAAABkIj7cGDJYKbnQalHot3qfNuAY1AMwMdKFP0kVQB6_8HRCSizDQpxMfspjx2S7t8rMWPSwYwg0-Yox2QM9E3KPWVq72YPl-cr8XwlDIn-ev9WjpmkmCtlOqOs0M0rOSvQggxdNB0xLx
2-gYiWUyMAYW6A1vXcup7Rvs8-YFetPKr2vGnY

[...]

Full log trace:

https://paste.openstack.org/show/bx9CmbgOsDNrIn16PfUW

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: api security

** Description changed:

  Noticed this while working on something else, if the API is going to
  return a non 2xx HTTP success status code, a lot of request details are
  logged including the user's unsanitized auth token. In the past,
  operators considered this to be a security issue despite logging only at
- level DEBUG. For this reason I am opening a bug for review.
+ level DEBUG. For this reason, I am opening a bug for review.
  
  Example for GET /servers HTTP 404:
  
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: INFO nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] HTTP exception thrown: Instance test could not be found.
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: DEBUG nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] Request method failure captured:
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]:   request: GET /compute/v2.1/servers/test HTTP/1.1
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Accept: application/json
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Accept-Encoding: gzip, deflate
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Connection: keep-alive
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Content-Length: 0
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Host: 192.168.44.11
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Openstack-System-Scope: None
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: User-Agent: python-novaclient
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: X-Auth-Token: gAAAAABkIj7cGDJYKbnQalHot3qfNuAY1AMwMdKFP0kVQB6_8HRCSizDQpxMfspjx2S7t8rMWPSwYwg0-Yox2QM9E3KPWVq72YPl-cr8XwlDIn-ev9WjpmkmCtlOqOs0M0rOSvQggxdNB0xLx
  2-gYiWUyMAYW6A1vXcup7Rvs8-YFetPKr2vGnY
  
  [...]
  
  Full log trace:
  
  https://paste.openstack.org/show/bx9CmbgOsDNrIn16PfUW

** Description changed:

  Noticed this while working on something else, if the API is going to
  return a non 2xx HTTP success status code, a lot of request details are
  logged including the user's unsanitized auth token. In the past,
  operators considered this to be a security issue despite logging only at
- level DEBUG. For this reason, I am opening a bug for review.
+ level DEBUG. For this reason I am opening a bug for review.
  
- Example for GET /servers HTTP 404:
+ This particular logging code was added in the Zed release:
+ 
+ https://review.opendev.org/c/openstack/nova/+/806683
+ 
+ Example log for GET /servers HTTP 404:
  
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: INFO nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] HTTP exception thrown: Instance test could not be found.
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: DEBUG nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] Request method failure captured:
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]:   request: GET /compute/v2.1/servers/test HTTP/1.1
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Accept: application/json
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Accept-Encoding: gzip, deflate
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Connection: keep-alive
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Content-Length: 0
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Host: 192.168.44.11
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Openstack-System-Scope: None
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: User-Agent: python-novaclient
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: X-Auth-Token: gAAAAABkIj7cGDJYKbnQalHot3qfNuAY1AMwMdKFP0kVQB6_8HRCSizDQpxMfspjx2S7t8rMWPSwYwg0-Yox2QM9E3KPWVq72YPl-cr8XwlDIn-ev9WjpmkmCtlOqOs0M0rOSvQggxdNB0xLx
  2-gYiWUyMAYW6A1vXcup7Rvs8-YFetPKr2vGnY
  
  [...]
  
  Full log trace:
  
  https://paste.openstack.org/show/bx9CmbgOsDNrIn16PfUW

** Description changed:

  Noticed this while working on something else, if the API is going to
  return a non 2xx HTTP success status code, a lot of request details are
  logged including the user's unsanitized auth token. In the past,
  operators considered this to be a security issue despite logging only at
  level DEBUG. For this reason I am opening a bug for review.
  
  This particular logging code was added in the Zed release:
  
  https://review.opendev.org/c/openstack/nova/+/806683
+ 
+ These are logged a lot when using OSC + server names because OSC always
+ tries to lookup a name as a UUID (which will fail with 404) before it
+ falls back on trying it as an ID. So commands such as 'openstack server
+ show MyVM' will produce debug logs like the following.
  
  Example log for GET /servers HTTP 404:
  
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: INFO nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] HTTP exception thrown: Instance test could not be found.
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: DEBUG nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] Request method failure captured:
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]:   request: GET /compute/v2.1/servers/test HTTP/1.1
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Accept: application/json
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Accept-Encoding: gzip, deflate
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Connection: keep-alive
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Content-Length: 0
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Host: 192.168.44.11
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Openstack-System-Scope: None
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: User-Agent: python-novaclient
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: X-Auth-Token: gAAAAABkIj7cGDJYKbnQalHot3qfNuAY1AMwMdKFP0kVQB6_8HRCSizDQpxMfspjx2S7t8rMWPSwYwg0-Yox2QM9E3KPWVq72YPl-cr8XwlDIn-ev9WjpmkmCtlOqOs0M0rOSvQggxdNB0xLx
  2-gYiWUyMAYW6A1vXcup7Rvs8-YFetPKr2vGnY
  
  [...]
  
  Full log trace:
  
  https://paste.openstack.org/show/bx9CmbgOsDNrIn16PfUW

** Description changed:

  Noticed this while working on something else, if the API is going to
  return a non 2xx HTTP success status code, a lot of request details are
  logged including the user's unsanitized auth token. In the past,
  operators considered this to be a security issue despite logging only at
  level DEBUG. For this reason I am opening a bug for review.
  
  This particular logging code was added in the Zed release:
  
  https://review.opendev.org/c/openstack/nova/+/806683
  
  These are logged a lot when using OSC + server names because OSC always
  tries to lookup a name as a UUID (which will fail with 404) before it
  falls back on trying it as an ID. So commands such as 'openstack server
- show MyVM' will produce debug logs like the following.
+ show MyServer' will produce debug logs like the following.
  
  Example log for GET /servers HTTP 404:
  
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: INFO nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] HTTP exception thrown: Instance test could not be found.
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: DEBUG nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] Request method failure captured:
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]:   request: GET /compute/v2.1/servers/test HTTP/1.1
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Accept: application/json
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Accept-Encoding: gzip, deflate
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Connection: keep-alive
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Content-Length: 0
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Host: 192.168.44.11
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Openstack-System-Scope: None
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: User-Agent: python-novaclient
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: X-Auth-Token: gAAAAABkIj7cGDJYKbnQalHot3qfNuAY1AMwMdKFP0kVQB6_8HRCSizDQpxMfspjx2S7t8rMWPSwYwg0-Yox2QM9E3KPWVq72YPl-cr8XwlDIn-ev9WjpmkmCtlOqOs0M0rOSvQggxdNB0xLx
  2-gYiWUyMAYW6A1vXcup7Rvs8-YFetPKr2vGnY
  
  [...]
  
  Full log trace:
  
  https://paste.openstack.org/show/bx9CmbgOsDNrIn16PfUW

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2012993

Title:
  Unredacted X-Auth-Token logged at level DEBUG in nova-api when HTTP
  status code != 2xx

Status in OpenStack Compute (nova):
  New

Bug description:
  Noticed this while working on something else, if the API is going to
  return a non 2xx HTTP success status code, a lot of request details
  are logged including the user's unsanitized auth token. In the past,
  operators considered this to be a security issue despite logging only
  at level DEBUG. For this reason I am opening a bug for review.

  This particular logging code was added in the Zed release:

  https://review.opendev.org/c/openstack/nova/+/806683

  These are logged a lot when using OSC + server names because OSC
  always tries to lookup a name as a UUID (which will fail with 404)
  before it falls back on trying it as an ID. So commands such as
  'openstack server show MyServer' will produce debug logs like the
  following.

  Example log for GET /servers HTTP 404:

  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: INFO nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] HTTP exception thrown: Instance test could not be found.
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: DEBUG nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] Request method failure captured:
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]:   request: GET /compute/v2.1/servers/test HTTP/1.1
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Accept: application/json
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Accept-Encoding: gzip, deflate
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Connection: keep-alive
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Content-Length: 0
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Host: 192.168.44.11
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: Openstack-System-Scope: None
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: User-Agent: python-novaclient
  Mar 28 01:11:57 ubuntu-focal devstack@n-api.service[3890335]: X-Auth-Token: gAAAAABkIj7cGDJYKbnQalHot3qfNuAY1AMwMdKFP0kVQB6_8HRCSizDQpxMfspjx2S7t8rMWPSwYwg0-Yox2QM9E3KPWVq72YPl-cr8XwlDIn-ev9WjpmkmCtlOqOs0M0rOSvQggxdNB0xLx
  2-gYiWUyMAYW6A1vXcup7Rvs8-YFetPKr2vGnY

  [...]

  Full log trace:

  https://paste.openstack.org/show/bx9CmbgOsDNrIn16PfUW

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2012993/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next