yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1172551] Re: connection refused / timed out under load

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Launchpad Bug Tracker <1172551@xxxxxxxxxxxxxxxxxx>
Date: Mon, 20 Jan 2014 04:17:50 -0000
Reply-to: Bug 1172551 <1172551@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
[Expired for Keystone because there has been no activity for 60 days.]

** Changed in: keystone
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Keystone.
https://bugs.launchpad.net/bugs/1172551

Title:
  connection refused / timed out under load

Status in OpenStack Identity (Keystone):
  Expired

Bug description:
  I'm working under folsom.
  I'm dong a test for provision vms at a 100vms/mintues rate.
  Then I get error in quantum-server such as:

  2013-04-24 17:06:21    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 110] ETIMEDOUT
  2013-04-24 17:06:21    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 110] ETIMEDOUT
  2013-04-24 17:06:21    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 110] ETIMEDOUT
  2013-04-24 17:06:21    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 110] ETIMEDOUT
  2013-04-24 17:06:21    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 110] ETIMEDOUT
  2013-04-24 17:06:22    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 110] ETIMEDOUT
  2013-04-24 17:06:22    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 110] ETIMEDOUT
  2013-04-24 17:06:40    ERROR [keystone.middleware.auth_token] HTTP connection exception: ''
  2013-04-24 17:06:40    ERROR [keystone.middleware.auth_token] HTTP connection exception: ''
  2013-04-24 17:06:40    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 111] ECONNREFUSED
  2013-04-24 17:06:40    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 111] ECONNREFUSED
  2013-04-24 17:06:40    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 111] ECONNREFUSED
  2013-04-24 17:06:40    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 111] ECONNREFUSED
  2013-04-24 17:06:40    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 111] ECONNREFUSED
  2013-04-24 17:06:40    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 111] ECONNREFUSED
  2013-04-24 17:06:40    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 111] ECONNREFUSED
  2013-04-24 17:06:40    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 111] ECONNREFUSED
  2013-04-24 17:06:40    ERROR [keystone.middleware.auth_token] HTTP connection exception: [Errno 111] ECONNREFUSED

  
  But, there is no error message in keystone, it works just normal:

  
  (keystone.policy.backends.rules): 2013-04-24 17:06:21,014 DEBUG enforce admin_required: {'tenant_id': u'88ca0eddb02045a6838aca2363df2736', 'user_id': u'e1d71f67533f46b08a49a986eeaa51a2', u'roles': [u'admin']}
  (eventlet.wsgi.server): 2013-04-24 17:06:21,034 DEBUG 192.168.11.100 - - [24/Apr/2013 17:06:21] "GET /v2.0/tokens/74e2167fffed481f807d4b8a7bbb856c HTTP/1.1" 200 2235 0.026920

  (keystone.policy.backends.rules): 2013-04-24 17:06:21,041 DEBUG enforce admin_required: {'tenant_id': u'88ca0eddb02045a6838aca2363df2736', 'user_id': u'5c96c0ca03994ff3b79ef369884e12f4', u'roles': [u'admin']}
  (eventlet.wsgi.server): 2013-04-24 17:06:21,062 DEBUG 192.168.11.101 - - [24/Apr/2013 17:06:21] "GET /v2.0/tokens/a794a66a521d4553a0bc548973d4147d HTTP/1.1" 200 2235 0.027285

  (keystone.policy.backends.rules): 2013-04-24 17:06:21,069 DEBUG enforce admin_required: {'tenant_id': u'88ca0eddb02045a6838aca2363df2736', 'user_id': u'e1d71f67533f46b08a49a986eeaa51a2', u'roles': [u'admin']}
  (eventlet.wsgi.server): 2013-04-24 17:06:21,088 DEBUG 192.168.11.100 - - [24/Apr/2013 17:06:21] "GET /v2.0/tokens/56091aeab0b34e23842ded579f10e6b1 HTTP/1.1" 200 2235 0.026018

  (keystone.policy.backends.rules): 2013-04-24 17:06:21,096 DEBUG enforce admin_required: {'tenant_id': u'88ca0eddb02045a6838aca2363df2736', 'user_id': u'5c96c0ca03994ff3b79ef369884e12f4', u'roles': [u'admin']}
  (eventlet.wsgi.server): 2013-04-24 17:06:21,116 DEBUG 192.168.11.101 - - [24/Apr/2013 17:06:21] "GET /v2.0/tokens/ab9667bde1fe4c27b57672b4f05e86c2 HTTP/1.1" 200 2235 0.027765

  (keystone.policy.backends.rules): 2013-04-24 17:06:21,124 DEBUG enforce admin_required: {'tenant_id': u'88ca0eddb02045a6838aca2363df2736', 'user_id': u'5c96c0ca03994ff3b79ef369884e12f4', u'roles': [u'admin']}
  (eventlet.wsgi.server): 2013-04-24 17:06:21,144 DEBUG 192.168.11.101 - - [24/Apr/2013 17:06:21] "GET /v2.0/tokens/3ea0601c034e4a52914163e8795065f7 HTTP/1.1" 200 2235 0.027198

  (keystone.policy.backends.rules): 2013-04-24 17:06:21,151 DEBUG enforce admin_required: {'tenant_id': u'88ca0eddb02045a6838aca2363df2736', 'user_id': u'e1d71f67533f46b08a49a986eeaa51a2', u'roles': [u'admin']}
  (eventlet.wsgi.server): 2013-04-24 17:06:21,171 DEBUG 192.168.11.100 - - [24/Apr/2013 17:06:21] "GET /v2.0/tokens/547faa524e7e45a69403842ca2a89501 HTTP/1.1" 200 2238 0.026907

  (keystone.policy.backends.rules): 2013-04-24 17:06:21,178 DEBUG enforce admin_required: {'tenant_id': u'88ca0eddb02045a6838aca2363df2736', 'user_id': u'e1d71f67533f46b08a49a986eeaa51a2', u'roles': [u'admin']}
  (eventlet.wsgi.server): 2013-04-24 17:06:21,198 DEBUG 192.168.11.100 - - [24/Apr/2013 17:06:21] "GET /v2.0/tokens/7c8fcd3f03d54d7c9c42f25411fc0806 HTTP/1.1" 200 2235 0.026686

  (keystone.policy.backends.rules): 2013-04-24 17:06:21,205 DEBUG enforce admin_required: {'tenant_id': u'88ca0eddb02045a6838aca2363df2736', 'user_id': u'5c96c0ca03994ff3b79ef369884e12f4', u'roles': [u'admin']}
  (eventlet.wsgi.server): 2013-04-24 17:06:21,281 DEBUG 192.168.11.101 - - [24/Apr/2013 17:06:21] "GET /v2.0/tokens/715cbdbc6f184a3f97637f7e8180d3ec HTTP/1.1" 200 2235 0.082189

  (keystone.policy.backends.rules): 2013-04-24 17:06:21,288 DEBUG enforce admin_required: {'tenant_id': u'88ca0eddb02045a6838aca2363df2736', 'user_id': u'5c96c0ca03994ff3b79ef369884e12f4', u'roles': [u'admin']}
  (eventlet.wsgi.server): 2013-04-24 17:06:21,307 DEBUG 192.168.11.101 - - [24/Apr/2013 17:06:21] "GET /v2.0/tokens/7bf4d7f287c4407bb33482c79ff19245 HTTP/1.1" 200 2235 0.026297

  (keystone.policy.backends.rules): 2013-04-24 17:06:21,314 DEBUG enforce admin_required: {'tenant_id': u'88ca0eddb02045a6838aca2363df2736', 'user_id': u'5c96c0ca03994ff3b79ef369884e12f4', u'roles': [u'admin']}
  (eventlet.wsgi.server): 2013-04-24 17:06:21,334 DEBUG 192.168.11.101 - - [24/Apr/2013 17:06:21] "GET /v2.0/tokens/9d5736ab2cf8474c83af3da72d70954f HTTP/1.1" 200 2235 0.025823

  (keystone.policy.backends.rules): 2013-04-24 17:06:21,341 DEBUG enforce admin_required: {'tenant_id': u'88ca0eddb02045a6838aca2363df2736', 'user_id': u'5c96c0ca03994ff3b79ef369884e12f4', u'roles': [u'admin']}
  (eventlet.wsgi.server): 2013-04-24 17:06:21,361 DEBUG 192.168.11.101 - - [24/Apr/2013 17:06:21] "GET /v2.0/tokens/5f8b9afbd02444fd9238ec24a3f225f9 HTTP/1.1" 200 2235 0.026556

  (keystone.policy.backends.rules): 2013-04-24 17:06:21,368 DEBUG enforce admin_required: {'tenant_id': u'88ca0eddb02045a6838aca2363df2736', 'user_id': u'5c96c0ca03994ff3b79ef369884e12f4', u'roles': [u'admin']}
  (eventlet.wsgi.server): 2013-04-24 17:06:21,388 DEBUG 192.168.11.101 - - [24/Apr/2013 17:06:21] "GET /v2.0/tokens/f89fd3062ace4ff988193e441176ff85 HTTP/1.1" 200 2235 0.026680

  (keystone.policy.backends.rules): 2013-04-24 17:06:21,394 DEBUG enforce admin_required: {'tenant_id': u'88ca0eddb02045a6838aca2363df2736', 'user_id': u'5c96c0ca03994ff3b79ef369884e12f4', u'roles': [u'admin']}
  (eventlet.wsgi.server): 2013-04-24 17:06:21,414 DEBUG 192.168.11.101 - - [24/Apr/2013 17:06:21] "GET /v2.0/tokens/fa8a9a3e232a49bdb922940e0f8c3d08 HTTP/1.1" 200 2235 0.026036

  (keystone.policy.backends.rules): 2013-04-24 17:06:21,421 DEBUG enforce admin_required: {'tenant_id': u'88ca0eddb02045a6838aca2363df2736', 'user_id': u'5c96c0ca03994ff3b79ef369884e12f4', u'roles': [u'admin']}
  (eventlet.wsgi.server): 2013-04-24 17:06:21,440 DEBUG 192.168.11.101 - - [24/Apr/2013 17:06:21] "GET /v2.0/tokens/6205e8e5f232450f9aaaf48b21f7a949 HTTP/1.1" 200 2235 0.025824

  (keystone.policy.backends.rules): 2013-04-24 17:06:21,447 DEBUG enforce admin_required: {'tenant_id': u'88ca0eddb02045a6838aca2363df2736', 'user_id': u'e1d71f67533f46b08a49a986eeaa51a2', u'roles': [u'admin']}
  (eventlet.wsgi.server): 2013-04-24 17:06:21,466 DEBUG 192.168.11.100 - - [24/Apr/2013 17:06:21] "GET /v2.0/tokens/770e876e42b842fb9f200af3c186641e HTTP/1.1" 200 2235 0.025708

  
  Anyone know why this happen??
  I

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1172551/+subscriptions