← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #76069

[Bug 1806377] [NEW] admin/internal/public endpoint creation failed due to to pymysql.err.IntegrityError in keystone service

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

This is hit with both Queens & Rocky branches

Request sent to create keystone endpoints, failed with error "An
unexpected error prevented the server from fulfilling your request.
(HTTP 500)"

TASK [create identity endpoints] ***********************************************
failed: [osctrl01] (item={u'url': u'http://60.0.25.189:5000/v3', u'type': u'public'}) => {"changed": true, "cmd": ["/opt/mhos/python/bin/openstack", "endpoint", "create", "--region", "SC2", "identity", "public", "http://60.0.25.189:5000/v3";], "delta": "0:00:00.610018", "end": "2018-12-01 07:15:56.894998", "failed": true, "item": {"type": "public", "url": "http://60.0.25.189:5000/v3"}, "rc": 1, "start": "2018-12-01 07:15:56.284980", "stderr": "An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-125826e3-7c54-4ee4-bb47-69c2a59e9d06)", "stdout": "", "stdout_lines": [], "warnings": []}

failed: [osctrl01] (item={u'url': u'http://60.0.25.188:5000/v3',
u'type': u'internal'}) => {"changed": true, "cmd":
["/opt/mhos/python/bin/openstack", "endpoint", "create", "--region",
"SC2", "identity", "internal", "http://60.0.25.188:5000/v3";], "delta":
"0:00:00.741814", "end": "2018-12-01 07:15:57.787091", "failed": true,
"item": {"type": "internal", "url": "http://60.0.25.188:5000/v3"}, "rc":
1, "start": "2018-12-01 07:15:57.045277", "stderr": "An unexpected error
prevented the server from fulfilling your request. (HTTP 500) (Request-
ID: req-ac52526c-3450-44d5-b13c-42d9040c2fc5)", "stdout": "",
"stdout_lines": [], "warnings": []}

failed: [osctrl01] (item={u'url': u'http://60.0.25.188:35357/v3', u'type': u'admin'}) => {"changed": true, "cmd": ["/opt/mhos/python/bin/openstack", "endpoint", "create", "--region", "SC2", "identity", "admin", "http://60.0.25.188:35357/v3";], "delta": "0:00:00.599662", "end": "2018-12-01 07:15:58.543126", "failed": true, "item": {"type": "admin", "url": "http://60.0.25.188:35357/v3"}, "rc": 1, "start": "2018-12-01 07:15:57.943464", "stderr": "An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-629c6181-433b-4e98-9220-a8028f1dc17a)", "stdout": "", "stdout_lines": [], "warnings": []}
	to retry, use: --limit @/tmp/ansible_os_run/3825619e-6adf-4a9e-a681-d03a0b7e8396/nsbu_cqe_openstack/initialize/site.retry

PLAY RECAP *********************************************************************
osctrl01                   : ok=8    changed=5    unreachable=0    failed=1   
osctrl02                   : ok=3    changed=3    unreachable=0    failed=0   


Keystone service unavailable server error:

TASK [create openstack id and add plugin] **************************************
changed: [osctrl01]

TASK [debug] *******************************************************************
ok: [osctrl01] => {
    "openstack_token_id.stdout.split()[-2]": "gAAAAABcAi1-eWXyCq7q47S4CpE1PQQAD6xo7l-ngoI6D2Re58lymp0goziGxi2bOm3G5j5_dnBpWHE6pd-0okoETLVtc9w1bDo5LGD69BL7wIW69hbzsmJPnrKKTzdnV2MIEeNU49PfMILD6t_6SNpVMQ_BN5RULVK0tq20mJ09qiQUAy94hQg"
}

TASK [uri] *********************************************************************
fatal: [osctrl01]: FAILED! => {"cache_control": "no-cache", "changed": false, "connection": "close", "content": "<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n", "content_type": "text/html", "failed": true, "msg": "Status code was not [201]: HTTP Error 503: Service Unavailable", "redirected": false, "status": 503, "url": "http://60.0.25.188:9696/v2.0/project-plugin-maps"}
	to retry, use: --limit @/tmp/ansible_os_run/3825619e-6adf-4a9e-a681-d03a0b7e8396/nsbu_cqe_openstack/initialize/site.retry

PLAY RECAP *********************************************************************
oscompute01                : ok=3    changed=2    unreachable=0    failed=0   
oscompute02                : ok=2    changed=2    unreachable=0    failed=0   
oscompute03                : ok=2    changed=2    unreachable=0    failed=0   
osctrl01                   : ok=33   changed=28   unreachable=0    failed=1   
osctrl02                   : ok=3    changed=3    unreachable=0    failed=0   
oskvm01                    : ok=3    changed=3    unreachable=0    failed=0   
oskvm02                    : ok=3    changed=3    unreachable=0    failed=0   

2018-12-01 12:12:04,124 INFO Exit code:512
2018-12-01 12:12:04,125 WARNING Either install or initialize failed
2018-12-01 12:12:04,125 INFO Executing cmd:sudo ansible-playbook -i etc/tb201_nsxtvd_ansible_invt_hosts etc/uninstall_site.yml -e "host_key_checking=False ansible_user=root ansible_ssh_pass=stack building=True ansible_ssh_common_args='-o StrictHostKeyChecking=no'"


keystone log trace,

admin.log:2018-12-01 07:15:57.735 4580 DEBUG keystone.middleware.auth [req-125826e3-7c54-4ee4-bb47-69c2a59e9d06 - - - - -] RBAC: auth_context: {} fill_context /opt/mhos/openstack/keystone/lib/python2.7/site-packages/keystone/middleware/auth.py:247
admin.log:2018-12-01 07:15:57.736 4580 INFO keystone.common.wsgi [req-125826e3-7c54-4ee4-bb47-69c2a59e9d06 - - - - -] POST http://60.0.25.188:35357/v3/endpoints
admin.log:2018-12-01 07:15:57.737 4580 WARNING keystone.common.controller [req-125826e3-7c54-4ee4-bb47-69c2a59e9d06 - - - - -] RBAC: Bypassing authorization
admin.log:2018-12-01 07:15:57.749 4580 ERROR keystone.common.wsgi [req-125826e3-7c54-4ee4-bb47-69c2a59e9d06 - - - - -] (pymysql.err.IntegrityError) (1452, u'Cannot add or update a child row: a foreign key constraint fails (`keystone`.`endpoint`, CONSTRAINT `fk_endpoint_region_id` FOREIGN KEY (`region_id`) REFERENCES `region` (`id`))') [SQL: u'INSERT INTO endpoint (id, legacy_endpoint_id, interface, region_id, service_id, url, enabled, extra) VALUES (%(id)s, %(legacy_endpoint_id)s, %(interface)s, %(region_id)s, %(service_id)s, %(url)s, %(enabled)s, %(extra)s)'] [parameters: {'region_id': u'SC2', 'extra': '{}', 'url': u'http://60.0.25.189:5000/v3', 'enabled': 1, 'legacy_endpoint_id': None, 'interface': u'public', 'service_id': u'bc2f72f76d3a44cf81cd05bc581c1cda', 'id': 'dea654331d4644b89689ec452edb9fed'}] (Background on this error at: http://sqlalche.me/e/gkpj): DBReferenceError: (pymysql.err.IntegrityError) (1452, u'Cannot add or update a child row: a foreign key constraint fails (`keystone`.`endpoint`, CONSTRAINT `fk_endpoint_region_id` FOREIGN KEY (`region_id`) REFERENCES `region` (`id`))') [SQL: u'INSERT INTO endpoint (id, legacy_endpoint_id, interface, region_id, service_id, url, enabled, extra) VALUES (%(id)s, %(legacy_endpoint_id)s, %(interface)s, %(region_id)s, %(service_id)s, %(url)s, %(enabled)s, %(extra)s)'] [parameters: {'region_id': u'SC2', 'extra': '{}', 'url': u'http://60.0.25.189:5000/v3', 'enabled': 1, 'legacy_endpoint_id': None, 'interface': u'public', 'service_id': u'bc2f72f76d3a44cf81cd05bc581c1cda', 'id': 'dea654331d4644b89689ec452edb9fed'}] (Background on this error at: http://sqlalche.me/e/gkpj)

** Affects: keystone
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1806377

Title:
  admin/internal/public endpoint creation failed due to to
  pymysql.err.IntegrityError in keystone service

Status in OpenStack Identity (keystone):
  New

Bug description:
  This is hit with both Queens & Rocky branches

  Request sent to create keystone endpoints, failed with error "An
  unexpected error prevented the server from fulfilling your request.
  (HTTP 500)"

  TASK [create identity endpoints] ***********************************************
  failed: [osctrl01] (item={u'url': u'http://60.0.25.189:5000/v3', u'type': u'public'}) => {"changed": true, "cmd": ["/opt/mhos/python/bin/openstack", "endpoint", "create", "--region", "SC2", "identity", "public", "http://60.0.25.189:5000/v3";], "delta": "0:00:00.610018", "end": "2018-12-01 07:15:56.894998", "failed": true, "item": {"type": "public", "url": "http://60.0.25.189:5000/v3"}, "rc": 1, "start": "2018-12-01 07:15:56.284980", "stderr": "An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-125826e3-7c54-4ee4-bb47-69c2a59e9d06)", "stdout": "", "stdout_lines": [], "warnings": []}

  failed: [osctrl01] (item={u'url': u'http://60.0.25.188:5000/v3',
  u'type': u'internal'}) => {"changed": true, "cmd":
  ["/opt/mhos/python/bin/openstack", "endpoint", "create", "--region",
  "SC2", "identity", "internal", "http://60.0.25.188:5000/v3";], "delta":
  "0:00:00.741814", "end": "2018-12-01 07:15:57.787091", "failed": true,
  "item": {"type": "internal", "url": "http://60.0.25.188:5000/v3"},
  "rc": 1, "start": "2018-12-01 07:15:57.045277", "stderr": "An
  unexpected error prevented the server from fulfilling your request.
  (HTTP 500) (Request-ID: req-ac52526c-3450-44d5-b13c-42d9040c2fc5)",
  "stdout": "", "stdout_lines": [], "warnings": []}

  failed: [osctrl01] (item={u'url': u'http://60.0.25.188:35357/v3', u'type': u'admin'}) => {"changed": true, "cmd": ["/opt/mhos/python/bin/openstack", "endpoint", "create", "--region", "SC2", "identity", "admin", "http://60.0.25.188:35357/v3";], "delta": "0:00:00.599662", "end": "2018-12-01 07:15:58.543126", "failed": true, "item": {"type": "admin", "url": "http://60.0.25.188:35357/v3"}, "rc": 1, "start": "2018-12-01 07:15:57.943464", "stderr": "An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-629c6181-433b-4e98-9220-a8028f1dc17a)", "stdout": "", "stdout_lines": [], "warnings": []}
  	to retry, use: --limit @/tmp/ansible_os_run/3825619e-6adf-4a9e-a681-d03a0b7e8396/nsbu_cqe_openstack/initialize/site.retry

  PLAY RECAP *********************************************************************
  osctrl01                   : ok=8    changed=5    unreachable=0    failed=1   
  osctrl02                   : ok=3    changed=3    unreachable=0    failed=0   


  Keystone service unavailable server error:

  TASK [create openstack id and add plugin] **************************************
  changed: [osctrl01]

  TASK [debug] *******************************************************************
  ok: [osctrl01] => {
      "openstack_token_id.stdout.split()[-2]": "gAAAAABcAi1-eWXyCq7q47S4CpE1PQQAD6xo7l-ngoI6D2Re58lymp0goziGxi2bOm3G5j5_dnBpWHE6pd-0okoETLVtc9w1bDo5LGD69BL7wIW69hbzsmJPnrKKTzdnV2MIEeNU49PfMILD6t_6SNpVMQ_BN5RULVK0tq20mJ09qiQUAy94hQg"
  }

  TASK [uri] *********************************************************************
  fatal: [osctrl01]: FAILED! => {"cache_control": "no-cache", "changed": false, "connection": "close", "content": "<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n", "content_type": "text/html", "failed": true, "msg": "Status code was not [201]: HTTP Error 503: Service Unavailable", "redirected": false, "status": 503, "url": "http://60.0.25.188:9696/v2.0/project-plugin-maps"}
  	to retry, use: --limit @/tmp/ansible_os_run/3825619e-6adf-4a9e-a681-d03a0b7e8396/nsbu_cqe_openstack/initialize/site.retry

  PLAY RECAP *********************************************************************
  oscompute01                : ok=3    changed=2    unreachable=0    failed=0   
  oscompute02                : ok=2    changed=2    unreachable=0    failed=0   
  oscompute03                : ok=2    changed=2    unreachable=0    failed=0   
  osctrl01                   : ok=33   changed=28   unreachable=0    failed=1   
  osctrl02                   : ok=3    changed=3    unreachable=0    failed=0   
  oskvm01                    : ok=3    changed=3    unreachable=0    failed=0   
  oskvm02                    : ok=3    changed=3    unreachable=0    failed=0   

  2018-12-01 12:12:04,124 INFO Exit code:512
  2018-12-01 12:12:04,125 WARNING Either install or initialize failed
  2018-12-01 12:12:04,125 INFO Executing cmd:sudo ansible-playbook -i etc/tb201_nsxtvd_ansible_invt_hosts etc/uninstall_site.yml -e "host_key_checking=False ansible_user=root ansible_ssh_pass=stack building=True ansible_ssh_common_args='-o StrictHostKeyChecking=no'"

  
  keystone log trace,

  admin.log:2018-12-01 07:15:57.735 4580 DEBUG keystone.middleware.auth [req-125826e3-7c54-4ee4-bb47-69c2a59e9d06 - - - - -] RBAC: auth_context: {} fill_context /opt/mhos/openstack/keystone/lib/python2.7/site-packages/keystone/middleware/auth.py:247
  admin.log:2018-12-01 07:15:57.736 4580 INFO keystone.common.wsgi [req-125826e3-7c54-4ee4-bb47-69c2a59e9d06 - - - - -] POST http://60.0.25.188:35357/v3/endpoints
  admin.log:2018-12-01 07:15:57.737 4580 WARNING keystone.common.controller [req-125826e3-7c54-4ee4-bb47-69c2a59e9d06 - - - - -] RBAC: Bypassing authorization
  admin.log:2018-12-01 07:15:57.749 4580 ERROR keystone.common.wsgi [req-125826e3-7c54-4ee4-bb47-69c2a59e9d06 - - - - -] (pymysql.err.IntegrityError) (1452, u'Cannot add or update a child row: a foreign key constraint fails (`keystone`.`endpoint`, CONSTRAINT `fk_endpoint_region_id` FOREIGN KEY (`region_id`) REFERENCES `region` (`id`))') [SQL: u'INSERT INTO endpoint (id, legacy_endpoint_id, interface, region_id, service_id, url, enabled, extra) VALUES (%(id)s, %(legacy_endpoint_id)s, %(interface)s, %(region_id)s, %(service_id)s, %(url)s, %(enabled)s, %(extra)s)'] [parameters: {'region_id': u'SC2', 'extra': '{}', 'url': u'http://60.0.25.189:5000/v3', 'enabled': 1, 'legacy_endpoint_id': None, 'interface': u'public', 'service_id': u'bc2f72f76d3a44cf81cd05bc581c1cda', 'id': 'dea654331d4644b89689ec452edb9fed'}] (Background on this error at: http://sqlalche.me/e/gkpj): DBReferenceError: (pymysql.err.IntegrityError) (1452, u'Cannot add or update a child row: a foreign key constraint fails (`keystone`.`endpoint`, CONSTRAINT `fk_endpoint_region_id` FOREIGN KEY (`region_id`) REFERENCES `region` (`id`))') [SQL: u'INSERT INTO endpoint (id, legacy_endpoint_id, interface, region_id, service_id, url, enabled, extra) VALUES (%(id)s, %(legacy_endpoint_id)s, %(interface)s, %(region_id)s, %(service_id)s, %(url)s, %(enabled)s, %(extra)s)'] [parameters: {'region_id': u'SC2', 'extra': '{}', 'url': u'http://60.0.25.189:5000/v3', 'enabled': 1, 'legacy_endpoint_id': None, 'interface': u'public', 'service_id': u'bc2f72f76d3a44cf81cd05bc581c1cda', 'id': 'dea654331d4644b89689ec452edb9fed'}] (Background on this error at: http://sqlalche.me/e/gkpj)

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1806377/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next