← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #93149

[Bug 2045287] [NEW] Error in charm when bootstrapping OpenStack with Sunbeam

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

When trying to bootstrap OpenStack using Sunbeam, there is an error in
the charm of neutron, which looks like the following:

* In app view:

neutron                                            waiting      1
neutron-k8s                2023.1/stable   53  10.152.183.36   no
installing agent

* In unit view:

neutron/0*                   blocked   idle   10.1.163.29
(workload) Error in charm (see logs): timed out waiting for change 2
(301 seconds)

* Looking at the logs for the error, they look like this:

2023-11-30T09:27:16.903Z [container-agent] 2023-11-30 09:27:16 INFO juju-log identity-service:85: Syncing database...
2023-11-30T09:32:18.000Z [container-agent] 2023-11-30 09:32:18 ERROR juju-log identity-service:85: Exception raised in section 'Bootstrapping': timed out waiting for change 2 (301 seconds)
2023-11-30T09:32:18.008Z [container-agent] 2023-11-30 09:32:18 ERROR juju-log identity-service:85: Traceback (most recent call last):
2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/guard.py", line 91, in guard
2023-11-30T09:32:18.008Z [container-agent]     yield
2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/charm.py", line 265, in configure_charm
2023-11-30T09:32:18.008Z [container-agent]     self.configure_unit(event)
2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/charm.py", line 479, in configure_unit
2023-11-30T09:32:18.008Z [container-agent]     self.run_db_sync()
2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/job_ctrl.py", line 74, in wrapped_f
2023-11-30T09:32:18.008Z [container-agent]     f(charm, *args, **kwargs)
2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/./src/charm.py", line 302, in run_db_sync
2023-11-30T09:32:18.008Z [container-agent]     super().run_db_sync()
2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/job_ctrl.py", line 74, in wrapped_f
2023-11-30T09:32:18.008Z [container-agent]     f(charm, *args, **kwargs)
2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/charm.py", line 549, in run_db_sync
2023-11-30T09:32:18.008Z [container-agent]     self._retry_db_sync(cmd)
2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/tenacity/__init__.py", line 289, in wrapped_f
2023-11-30T09:32:18.008Z [container-agent]     return self(f, *args, **kw)
2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/tenacity/__init__.py", line 379, in __call__
2023-11-30T09:32:18.008Z [container-agent]     do = self.iter(retry_state=retry_state)
2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/tenacity/__init__.py", line 314, in iter
2023-11-30T09:32:18.008Z [container-agent]     return fut.result()
2023-11-30T09:32:18.008Z [container-agent]   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2023-11-30T09:32:18.008Z [container-agent]     return self.__get_result()
2023-11-30T09:32:18.008Z [container-agent]   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-11-30T09:32:18.008Z [container-agent]     raise self._exception
2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/tenacity/__init__.py", line 382, in __call__
2023-11-30T09:32:18.008Z [container-agent]     result = fn(*args, **kwargs)
2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/charm.py", line 529, in _retry_db_sync
2023-11-30T09:32:18.008Z [container-agent]     out, warnings = process.wait_output()
2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops/pebble.py", line 1354, in wait_output
2023-11-30T09:32:18.008Z [container-agent]     exit_code: int = self._wait()
2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops/pebble.py", line 1294, in _wait
2023-11-30T09:32:18.008Z [container-agent]     change = self._client.wait_change(self._change_id, timeout=timeout)
2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops/pebble.py", line 1794, in wait_change
2023-11-30T09:32:18.008Z [container-agent]     return self._wait_change_using_wait(change_id, timeout)
2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops/pebble.py", line 1820, in _wait_change_using_wait
2023-11-30T09:32:18.008Z [container-agent]     raise TimeoutError(f'timed out waiting for change {change_id} ({timeout} seconds)')
2023-11-30T09:32:18.008Z [container-agent] ops.pebble.TimeoutError: timed out waiting for change 2 (301 seconds)
2023-11-30T09:32:18.008Z [container-agent]
2023-11-30T09:33:09.704Z [container-agent] 2023-11-30 09:33:09 INFO juju.worker.uniter.operation runhook.go:186 ran "identity-service-relation-changed" hook (via hook dispatching script: dispatch)
2023-11-30T09:33:14.730Z [container-agent] 2023-11-30 09:33:14 INFO juju.worker.uniter.operation runhook.go:186 ran "update-status" hook (via hook dispatching script: dispatch)

* This has not allowed me to get past the bootstrapping process to start
using OpenStack as 22 of the 23 dependancies are up only.

I am running everything with the following commands:
--> sudo snap install openstack
--> sunbeam prepare-node-script | bash -x && newgrp snap_daemon
--> sunbeam cluster bootstrap --accept-defaults

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2045287

Title:
  Error in charm when bootstrapping OpenStack with Sunbeam

Status in neutron:
  New

Bug description:
  When trying to bootstrap OpenStack using Sunbeam, there is an error in
  the charm of neutron, which looks like the following:

  * In app view:

  neutron                                            waiting      1
  neutron-k8s                2023.1/stable   53  10.152.183.36   no
  installing agent

  * In unit view:

  neutron/0*                   blocked   idle   10.1.163.29
  (workload) Error in charm (see logs): timed out waiting for change 2
  (301 seconds)

  * Looking at the logs for the error, they look like this:

  2023-11-30T09:27:16.903Z [container-agent] 2023-11-30 09:27:16 INFO juju-log identity-service:85: Syncing database...
  2023-11-30T09:32:18.000Z [container-agent] 2023-11-30 09:32:18 ERROR juju-log identity-service:85: Exception raised in section 'Bootstrapping': timed out waiting for change 2 (301 seconds)
  2023-11-30T09:32:18.008Z [container-agent] 2023-11-30 09:32:18 ERROR juju-log identity-service:85: Traceback (most recent call last):
  2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/guard.py", line 91, in guard
  2023-11-30T09:32:18.008Z [container-agent]     yield
  2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/charm.py", line 265, in configure_charm
  2023-11-30T09:32:18.008Z [container-agent]     self.configure_unit(event)
  2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/charm.py", line 479, in configure_unit
  2023-11-30T09:32:18.008Z [container-agent]     self.run_db_sync()
  2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/job_ctrl.py", line 74, in wrapped_f
  2023-11-30T09:32:18.008Z [container-agent]     f(charm, *args, **kwargs)
  2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/./src/charm.py", line 302, in run_db_sync
  2023-11-30T09:32:18.008Z [container-agent]     super().run_db_sync()
  2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/job_ctrl.py", line 74, in wrapped_f
  2023-11-30T09:32:18.008Z [container-agent]     f(charm, *args, **kwargs)
  2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/charm.py", line 549, in run_db_sync
  2023-11-30T09:32:18.008Z [container-agent]     self._retry_db_sync(cmd)
  2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/tenacity/__init__.py", line 289, in wrapped_f
  2023-11-30T09:32:18.008Z [container-agent]     return self(f, *args, **kw)
  2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/tenacity/__init__.py", line 379, in __call__
  2023-11-30T09:32:18.008Z [container-agent]     do = self.iter(retry_state=retry_state)
  2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/tenacity/__init__.py", line 314, in iter
  2023-11-30T09:32:18.008Z [container-agent]     return fut.result()
  2023-11-30T09:32:18.008Z [container-agent]   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
  2023-11-30T09:32:18.008Z [container-agent]     return self.__get_result()
  2023-11-30T09:32:18.008Z [container-agent]   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
  2023-11-30T09:32:18.008Z [container-agent]     raise self._exception
  2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/tenacity/__init__.py", line 382, in __call__
  2023-11-30T09:32:18.008Z [container-agent]     result = fn(*args, **kwargs)
  2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops_sunbeam/charm.py", line 529, in _retry_db_sync
  2023-11-30T09:32:18.008Z [container-agent]     out, warnings = process.wait_output()
  2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops/pebble.py", line 1354, in wait_output
  2023-11-30T09:32:18.008Z [container-agent]     exit_code: int = self._wait()
  2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops/pebble.py", line 1294, in _wait
  2023-11-30T09:32:18.008Z [container-agent]     change = self._client.wait_change(self._change_id, timeout=timeout)
  2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops/pebble.py", line 1794, in wait_change
  2023-11-30T09:32:18.008Z [container-agent]     return self._wait_change_using_wait(change_id, timeout)
  2023-11-30T09:32:18.008Z [container-agent]   File "/var/lib/juju/agents/unit-neutron-0/charm/venv/ops/pebble.py", line 1820, in _wait_change_using_wait
  2023-11-30T09:32:18.008Z [container-agent]     raise TimeoutError(f'timed out waiting for change {change_id} ({timeout} seconds)')
  2023-11-30T09:32:18.008Z [container-agent] ops.pebble.TimeoutError: timed out waiting for change 2 (301 seconds)
  2023-11-30T09:32:18.008Z [container-agent]
  2023-11-30T09:33:09.704Z [container-agent] 2023-11-30 09:33:09 INFO juju.worker.uniter.operation runhook.go:186 ran "identity-service-relation-changed" hook (via hook dispatching script: dispatch)
  2023-11-30T09:33:14.730Z [container-agent] 2023-11-30 09:33:14 INFO juju.worker.uniter.operation runhook.go:186 ran "update-status" hook (via hook dispatching script: dispatch)

  * This has not allowed me to get past the bootstrapping process to
  start using OpenStack as 22 of the 23 dependancies are up only.

  I am running everything with the following commands:
  --> sudo snap install openstack
  --> sunbeam prepare-node-script | bash -x && newgrp snap_daemon
  --> sunbeam cluster bootstrap --accept-defaults

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2045287/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next