← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 2099911] [NEW] Castellan as config source fails with "greenlet.error: cannot switch to a different thread"

 

Public bug reported:

1) Setup Castellan as config source
2) Start Neutron
3) DHCP and L3 agents fail to start

Output:

Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
    timer()
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/timer.py", line 59, in __call__
    cb(*args, **kw)
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
    waiter.switch()
greenlet.error: cannot switch to a different thread

Log:

2025-02-18 15:22:28.969 153057 INFO castellan._config_driver [-] option '[DEFAULT] default_log_levels' not present in '[castellan_configsource] mapping_file'
2025-02-18 15:22:28.973 153057 INFO castellan._config_driver [-] option '[DEFAULT] rate_limit_burst' not present in '[castellan_configsource] mapping_file'
2025-02-18 15:22:28.974 153057 INFO neutron.common.config [-] Logging enabled!
2025-02-18 15:22:28.974 153057 INFO neutron.common.config [-] /var/lib/kolla/venv/bin/neutron-l3-agent version 24.1.1.dev11
2025-02-18 15:22:28.975 153057 INFO castellan._config_driver [-] option '[AGENT] report_interval' not present in '[castellan_configsource] mapping_file'
2025-02-18 15:22:28.975 153057 INFO castellan._config_driver [-] option '[DEFAULT] host' not present in '[castellan_configsource] mapping_file'
2025-02-18 15:22:28.976 153057 INFO castellan._config_driver [-] option '[DEFAULT] periodic_interval' not present in '[castellan_configsource] mapping_file'
2025-02-18 15:22:28.976 153057 INFO castellan._config_driver [-] option '[DEFAULT] periodic_fuzzy_delay' not present in '[castellan_configsource] mapping_file'
2025-02-18 15:22:29.499 153057 INFO castellan._config_driver [-] option '[DEFAULT] cleanup_on_shutdown' not present in '[castellan_configsource] mapping_file'
2025-02-18 15:22:29.500 153057 INFO castellan._config_driver [-] option '[DEFAULT] ipv6_gateway' not present in '[castellan_configsource] mapping_file'
2025-02-18 15:22:29.500 153057 INFO castellan._config_driver [-] option '[AGENT] check_child_processes_interval' not present in '[castellan_configsource] mapping_file'
2025-02-18 15:22:29.501 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_default_rule' not present in '[castellan_configsource] mapping_file'
2025-02-18 15:22:29.501 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_file' not present in '[castellan_configsource] mapping_file'
2025-02-18 15:22:29.502 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_file' not present in '[castellan_configsource] mapping_file'
2025-02-18 15:22:29.503 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_dirs' not present in '[castellan_configsource] mapping_file'
2025-02-18 15:22:29.539 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[OVS] of_inactivity_probe' not present in '[castellan_configsource] mapping_file'
2025-02-18 15:22:29.545 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[DEFAULT] ovs_use_veth' not present in '[castellan_configsource] mapping_file'
2025-02-18 15:22:29.547 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_messaging_metrics] metrics_socket_file' not present in '[castellan_configsource] mapping_file'
2025-02-18 15:22:29.547 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_messaging_metrics] metrics_buffer_size' not present in '[castellan_configsource] mapping_file'
2025-02-18 15:22:29.548 153057 INFO castellan._config_driver [-] option '[oslo_messaging_metrics] metrics_thread_stop_timeout' not present in '[castellan_configsource] mapping_file'


4) press Ctrl+C.

Output:

^C^CError in sys.excepthook:
Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 115, in acquire
    hubs.get_hub().switch()
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
    return self.greenlet.switch()
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
    self.wait(sleep_time)
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/poll.py", line 77, in wait
    time.sleep(seconds)
KeyboardInterrupt

Original exception was:
Traceback (most recent call last):
  File "/var/lib/kolla/venv/bin/neutron-l3-agent", line 8, in <module>
    sys.exit(main())
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/cmd/eventlet/agents/l3.py", line 24, in main
    l3_agent.main()
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3_agent.py", line 56, in main
    server = neutron_service.Service.create(
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/service.py", line 421, in create
    service_obj = cls(host, binary, topic, manager,
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/service.py", line 356, in __init__
    self.manager = manager_class(host=host, *args, **kwargs)
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3/agent.py", line 1002, in __init__
    super(L3NATAgentWithStateReport, self).__init__(host=host, conf=conf)
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3/agent.py", line 307, in __init__
    self.plugin_rpc.get_host_ha_router_count(self.context))
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/timeutils.py", line 333, in wrapper
    result = func(*args, **kwargs)
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/common/utils.py", line 962, in wrapper
    ret = f(*args, **kwargs)
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3/agent.py", line 199, in get_host_ha_router_count
    return cctxt.call(context, 'get_host_ha_router_count', host=self.host)
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/rpc.py", line 149, in call
    return self._original_context.call(ctxt, method, **kwargs)
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/rpc/client.py", line 186, in call
    with metrics.get_collector(self.conf, "rpc_client",
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/_metrics/client.py", line 82, in __enter__
    if not self.conf.metrics_enabled:
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_config/cfg.py", line 3172, in __getattr__
    return self._conf._get(name, self._group)
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_config/cfg.py", line 2664, in _get
    value, loc = self._do_get(name, group, namespace)
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_config/cfg.py", line 2752, in _do_get
    val = source.get(group_name, name, opt)
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/castellan/_config_driver.py", line 126, in get
    LOG.info("option '[%s] %s' not present in '[%s] mapping_file'",
  File "/usr/lib64/python3.9/logging/__init__.py", line 1806, in info
    self.log(INFO, msg, *args, **kwargs)
  File "/usr/lib64/python3.9/logging/__init__.py", line 1844, in log
    self.logger.log(level, msg, *args, **kwargs)
  File "/usr/lib64/python3.9/logging/__init__.py", line 1512, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/lib64/python3.9/logging/__init__.py", line 1589, in _log
    self.handle(record)
  File "/usr/lib64/python3.9/logging/__init__.py", line 1599, in handle
    self.callHandlers(record)
  File "/usr/lib64/python3.9/logging/__init__.py", line 1661, in callHandlers
    hdlr.handle(record)
  File "/usr/lib64/python3.9/logging/__init__.py", line 950, in handle
    self.acquire()
  File "/usr/lib64/python3.9/logging/__init__.py", line 901, in acquire
    self.lock.acquire()
  File "/usr/lib64/python3.9/threading.py", line 160, in acquire
    rc = self._block.acquire(blocking, timeout)
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 115, in acquire
    hubs.get_hub().switch()
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
    return self.greenlet.switch()
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
    self.wait(sleep_time)
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/poll.py", line 77, in wait
    time.sleep(seconds)
KeyboardInterrupt
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib64/python3.9/logging/__init__.py", line 2140, in shutdown
    h.acquire()
  File "/usr/lib64/python3.9/logging/__init__.py", line 901, in acquire
    self.lock.acquire()
  File "/usr/lib64/python3.9/threading.py", line 160, in acquire
    rc = self._block.acquire(blocking, timeout)
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 115, in acquire
    hubs.get_hub().switch()
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
    return self.greenlet.switch()
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
    self.wait(sleep_time)
  File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/poll.py", line 77, in wait
    time.sleep(seconds)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.9/logging/__init__.py", line 2150, in shutdown
    h.release()
  File "/usr/lib64/python3.9/logging/__init__.py", line 908, in release
    self.lock.release()
  File "/usr/lib64/python3.9/threading.py", line 185, in release
    raise RuntimeError("cannot release un-acquired lock")
RuntimeError: cannot release un-acquired lock

** Affects: neutron
     Importance: Undecided
     Assignee: Mitya Eremeev (mitos)
         Status: In Progress

** Description changed:

  1) Setup Castellan as config source
  2) Start Neutron
  3) DHCP and L3 agents fail to start
  
  Output:
  
  Traceback (most recent call last):
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
-     timer()
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/timer.py", line 59, in __call__
-     cb(*args, **kw)
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
-     waiter.switch()
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
+     timer()
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/timer.py", line 59, in __call__
+     cb(*args, **kw)
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
+     waiter.switch()
  greenlet.error: cannot switch to a different thread
- 
  
  press Ctrl+C:
  
  ^C^CError in sys.excepthook:
  Traceback (most recent call last):
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 115, in acquire
-     hubs.get_hub().switch()
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
-     return self.greenlet.switch()
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
-     self.wait(sleep_time)
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/poll.py", line 77, in wait
-     time.sleep(seconds)
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 115, in acquire
+     hubs.get_hub().switch()
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
+     return self.greenlet.switch()
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
+     self.wait(sleep_time)
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/poll.py", line 77, in wait
+     time.sleep(seconds)
  KeyboardInterrupt
  
  Original exception was:
  Traceback (most recent call last):
-   File "/var/lib/kolla/venv/bin/neutron-l3-agent", line 8, in <module>
-     sys.exit(main())
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/cmd/eventlet/agents/l3.py", line 24, in main
-     l3_agent.main()
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3_agent.py", line 56, in main
-     server = neutron_service.Service.create(
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/service.py", line 421, in create
-     service_obj = cls(host, binary, topic, manager,
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/service.py", line 356, in __init__
-     self.manager = manager_class(host=host, *args, **kwargs)
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3/agent.py", line 1002, in __init__
-     super(L3NATAgentWithStateReport, self).__init__(host=host, conf=conf)
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3/agent.py", line 307, in __init__
-     self.plugin_rpc.get_host_ha_router_count(self.context))
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/timeutils.py", line 333, in wrapper
-     result = func(*args, **kwargs)
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/common/utils.py", line 962, in wrapper
-     ret = f(*args, **kwargs)
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3/agent.py", line 199, in get_host_ha_router_count
-     return cctxt.call(context, 'get_host_ha_router_count', host=self.host)
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/rpc.py", line 149, in call
-     return self._original_context.call(ctxt, method, **kwargs)
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/rpc/client.py", line 186, in call
-     with metrics.get_collector(self.conf, "rpc_client",
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/_metrics/client.py", line 82, in __enter__
-     if not self.conf.metrics_enabled:
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_config/cfg.py", line 3172, in __getattr__
-     return self._conf._get(name, self._group)
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_config/cfg.py", line 2664, in _get
-     value, loc = self._do_get(name, group, namespace)
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_config/cfg.py", line 2752, in _do_get
-     val = source.get(group_name, name, opt)
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/castellan/_config_driver.py", line 126, in get
-     LOG.info("option '[%s] %s' not present in '[%s] mapping_file'",
-   File "/usr/lib64/python3.9/logging/__init__.py", line 1806, in info
-     self.log(INFO, msg, *args, **kwargs)
-   File "/usr/lib64/python3.9/logging/__init__.py", line 1844, in log
-     self.logger.log(level, msg, *args, **kwargs)
-   File "/usr/lib64/python3.9/logging/__init__.py", line 1512, in log
-     self._log(level, msg, args, **kwargs)
-   File "/usr/lib64/python3.9/logging/__init__.py", line 1589, in _log
-     self.handle(record)
-   File "/usr/lib64/python3.9/logging/__init__.py", line 1599, in handle
-     self.callHandlers(record)
-   File "/usr/lib64/python3.9/logging/__init__.py", line 1661, in callHandlers
-     hdlr.handle(record)
-   File "/usr/lib64/python3.9/logging/__init__.py", line 950, in handle
-     self.acquire()
-   File "/usr/lib64/python3.9/logging/__init__.py", line 901, in acquire
-     self.lock.acquire()
-   File "/usr/lib64/python3.9/threading.py", line 160, in acquire
-     rc = self._block.acquire(blocking, timeout)
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 115, in acquire
-     hubs.get_hub().switch()
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
-     return self.greenlet.switch()
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
-     self.wait(sleep_time)
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/poll.py", line 77, in wait
-     time.sleep(seconds)
+   File "/var/lib/kolla/venv/bin/neutron-l3-agent", line 8, in <module>
+     sys.exit(main())
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/cmd/eventlet/agents/l3.py", line 24, in main
+     l3_agent.main()
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3_agent.py", line 56, in main
+     server = neutron_service.Service.create(
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/service.py", line 421, in create
+     service_obj = cls(host, binary, topic, manager,
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/service.py", line 356, in __init__
+     self.manager = manager_class(host=host, *args, **kwargs)
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3/agent.py", line 1002, in __init__
+     super(L3NATAgentWithStateReport, self).__init__(host=host, conf=conf)
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3/agent.py", line 307, in __init__
+     self.plugin_rpc.get_host_ha_router_count(self.context))
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/timeutils.py", line 333, in wrapper
+     result = func(*args, **kwargs)
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/common/utils.py", line 962, in wrapper
+     ret = f(*args, **kwargs)
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3/agent.py", line 199, in get_host_ha_router_count
+     return cctxt.call(context, 'get_host_ha_router_count', host=self.host)
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/rpc.py", line 149, in call
+     return self._original_context.call(ctxt, method, **kwargs)
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/rpc/client.py", line 186, in call
+     with metrics.get_collector(self.conf, "rpc_client",
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/_metrics/client.py", line 82, in __enter__
+     if not self.conf.metrics_enabled:
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_config/cfg.py", line 3172, in __getattr__
+     return self._conf._get(name, self._group)
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_config/cfg.py", line 2664, in _get
+     value, loc = self._do_get(name, group, namespace)
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_config/cfg.py", line 2752, in _do_get
+     val = source.get(group_name, name, opt)
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/castellan/_config_driver.py", line 126, in get
+     LOG.info("option '[%s] %s' not present in '[%s] mapping_file'",
+   File "/usr/lib64/python3.9/logging/__init__.py", line 1806, in info
+     self.log(INFO, msg, *args, **kwargs)
+   File "/usr/lib64/python3.9/logging/__init__.py", line 1844, in log
+     self.logger.log(level, msg, *args, **kwargs)
+   File "/usr/lib64/python3.9/logging/__init__.py", line 1512, in log
+     self._log(level, msg, args, **kwargs)
+   File "/usr/lib64/python3.9/logging/__init__.py", line 1589, in _log
+     self.handle(record)
+   File "/usr/lib64/python3.9/logging/__init__.py", line 1599, in handle
+     self.callHandlers(record)
+   File "/usr/lib64/python3.9/logging/__init__.py", line 1661, in callHandlers
+     hdlr.handle(record)
+   File "/usr/lib64/python3.9/logging/__init__.py", line 950, in handle
+     self.acquire()
+   File "/usr/lib64/python3.9/logging/__init__.py", line 901, in acquire
+     self.lock.acquire()
+   File "/usr/lib64/python3.9/threading.py", line 160, in acquire
+     rc = self._block.acquire(blocking, timeout)
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 115, in acquire
+     hubs.get_hub().switch()
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
+     return self.greenlet.switch()
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
+     self.wait(sleep_time)
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/poll.py", line 77, in wait
+     time.sleep(seconds)
  KeyboardInterrupt
  ^CError in atexit._run_exitfuncs:
  Traceback (most recent call last):
-   File "/usr/lib64/python3.9/logging/__init__.py", line 2140, in shutdown
-     h.acquire()
-   File "/usr/lib64/python3.9/logging/__init__.py", line 901, in acquire
-     self.lock.acquire()
-   File "/usr/lib64/python3.9/threading.py", line 160, in acquire
-     rc = self._block.acquire(blocking, timeout)
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 115, in acquire
-     hubs.get_hub().switch()
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
-     return self.greenlet.switch()
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
-     self.wait(sleep_time)
-   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/poll.py", line 77, in wait
-     time.sleep(seconds)
+   File "/usr/lib64/python3.9/logging/__init__.py", line 2140, in shutdown
+     h.acquire()
+   File "/usr/lib64/python3.9/logging/__init__.py", line 901, in acquire
+     self.lock.acquire()
+   File "/usr/lib64/python3.9/threading.py", line 160, in acquire
+     rc = self._block.acquire(blocking, timeout)
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 115, in acquire
+     hubs.get_hub().switch()
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
+     return self.greenlet.switch()
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
+     self.wait(sleep_time)
+   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/poll.py", line 77, in wait
+     time.sleep(seconds)
  KeyboardInterrupt
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
-   File "/usr/lib64/python3.9/logging/__init__.py", line 2150, in shutdown
-     h.release()
-   File "/usr/lib64/python3.9/logging/__init__.py", line 908, in release
-     self.lock.release()
-   File "/usr/lib64/python3.9/threading.py", line 185, in release
-     raise RuntimeError("cannot release un-acquired lock")
+   File "/usr/lib64/python3.9/logging/__init__.py", line 2150, in shutdown
+     h.release()
+   File "/usr/lib64/python3.9/logging/__init__.py", line 908, in release
+     self.lock.release()
+   File "/usr/lib64/python3.9/threading.py", line 185, in release
+     raise RuntimeError("cannot release un-acquired lock")
  RuntimeError: cannot release un-acquired lock
- 
  
  Log:
  
  2025-02-18 15:22:28.969 153057 INFO castellan._config_driver [-] option '[DEFAULT] default_log_levels' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:28.973 153057 INFO castellan._config_driver [-] option '[DEFAULT] rate_limit_burst' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:28.974 153057 INFO neutron.common.config [-] Logging enabled!
  2025-02-18 15:22:28.974 153057 INFO neutron.common.config [-] /var/lib/kolla/venv/bin/neutron-l3-agent version 24.1.1.dev11
  2025-02-18 15:22:28.975 153057 INFO castellan._config_driver [-] option '[AGENT] report_interval' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:28.975 153057 INFO castellan._config_driver [-] option '[DEFAULT] host' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:28.976 153057 INFO castellan._config_driver [-] option '[DEFAULT] periodic_interval' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:28.976 153057 INFO castellan._config_driver [-] option '[DEFAULT] periodic_fuzzy_delay' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.499 153057 INFO castellan._config_driver [-] option '[DEFAULT] cleanup_on_shutdown' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.500 153057 INFO castellan._config_driver [-] option '[DEFAULT] ipv6_gateway' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.500 153057 INFO castellan._config_driver [-] option '[AGENT] check_child_processes_interval' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.501 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_default_rule' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.501 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_file' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.502 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_file' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.503 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_dirs' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.539 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[OVS] of_inactivity_probe' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.545 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[DEFAULT] ovs_use_veth' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.547 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_messaging_metrics] metrics_socket_file' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.547 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_messaging_metrics] metrics_buffer_size' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.548 153057 INFO castellan._config_driver [-] option '[oslo_messaging_metrics] metrics_thread_stop_timeout' not present in '[castellan_configsource] mapping_file'

** Summary changed:

- Castellan as config source failes with "greenlet.error: cannot switch to a different thread"
+ Castellan as config source fails with "greenlet.error: cannot switch to a different thread"

** Description changed:

  1) Setup Castellan as config source
  2) Start Neutron
  3) DHCP and L3 agents fail to start
  
  Output:
  
  Traceback (most recent call last):
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
      timer()
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/timer.py", line 59, in __call__
      cb(*args, **kw)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
      waiter.switch()
  greenlet.error: cannot switch to a different thread
  
- press Ctrl+C:
+ Log:
+ 
+ 2025-02-18 15:22:28.969 153057 INFO castellan._config_driver [-] option '[DEFAULT] default_log_levels' not present in '[castellan_configsource] mapping_file'
+ 2025-02-18 15:22:28.973 153057 INFO castellan._config_driver [-] option '[DEFAULT] rate_limit_burst' not present in '[castellan_configsource] mapping_file'
+ 2025-02-18 15:22:28.974 153057 INFO neutron.common.config [-] Logging enabled!
+ 2025-02-18 15:22:28.974 153057 INFO neutron.common.config [-] /var/lib/kolla/venv/bin/neutron-l3-agent version 24.1.1.dev11
+ 2025-02-18 15:22:28.975 153057 INFO castellan._config_driver [-] option '[AGENT] report_interval' not present in '[castellan_configsource] mapping_file'
+ 2025-02-18 15:22:28.975 153057 INFO castellan._config_driver [-] option '[DEFAULT] host' not present in '[castellan_configsource] mapping_file'
+ 2025-02-18 15:22:28.976 153057 INFO castellan._config_driver [-] option '[DEFAULT] periodic_interval' not present in '[castellan_configsource] mapping_file'
+ 2025-02-18 15:22:28.976 153057 INFO castellan._config_driver [-] option '[DEFAULT] periodic_fuzzy_delay' not present in '[castellan_configsource] mapping_file'
+ 2025-02-18 15:22:29.499 153057 INFO castellan._config_driver [-] option '[DEFAULT] cleanup_on_shutdown' not present in '[castellan_configsource] mapping_file'
+ 2025-02-18 15:22:29.500 153057 INFO castellan._config_driver [-] option '[DEFAULT] ipv6_gateway' not present in '[castellan_configsource] mapping_file'
+ 2025-02-18 15:22:29.500 153057 INFO castellan._config_driver [-] option '[AGENT] check_child_processes_interval' not present in '[castellan_configsource] mapping_file'
+ 2025-02-18 15:22:29.501 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_default_rule' not present in '[castellan_configsource] mapping_file'
+ 2025-02-18 15:22:29.501 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_file' not present in '[castellan_configsource] mapping_file'
+ 2025-02-18 15:22:29.502 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_file' not present in '[castellan_configsource] mapping_file'
+ 2025-02-18 15:22:29.503 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_dirs' not present in '[castellan_configsource] mapping_file'
+ 2025-02-18 15:22:29.539 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[OVS] of_inactivity_probe' not present in '[castellan_configsource] mapping_file'
+ 2025-02-18 15:22:29.545 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[DEFAULT] ovs_use_veth' not present in '[castellan_configsource] mapping_file'
+ 2025-02-18 15:22:29.547 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_messaging_metrics] metrics_socket_file' not present in '[castellan_configsource] mapping_file'
+ 2025-02-18 15:22:29.547 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_messaging_metrics] metrics_buffer_size' not present in '[castellan_configsource] mapping_file'
+ 2025-02-18 15:22:29.548 153057 INFO castellan._config_driver [-] option '[oslo_messaging_metrics] metrics_thread_stop_timeout' not present in '[castellan_configsource] mapping_file'
+ 
+ 
+ 4) press Ctrl+C.
+ 
+ Output:
  
  ^C^CError in sys.excepthook:
  Traceback (most recent call last):
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 115, in acquire
      hubs.get_hub().switch()
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
      return self.greenlet.switch()
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
      self.wait(sleep_time)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/poll.py", line 77, in wait
      time.sleep(seconds)
  KeyboardInterrupt
  
  Original exception was:
  Traceback (most recent call last):
    File "/var/lib/kolla/venv/bin/neutron-l3-agent", line 8, in <module>
      sys.exit(main())
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/cmd/eventlet/agents/l3.py", line 24, in main
      l3_agent.main()
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3_agent.py", line 56, in main
      server = neutron_service.Service.create(
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/service.py", line 421, in create
      service_obj = cls(host, binary, topic, manager,
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/service.py", line 356, in __init__
      self.manager = manager_class(host=host, *args, **kwargs)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3/agent.py", line 1002, in __init__
      super(L3NATAgentWithStateReport, self).__init__(host=host, conf=conf)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3/agent.py", line 307, in __init__
      self.plugin_rpc.get_host_ha_router_count(self.context))
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/timeutils.py", line 333, in wrapper
      result = func(*args, **kwargs)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/common/utils.py", line 962, in wrapper
      ret = f(*args, **kwargs)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3/agent.py", line 199, in get_host_ha_router_count
      return cctxt.call(context, 'get_host_ha_router_count', host=self.host)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/rpc.py", line 149, in call
      return self._original_context.call(ctxt, method, **kwargs)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/rpc/client.py", line 186, in call
      with metrics.get_collector(self.conf, "rpc_client",
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/_metrics/client.py", line 82, in __enter__
      if not self.conf.metrics_enabled:
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_config/cfg.py", line 3172, in __getattr__
      return self._conf._get(name, self._group)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_config/cfg.py", line 2664, in _get
      value, loc = self._do_get(name, group, namespace)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_config/cfg.py", line 2752, in _do_get
      val = source.get(group_name, name, opt)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/castellan/_config_driver.py", line 126, in get
      LOG.info("option '[%s] %s' not present in '[%s] mapping_file'",
    File "/usr/lib64/python3.9/logging/__init__.py", line 1806, in info
      self.log(INFO, msg, *args, **kwargs)
    File "/usr/lib64/python3.9/logging/__init__.py", line 1844, in log
      self.logger.log(level, msg, *args, **kwargs)
    File "/usr/lib64/python3.9/logging/__init__.py", line 1512, in log
      self._log(level, msg, args, **kwargs)
    File "/usr/lib64/python3.9/logging/__init__.py", line 1589, in _log
      self.handle(record)
    File "/usr/lib64/python3.9/logging/__init__.py", line 1599, in handle
      self.callHandlers(record)
    File "/usr/lib64/python3.9/logging/__init__.py", line 1661, in callHandlers
      hdlr.handle(record)
    File "/usr/lib64/python3.9/logging/__init__.py", line 950, in handle
      self.acquire()
    File "/usr/lib64/python3.9/logging/__init__.py", line 901, in acquire
      self.lock.acquire()
    File "/usr/lib64/python3.9/threading.py", line 160, in acquire
      rc = self._block.acquire(blocking, timeout)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 115, in acquire
      hubs.get_hub().switch()
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
      return self.greenlet.switch()
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
      self.wait(sleep_time)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/poll.py", line 77, in wait
      time.sleep(seconds)
  KeyboardInterrupt
  ^CError in atexit._run_exitfuncs:
  Traceback (most recent call last):
    File "/usr/lib64/python3.9/logging/__init__.py", line 2140, in shutdown
      h.acquire()
    File "/usr/lib64/python3.9/logging/__init__.py", line 901, in acquire
      self.lock.acquire()
    File "/usr/lib64/python3.9/threading.py", line 160, in acquire
      rc = self._block.acquire(blocking, timeout)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 115, in acquire
      hubs.get_hub().switch()
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
      return self.greenlet.switch()
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
      self.wait(sleep_time)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/poll.py", line 77, in wait
      time.sleep(seconds)
  KeyboardInterrupt
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/usr/lib64/python3.9/logging/__init__.py", line 2150, in shutdown
      h.release()
    File "/usr/lib64/python3.9/logging/__init__.py", line 908, in release
      self.lock.release()
    File "/usr/lib64/python3.9/threading.py", line 185, in release
      raise RuntimeError("cannot release un-acquired lock")
  RuntimeError: cannot release un-acquired lock
- 
- Log:
- 
- 2025-02-18 15:22:28.969 153057 INFO castellan._config_driver [-] option '[DEFAULT] default_log_levels' not present in '[castellan_configsource] mapping_file'
- 2025-02-18 15:22:28.973 153057 INFO castellan._config_driver [-] option '[DEFAULT] rate_limit_burst' not present in '[castellan_configsource] mapping_file'
- 2025-02-18 15:22:28.974 153057 INFO neutron.common.config [-] Logging enabled!
- 2025-02-18 15:22:28.974 153057 INFO neutron.common.config [-] /var/lib/kolla/venv/bin/neutron-l3-agent version 24.1.1.dev11
- 2025-02-18 15:22:28.975 153057 INFO castellan._config_driver [-] option '[AGENT] report_interval' not present in '[castellan_configsource] mapping_file'
- 2025-02-18 15:22:28.975 153057 INFO castellan._config_driver [-] option '[DEFAULT] host' not present in '[castellan_configsource] mapping_file'
- 2025-02-18 15:22:28.976 153057 INFO castellan._config_driver [-] option '[DEFAULT] periodic_interval' not present in '[castellan_configsource] mapping_file'
- 2025-02-18 15:22:28.976 153057 INFO castellan._config_driver [-] option '[DEFAULT] periodic_fuzzy_delay' not present in '[castellan_configsource] mapping_file'
- 2025-02-18 15:22:29.499 153057 INFO castellan._config_driver [-] option '[DEFAULT] cleanup_on_shutdown' not present in '[castellan_configsource] mapping_file'
- 2025-02-18 15:22:29.500 153057 INFO castellan._config_driver [-] option '[DEFAULT] ipv6_gateway' not present in '[castellan_configsource] mapping_file'
- 2025-02-18 15:22:29.500 153057 INFO castellan._config_driver [-] option '[AGENT] check_child_processes_interval' not present in '[castellan_configsource] mapping_file'
- 2025-02-18 15:22:29.501 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_default_rule' not present in '[castellan_configsource] mapping_file'
- 2025-02-18 15:22:29.501 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_file' not present in '[castellan_configsource] mapping_file'
- 2025-02-18 15:22:29.502 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_file' not present in '[castellan_configsource] mapping_file'
- 2025-02-18 15:22:29.503 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_dirs' not present in '[castellan_configsource] mapping_file'
- 2025-02-18 15:22:29.539 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[OVS] of_inactivity_probe' not present in '[castellan_configsource] mapping_file'
- 2025-02-18 15:22:29.545 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[DEFAULT] ovs_use_veth' not present in '[castellan_configsource] mapping_file'
- 2025-02-18 15:22:29.547 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_messaging_metrics] metrics_socket_file' not present in '[castellan_configsource] mapping_file'
- 2025-02-18 15:22:29.547 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_messaging_metrics] metrics_buffer_size' not present in '[castellan_configsource] mapping_file'
- 2025-02-18 15:22:29.548 153057 INFO castellan._config_driver [-] option '[oslo_messaging_metrics] metrics_thread_stop_timeout' not present in '[castellan_configsource] mapping_file'

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2099911

Title:
  Castellan as config source fails with "greenlet.error: cannot switch
  to a different thread"

Status in neutron:
  In Progress

Bug description:
  1) Setup Castellan as config source
  2) Start Neutron
  3) DHCP and L3 agents fail to start

  Output:

  Traceback (most recent call last):
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
      timer()
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/timer.py", line 59, in __call__
      cb(*args, **kw)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
      waiter.switch()
  greenlet.error: cannot switch to a different thread

  Log:

  2025-02-18 15:22:28.969 153057 INFO castellan._config_driver [-] option '[DEFAULT] default_log_levels' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:28.973 153057 INFO castellan._config_driver [-] option '[DEFAULT] rate_limit_burst' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:28.974 153057 INFO neutron.common.config [-] Logging enabled!
  2025-02-18 15:22:28.974 153057 INFO neutron.common.config [-] /var/lib/kolla/venv/bin/neutron-l3-agent version 24.1.1.dev11
  2025-02-18 15:22:28.975 153057 INFO castellan._config_driver [-] option '[AGENT] report_interval' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:28.975 153057 INFO castellan._config_driver [-] option '[DEFAULT] host' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:28.976 153057 INFO castellan._config_driver [-] option '[DEFAULT] periodic_interval' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:28.976 153057 INFO castellan._config_driver [-] option '[DEFAULT] periodic_fuzzy_delay' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.499 153057 INFO castellan._config_driver [-] option '[DEFAULT] cleanup_on_shutdown' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.500 153057 INFO castellan._config_driver [-] option '[DEFAULT] ipv6_gateway' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.500 153057 INFO castellan._config_driver [-] option '[AGENT] check_child_processes_interval' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.501 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_default_rule' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.501 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_file' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.502 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_file' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.503 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_policy] policy_dirs' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.539 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[OVS] of_inactivity_probe' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.545 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[DEFAULT] ovs_use_veth' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.547 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_messaging_metrics] metrics_socket_file' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.547 153057 INFO castellan._config_driver [req-5f029160-6cff-4834-9c7f-ac5d418d8356 - - - - - -] option '[oslo_messaging_metrics] metrics_buffer_size' not present in '[castellan_configsource] mapping_file'
  2025-02-18 15:22:29.548 153057 INFO castellan._config_driver [-] option '[oslo_messaging_metrics] metrics_thread_stop_timeout' not present in '[castellan_configsource] mapping_file'

  
  4) press Ctrl+C.

  Output:

  ^C^CError in sys.excepthook:
  Traceback (most recent call last):
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 115, in acquire
      hubs.get_hub().switch()
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
      return self.greenlet.switch()
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
      self.wait(sleep_time)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/poll.py", line 77, in wait
      time.sleep(seconds)
  KeyboardInterrupt

  Original exception was:
  Traceback (most recent call last):
    File "/var/lib/kolla/venv/bin/neutron-l3-agent", line 8, in <module>
      sys.exit(main())
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/cmd/eventlet/agents/l3.py", line 24, in main
      l3_agent.main()
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3_agent.py", line 56, in main
      server = neutron_service.Service.create(
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/service.py", line 421, in create
      service_obj = cls(host, binary, topic, manager,
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/service.py", line 356, in __init__
      self.manager = manager_class(host=host, *args, **kwargs)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3/agent.py", line 1002, in __init__
      super(L3NATAgentWithStateReport, self).__init__(host=host, conf=conf)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3/agent.py", line 307, in __init__
      self.plugin_rpc.get_host_ha_router_count(self.context))
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/timeutils.py", line 333, in wrapper
      result = func(*args, **kwargs)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/common/utils.py", line 962, in wrapper
      ret = f(*args, **kwargs)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/l3/agent.py", line 199, in get_host_ha_router_count
      return cctxt.call(context, 'get_host_ha_router_count', host=self.host)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/rpc.py", line 149, in call
      return self._original_context.call(ctxt, method, **kwargs)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/rpc/client.py", line 186, in call
      with metrics.get_collector(self.conf, "rpc_client",
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/_metrics/client.py", line 82, in __enter__
      if not self.conf.metrics_enabled:
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_config/cfg.py", line 3172, in __getattr__
      return self._conf._get(name, self._group)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_config/cfg.py", line 2664, in _get
      value, loc = self._do_get(name, group, namespace)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_config/cfg.py", line 2752, in _do_get
      val = source.get(group_name, name, opt)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/castellan/_config_driver.py", line 126, in get
      LOG.info("option '[%s] %s' not present in '[%s] mapping_file'",
    File "/usr/lib64/python3.9/logging/__init__.py", line 1806, in info
      self.log(INFO, msg, *args, **kwargs)
    File "/usr/lib64/python3.9/logging/__init__.py", line 1844, in log
      self.logger.log(level, msg, *args, **kwargs)
    File "/usr/lib64/python3.9/logging/__init__.py", line 1512, in log
      self._log(level, msg, args, **kwargs)
    File "/usr/lib64/python3.9/logging/__init__.py", line 1589, in _log
      self.handle(record)
    File "/usr/lib64/python3.9/logging/__init__.py", line 1599, in handle
      self.callHandlers(record)
    File "/usr/lib64/python3.9/logging/__init__.py", line 1661, in callHandlers
      hdlr.handle(record)
    File "/usr/lib64/python3.9/logging/__init__.py", line 950, in handle
      self.acquire()
    File "/usr/lib64/python3.9/logging/__init__.py", line 901, in acquire
      self.lock.acquire()
    File "/usr/lib64/python3.9/threading.py", line 160, in acquire
      rc = self._block.acquire(blocking, timeout)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 115, in acquire
      hubs.get_hub().switch()
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
      return self.greenlet.switch()
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
      self.wait(sleep_time)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/poll.py", line 77, in wait
      time.sleep(seconds)
  KeyboardInterrupt
  ^CError in atexit._run_exitfuncs:
  Traceback (most recent call last):
    File "/usr/lib64/python3.9/logging/__init__.py", line 2140, in shutdown
      h.acquire()
    File "/usr/lib64/python3.9/logging/__init__.py", line 901, in acquire
      self.lock.acquire()
    File "/usr/lib64/python3.9/threading.py", line 160, in acquire
      rc = self._block.acquire(blocking, timeout)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/semaphore.py", line 115, in acquire
      hubs.get_hub().switch()
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
      return self.greenlet.switch()
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
      self.wait(sleep_time)
    File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/hubs/poll.py", line 77, in wait
      time.sleep(seconds)
  KeyboardInterrupt

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/usr/lib64/python3.9/logging/__init__.py", line 2150, in shutdown
      h.release()
    File "/usr/lib64/python3.9/logging/__init__.py", line 908, in release
      self.lock.release()
    File "/usr/lib64/python3.9/threading.py", line 185, in release
      raise RuntimeError("cannot release un-acquired lock")
  RuntimeError: cannot release un-acquired lock

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2099911/+subscriptions