yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1782575] [NEW] Crash due to a lack of a root-fs

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From:  Christian Ehrhardt  <1782575@xxxxxxxxxxxxxxxxxx>
Date: Thu, 19 Jul 2018 13:41:37 -0000
Reply-to: Bug 1782575 <1782575@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Hey, this certainly isn't the most important case.
But I wanted to report it to make you aware.

For bug 1732028 I was trying different configs.
One was an empty /etc/fstab with the kernel providing the iscsi root.

In that case cloud init runs and since it does not check on RW for / it
crashes a bit less soft than it could be.

[ 1304.092816] cloud-init[416]: Traceback (most recent call last):
[ 1305.090202] cloud-init[416]:   File "/usr/bin/cloud-init", line 11, in <module>
[ 1305.591502] cloud-init[416]:     load_entry_point('cloud-init==18.3', 'console_scripts', 'cloud-init')()
[ 1306.590627] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 882, in main[  OK  ] Started ebtables ruleset.

[ 1308.196853] cloud-init[416]:     get_uptime=True, func=functor, args=(name, args))
[ 1308.595178] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2482, in log_time
[ 1309.397478] cloud-init[416]:     ret = func(*args, **kwargs)
[ 1309.495795] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 603, in status_wrapper
[ 1309.591048] cloud-init[416]:     util.ensure_dirs((data_d, link_d,))
[ 1309.594596] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1540, in ensure_dirs
[ 1309.598989] cloud-init[416]:     ensure_dir(d, mode)
[ 1310.792843] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1580, in ensure_dir
[ 1310.800009] cloud-init[416]:     os.makedirs(path)
[ 1311.194429] cloud-init[416]:   File "/usr/lib/python3.6/os.py", line 210, in makedirs
[ 1311.199855] cloud-init[416]:     makedirs(head, mode, exist_ok)
[ 1311.594192] cloud-init[416]:   File "/usr/lib/python3.6/os.py", line 220, in makedirs
[ 1311.597483] cloud-init[416]:     mkdir(name, mode)
[ 1311.599771] cloud-init[416]: OSError: [Errno 30] Read-only file system: '/var/lib/cloud'
[ 1312.098866] cloud-init[416]: Error in sys.excepthook:
[ 1312.494095] cloud-init[416]: Traceback (most recent call last):
[ 1312.499649] cloud-init[416]:   File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 145, in apport_excepthook
[ 1312.895912] cloud-init[416]:     os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o640), 'wb') as f:
[ 1312.990471] cloud-init[416]: OSError: [Errno 30] Read-only file system: '/var/crash/_usr_bin_cloud-init.0.crash'
[ 1312.994822] cloud-init[416]: Original exception was:
[ 1312.998082] cloud-init[416]: Traceback (most recent call last):
[ 1313.389856] cloud-init[416]:   File "/usr/bin/cloud-init", line 11, in <module>
[ 1313.795775] cloud-init[416]:     load_entry_point('cloud-init==18.3', 'console_scripts', 'cloud-init')()
[ 1314.193905] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 882, in main
[ 1314.199596] cloud-init[416]:     get_uptime=True, func=functor, args=(name, args))
[ 1314.997439] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2482, in log_time
[ 1316.190352] cloud-init[416]:     ret = func(*args, **kwargs)
[ 1316.198620] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 603, in status_wrapper
[ 1316.290214] cloud-init[416]:     util.ensure_dirs((data_d, link_d,))
[ 1316.295751] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1540, in ensure_dirs
[ 1316.690112] cloud-init[416]:     ensure_dir(d, mode)
[ 1316.696509] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1580, in ensure_dir
[ 1316.893821] cloud-init[416]:     os.makedirs(path)
[ 1318.097629] cloud-init[416]:   File "/usr/lib/python3.6/os.py", line 210, in makedirs
[ 1318.992338] cloud-init[416]:     makedirs(head, mode, exist_ok)
[ 1318.998705] cloud-init[416]:   File "/usr/lib/python3.6/os.py", line 220, in makedirs
[ 1320.804643] cloud-init[416]:     mkdir(name, mode)
[ 1323.795736] cloud-init[416]: OSError: [Errno 30] Read-only file system: '/var/lib/cloud'
[FAILED] Failed to start Initial cloud-init job (pre-networking).
See 'systemctl status cloud-init-local.service' for details.

OTOH while the stacktrace is a bit much for this case the message in between is sort of clear
  [ 1311.599771] cloud-init[416]: OSError: [Errno 30] Read-only file system: '/var/lib/cloud'

As I said I only wanted to let you know if you want to adapt that to not even try to run in those cases and maybe throw less text at the user.
Feel free to just mark it invalid if you are fine with that being the messaging in this case.

** Affects: cloud-init
     Importance: Low
         Status: New

** Changed in: cloud-init
   Importance: Undecided => Low

** Description changed:

  Hey, this certainly isn't the most important case.
  But I wanted to report it to make you aware.
  
  For bug 1732028 I was trying different configs.
  One was an empty /etc/fstab with the kernel providing the iscsi root.
  
  In that case cloud init runs and since it does not check on RW for / it
  crashes a bit less soft than it could be.
  
  [ 1304.092816] cloud-init[416]: Traceback (most recent call last):
  [ 1305.090202] cloud-init[416]:   File "/usr/bin/cloud-init", line 11, in <module>
  [ 1305.591502] cloud-init[416]:     load_entry_point('cloud-init==18.3', 'console_scripts', 'cloud-init')()
  [ 1306.590627] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 882, in main[  OK  ] Started ebtables ruleset.
  
  [ 1308.196853] cloud-init[416]:     get_uptime=True, func=functor, args=(name, args))
  [ 1308.595178] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2482, in log_time
  [ 1309.397478] cloud-init[416]:     ret = func(*args, **kwargs)
  [ 1309.495795] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 603, in status_wrapper
  [ 1309.591048] cloud-init[416]:     util.ensure_dirs((data_d, link_d,))
  [ 1309.594596] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1540, in ensure_dirs
  [ 1309.598989] cloud-init[416]:     ensure_dir(d, mode)
  [ 1310.792843] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1580, in ensure_dir
  [ 1310.800009] cloud-init[416]:     os.makedirs(path)
  [ 1311.194429] cloud-init[416]:   File "/usr/lib/python3.6/os.py", line 210, in makedirs
  [ 1311.199855] cloud-init[416]:     makedirs(head, mode, exist_ok)
  [ 1311.594192] cloud-init[416]:   File "/usr/lib/python3.6/os.py", line 220, in makedirs
  [ 1311.597483] cloud-init[416]:     mkdir(name, mode)
  [ 1311.599771] cloud-init[416]: OSError: [Errno 30] Read-only file system: '/var/lib/cloud'
  [ 1312.098866] cloud-init[416]: Error in sys.excepthook:
  [ 1312.494095] cloud-init[416]: Traceback (most recent call last):
  [ 1312.499649] cloud-init[416]:   File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 145, in apport_excepthook
  [ 1312.895912] cloud-init[416]:     os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o640), 'wb') as f:
  [ 1312.990471] cloud-init[416]: OSError: [Errno 30] Read-only file system: '/var/crash/_usr_bin_cloud-init.0.crash'
  [ 1312.994822] cloud-init[416]: Original exception was:
  [ 1312.998082] cloud-init[416]: Traceback (most recent call last):
  [ 1313.389856] cloud-init[416]:   File "/usr/bin/cloud-init", line 11, in <module>
  [ 1313.795775] cloud-init[416]:     load_entry_point('cloud-init==18.3', 'console_scripts', 'cloud-init')()
  [ 1314.193905] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 882, in main
  [ 1314.199596] cloud-init[416]:     get_uptime=True, func=functor, args=(name, args))
  [ 1314.997439] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2482, in log_time
  [ 1316.190352] cloud-init[416]:     ret = func(*args, **kwargs)
  [ 1316.198620] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 603, in status_wrapper
  [ 1316.290214] cloud-init[416]:     util.ensure_dirs((data_d, link_d,))
  [ 1316.295751] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1540, in ensure_dirs
  [ 1316.690112] cloud-init[416]:     ensure_dir(d, mode)
  [ 1316.696509] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1580, in ensure_dir
  [ 1316.893821] cloud-init[416]:     os.makedirs(path)
  [ 1318.097629] cloud-init[416]:   File "/usr/lib/python3.6/os.py", line 210, in makedirs
  [ 1318.992338] cloud-init[416]:     makedirs(head, mode, exist_ok)
  [ 1318.998705] cloud-init[416]:   File "/usr/lib/python3.6/os.py", line 220, in makedirs
  [ 1320.804643] cloud-init[416]:     mkdir(name, mode)
  [ 1323.795736] cloud-init[416]: OSError: [Errno 30] Read-only file system: '/var/lib/cloud'
  [FAILED] Failed to start Initial cloud-init job (pre-networking).
  See 'systemctl status cloud-init-local.service' for details.
  
+ OTOH while the stacktrace is a bit much for this case the message in between is sort of clear
+   [ 1311.599771] cloud-init[416]: OSError: [Errno 30] Read-only file system: '/var/lib/cloud'
  
- OTOH while the stacktrace is a bit much for this case the message in between is sort of clear
-   [ 1311.599771] cloud-init[416]: OSError: [Errno 30] Read-only file system: '/var/lib/cloud'
- 
- As I said I only wanted to let you know if you want to adapt that to not
- even try to run in those cases and maybe. Feel free to just mark it
- invalid if you are fine with that being the messaging in this case.
+ As I said I only wanted to let you know if you want to adapt that to not even try to run in those cases and maybe throw less text at the user.
+ Feel free to just mark it invalid if you are fine with that being the messaging in this case.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1782575

Title:
  Crash due to a lack of a root-fs

Status in cloud-init:
  New

Bug description:
  Hey, this certainly isn't the most important case.
  But I wanted to report it to make you aware.

  For bug 1732028 I was trying different configs.
  One was an empty /etc/fstab with the kernel providing the iscsi root.

  In that case cloud init runs and since it does not check on RW for /
  it crashes a bit less soft than it could be.

  [ 1304.092816] cloud-init[416]: Traceback (most recent call last):
  [ 1305.090202] cloud-init[416]:   File "/usr/bin/cloud-init", line 11, in <module>
  [ 1305.591502] cloud-init[416]:     load_entry_point('cloud-init==18.3', 'console_scripts', 'cloud-init')()
  [ 1306.590627] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 882, in main[  OK  ] Started ebtables ruleset.

  [ 1308.196853] cloud-init[416]:     get_uptime=True, func=functor, args=(name, args))
  [ 1308.595178] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2482, in log_time
  [ 1309.397478] cloud-init[416]:     ret = func(*args, **kwargs)
  [ 1309.495795] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 603, in status_wrapper
  [ 1309.591048] cloud-init[416]:     util.ensure_dirs((data_d, link_d,))
  [ 1309.594596] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1540, in ensure_dirs
  [ 1309.598989] cloud-init[416]:     ensure_dir(d, mode)
  [ 1310.792843] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1580, in ensure_dir
  [ 1310.800009] cloud-init[416]:     os.makedirs(path)
  [ 1311.194429] cloud-init[416]:   File "/usr/lib/python3.6/os.py", line 210, in makedirs
  [ 1311.199855] cloud-init[416]:     makedirs(head, mode, exist_ok)
  [ 1311.594192] cloud-init[416]:   File "/usr/lib/python3.6/os.py", line 220, in makedirs
  [ 1311.597483] cloud-init[416]:     mkdir(name, mode)
  [ 1311.599771] cloud-init[416]: OSError: [Errno 30] Read-only file system: '/var/lib/cloud'
  [ 1312.098866] cloud-init[416]: Error in sys.excepthook:
  [ 1312.494095] cloud-init[416]: Traceback (most recent call last):
  [ 1312.499649] cloud-init[416]:   File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 145, in apport_excepthook
  [ 1312.895912] cloud-init[416]:     os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o640), 'wb') as f:
  [ 1312.990471] cloud-init[416]: OSError: [Errno 30] Read-only file system: '/var/crash/_usr_bin_cloud-init.0.crash'
  [ 1312.994822] cloud-init[416]: Original exception was:
  [ 1312.998082] cloud-init[416]: Traceback (most recent call last):
  [ 1313.389856] cloud-init[416]:   File "/usr/bin/cloud-init", line 11, in <module>
  [ 1313.795775] cloud-init[416]:     load_entry_point('cloud-init==18.3', 'console_scripts', 'cloud-init')()
  [ 1314.193905] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 882, in main
  [ 1314.199596] cloud-init[416]:     get_uptime=True, func=functor, args=(name, args))
  [ 1314.997439] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2482, in log_time
  [ 1316.190352] cloud-init[416]:     ret = func(*args, **kwargs)
  [ 1316.198620] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 603, in status_wrapper
  [ 1316.290214] cloud-init[416]:     util.ensure_dirs((data_d, link_d,))
  [ 1316.295751] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1540, in ensure_dirs
  [ 1316.690112] cloud-init[416]:     ensure_dir(d, mode)
  [ 1316.696509] cloud-init[416]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1580, in ensure_dir
  [ 1316.893821] cloud-init[416]:     os.makedirs(path)
  [ 1318.097629] cloud-init[416]:   File "/usr/lib/python3.6/os.py", line 210, in makedirs
  [ 1318.992338] cloud-init[416]:     makedirs(head, mode, exist_ok)
  [ 1318.998705] cloud-init[416]:   File "/usr/lib/python3.6/os.py", line 220, in makedirs
  [ 1320.804643] cloud-init[416]:     mkdir(name, mode)
  [ 1323.795736] cloud-init[416]: OSError: [Errno 30] Read-only file system: '/var/lib/cloud'
  [FAILED] Failed to start Initial cloud-init job (pre-networking).
  See 'systemctl status cloud-init-local.service' for details.

  OTOH while the stacktrace is a bit much for this case the message in between is sort of clear
    [ 1311.599771] cloud-init[416]: OSError: [Errno 30] Read-only file system: '/var/lib/cloud'

  As I said I only wanted to let you know if you want to adapt that to not even try to run in those cases and maybe throw less text at the user.
  Feel free to just mark it invalid if you are fine with that being the messaging in this case.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1782575/+subscriptions
Follow ups

[Bug 1782575] Re: Crash due to a lack of a root-fs
From: James Falcon, 2023-05-11