yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #52667
[Bug 1581553] Re: MAAS fails w/ bad hwclock & no NTP access
*** This bug is a duplicate of bug 1511589 ***
https://bugs.launchpad.net/bugs/1511589
** This bug has been marked a duplicate of bug 1511589
maas provider, hwclock out of sync means juju will not work
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1581553
Title:
MAAS fails w/ bad hwclock & no NTP access
Status in cloud-init:
New
Status in curtin:
New
Status in MAAS:
Invalid
Bug description:
mmm, let me try this again :)
We have been testing maas 2.0 Beta 4 on some enablement hardware and
have been seeing a problem in which during enlistment, a clock skew is
reported on the host console:
[ 288.753993] cloud-init[1464]: Success
[ 290.910846] cloud-init[1464]: updated clock skew to 7853431
[ 290.911437] cloud-init[1464]: request to http://10.246.48.112/MAAS/metadata//2012-03-01/ failed. sleeping 1.: HTTP Error 401: OK
[ 290.911929] cloud-init[1464]: Success
[ 292.752177] cloud-init[1464]: updated clock skew to 7853431
[ 292.752746] cloud-init[1464]: request to http://10.246.48.112/MAAS/metadata//2012-03-01/ failed. sleeping 1.: HTTP Error 401: OK
[ 292.753234] cloud-init[1464]: Success
[ 337.916546] cloud-init[1464]: updated clock skew to 7853431
[ 337.917122] cloud-init[1464]: request to http://10.246.48.112/
This happens a number of times. And as you mentioned, this is cloud
init fixing the clock skew.
The enlistment will complete and we are able to successfully finish
commissioning the Host. The host will appear in a ready state via
the MAAS UI.
Now before I go further, As mentioned earlier. I always ensure that
both HOST AND CLIENT dates & time match in UEFI prior to starting
enlistment. If anything the times on all of the hosts are offset by
+/- 2 minutes.
Moving onto deployment:
When Deploying Xenial on these hosts This is where I get stuck, due to
the clock-skew, which is fixed via cloud-init, tar will report time
stamps approx 1 month into the future, while copying the disk image
and eventually cause the deployment to fail due to a timeout. (tar is
still extracting the root image)
Does setting the ntp host in maas settings have any affect on this,
(for example: if ntp.ubuntu.com was unavailable ? )
We have been triaging this on our end of the stick, however would like
some insight from the maas team.
[ 19.353487] cloud-init[1207]: Cloud-init v. 0.7.7 running 'init' at Thu, 11 Feb 2016 16:28:06 +0000. Up 19.03 seconds.
[ 19.368566] cloud-init[1207]: ci-info: ++++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++++
[ 19.388533] cloud-init[1207]: ci-info: +------------+-------+------------------------------+-------------+-------+-------------------+
[ 19.408484] cloud-init[1207]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
[ 19.428484] cloud-init[1207]: ci-info: +------------+-------+------------------------------+-------------+-------+-------------------+
[ 19.444480] cloud-init[1207]: ci-info: | enP2p1s0f1 | False | . | . | . | 40:8d:5c:ba:b9:10 |
[ 19.464490] cloud-init[1207]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | . | . |
[ 19.484480] cloud-init[1207]: ci-info: | lo | True | ::1/128 | . | host | . |
[ 19.500475] cloud-init[1207]: ci-info: | enP2p1s0f3 | False | . | . | . | 40:8d:5c:ba:b9:12 |
[ 19.520503] cloud-init[1207]: ci-info: | enP2p1s0f2 | True | 10.246.48.3 | 255.255.0.0 | . | 40:8d:5c:ba:b9:11 |
[ 19.536497] cloud-init[1207]: ci-info: | enP2p1s0f2 | True | fe80::428d:5cff:feba:b911/64 | . | link | 40:8d:5c:ba:b9:11 |
[ 19.556478] cloud-init[1207]: ci-info: +------------+-------+------------------------------+-------------+-------+-------------------+
[ 19.576514] cloud-init[1207]: ci-info: ++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++
[ 19.592494] cloud-init[1207]: ci-info: +-------+-------------+-------------+-------------+------------+-------+
[ 19.608504] cloud-init[1207]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
[ 19.624535] cloud-init[1207]: ci-info: +-------+-------------+-------------+-------------+------------+-------+
[ 19.640528] cloud-init[1207]: ci-info: | 0 | 0.0.0.0 | 10.246.48.1 | 0.0.0.0 | enP2p1s0f2 | UG |
[ 19.656495] cloud-init[1207]: ci-info: | 1 | 10.246.0.0 | 0.0.0.0 | 255.255.0.0 | enP2p1s0f2 | U |
[ 19.672513] cloud-init[1207]: ci-info: +-------+-------------+-------------+-------------+------------+-------+
[ 19.688941] cloud-init[1207]: 2016-02-11 16:28:07,035 - url_helper.py[WARNING]: Setting oauth clockskew for 10.246.48.112 to 7938974
[ 19.704762] cloud-init[1207]: 2016-02-11 16:28:07,036 - handlers.py[WARNING]: failed posting event: start: init-network/check-cache: attempting to read from cache [trust]
[ OK ] Started Seed the pseudo random number generator on first boot.
[ OK ] Started System Logging Service.
[ 30.077613] cloud-init[1207]: 2016-02-11 16:28:17,758 - cc_resizefs.py[WARNING]: Device 'overlayroot' did not exist. cannot resize: dev=overlayroot mnt_point=/ path=/
Stopping System Logging Service...
[ OK ] Stopped System Logging Service.
Starting System Logging Service...
[ OK ] Started System Logging Service.
[ 45.015651] cloud-init[1207]: Generating public/private rsa key pair.
[ 45.024537] cloud-init[1207]: Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
[ 45.036528] cloud-init[1207]: Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.
[ 45.048531] cloud-init[1207]: The key fingerprint is:
[ 45.056494] cloud-init[1207]: SHA256:I+KDEsp0Fe54XuAYVaP2sqXyZqt6CM++/tgX7NQat18 root@cvm15
[ 45.068940] cloud-init[1207]: The key's randomart image is:
[ 45.076665] cloud-init[1207]: +---[RSA 2048]----+
[ 45.088500] cloud-init[1207]: | o.o |
[ 45.096499] cloud-init[1207]: | o o . |
[ 45.104568] cloud-init[1207]: | . * |
[ 45.112575] cloud-init[1207]: | O o |
[ 45.120535] cloud-init[1207]: |.. +.*.=S |
[ 45.128452] cloud-init[1207]: |=..oo.@.o. |
[ OK ] Started Initial cloud-init job (metadata service crawler).
[ 45.136445] cloud-init[1207]: |++o.o* = . E |
[ OK ] Reached target Network is Online.
[ 45.156253] cloud-init[1207]: | oo+o++ . . |
[ 45.172228] cloud-init[1207]: | +B+*+. .. |
[ 45.180217] cloud-init[1207]: +----[SHA256]-----+
Starting /etc/rc.local Compatibility...
[ 45.188245] cloud-init[1207]: Generating public/private dsa key pair.
Starting iSCSI initiator daemon (iscsid)...
[ 45.208228] cloud-init[1207]: Your identification has been saved in /etc/ssh/ssh_host_dsa_key.
Starting OpenBSD Secure Shell server...
[ 45.228241] cloud-init[1207]: [Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub. OK
] Reached target Cloud-config availability.
[ 45.240327] cloud-init[1207]: The key fingerprint is:
[ 45.272221] cloud-init[1207]: SHA256:WE+uOqGNjKmwyhiQEzYaSEZ/0aRSyPTkrpjyPQApBpE root@cvm15
[ 45.284217] cloud-init[1207]: The key's randomart image is:
[ 45.292209] cloud-init[1207]: +---[DSA 1024]----+
Starting Apply the settings specified in cloud-config...
[ 45.300245] cloud-init[1207]: |o*o..+o. |
[ OK ] Started /etc/rc.local Compatibility.
[ 45.320227] cloud-init[1207]: |E .o= o. |
[ 45.336316] cloud-init[1207]: |=+ o = . . |
[ OK ] Started iSCSI initiator daemon (iscsid).
[ 45.344249] cloud-init[1207]: |*=. + o + |
[ 45.360228] cloud-init[1207]: |B. .. S o |
[ 45.368226] cloud-init[1207]: |..+ . . . |
[ 45.376226] cloud-init[1207]: |+o * + .. |
[ 45.384260] cloud-init[1207]: |=+o.= o. |
[ 45.392228] cloud-init[1207]: |*o. .... |
[ 45.400209] cloud-init[1207]: +----[SHA256]-----+
[ 45.408233] cloud-init[1207]: Generating public/private ecdsa key pair.
Starting Login to default iSCSI targets...
[ 45.420209] cloud-init[1207]: Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key.
[ 45.440316] cloud-init[1207]: Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub.
[ 45.452238] cloud-init[1207]: The key fingerprint is:
[ 45.460232] cloud-init[1207]: SHA256:DddLJKqvTZHzKPJ65EN9aqYavYD/wd+VuJz1eqnbgoI root@cvm15
[ 45.472220] cloud-init[1207]: The key's randomart image is:
[ 45.480211] cloud-init[1207]: +---[ECDSA 256]---+
[ 45.488276] cloud-init[1207]: | . . |
[ 45.496207] cloud-init[1207]: | . + |
[ 45.504208] cloud-init[1207]: | o . o |
[ 45.512211] cloud-init[1207]: | . = . . |
[ 45.520203] cloud-init[1207]: | ..S . . |
[ 45.528233] cloud-init[1207]: | . oo..=.. . |
[ 45.536201] cloud-init[1207]: | . ++=.+oo.+ . |
[ 45.544224] cloud-init[1207]: | . =EO=o.=.oo |
[ 45.552204] cloud-init[1207]: | +=*=o.= +*o |
[ 45.560215] cloud-init[1207]: +----[SHA256]-----+
[ 45.568201] cloud-init[1207]: Generating public/private ed25519 key pair.
[ 45.580205] cloud-init[1207]: Your identification has been saved in /etc/ssh/ssh_host_ed25519_key.
[ 45.592205] cloud-init[1207]: Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub.
[ 45.604207] cloud-init[1207]: The key fingerprint is:
[ 45.612201] cloud-init[1207]: SHA256:E5SSzahN/DmxED4slp4ynrvH052zQXFFkok/MQ8r5yw root@cvm15
[ 45.624243] cloud-init[1207]: The key's randomart image is:
[ 45.632201] cloud-init[1207]: +--[ED25519 256]--+
[ 45.640211] cloud-init[1207]: | ..*.o.+o |
[ 45.648209] cloud-init[1207]: | +B.* *o |
[ 45.656205] cloud-init[1207]: | ++++o=.* |
[ 45.664206] cloud-init[1207]: | o.o..*+= . |
[ 45.672204] cloud-init[1207]: | o o S* . |
[ 45.680223] cloud-init[1207]: | . + .E.o |
[ 45.688208] cloud-init[1207]: | o. . ..o |
[ 45.696226] cloud-init[1207]: | .+ . +. |
[ 45.704225] cloud-init[1207]: | oo . .o |
[ 45.712233] cloud-init[1207]: +----[SHA256]-----+
[ OK ] Started OpenBSD Secure Shell server.
[ OK ] Started Login to default iSCSI targets.
[ OK ] Reached target Remote File Systems (Pre).
[ OK ] Reached target Remote File Systems.
Starting LSB: daemon to balance interrupts for SMP systems...
Starting Permit User Sessions...
Starting LSB: automatic crash report generation...
Starting LSB: Set the CPU Frequency Scaling governor to "ondemand"...
[ OK ] Started Permit User Sessions.
Starting Terminate Plymouth Boot Screen...
Starting Hold until boot process finishes up...
[ OK ] Started LSB: automatic crash report generation.
[ OK ] Started LSB: Set the CPU Frequency Scaling governor to "ondemand".
[ OK ] Started Terminate Plymouth Boot Screen.
[ OK ] Started Hold until boot process finishes up.
[ OK ] Started Serial Getty on ttyAMA0.
[ OK ] Started Getty on tty1.
[ OK ] Reached target Login Prompts.
[ OK ] Started LSB: daemon to balance interrupts for SMP systems.
[ 50.198320] cloud-init[1352]: Generating locales (this might take a while)...
[ 52.582921] cloud-init[1352]: en_US.UTF-8... done
[ 52.592665] cloud-init[1352]: Generation complete.
Ubuntu 16.04 LTS cvm15 ttyAMA0
cvm15 login: [ 131.235429] cloud-init[1352]: Cloud-init v. 0.7.7 running 'modules:config' at Thu, 11 Feb 2016 16:28:34 +0000. Up 46.61 seconds.
[ 135.455232] cloud-init[1459]: tar: ./curtin/deps/__init__.py: time stamp 2016-03-18 14:16:44 is 3102400.8621135 s in the future
[ 135.455801] cloud-init[1459]: tar: ./curtin/deps/check.py: time stamp 2016-03-18 14:16:44 is 3102400.86187222 s in the future
[ 135.456327] cloud-init[1459]: tar: ./curtin/deps/install.py: time stamp 2016-03-18 14:16:44 is 3102400.86172929 s in the future
[ 135.456824] cloud-init[1459]: tar: ./curtin/deps: time stamp 2016-05-12 12:42:12 is 7848728.86163933 s in the future
[ 135.457315] cloud-init[1459]: tar: ./curtin/__init__.py: time stamp 2016-03-18 14:16:44 is 3102400.86152015 s in the future
[ 135.457805] cloud-init[1459]: tar: ./curtin/udev.py: time stamp 2016-03-18 14:16:44 is 3102400.86139546 s in the future
[ 135.458353] cloud-init[1459]: tar: ./curtin/config.py: time stamp 2016-03-18 14:16:44 is 3102400.86125462 s in the future
[ 135.458847] cloud-init[1459]: tar: ./curtin/commands/block_meta.py: time stamp 2016-03-18 14:16:44 is 3102400.86097206 s in the future
[ 135.459338] cloud-init[1459]: tar: ./curtin/commands/apply_net.py: time stamp 2016-03-18 14:16:44 is 3102400.86084094 s in the future
[ 135.459835] cloud-init[1459]: tar: ./curtin/commands/extract.py: time stamp 2016-03-18 14:16:44 is 3102400.86071796 s in the future
[ 135.460352] cloud-init[1459]: tar: ./curtin/commands/__init__.py: time stamp 2016-03-18 14:16:44 is 3102400.8605888 s in the future
[ 135.460848] cloud-init[1459]: tar: ./curtin/commands/hook.py: time stamp 2016-03-18 14:16:44 is 3102400.86046616 s in the future
[ 135.461479] cloud-init[1459]: tar: ./curtin/commands/mkfs.py: time stamp 2016-03-18 14:16:44 is 3102400.8603415 s in the future
[ 135.461969] cloud-init[1459]: tar: ./curtin/commands/swap.py: time stamp 2016-03-18 14:16:44 is 3102400.86021895 s in the future
[ 135.462461] cloud-init[1459]: tar: ./curtin/commands/curthooks.py: time stamp 2016-03-18 14:16:44 is 3102400.86004385 s in the future
[ 135.462980] cloud-init[1459]: tar: ./curtin/commands/net_meta.py: time stamp 2016-03-18 14:16:44 is 3102400.85990484 s in the future
[ 135.463591] cloud-init[1459]: tar: ./curtin/commands/in_target.py: time stamp 2016-03-18 14:16:44 is 3102400.85977304 s in the future
[ 135.464143] cloud-init[1459]: tar: ./curtin/commands/system_upgrade.py: time stamp 2016-03-18 14:16:44 is 3102400.85964961 s in the future
[ 135.464718] cloud-init[1459]: tar: ./curtin/commands/install.py: time stamp 2016-03-18 14:16:44 is 3102400.85948571 s in the future
[ 135.465299] cloud-init[1459]: tar: ./curtin/commands/pack.py: time stamp 2016-03-18 14:16:44 is 3102400.85935982 s in the future
[ 135.465940] cloud-init[1459]: tar: ./curtin/commands/main.py: time stamp 2016-03-18 14:16:44 is 3102400.85921463 s in the future
[ 135.466576] cloud-init[1459]: tar: ./curtin/commands/system_install.py: time stamp 2016-03-18 14:16:44 is 3102400.85909071 s in the future
[ 135.467188] cloud-init[1459]: tar: ./curtin/commands: time stamp 2016-05-12 12:42:12 is 7848728.85900553 s in the future
[ 135.467857] cloud-init[1459]: tar: ./curtin/futil.py: time stamp 2016-03-18 14:16:44 is 3102400.85888899 s in the future
[ 135.468575] cloud-init[1459]: tar: ./curtin/log.py: time stamp 2016-03-18 14:16:44 is 3102400.85876914 s in the future
[ 135.469142] cloud-init[1459]: tar: ./curtin/swap.py: time stamp 2016-03-18 14:16:44 is 3102400.85864809 s in the future
[ 135.469725] cloud-init[1459]: tar: ./curtin/net/__init__.py: time stamp 2016-03-18 14:16:44 is 3102400.85846213 s in the future
[ 135.470302] cloud-init[1459]: tar: ./curtin/net/network_state.py: time stamp 2016-03-18 14:16:44 is 3102400.85830302 s in the future
[ 135.470891] cloud-init[1459]: tar: ./curtin/net: time stamp 2016-05-12 12:42:12 is 7848728.8582177 s in the future
[ 135.471392] cloud-init[1459]: tar: ./curtin/url_helper.py: time stamp 2016-03-18 14:16:44 is 3102400.85809278 s in the future
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1581553/+subscriptions
