maas-builds team mailing list archive

Thread
Date
Jenkins Still Failing - trusty-adt-maas-daily 55

To: maas-builds@xxxxxxxxxxxxxxxxxxx
From: Jenkins Notification <devnull@xxxxxxxxxxxxx>
Date: Tue, 21 Jan 2014 19:11:37 +0000 (UTC)
In-reply-to: <579168599.440.1390316145938.JavaMail.jenkins@tachash>
See http://d-jenkins.ubuntu-ci:8080/job/trusty-adt-maas-daily/55/

[...truncated 7092 lines...]
maas-integration.TestMAASIntegration.test_restart_dbus_avahi ... SKIP: Avahi/DBUS are not used anymore
maas-integration.TestMAASIntegration.test_update_maas_url ... ok
maas-integration.TestMAASIntegration.test_restart_provisioning_server ... ok
maas-integration.TestMAASIntegration.test_check_initial_services ... ok
maas-integration.TestMAASIntegration.test_update_pxe_config ... ok
maas-integration.TestMAASIntegration.test_import_pxe_files ... ok
maas-integration.TestMAASIntegration.test_update_preseed_arm ... ok
maas-integration.TestMAASIntegration.test_login_api ... ok
maas-integration.TestMAASIntegration.test_set_http_proxy ... SKIP: Not testing proxy.
maas-integration.TestMAASIntegration.test_cluster_connected ... ok
maas-integration.TestMAASIntegration.test_set_up_dhcp_region ... ok
maas-integration.TestMAASIntegration.test_check_dhcp_service ... ok
maas-integration.TestMAASIntegration.test_set_up_dhcp_cluster ... SKIP: Not testing cluster controller
maas-integration.TestMAASIntegration.test_update_dns_config ... ok
maas-integration.TestMAASIntegration.test_boot_nodes_enlist ... ok
maas-integration.TestMAASIntegration.test_check_nodes_declared ... ok
maas-integration.TestMAASIntegration.test_set_nodes_ipmi_config ... ok
maas-integration.TestMAASIntegration.test_commission_nodes ... ok
maas-integration.TestMAASIntegration.test_check_nodes_ready ... ok
maas-integration.TestMAASIntegration.test_apply_tag_to_all_nodes ... ok
maas-integration.TestMAASIntegration.test_check_tag_applied_to_all_nodes ... ok
maas-integration.TestMAASIntegration.test_configure_juju ... ok
maas-integration.TestMAASIntegration.test_juju_bootstrap ... ERROR
SKIP: Not testing Cluster controller

======================================================================
ERROR: maas-integration.TestMAASIntegration.test_juju_bootstrap
----------------------------------------------------------------------
_StringException: juju --debug bootstrap --upload-tools: {{{
2014-01-21 18:22:00 DEBUG juju.environs.configstore disk.go:77 Making /root/.juju/environments
2014-01-21 18:22:00 DEBUG juju.environs.configstore disk.go:67 Making /root/.juju/environments owned by 1000:1000
2014-01-21 18:22:00 DEBUG juju.environs.configstore disk.go:67 Making /root/.juju/environments/maas.jenv owned by 1000:1000
2014-01-21 18:22:00 DEBUG juju.provider.maas environprovider.go:33 opening environment "maas".
2014-01-21 18:22:00 DEBUG juju.environs.configstore disk.go:67 Making /root/.juju/environments/maas.jenv owned by 1000:1000
2014-01-21 18:22:00 DEBUG juju.environs.sync sync.go:211 Uploading tools for [trusty precise]
2014-01-21 18:22:00 DEBUG juju.environs.tools build.go:115 looking for: juju
2014-01-21 18:22:00 DEBUG juju.environs.tools build.go:156 checking: /usr/bin/jujud
2014-01-21 18:22:00 INFO juju.environs.tools build.go:162 found existing jujud
2014-01-21 18:22:00 INFO juju.environs.tools build.go:172 target: /tmp/adt-run.Imxr9m/ubtree0t-maas-package-test-testtmp/tmpdir/juju-tools537419253/jujud
2014-01-21 18:22:01 DEBUG juju.environs.tools build.go:223 forcing version to 1.17.0.1
2014-01-21 18:22:01 DEBUG juju.environs.tools build.go:39 adding entry: &tar.Header{Name:"FORCE-VERSION", Mode:420, Uid:0, Gid:0, Size:8, ModTime:time.Time{sec:63525925321, nsec:0x69832c0, loc:(*time.Location)(0x1701340)}, Typeflag:0x30, Linkname:"", Uname:"ubuntu", Gname:"ubuntu", Devmajor:0, Devminor:0, AccessTime:time.Time{sec:63525925321, nsec:0x69832c0, loc:(*time.Location)(0x1701340)}, ChangeTime:time.Time{sec:63525925321, nsec:0x69832c0, loc:(*time.Location)(0x1701340)}}
2014-01-21 18:22:01 DEBUG juju.environs.tools build.go:39 adding entry: &tar.Header{Name:"jujud", Mode:493, Uid:0, Gid:0, Size:32749104, ModTime:time.Time{sec:63525925321, nsec:0x69832c0, loc:(*time.Location)(0x1701340)}, Typeflag:0x30, Linkname:"", Uname:"ubuntu", Gname:"ubuntu", Devmajor:0, Devminor:0, AccessTime:time.Time{sec:63525925321, nsec:0x69832c0, loc:(*time.Location)(0x1701340)}, ChangeTime:time.Time{sec:63525925321, nsec:0x69832c0, loc:(*time.Location)(0x1701340)}}
2014-01-21 18:22:04 INFO juju.environs.sync sync.go:230 built 1.17.0.1-trusty-amd64 (6562kB)
2014-01-21 18:22:04 INFO juju.environs.sync sync.go:71 listing available tools
2014-01-21 18:22:04 DEBUG juju.environs.tools storage.go:35 reading v1.* tools
2014-01-21 18:22:04 DEBUG juju.environs.tools storage.go:62 found 1.17.0.1-precise-amd64
2014-01-21 18:22:04 DEBUG juju.environs.tools storage.go:62 found 1.17.0.1-trusty-amd64
2014-01-21 18:22:04 INFO juju.environs.sync sync.go:95 found 2 tools
2014-01-21 18:22:04 DEBUG juju.environs.sync sync.go:102 found source tool: &{1.17.0.1-precise-amd64 file:///tmp/adt-run.Imxr9m/ubtree0t-maas-package-test-testtmp/tmpdir/597527760/tools/releases/juju-1.17.0.1-precise-amd64.tgz  0}
2014-01-21 18:22:04 DEBUG juju.environs.sync sync.go:102 found source tool: &{1.17.0.1-trusty-amd64 file:///tmp/adt-run.Imxr9m/ubtree0t-maas-package-test-testtmp/tmpdir/597527760/tools/releases/juju-1.17.0.1-trusty-amd64.tgz  0}
2014-01-21 18:22:04 INFO juju.environs.sync sync.go:105 listing target bucket
2014-01-21 18:22:04 DEBUG juju.environs.tools storage.go:35 reading v1.* tools
2014-01-21 18:22:05 INFO juju.environs.sync sync.go:118 found 0 tools in target; 2 tools to be copied
2014-01-21 18:22:05 INFO juju.environs.sync sync.go:156 copying 1.17.0.1-precise-amd64 from file:///tmp/adt-run.Imxr9m/ubtree0t-maas-package-test-testtmp/tmpdir/597527760/tools/releases/juju-1.17.0.1-precise-amd64.tgz
2014-01-21 18:22:05 INFO juju.environs.sync sync.go:170 copying tools/releases/juju-1.17.0.1-precise-amd64.tgz
2014-01-21 18:22:05 INFO juju.environs.sync sync.go:184 downloaded tools/releases/juju-1.17.0.1-precise-amd64.tgz (6562kB), uploading
2014-01-21 18:22:05 INFO juju.environs.sync sync.go:185 download 6562kB, uploading
2014-01-21 18:22:05 INFO juju.environs.sync sync.go:156 copying 1.17.0.1-trusty-amd64 from file:///tmp/adt-run.Imxr9m/ubtree0t-maas-package-test-testtmp/tmpdir/597527760/tools/releases/juju-1.17.0.1-trusty-amd64.tgz
2014-01-21 18:22:05 INFO juju.environs.sync sync.go:170 copying tools/releases/juju-1.17.0.1-trusty-amd64.tgz
2014-01-21 18:22:05 INFO juju.environs.sync sync.go:184 downloaded tools/releases/juju-1.17.0.1-trusty-amd64.tgz (6562kB), uploading
2014-01-21 18:22:05 INFO juju.environs.sync sync.go:185 download 6562kB, uploading
2014-01-21 18:22:06 INFO juju.environs.sync sync.go:123 copied 2 tools
2014-01-21 18:22:06 INFO juju.environs.sync sync.go:125 generating tools metadata
2014-01-21 18:22:06 DEBUG juju.environs.tools tools.go:44 no architecture specified when finding tools, looking for any
2014-01-21 18:22:06 DEBUG juju.environs.tools tools.go:55 no series specified when finding tools, looking for any
2014-01-21 18:22:06 DEBUG juju.environs.simplestreams simplestreams.go:481 fetchData failed for "tools/streams/v1/index.sjson": file 'b80bdfec-8317-4845-823c-8a3ed5893999-tools/streams/v1/index.sjson' not found not found
2014-01-21 18:22:06 DEBUG juju.environs.simplestreams simplestreams.go:458 cannot load index "streams/v1/index.sjson": invalid URL "tools/streams/v1/index.sjson" not found
2014-01-21 18:22:06 DEBUG juju.environs.simplestreams simplestreams.go:481 fetchData failed for "tools/streams/v1/index.json": file 'b80bdfec-8317-4845-823c-8a3ed5893999-tools/streams/v1/index.json' not found not found
2014-01-21 18:22:06 DEBUG juju.environs.simplestreams simplestreams.go:458 cannot load index "streams/v1/index.json": invalid URL "tools/streams/v1/index.json" not found
2014-01-21 18:22:06 INFO juju.environs.tools simplestreams.go:357 Writing tools/streams/v1/index.json
2014-01-21 18:22:06 INFO juju.environs.tools simplestreams.go:357 Writing tools/streams/v1/com.ubuntu.juju:released:tools.json
2014-01-21 18:22:06 INFO juju.environs.sync sync.go:137 tools metadata written
2014-01-21 18:22:08 INFO juju.environs.bootstrap bootstrap.go:46 bootstrapping environment "maas"
2014-01-21 18:22:08 DEBUG juju.environs.bootstrap synctools.go:91 looking for bootstrap tools: series="precise", arch=<nil>, version=1.17.0.1
2014-01-21 18:22:08 INFO juju.environs.tools tools.go:85 reading tools with major.minor version 1.17
2014-01-21 18:22:08 INFO juju.environs.tools tools.go:93 filtering tools by version: 1.17.0.1
2014-01-21 18:22:08 INFO juju.environs.tools tools.go:96 filtering tools by series: precise
2014-01-21 18:22:08 DEBUG juju.environs.tools tools.go:44 no architecture specified when finding tools, looking for any
2014-01-21 18:22:08 DEBUG juju.environs.simplestreams simplestreams.go:481 fetchData failed for "tools/streams/v1/index.sjson": file 'b80bdfec-8317-4845-823c-8a3ed5893999-tools/streams/v1/index.sjson' not found not found
2014-01-21 18:22:08 DEBUG juju.environs.simplestreams simplestreams.go:458 cannot load index "streams/v1/index.sjson": invalid URL "tools/streams/v1/index.sjson" not found
2014-01-21 18:22:09 DEBUG juju.environs.simplestreams simplestreams.go:481 fetchData failed for "tools/streams/v1/mirrors.json": file 'b80bdfec-8317-4845-823c-8a3ed5893999-tools/streams/v1/mirrors.json' not found not found
2014-01-21 18:22:09 DEBUG juju.environs.simplestreams simplestreams.go:557 no mirror index file found
2014-01-21 18:22:09 DEBUG juju.environs.simplestreams simplestreams.go:538 no mirror information available for { }: mirror data for "com.ubuntu.juju:released:tools" not found
2014-01-21 18:22:09 DEBUG juju.environs.simplestreams simplestreams.go:462 read metadata index at "http://192.168.21.5:80/MAAS/api/1.0/files/?key=ef21fb9a-82c8-11e3-bfee-525400123456&op=get_by_key";
2014-01-21 18:22:09 DEBUG juju.environs.simplestreams simplestreams.go:625 candidate matches for products ["com.ubuntu.juju:12.04:amd64" "com.ubuntu.juju:12.04:i386" "com.ubuntu.juju:12.04:arm"] are [{Tue, 21 Jan 2014 13:22:06 -0500 products:1.0 content-download  [] streams/v1/com.ubuntu.juju:released:tools.json [com.ubuntu.juju:12.04:amd64 com.ubuntu.juju:14.04:amd64]}]
2014-01-21 18:22:09 DEBUG juju.environs.simplestreams simplestreams.go:937 finding products at path "streams/v1/com.ubuntu.juju:released:tools.json"
2014-01-21 18:22:09 DEBUG juju.environs.simplestreams simplestreams.go:974 metadata: &{map[com.ubuntu.juju:12.04:amd64:{ 1.17.0.1 amd64   map[20140121:0xc212d11780]} com.ubuntu.juju:14.04:amd64:{ 1.17.0.1 amd64   map[20140121:0xc212d118a0]}] map[] Tue, 21 Jan 2014 13:22:06 -0500 products:1.0 com.ubuntu.juju:released:tools}
2014-01-21 18:22:10 INFO juju.environs.bootstrap bootstrap.go:58 picked newest version: 1.17.0.1
Launching instance
2014-01-21 18:22:10 WARNING juju.provider.maas environ.go:193 picked arbitrary tools &{"1.17.0.1-precise-amd64" "http://192.168.21.5:80/MAAS/api/1.0/files/?key=ee3331c2-82c8-11e3-bfee-525400123456&op=get_by_key"; "11b358171b379889c7b23bb8fb2265a55b6c4208925e9061ae16e68b1ec99ea7" %!q(int64=6719588)}
2014-01-21 18:22:10 DEBUG juju.provider.maas environ.go:279 maas user data; 717 bytes
2014-01-21 18:22:11 DEBUG juju.provider.maas environ.go:285 started instance "/MAAS/api/1.0/nodes/node-58fdd382-82c8-11e3-bfee-525400123456/"
 - /MAAS/api/1.0/nodes/node-58fdd382-82c8-11e3-bfee-525400123456/
Waiting for address
Attempting to connect to dqed6.master:22
Attempting to connect to 192.168.21.11:22
2014-01-21 18:32:11 ERROR juju.provider.common bootstrap.go:85 bootstrap failed: waited for 10m0s without being able to connect: ssh: connect to host 192.168.21.11 port 22: Connection refused
Stopping instance...
2014-01-21 18:32:11 ERROR juju.cmd supercommand.go:294 waited for 10m0s without being able to connect: ssh: connect to host 192.168.21.11 port 22: Connection refused
}}}

juju --debug status: {{{
2014-01-21 19:11:24 DEBUG juju.provider.maas environprovider.go:33 opening environment "maas".
2014-01-21 19:11:24 WARNING juju api.go:185 discarding API open error: <nil>
2014-01-21 19:11:24 ERROR juju.cmd supercommand.go:294 Unable to connect to environment "".
Please check your credentials or use 'juju bootstrap' to create a new environment.

Error details:
environment is not bootstrapped
}}}

Traceback (most recent call last):
  File "/tmp/adt-run.Imxr9m/ubtree0-build/real-tree/debian/tests/utils.py", line 69, in wrapper
    result = func(*args, **kwargs)
  File "/tmp/adt-run.Imxr9m/ubtree0-build/real-tree/debian/tests/maas-integration.py", line 618, in test_juju_bootstrap
    self._wait_machines_running(1)
  File "/tmp/adt-run.Imxr9m/ubtree0-build/real-tree/debian/tests/maas-integration.py", line 632, in _wait_machines_running
    sleep(20)
  File "/tmp/adt-run.Imxr9m/ubtree0-build/real-tree/debian/tests/utils.py", line 63, in _handle_timeout
    raise TimeoutError(error_message)
TimeoutError: Timer expired


maas-integration.TestMAASIntegration.test_update_pxe_config: 0.0005s
maas-integration.TestMAASIntegration.test_check_dhcp_service: 0.0205s
maas-integration.TestMAASIntegration.test_update_preseed_arm: 0.0396s
maas-integration.TestMAASIntegration.test_check_initial_services: 0.0496s
maas-integration.TestMAASIntegration.test_restart_provisioning_server: 0.0736s
maas-integration.TestMAASIntegration.test_create_admin: 0.1153s
maas-integration.TestMAASIntegration.test_cluster_connected: 0.2113s
maas-integration.TestMAASIntegration.test_apply_tag_to_all_nodes: 0.2445s
maas-integration.TestMAASIntegration.test_check_tag_applied_to_all_nodes: 0.2488s
maas-integration.TestMAASIntegration.test_commission_nodes: 0.4184s
maas-integration.TestMAASIntegration.test_boot_nodes_enlist: 0.5231s
maas-integration.TestMAASIntegration.test_update_maas_url: 1.1216s
maas-integration.TestMAASIntegration.test_update_dns_config: 1.3600s
maas-integration.TestMAASIntegration.test_login_api: 1.3933s
maas-integration.TestMAASIntegration.test_set_up_dhcp_region: 2.5348s
maas-integration.TestMAASIntegration.test_set_nodes_ipmi_config: 2.7386s
maas-integration.TestMAASIntegration.test_configure_juju: 5.8804s
maas-integration.TestMAASIntegration.test_check_nodes_declared: 162.7519s
maas-integration.TestMAASIntegration.test_check_nodes_ready: 199.8625s
maas-integration.TestMAASIntegration.test_import_pxe_files: 954.0842s
maas-integration.TestMAASIntegration.test_juju_bootstrap: 3000.0233s
----------------------------------------------------------------------
Ran 24 tests in 4333.858s

FAILED (SKIP=4, errors=1)
adt-run1: testbed executing test finished with exit status 1
adt-run: & ubtree0t-maas-package-test: ----------------------------------------]
adt-run: & ubtree0t-maas-package-test:  - - - - - - - - - - results - - - - - - - - - -
ubtree0t-maas-package-test FAIL non-zero exit status 1
adt-run1: ** needs_reset, previously=False
adt-run: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ tests done.
adt-run1: ** stop
adt-run1: ** close, scratch=/tmp/adt-run.Imxr9m
+ RC=4
+ [ 4 -eq 20 ]
+ [ 0 -eq 1 ]
+ [ -x /home/ubuntu/adt-export-result ]
+ RES=PASS
+ [ 4 -gt 0 ]
+ RES=FAIL
+ /home/ubuntu/adt-export-result -D /root/adt-log maas FAIL
+ chown -R ubuntu /root/adt-log /var/tmp/testresults
+ chmod og+r /var/log/syslog
+ ls /var/crash/
+ [ -n  ]
+ exit 4
Connection to localhost closed.
+ RET=4
+ [ 0 -eq 1 ]
+ [ 4 -gt 0 ]
+ log_failure_msg adt-run exited with status 4.
+ log_msg Failure: adt-run exited with status 4.\n
+ date +%F %X
+ printf 2014-01-21 14:11:34: Failure: adt-run exited with status 4.\n
2014-01-21 14:11:34: Failure: adt-run exited with status 4.
+ [ 0 -eq 0 ]
+ mkdir -p /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results
+ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o CheckHostIP=no -i /var/tmp/adt/disks/adtkey -p 54323 -tt -o BatchMode=yes -l ubuntu localhost sudo chown -R ubuntu /root/adt-log; find /root/adt-log -type f -empty | xargs rm 2>/dev/null
Warning: Permanently added '[localhost]:54323' (ECDSA) to the list of known hosts.
find: `/root/adt-log': Permission denied
Connection to localhost closed.
+ true
+ scp -r -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o CheckHostIP=no -i /var/tmp/adt/disks/adtkey -P 54323 ubuntu@localhost:/root/adt-log/* /var/crash/*crash /var/log/syslog /var/tmp/testresults /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results
+ true
+ log_info_msg Test artifacts copied to /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results
+ log_msg Info: Test artifacts copied to /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results\n
+ date +%F %X
+ printf 2014-01-21 14:11:36: Info: Test artifacts copied to /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results\n
2014-01-21 14:11:36: Info: Test artifacts copied to /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results
+ [ -f /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results/summary.log ]
+ [ -n 2014-01-21_17-49-58 ]
+ + ls -tr /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results/*.result
tail -1
+ resfile=
+ [ -n  ]
+ log_failure_msg Test didn't end normally. Generating error file
+ log_msg Failure: Test didn't end normally. Generating error file\n
+ date +%F %X
+ printf 2014-01-21 14:11:36: Failure: Test didn't end normally. Generating error file\n
2014-01-21 14:11:36: Failure: Test didn't end normally. Generating error file
+ date +%Y%m%d-%H%M%S
+ errfile=/home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results/trusty_amd64_maas_20140121-141136.error
+ echo trusty amd64 maas
+ rsync -a /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results/trusty_amd64_maas_20140121-141136.error /trusty/tmp/
rsync: mkdir "/trusty/tmp" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(674) [Receiver=3.1.0]
+ true
+ [ 0 -eq 0 ]
+ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o CheckHostIP=no -i /var/tmp/adt/disks/adtkey -p 54323 -tt -o BatchMode=yes -l ubuntu localhost sudo poweroff
Warning: Permanently added '[localhost]:54323' (ECDSA) to the list of known hosts.


Broadcast message from ubuntu@autopkgtest

	(/dev/pts/0) at 14:11 ...




The system is going down for power off NOW!

Connection to localhost closed.
+ exit 4
+ on_exit
+ log_info_msg Cleaning up
+ log_msg Info: Cleaning up\n
+ date +%F %X
+ printf 2014-01-21 14:11:36: Info: Cleaning up\n
2014-01-21 14:11:36: Info: Cleaning up
+ [ -f /var/tmp/adt/disks/run/trusty-amd64-maas-20140121_125029.ueKp8x.img.pid ]
+ cat /var/tmp/adt/disks/run/trusty-amd64-maas-20140121_125029.ueKp8x.img.pid
+ kill -9 6783
+ rm -f /var/tmp/adt/disks/run/trusty-amd64-maas-20140121_125029.ueKp8x.img.pid
+ rm -f /var/tmp/adt/disks/run/trusty-amd64-maas-20140121_125029.ueKp8x.img
+ rm -f /var/tmp/adt/disks/run/trusty-amd64-maas-20140121_125029.ueKp8x.img.monitor /var/tmp/adt/disks/run/trusty-amd64-maas-20140121_125029.ueKp8x.img.serial
+ rm -f /var/lock/adt/ssh.54323.lock
+ rm -f /var/lock/adt/vnc.5911.lock
+ [ -d /tmp/adt-amd64.QIpo2U ]
+ rm -Rf /tmp/adt-amd64.QIpo2U
+ rm -f /var/tmp/adt/disks/run/trusty-amd64-maas-*.img*
+ find /var/lock/adt -name *.lock -mtime +1
+ exit 4
Build step 'Execute shell' marked build as failure
Archiving artifacts
Email was triggered for: Failure
Sending email for trigger: Failure
Follow ups

Jenkins Still Failing - trusty-adt-maas-daily 56
From: Jenkins Notification, 2014-01-22
References

Jenkins Still Failing - trusty-adt-maas-daily 54
From: Jenkins Notification, 2014-01-21