maas-builds team mailing list archive

Thread
Date
Jenkins Still Failing - trusty-adt-maas-daily 37

To: maas-builds@xxxxxxxxxxxxxxxxxxx
From: Jenkins Notification <devnull@xxxxxxxxxxxxx>
Date: Tue, 14 Jan 2014 18:27:08 +0000 (UTC)
In-reply-to: <1453028704.551.1389664926829.JavaMail.jenkins@tachash>
See http://d-jenkins.ubuntu-ci:8080/job/trusty-adt-maas-daily/37/

[...truncated 6654 lines...]

    CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']

You must only enable the serializers that you will actually use.


  warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))

[2014-01-14 13:05:41,583: INFO/Beat] beat: Starting...
[2014-01-14 13:05:41,599: INFO/MainProcess] Connected to amqp://guest@localhost:5672//
[2014-01-14 13:05:41,615: WARNING/Beat] Reset: Account for new __version__ field
[2014-01-14 13:05:41,616: INFO/MainProcess] mingle: searching for neighbors
[2014-01-14 13:05:41,615: WARNING/Beat] Reset: Account for new tz field
[2014-01-14 13:05:41,617: WARNING/Beat] Reset: Account for new utc_enabled field
[2014-01-14 13:05:42,627: INFO/MainProcess] mingle: all alone
[2014-01-14 13:05:42,665: WARNING/MainProcess] celery@autopkgtest ready.
[2014-01-14 13:05:49,542: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/celery/worker/consumer.py", line 270, in start
    blueprint.start(self)
  File "/usr/lib/python2.7/dist-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/usr/lib/python2.7/dist-packages/celery/worker/consumer.py", line 790, in start
    c.loop(*c.loop_args())
  File "/usr/lib/python2.7/dist-packages/celery/worker/loops.py", line 72, in asynloop
    next(loop)
  File "/usr/lib/python2.7/dist-packages/kombu/async/hub.py", line 333, in create_loop
    cb(*cbargs)
  File "/usr/lib/python2.7/dist-packages/kombu/transport/base.py", line 156, in on_readable
    reader(loop)
  File "/usr/lib/python2.7/dist-packages/kombu/transport/base.py", line 141, in _read
    drain_events(timeout=0)
  File "/usr/lib/python2.7/dist-packages/librabbitmq/__init__.py", line 205, in drain_events
    self._basic_recv(timeout)
ConnectionError: Bad frame read
[2014-01-14 13:05:49,551: ERROR/MainProcess] consumer: Cannot connect to amqp://guest@localhost:5672//: Error opening socket: Unknown error 536871023.
Trying again in 2.00 seconds...

[2014-01-14 13:05:51,563: ERROR/MainProcess] consumer: Cannot connect to amqp://guest@localhost:5672//: Error opening socket: Unknown error 536871023.
Trying again in 4.00 seconds...

[2014-01-14 13:05:55,587: ERROR/MainProcess] Unrecoverable error: ValueError('I/O operation on closed epoll fd',)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/celery/worker/__init__.py", line 212, in start
    self.blueprint.start(self)
  File "/usr/lib/python2.7/dist-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/usr/lib/python2.7/dist-packages/celery/bootsteps.py", line 373, in start
    return self.obj.start()
  File "/usr/lib/python2.7/dist-packages/celery/worker/consumer.py", line 270, in start
    blueprint.start(self)
  File "/usr/lib/python2.7/dist-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/usr/lib/python2.7/dist-packages/celery/worker/consumer.py", line 467, in start
    c.connection = c.connect()
  File "/usr/lib/python2.7/dist-packages/celery/worker/consumer.py", line 369, in connect
    conn.transport.register_with_event_loop(conn.connection, self.hub)
  File "/usr/lib/python2.7/dist-packages/kombu/transport/librabbitmq.py", line 147, in register_with_event_loop
    connection.fileno(), self.on_readable, connection, loop,
  File "/usr/lib/python2.7/dist-packages/kombu/async/hub.py", line 214, in add_reader
    return self.add(fds, callback, READ | ERR, args)
  File "/usr/lib/python2.7/dist-packages/kombu/async/hub.py", line 165, in add
    self.poller.register(fd, flags)
  File "/usr/lib/python2.7/dist-packages/kombu/utils/eventio.py", line 78, in register
    self._epoll.register(fd, events)
ValueError: I/O operation on closed epoll fd
[2014-01-14 13:05:55,588: INFO/MainProcess] beat: Shutting down...
[2014-01-14 13:05:57,936: WARNING/MainProcess] /usr/lib/python2.7/dist-packages/celery/apps/worker.py:159: CDeprecationWarning: 
Starting from version 3.2 Celery will refuse to accept pickle by default.

The pickle serializer is a security concern as it may give attackers
the ability to execute any command.  It's important to secure
your broker from unauthorized access when using pickle, so we think
that enabling pickle should require a deliberate action and not be
the default choice.

If you depend on pickle then you should set a setting to disable this
warning and to be sure that everything will continue working
when you upgrade to Celery 3.2::

    CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']

You must only enable the serializers that you will actually use.


  warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))

[2014-01-14 13:05:57,957: INFO/Beat] beat: Starting...
[2014-01-14 13:05:57,960: INFO/MainProcess] Connected to amqp://maas_workers@10.0.2.15:5672//maas_workers
[2014-01-14 13:05:57,975: INFO/MainProcess] mingle: searching for neighbors
[2014-01-14 13:05:58,987: INFO/MainProcess] mingle: all alone
[2014-01-14 13:05:59,024: WARNING/MainProcess] celery@autopkgtest ready.
[2014-01-14 13:06:08,142: INFO/MainProcess] Received task: provisioningserver.tasks.refresh_secrets[c6da2c33-31b3-46d9-ace8-d951cd37b92b]
[2014-01-14 13:06:08,145: INFO/MainProcess] Task provisioningserver.tasks.refresh_secrets[c6da2c33-31b3-46d9-ace8-d951cd37b92b] succeeded in 0.001664963s: None
[2014-01-14 13:06:14,175: INFO/MainProcess] Received task: provisioningserver.tasks.refresh_secrets[1157d4aa-fc31-4fb9-965b-4ab845f14795]
[2014-01-14 13:06:14,177: INFO/MainProcess] Task provisioningserver.tasks.refresh_secrets[1157d4aa-fc31-4fb9-965b-4ab845f14795] succeeded in 0.000876173999984s: None
[2014-01-14 13:06:19,434: INFO/MainProcess] Received task: provisioningserver.tasks.refresh_secrets[1b7c6262-4d3a-44a6-9406-545d8d7db4a3]
[2014-01-14 13:06:19,438: INFO/MainProcess] Task provisioningserver.tasks.refresh_secrets[1b7c6262-4d3a-44a6-9406-545d8d7db4a3] succeeded in 0.00224746399999s: None
[2014-01-14 13:06:19,478: INFO/MainProcess] Received task: provisioningserver.tasks.refresh_secrets[563da12d-99bf-4149-b1af-59561d2f9334]
[2014-01-14 13:06:19,479: INFO/MainProcess] Task provisioningserver.tasks.refresh_secrets[563da12d-99bf-4149-b1af-59561d2f9334] succeeded in 0.000635513999953s: None
[2014-01-14 13:21:56,650: INFO/MainProcess] Received task: provisioningserver.tasks.write_full_dns_config[e93652b7-7bb4-4557-9bb7-869507a8134e]
[2014-01-14 13:21:56,953: INFO/MainProcess] Received task: provisioningserver.tasks.rndc_command[2dc54506-d598-49f6-93b5-0ec158dd4d24]
[2014-01-14 13:21:56,957: INFO/MainProcess] Task provisioningserver.tasks.write_full_dns_config[e93652b7-7bb4-4557-9bb7-869507a8134e] succeeded in 0.227183831s: None
[2014-01-14 13:21:57,284: INFO/MainProcess] Task provisioningserver.tasks.rndc_command[2dc54506-d598-49f6-93b5-0ec158dd4d24] succeeded in 0.323064958s: None
}}}

retcode for maas-cli maas ['node-group-interface', 'update', 'a5178092-9d35-4ce1-ba00-98eefb9a5bf2', 'eth1', 'subnet_mask=255.255.255.0', 'ip_range_low=192.168.21.10', 'broadcast_ip=192.168.21.255', 'router_ip=192.168.21.1', 'ip_range_high=192.168.21.30', 'interface=eth1', 'ip=192.168.21.5', 'management=2']: {{{0}}}
retcode for maas-cli maas ['node-groups', 'list']: {{{0}}}
stdout for maas-cli maas ['node-group-interface', 'update', 'a5178092-9d35-4ce1-ba00-98eefb9a5bf2', 'eth1', 'subnet_mask=255.255.255.0', 'ip_range_low=192.168.21.10', 'broadcast_ip=192.168.21.255', 'router_ip=192.168.21.1', 'ip_range_high=192.168.21.30', 'interface=eth1', 'ip=192.168.21.5', 'management=2']: {{{
{
    "ip_range_high": "192.168.21.30", 
    "ip_range_low": "192.168.21.10", 
    "broadcast_ip": "192.168.21.255", 
    "ip": "192.168.21.5", 
    "subnet_mask": "255.255.255.0", 
    "management": 2, 
    "interface": "eth1"
}
}}}

stdout for maas-cli maas ['node-groups', 'list']: {{{
[
    {
        "cluster_name": "Cluster master", 
        "status": 1, 
        "name": "master", 
        "uuid": "a5178092-9d35-4ce1-ba00-98eefb9a5bf2"
    }
]
}}}

Traceback (most recent call last):
  File "/tmp/adt-run.twCYJH/ubtree0-build/real-tree/debian/tests/utils.py", line 67, in wrapper
    result = func(*args, **kwargs)
  File "/tmp/adt-run.twCYJH/ubtree0-build/real-tree/debian/tests/maas-integration.py", line 423, in test_set_up_dhcp_region
    sleep(2)
  File "/tmp/adt-run.twCYJH/ubtree0-build/real-tree/debian/tests/utils.py", line 61, in _handle_timeout
    raise TimeoutError(error_message)
TimeoutError: Timer expired


maas-integration.TestMAASIntegration.test_update_pxe_config: 0.0003s
maas-integration.TestMAASIntegration.test_restart_provisioning_server: 0.0651s
maas-integration.TestMAASIntegration.test_check_initial_services: 0.0958s
maas-integration.TestMAASIntegration.test_update_preseed_arm: 0.1053s
maas-integration.TestMAASIntegration.test_create_admin: 0.1083s
maas-integration.TestMAASIntegration.test_restart_dbus_avahi: 0.1422s
maas-integration.TestMAASIntegration.test_cluster_connected: 0.4647s
maas-integration.TestMAASIntegration.test_update_maas_url: 1.2243s
maas-integration.TestMAASIntegration.test_login_api: 1.9818s
maas-integration.TestMAASIntegration.test_set_up_dhcp_region: 300.0347s
maas-integration.TestMAASIntegration.test_import_pxe_files: 934.4676s
----------------------------------------------------------------------
Ran 12 tests in 1240.251s

FAILED (SKIP=2, errors=1)
adt-run1: testbed executing test finished with exit status 1
adt-run: & ubtree0t-maas-package-test: ----------------------------------------]
adt-run: & ubtree0t-maas-package-test:  - - - - - - - - - - results - - - - - - - - - -
ubtree0t-maas-package-test FAIL non-zero exit status 1
adt-run1: ** needs_reset, previously=False
adt-run: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ tests done.
adt-run1: ** stop
adt-run1: ** close, scratch=/tmp/adt-run.twCYJH
+ RC=4
+ [ 4 -eq 20 ]
+ [ 0 -eq 1 ]
+ [ -x /home/ubuntu/adt-export-result ]
+ RES=PASS
+ [ 4 -gt 0 ]
+ RES=FAIL
+ /home/ubuntu/adt-export-result -D /root/adt-log maas FAIL
+ chown -R ubuntu /root/adt-log /var/tmp/testresults
+ chmod og+r /var/log/syslog
+ ls /var/crash/
+ [ -n  ]
+ exit 4
Connection to localhost closed.
+ RET=4
+ [ 0 -eq 1 ]
+ [ 4 -gt 0 ]
+ log_failure_msg adt-run exited with status 4.
+ log_msg Failure: adt-run exited with status 4.\n
+ date +%F %X
+ printf 2014-01-14 13:26:55: Failure: adt-run exited with status 4.\n
2014-01-14 13:26:55: Failure: adt-run exited with status 4.
+ [ 0 -eq 0 ]
+ mkdir -p /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results
+ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o CheckHostIP=no -i /var/tmp/adt/disks/adtkey -p 54323 -tt -o BatchMode=yes -l ubuntu localhost sudo chown -R ubuntu /root/adt-log; find /root/adt-log -type f -empty | xargs rm 2>/dev/null
Warning: Permanently added '[localhost]:54323' (ECDSA) to the list of known hosts.
find: `/root/adt-log': Permission denied
Connection to localhost closed.
+ true
+ scp -r -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o CheckHostIP=no -i /var/tmp/adt/disks/adtkey -P 54323 ubuntu@localhost:/root/adt-log/* /var/crash/*crash /var/log/syslog /var/tmp/testresults /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results
+ true
+ log_info_msg Test artifacts copied to /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results
+ log_msg Info: Test artifacts copied to /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results\n
+ date +%F %X
+ printf 2014-01-14 13:26:59: Info: Test artifacts copied to /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results\n
2014-01-14 13:26:59: Info: Test artifacts copied to /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results
+ [ -f /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results/summary.log ]
+ [ -n 2014-01-14_17-58-37 ]
+ ls -tr /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results/*.result
+ tail -1
+ resfile=
+ [ -n  ]
+ log_failure_msg Test didn't end normally. Generating error file
+ log_msg Failure: Test didn't end normally. Generating error file\n
+ date +%F %X
+ printf 2014-01-14 13:26:59: Failure: Test didn't end normally. Generating error file\n
2014-01-14 13:26:59: Failure: Test didn't end normally. Generating error file
+ date +%Y%m%d-%H%M%S
+ errfile=/home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results/trusty_amd64_maas_20140114-132659.error
+ echo trusty amd64 maas
+ rsync -a /home/ubuntu/jenkins-jobs/workspace/trusty-adt-maas-daily/results/trusty_amd64_maas_20140114-132659.error /trusty/tmp/
rsync: mkdir "/trusty/tmp" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(674) [Receiver=3.1.0]
+ true
+ [ 0 -eq 0 ]
+ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o CheckHostIP=no -i /var/tmp/adt/disks/adtkey -p 54323 -tt -o BatchMode=yes -l ubuntu localhost sudo poweroff
Warning: Permanently added '[localhost]:54323' (ECDSA) to the list of known hosts.


Broadcast message from ubuntu@autopkgtest
Connection to localhost closed.
+ exit 4
+ on_exit
+ log_info_msg Cleaning up
+ log_msg Info: Cleaning up\n
+ date +%F %X
+ printf 2014-01-14 13:27:00: Info: Cleaning up\n
2014-01-14 13:27:00: Info: Cleaning up
+ [ -f /var/tmp/adt/disks/run/trusty-amd64-maas-20140114_125859.TpXIFC.img.pid ]
+ cat /var/tmp/adt/disks/run/trusty-amd64-maas-20140114_125859.TpXIFC.img.pid
+ kill -9 27021
+ rm -f /var/tmp/adt/disks/run/trusty-amd64-maas-20140114_125859.TpXIFC.img.pid
+ rm -f /var/tmp/adt/disks/run/trusty-amd64-maas-20140114_125859.TpXIFC.img
+ rm -f /var/tmp/adt/disks/run/trusty-amd64-maas-20140114_125859.TpXIFC.img.monitor /var/tmp/adt/disks/run/trusty-amd64-maas-20140114_125859.TpXIFC.img.serial
+ rm -f /var/lock/adt/ssh.54323.lock
+ rm -f /var/lock/adt/vnc.5911.lock
+ [ -d /tmp/adt-amd64.kue8UM ]
+ rm -Rf /tmp/adt-amd64.kue8UM
+ rm -f /var/tmp/adt/disks/run/trusty-amd64-maas-*.img*
+ find /var/lock/adt -name *.lock -mtime +1
+ exit 4
Build step 'Execute shell' marked build as failure
Archiving artifacts
Email was triggered for: Failure
Sending email for trigger: Failure
Follow ups

Jenkins Still Failing - trusty-adt-maas-daily 38
From: Jenkins Notification, 2014-01-14
References

Jenkins Still Failing - trusty-adt-maas-daily 36
From: Jenkins Notification, 2014-01-14