← Back to team overview

ubuntu-phone team mailing list archive

Re: Status of CI Services - December 11, 2014

 

Just would like to inform that the devices have been restored and MP testing on the phones have been re-enabled.

On 12/12/14 13:49, psivaa wrote:
The phones that are attached to the s-jenkins are unusable again.

We're having to disable the MP testing on the phones again whilst we work with
the IS to restore them.

Thanks

On 12/12/14 02:24, Francis Ginther wrote:
The MP phone testing has been re-enabled, but there is still some work to do to fully enable the remote recovery. Big thanks to the IS engineers for
working to get this up and running again.

Francis

On Thu, Dec 11, 2014 at 4:23 PM, Francis Ginther <
francis.ginther@xxxxxxxxxxxxx> wrote:

We need to disable the MP phone testing again. We've temporarily lost the ability to remote recover the devices and as a result they are all offline now. As before, only the projects that use phone testing will be disabled.

Other updates:
  - The VMs for some specific unity8 and ubuntu-settings-components MP
tests are back up. The s-jenkins restart was successful.
- All of the test slaves used for proposed-migration should be available
and running tests again.

Francis


On Thu, Dec 11, 2014 at 10:49 AM, Francis Ginther <
francis.ginther@xxxxxxxxxxxxx> wrote:

The MP testing service, s-jenkins, will need a restart at 18:00 UTC. The
restart should take less than 15 minutes.

Also, please be aware that the VPN to access the internal CI services has
changed. For the new VPN go to
https://wiki.canonical.com/InformationInfrastructure/IS/HowTo/CompanyOpenVPN
for instructions.

Updates:
- The rtm-dashboard is up again. Results from older test runs are still
being populated.
  - The MP testing backlog has been processed. All projects including
those needing phones to test are enabled. The armhf builders are all back
up.

What’s still down:
- Emails from jenkins jobs, etc. are not being forwarded (IS has been
notified).
  - The VMs for some specific unity8 and ubuntu-settings-components MP
tests are unavailable. This is expected to be fixed by the s-jenkins
restart. The test jobs have been disabled until then.
- Some of the non-x86 test slaves used for proposed-migration are still
down (IS has been notified).
- The bootspeed and unity7 test systems are still awaiting the correct
power bricks and/or power cords (IS is working the issue).

Francis

On Wed, Dec 10, 2014 at 6:07 PM, Francis Ginther <
francis.ginther@xxxxxxxxxxxxx> wrote:

Since the last update:
- MP testing has been fully enabled. I expect to see some long build queues as the backlog is worked down and we're only at half our normal
number of phones available for testing.
  - The proposed migration slaves can now talk to ftpmaster.internal
which was causing failures earlier today.


What's still down:
- The rtm-dashboard for looking at krillin smoke testing results (still
a work in progress).
- Some of the non-x86 test slaves used for proposed-migration are still
down.


Francis

On Wed, Dec 10, 2014 at 12:00 PM, Francis Ginther <
francis.ginther@xxxxxxxxxxxxx> wrote:

Here's the latest on what is now working:

- The krillin build process is now fully enabled and builds are working.

- Smoke testing is running for both mako and krilin devices. Capacity is limited as we have several makos that need attention from the Data
Center Engineers (who are also busy moving other kit).


And what's still offline:

- MP testing is disabled, but should be re-enabled in the next 2-3
hours. This will initially be at very low capacity as several of the armhf builders were lost during the move. They are being rebuilt. Phones for
testing MPs are also limited.

- The rtm-dashboard for looking at krillin smoke testing results (work
in progress).

- The proposed-migration test slaves are still having network
connectivity issues (IS has been notified).

Francis

On Tue, Dec 9, 2014 at 5:57 PM, Francis Ginther <
francis.ginther@xxxxxxxxxxxxx> wrote:

The CI services are starting to come back online. Much is still
disabled, but I expect to get the smoke testing enabled tonight. The krillin builds are also partially enabled, just need to see if the rest of the system-image process is properly networked. The krillin builds are
currently a priority.

The MP testing is currently still disabled while we work with IS to
update the network and firewall connections to the build nodes. We also need to rebuild some of the armhf build nodes. The current ETA is to have
this enabled tomorrow.

Francis
--
Francis Ginther
Canonical - Ubuntu Engineering - Continuous Integration Team



--
Francis Ginther
Canonical - Ubuntu Engineering - Continuous Integration Team



--
Francis Ginther
Canonical - Ubuntu Engineering - Continuous Integration Team



--
Francis Ginther
Canonical - Ubuntu Engineering - Continuous Integration Team



--
Francis Ginther
Canonical - Ubuntu Engineering - Continuous Integration Team










References