Hey,
We took a big hit for multiple reasons since Friday. Our goal is to go
back to a green baseline as soon as possible. The current image status
isn't great[1], more details below.
Consequently, we do *not* attribute any CI train silo nor landing
which isn't targeted to get to a greener image or fix android 4.4
issue. Desktop-only impact landings can still land as usual. If you
would like to help us getting greener faster, please see the "CALL TO
ACTION" section below.
In the meantime, we never had working image for flo and the emulator
with 4.4, so those 2 were promoted on latest image (#206).
First, we didn't have full image testing results since Friday (apart
from latest one which was rerun, #206), due to issues Paul mentioned
on the mailing list. This doesn't allow us to know if any regressions
are coming from one of the 3 big landings: autopilot, unity8 and
android 4.4. Fortunately, the autopilot and unity8 landings are small
enough to hope they are not the root cause of any big issues. In
addition to that, a lot of core apps have been updating and affect the
global result as well.
Let's sum up image by image:
# 200:
* this one mostly contains the autopilot change + what was discussed
on Friday
# 201:
* mostly about new unity8 to isolate downloads in separate threads,
card background support and a bigger sidestage threshold.
* some core apps updates
#202:
* we got the click update UI back on the image
-> we didn't get test results due to CI infrastructure issues.
Then, we had the switch to the new android 4.4 kernel.
#203:
* First tentative version with the new android version, including
hybris support
# 204:
* Minor changes, mostly some dropped dependencies on zeigeist
# 205:
* Some android configuration change for flo.
# 206:
* First real 4.4 version being able to be flash recovery and boot on
your devices.
* New music and terminal application. Music introduced a bug on first
launch[2]. This wasn't in previous version of music-apps. Alan
reverted it. We'll need upstream as well to ensure they write an
integration test for that case.
-> We didn't get until this morning any results on those image as the
CI test infrastructure needed to be changed to take into account due
to the new flashing image code being the only working for the 4.4
switch. 206 was rerun this morning and we see a lot of system-settle
issues.
It took a big part of the day to clear that out: idle definition
changed in the kernel (it's based on all active CPU and not on all
active + shutdown CPUs) as in the past. That's why we started to see
pulseaudio and unity-system-compositor reporting higher CPU usage than
they used to. We need to the CI team to accord to those new ways of
reporting idle values. Thanks to Sivaa, Lukasz, Olivier and Andy to
have helped clearing that out.
Also, in parallel, Paul is looking at getting then real results in
term of what tests are actually failing or not out of the system
settle noise. We know that there were some that will need to be
addressed by the US landing team.
-------
CALL TO ACTION, we need to have every upstream teams working to get
back to a green image on mako at least:
All results are available at
http://ci.ubuntu.com/smokeng/trusty/touch/mako/206:20140224:20140224.
We have an awesome team of upstream teams/landing team pairing up to
examine all tests failures and tell us if the issue is in the tests
themselves, a regression in the app, or a global platform issue:
- ubuntu_terminal_app (Nicholas & Robert)
- test_circle_menu_shows
- ubuntu_weather_app (Nicholas & Robert)
- test_add_location_keyboard
- test_switch_scale
- ubuntu_clock_app (Nicholas & Robert)
- test_delete_alarm_must_delete_from_alarm_list
- ubuntu_calculator_app (Nicholas & Robert)
- test_three_negative_numbers_multiplication
- music_app (Nicholas & Robert)
- test_add_songs_to_queue_from_songs_tab_and_play
- test_show_albums_sheet
- camera_app (Bill & Mathieu)
- test_move_focus_ring
- unity8 (Lukasz & Kevin)
- test_pin_screen_wrong_code
- test_icon_summary_body
- test_swipe_out_application_started_by_url_dispatcher
- test_open_preview
- test_append_hint
- gallery_app (Bill & Mathieu)
- test_album_title_fields
- dialer_app (Bill & Mathieu)
- test_outgoing_answer_local_hangup
- test_call_log_item_opens_messaging
- test_outgoing_answer_remote_hangup
- test_outgoing_noanswer
Note that rssreader app failing was due to rename to short, Paul is
working in updating the test infrastructure according to it.
----
Ricardo also mentioned a couple of regressions that needs to be fixed
before being able to promote an image:
* Voice call not working properly after the first call [3]
* Camera freezes after taking photo on Nexus 4 [4]
* A pulseaudio high CPU usage due to switch on and off CPU
* An ophono-sim installed (during tests) which is making
network-manager quite verbose.
Note as well that the sound hardware buttons not working that Nicholas
reported on image 200 wasn't reproducible in latest image. So we put
that one on side and concentrate on the other issues first.
We need to have all that solved and the dashboard + dogfooding in an
acceptable state again before processing further. Let's get the
dream's team working together :)
Cheers,
Didier
[1]
http://ci.ubuntu.com/smokeng/trusty/touch/mako/206:20140224:20140224/6796/
[2] https://bugs.launchpad.net/music-app/+bug/1284025
[3] https://bugs.launchpad.net/ubuntu/+source/android/+bug/1283818
[4] https://bugs.launchpad.net/ubuntu/+source/android/+bug/1282701
