ubuntu-phone team mailing list archive

[Łukasz 'sil2100' Zemczak <lukasz.zemczak@xxxxxxxxxxxxx>]

Hello,

Sadly, regarding unity8 test flakyness, we don't have any good news. I
have asked Saviq to help me out tracing down the underlying problem - it
seems that sometimes during some random tests unity8 fails to start,
causing the given tests to fail. It's reproducible locally. It seem only
rare unity8 test-suite-runs pass completely - in every other run there
is at least a few failures caused by this problem.

The bad news is that we have still not identified the underlying
problem. The crash seems to be happening due to the new Android bits,
but debugging these kind of failures is a bit more time-consuming as we
don't have direct access to the underlying android symbols. Michał is
currently working on getting useful information from the stacktrace, but
we suspect that the fix for this problem might have to be done elsewere.

We might need the assistance of Ricardo sooner or later anyway.

On 24.02.2014 18:55, Didier Roche wrote:
> With all CC so that everyone knows where to answer :)
> 
> Le 24/02/2014 18:54, Didier Roche a écrit :
>> Hey,
>>
>> We took a big hit for multiple reasons since Friday. Our goal is to go
>> back to a green baseline as soon as possible. The current image status
>> isn't great[1], more details below.
>>
>> Consequently, we do *not* attribute any CI train silo nor landing
>> which isn't targeted to get to a greener image or fix android 4.4
>> issue. Desktop-only impact landings can still land as usual. If you
>> would like to help us getting greener faster, please see the "CALL TO
>> ACTION" section below.
>>
>> In the meantime, we never had working image for flo and the emulator
>> with 4.4, so those 2 were promoted on latest image (#206).
>>
>> First, we didn't have full image testing results since Friday (apart
>> from latest one which was rerun, #206), due to issues Paul mentioned
>> on the mailing list. This doesn't allow us to know if any regressions
>> are coming from one of the 3 big landings: autopilot, unity8 and
>> android 4.4. Fortunately, the autopilot and unity8 landings are small
>> enough to hope they are not the root cause of any big issues. In
>> addition to that, a lot of core apps have been updating and affect the
>> global result as well.
>>
>> Let's sum up image by image:
>> # 200:
>> * this one mostly contains the autopilot change + what was discussed
>> on Friday
>>
>> # 201:
>> * mostly about new unity8 to isolate downloads in separate threads,
>> card background support and a bigger sidestage threshold.
>> * some core apps updates
>>
>> #202:
>> * we got the click update UI back on the image
>>
>> -> we didn't get test results due to CI infrastructure issues.
>>
>> Then, we had the switch to the new android 4.4 kernel.
>>
>> #203:
>> * First tentative version with the new android version, including
>> hybris support
>>
>> # 204:
>> * Minor changes, mostly some dropped dependencies on zeigeist
>>
>> # 205:
>> * Some android configuration change for flo.
>>
>> # 206:
>> * First real 4.4 version being able to be flash recovery and boot on
>> your devices.
>> * New music and terminal application. Music introduced a bug on first
>> launch[2]. This wasn't in previous version of music-apps. Alan
>> reverted it. We'll need upstream as well to ensure they write an
>> integration test for that case.
>>
>> -> We didn't get until this morning any results on those image as the
>> CI test infrastructure needed to be changed to take into account due
>> to the new flashing image code being the only working for the 4.4
>> switch. 206 was rerun this morning and we see a lot of system-settle
>> issues.
>>
>> It took a big part of the day to clear that out: idle definition
>> changed in the kernel (it's based on all active CPU and not on all
>> active + shutdown CPUs) as in the past. That's why we started to see
>> pulseaudio and unity-system-compositor reporting higher CPU usage than
>> they used to. We need to the CI team to accord to those new ways of
>> reporting idle values. Thanks to Sivaa, Lukasz, Olivier and Andy to
>> have helped clearing that out.
>>
>> Also, in parallel, Paul is looking at getting then real results in
>> term of what tests are actually failing or not out of the system
>> settle noise. We know that there were some that will need to be
>> addressed by the US landing team.
>>
>> -------
>> CALL TO ACTION, we need to have every upstream teams working to get
>> back to a green image on mako at least:
>>
>> All results are available at
>> http://ci.ubuntu.com/smokeng/trusty/touch/mako/206:20140224:20140224.
>> We have an awesome team of upstream teams/landing team pairing up to
>> examine all tests failures and tell us if the issue is in the tests
>> themselves, a regression in the app, or a global platform issue:
>>
>> - ubuntu_terminal_app (Nicholas & Robert)
>>   - test_circle_menu_shows
>> - ubuntu_weather_app (Nicholas & Robert)
>>   - test_add_location_keyboard
>>   - test_switch_scale
>> - ubuntu_clock_app (Nicholas & Robert)
>>   - test_delete_alarm_must_delete_from_alarm_list
>> - ubuntu_calculator_app (Nicholas & Robert)
>>   - test_three_negative_numbers_multiplication
>> - music_app (Nicholas & Robert)
>>   - test_add_songs_to_queue_from_songs_tab_and_play
>>   - test_show_albums_sheet
>> - camera_app (Bill & Mathieu)
>>   - test_move_focus_ring
>> - unity8 (Lukasz & Kevin)
>>   - test_pin_screen_wrong_code
>>   - test_icon_summary_body
>>   - test_swipe_out_application_started_by_url_dispatcher
>>   - test_open_preview
>>   - test_append_hint
>> - gallery_app (Bill & Mathieu)
>>   - test_album_title_fields
>> - dialer_app (Bill & Mathieu)
>>   - test_outgoing_answer_local_hangup
>>   - test_call_log_item_opens_messaging
>>   - test_outgoing_answer_remote_hangup
>>   - test_outgoing_noanswer
>>
>> Note that rssreader app failing was due to rename to short, Paul is
>> working in updating the test infrastructure according to it.
>> ----
>>
>> Ricardo also mentioned a couple of regressions that needs to be fixed
>> before being able to promote an image:
>> * Voice call not working properly after the first call [3]
>> * Camera freezes after taking photo on Nexus 4 [4]
>> * A pulseaudio high CPU usage due to switch on and off CPU
>> * An ophono-sim installed (during tests) which is making
>> network-manager quite verbose.
>>
>> Note as well that the sound hardware buttons not working that Nicholas
>> reported on image 200 wasn't reproducible in latest image. So we put
>> that one on side and concentrate on the other issues first.
>>
>>
>> We need to have all that solved and the dashboard + dogfooding in an
>> acceptable state again before processing further. Let's get the
>> dream's team working together :)
>> Cheers,
>> Didier
>>
>>
>> [1]
>> http://ci.ubuntu.com/smokeng/trusty/touch/mako/206:20140224:20140224/6796/
>>
>> [2] https://bugs.launchpad.net/music-app/+bug/1284025
>> [3] https://bugs.launchpad.net/ubuntu/+source/android/+bug/1283818
>> [4] https://bugs.launchpad.net/ubuntu/+source/android/+bug/1282701
>>
> 

Best regards,

-- 
Łukasz 'sil2100' Zemczak
 lukasz.zemczak@xxxxxxxxxxxxx
 www.canonical.com