canonical-ubuntu-qa team mailing list archive

[Skia <2062537@xxxxxxxxxxxxxxxxxx>]

I've made some measurements on the last 12 hours of running tests, the
proportion of upstream tests seems appropriate, so I'll mark this as
"Fix released"

** Changed in: auto-package-testing
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of
Canonical's Ubuntu QA, which is subscribed to Auto Package Testing.
https://bugs.launchpad.net/bugs/2062537

Title:
  Change the way upstream jobs percentage is done

Status in Auto Package Testing:
  Fix Released

Bug description:
  This code[1] is mostly broken: it's only when connecting to the queues that we  decide if a worker is going to take `upstream` jobs, but this is based on randomness. I see the following flaws:
  * with low number of workers (20 is low), there is a high chance that the distribution will not be uniform at all, meaning you can end up with far more or far less workers taking upstream jobs than intended.
  * the dice is rolled only when the worker starts, and that means that is you end up with too many/few workers taking upstream jobs, the situation is going to stay until the workers are restarted.

  Here is a quick Python one-liner to play with. It gives the number of workers that will pick up upstream jobs, over a total of 20, with a chance of 50%:
  `len([r for r in [random.randint(1, 100) for _ in range(20)] if r > 50])`
  For only 20 workers and 50% chances, it's common to see a run with a deviation of more than 3 points from the objective (objective: 10, we often see <=7 or >=13). This gets worse when setting a lower threshold (eg 15), because we more often hit the extreme where no worker at all would pick upstream jobs.

  
  A better solution would be to roll the dice in the `request` function callback [2] so that for every upstream test request, the chance of processing get calculated. This will make the number of rolls proportionate with the number of jobs processed, and that means far greater numbers, which means far more accurate percentage of jobs filtered by the threshold over time.

  [1]: https://git.launchpad.net/autopkgtest-cloud/tree/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker#n1482
  [2]: around here: https://git.launchpad.net/autopkgtest-cloud/tree/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker#n640

To manage notifications about this bug go to:
https://bugs.launchpad.net/auto-package-testing/+bug/2062537/+subscriptions