ubuntu-appstore-developers team mailing list archive

[Dave Morley <davmor@xxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 17/06/13 14:12, Daniel Holbach wrote:
> Hello everybody,
> 
> we had a discussion about the process we could use for reviewing
> the incoming apps until an automated review mechanism is in place.
> Here's a quick summary. Please weigh in, so we can document this
> and start with the implementation.
> 
> 
> 
> Goal ====
> 
> The ultimate goal is to have a fully automated process, so that
> manual review is only the exception in those cases where there are
> needs for conflict resolution or upload revocation (e.g. failure to
> comply with the ToS). However, in the interim, we will need a team
> of human reviewers to approve app uploads until full automation has
> been implemented.
> 
> 
> Lessons learnt ==============
> 
> * Human review does not scale very well. * Arduous task. * Ensuring
> Debian best practices are upheld is tiresome. * Relaxing review
> policy was only possible up to a small degree. * Feedback ping-pong
> exhausts everybody’s patience, both reviewers and developers.
> 
> Requirements ============
> 
> * App upload should be a fully automated process * A human team of
> reviewers should only be required for maintenance and conflict
> resolution. * App review process: temporary human review period.
> 
> Queue handling ==============
> 
> Having automated review tools is a must.
> 
> Review policy ------------- * ARB review guidelines 
> https://wiki.ubuntu.com/AppReviewBoard/Review/Guidelines *
> Implementation idea: have a tool which downloads everything from 
> unreviewed queue, run lint tool over all of them, spit out list of 
> apps for approval and rejection. * David’s idea: “listadmin for
> click packages” (Daniel: +1)
> 
> Checks we could run ===================
> 
> * General: - Description + app details - Price of the app. - Do we
> want to check the screenshot? yes - Check icon/screenshot for
> noticable trademark infringement (ie, adobe icon or facebook icon,
> etc) - Description? yes - Contact line / support URL - copyright? *
> Submission - Tarball? Other file we don’t accept? - Submission
> should be defined click format only - no other option - Verify
> developer signature * Lint tool - Manifest - Size - Architectures -
> Security profile - are the specified perms supported by our
> tools/policy? (automatable) - do the specified perms make sense
> relative to description/developer justification? (manual review) *
> QA - App is installable - App is startable (perhaps just handle by
> user reviews? - this is how android does it)

If we are testing that the app is installable testing that it is
runable is a 30seconds more work and can possibly be automated, ie
start a vm on server, allow all ppas/package repos (however click is
going to work), start app, sleep 20 seconds, does appname appear in ps
aux, if true close app, sleep 20 seconds, does it appear in ps aux, if
false publish app.  If any app fails the test, they are then viewed
manually, so you need a system to ping you when one fails. :)

> - App operates correctly - doesn’t scale, handle by user reviews

When you say handled by user reviews, who looks at the reviews to see
if apps need removing?  Currently I don't think anyone is responsible
for clearing application review issues from reviews.ubuntu.com
currently, let alone looking to see if applications should be removed.

> * Non-automatable - Are the requested security features necessary? 
> - Is it a demo app? - Are apps generating revenue? (In-app
> purchases.)
> 
> 
> Review tools ============
> 
> We wrote arb-lint (https://launchpad.net/arb-lint) and some of its 
> functionality has been merged into lintian or could be used from
> there. (Mail from Niels Thykier, one of its Debian maintainers: 
> https://lists.ubuntu.com/archives/app-review-board/2012-November/002569.html)
>
> 
> 
> Staffing of the review team ===========================
> 
> Daniel: It’d be good if we could have working in shifts. So we’d
> have people assigned for each week day, much like the archive
> admins (https://wiki.ubuntu.com/ArchiveAdministration#Archive_days)
> do.
> 
> Plan ahead ==========
> 
> Daniel: it’d be good if we planned ahead and kept the period with
> manual review to an absolute minimum. This of course depends on how
> much the automated checking tools can do for us and how easily they
> can be integrated into an automatic review queue handler in the app
> store. Future review team: just deal with conflict resolution.
> 
> What needs to be done to automate review 
> ----------------------------------------
> 
> 1. Collect problematic cases during reviews, note them down, work 
> on automating their reviews. 2. Accept uploads, put them into a
> queue. 3. Process queue with automatic review tools. 4. Feedback by
> email, available on web as well.
> 
> 
> Open questions ==============
> 
> - Resolve namespacing discussion - Public open queue? (Developers
> might not enjoy this.) - Need to create better guidance on
> rejection of apps vs simply asking for changes to fix problems. -
> Current review process lacks methods for adequate informative 
> dialogue.
> 
> 
> 
> Comments and feedback welcome!
> 
> Have a great day, Daniel
> 


- -- 
You make it, I'll break it!

I love my job :)
http://www.ubuntu.com
http://www.canonical.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlG/ENIACgkQT5xqyT+h3OjyAACdFWAfPdTZ+4DQtLrU5RgpzXy1
MkgAoL9zS5XqLgDy1HYuJfdTRMCuAaVL
=XRDj
-----END PGP SIGNATURE-----