ubuntu-appstore-developers team mailing list archive

[Daniel Holbach <daniel.holbach@xxxxxxxxxx>]

Hello everybody,

we had a discussion about the process we could use for reviewing the
incoming apps until an automated review mechanism is in place. Here's a
quick summary. Please weigh in, so we can document this and start with
the implementation.



Goal
====

The ultimate goal is to have a fully automated process, so that manual
review is only the exception in those cases where there are needs for
conflict resolution or upload revocation (e.g. failure to comply with
the ToS). However, in the interim, we will need a team of human
reviewers to approve app uploads until full automation has been implemented.


Lessons learnt
==============

 * Human review does not scale very well.
 * Arduous task.
 * Ensuring Debian best practices are upheld is tiresome.
 * Relaxing review policy was only possible up to a small degree.
 * Feedback ping-pong exhausts everybody’s patience, both reviewers
   and developers.

Requirements
============

 * App upload should be a fully automated process
 * A human team of reviewers should only be required for maintenance
   and conflict resolution.
 * App review process: temporary human review period.

Queue handling
==============

Having automated review tools is a must.

Review policy
-------------
 * ARB review guidelines
   https://wiki.ubuntu.com/AppReviewBoard/Review/Guidelines
 * Implementation idea: have a tool which downloads everything from
   unreviewed queue, run lint tool over all of them, spit out list of
   apps for approval and rejection.
 * David’s idea: “listadmin for click packages” (Daniel: +1)

Checks we could run
===================

 * General:
   - Description + app details
   - Price of the app.
   - Do we want to check the screenshot? yes
   - Check icon/screenshot for noticable trademark infringement (ie,
     adobe icon or facebook icon, etc)
   - Description? yes
   - Contact line / support URL
   - copyright?
 * Submission
   - Tarball? Other file we don’t accept?
   - Submission should be defined click format only - no other option
   - Verify developer signature
 * Lint tool
   - Manifest
   - Size
   - Architectures
   - Security profile
   - are the specified perms supported by our tools/policy?
     (automatable)
   - do the specified perms make sense relative to
     description/developer justification? (manual review)
 * QA
   - App is installable
   - App is startable (perhaps just handle by user reviews? - this is
     how android does it)
   - App operates correctly - doesn’t scale, handle by user reviews
 * Non-automatable
   - Are the requested security features necessary?
   - Is it a demo app?
   - Are apps generating revenue? (In-app purchases.)


Review tools
============

We wrote arb-lint (https://launchpad.net/arb-lint) and some of its
functionality has been merged into lintian or could be used from there.
(Mail from Niels Thykier, one of its Debian maintainers:
https://lists.ubuntu.com/archives/app-review-board/2012-November/002569.html)


Staffing of the review team
===========================

Daniel: It’d be good if we could have working in shifts. So we’d have
people assigned for each week day, much like the archive admins
(https://wiki.ubuntu.com/ArchiveAdministration#Archive_days) do.

Plan ahead
==========

Daniel: it’d be good if we planned ahead and kept the period with manual
review to an absolute minimum. This of course depends on how much the
automated checking tools can do for us and how easily they can be
integrated into an automatic review queue handler in the app store.
Future review team: just deal with conflict resolution.

What needs to be done to automate review
----------------------------------------

 1. Collect problematic cases during reviews, note them down, work
    on automating their reviews.
 2. Accept uploads, put them into a queue.
 3. Process queue with automatic review tools.
 4. Feedback by email, available on web as well.


Open questions
==============

 - Resolve namespacing discussion
 - Public open queue? (Developers might not enjoy this.)
 - Need to create better guidance on rejection of apps vs
   simply asking for changes to fix problems.
 - Current review process lacks methods for adequate informative
   dialogue.



Comments and feedback welcome!

Have a great day,
 Daniel

-- 
Get involved in Ubuntu development! developer.ubuntu.com/packaging
Follow @ubuntudev on identi.ca/twitter.com/facebook.com/G+