ubuntu-appstore-developers team mailing list archive

[Dave Morley <davmor@xxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 17/06/13 14:12, Daniel Holbach wrote:
> Hello everybody,
> 
> we had a discussion about the process we could use for reviewing
> the incoming apps until an automated review mechanism is in place.
> Here's a quick summary. Please weigh in, so we can document this
> and start with the implementation.
> 
> 
> 
> Goal ====
> 
> The ultimate goal is to have a fully automated process, so that
> manual review is only the exception in those cases where there are
> needs for conflict resolution or upload revocation (e.g. failure to
> comply with the ToS). However, in the interim, we will need a team
> of human reviewers to approve app uploads until full automation has
> been implemented.
> 
> 
> Lessons learnt ==============
> 
> * Human review does not scale very well. * Arduous task. * Ensuring
> Debian best practices are upheld is tiresome. * Relaxing review
> policy was only possible up to a small degree. * Feedback ping-pong
> exhausts everybody’s patience, both reviewers and developers.
> 
> Requirements ============
> 
> * App upload should be a fully automated process * A human team of
> reviewers should only be required for maintenance and conflict
> resolution. * App review process: temporary human review period.
> 
> Queue handling ==============
> 
> Having automated review tools is a must.
> 
> Review policy ------------- * ARB review guidelines 
> https://wiki.ubuntu.com/AppReviewBoard/Review/Guidelines *
> Implementation idea: have a tool which downloads everything from 
> unreviewed queue, run lint tool over all of them, spit out list of 
> apps for approval and rejection. * David’s idea: “listadmin for
> click packages” (Daniel: +1)
> 
> Checks we could run ===================
> 
> * General: - Description + app details

/!\Critical, trying to cut out bad language, trademarked/copyrighted
phrases and names is very important.

> - Price of the app.
No review of this is necessary currently the only field a dev can
modify without approval is the price I'm assuming this will remain the
case.
> - Do we want to check the screenshot? yes

/!\Critical, Disturbing screenshots, nudity etc can not be allowed.
Because we have no age restrition on what is displayed we need to
filter these out, on the whole devs are pretty good at this but you
still need to be weary.

> - Check icon/screenshot for noticable trademark infringement (ie, 
> adobe icon or facebook icon, etc)

/!\Critical, Needs to no be trademarks/copyrighted.

> - Description? yes
See description above

> - Contact line / support URL

Great candidate for automation, Things to check does it exist, maybe
screenshot the page and display it along side for the review.
Automated response of "I'm sorry we can not accept your application at
this time as the urls you provided are not accessible.  Please resolve
this issue and resubmit your application.  Many thanks"

> - copyright?

One for John Pugh,  If I'm not 100% I pass it onto John who has more
experience digging into these issues.


> * Submission - Tarball? Other file we don’t accept? - Submission
> should be defined click format only - no other option
I thought the general goal here was to only accept click packages.  If
you accept tarballs then you open up a world of pain.

What happens about supporting more than one architecture, I'm assuming
that for now these click packages with be ARM only but what happens on
full divergence when you have arm i386 and amd64.

> - Verify developer signature
/!\ If we are checking dev sigs then we need to ensure that a
developer knows how to create a signiture on all platforms, we get
submissions from all platforms currently so we need to cater for that
and not just assume that they are on Ubuntu (even though that would be
nice).

> * Lint tool - Manifest - Size - Architectures - Security profile -
> are the specified perms supported by our tools/policy? 
> (automatable) - do the specified perms make sense relative to 
> description/developer justification? (manual review)
No idea on this

> * QA - App is installable - App is startable (perhaps just handle
> by user reviews? - this is how android does it) - App operates
> correctly - doesn’t scale, handle by user reviews

Currently this is non-automatable due to the wonderful way in which
payments happen and how the change from a standard window to a webkit
frame inside the standard window is handled.  I'm hoping it will be
easier with in-dash payments, but it is having issues there too currently.

> * Non-automatable - Are the requested security features necessary? 
> - Is it a demo app? - Are apps generating revenue? (In-app
> purchases.)
This should be obvious from some of the SCA admin pages
Also one of the tests run whenever pay is rolled out is that you can
purchase stuff from any department the uses it.  Between these too
purchasing should be covered.

> 
> 
> Review tools ============
> 
> We wrote arb-lint (https://launchpad.net/arb-lint) and some of its 
> functionality has been merged into lintian or could be used from
> there. (Mail from Niels Thykier, one of its Debian maintainers: 
> https://lists.ubuntu.com/archives/app-review-board/2012-November/002569.html)
>
> 
> 
> Staffing of the review team ===========================
> 
> Daniel: It’d be good if we could have working in shifts. So we’d
> have people assigned for each week day, much like the archive
> admins (https://wiki.ubuntu.com/ArchiveAdministration#Archive_days)
> do.

Depends how this is going to work.  If it follows the existing path of
ARB vs Commercial then there will likely be one possibly 2 people
within canonical that will handle the commercial queue.  If it is one
queue that handles both ARB style and Commercial applications then a
mixed team of canonical and non-canonical may cause issues with ndas
something to consider at least.

> 
> Plan ahead ==========
> 
> Daniel: it’d be good if we planned ahead and kept the period with
> manual review to an absolute minimum. This of course depends on how
> much the automated checking tools can do for us and how easily they
> can be integrated into an automatic review queue handler in the app
> store. Future review team: just deal with conflict resolution.
> 
> What needs to be done to automate review 
> ----------------------------------------
> 
> 1. Collect problematic cases during reviews, note them down, work 
> on automating their reviews.
I can't see how to check for copyright/trademark infringements.

> 2. Accept uploads, put them into a queue.
Automating that the package follows strict guidlines is doable if we
assume that we only accept click packages,  ie if the manifest ect
don't follow the guidelines it can be flagged for manual review.

> 3. Process queue with automatic review tools. 4. Feedback by email,
> available on web as well.
> 
> 
> Open questions ==============
> 
> - Resolve namespacing discussion
Is this the <devname>-<appname>-version or something different?

> - Public open queue? (Developers might not enjoy this.)
Nda's agreements certainly won't.

> - Need to create better guidance on rejection of apps vs simply
> asking for changes to fix problems.
No app should be rejected unless it continues to be submitted broken
or continues to breech trademark/copyright or is reported for security
issues.
Ask more info drops it out of the review queue and off everyones
radar.  Maybe add a more info for apps that are reported too.  ie a
user notices unusual activity tracks it down to an app that shouldn't
be using the network reports it we reject it, but if an app crashes on
startup all the time we have a report error to the developer that
drops the app back into more info mode with the issue maybe?

> - Current review process lacks methods for adequate informative 
> dialogue.
> 
> 
> 
> Comments and feedback welcome!
> 
> Have a great day, Daniel
> 


- -- 
You make it, I'll break it!

I love my job :)
http://www.ubuntu.com
http://www.canonical.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlHNn5oACgkQT5xqyT+h3OiTcgCaAtI+8m95gplT9tJuLeJqYZ+s
EyMAnRPK9+ShVqdjbi8cQs5bdbNSeE8P
=ipd/
-----END PGP SIGNATURE-----