ubuntu-appstore-developers team mailing list archive

[Jamie Strandboge <jamie@xxxxxxxxxxxxx>]

On 09/17/2014 09:53 AM, Ricardo Kirkner wrote:
> 
> 
> On Wed, Sep 17, 2014 at 11:48 AM, Jamie Strandboge <jamie@xxxxxxxxxxxxx
> <mailto:jamie@xxxxxxxxxxxxx>> wrote:
> 
>     On 09/17/2014 09:31 AM, Jamie Strandboge wrote:
>     ...
> 
>     >> All of this sounds pretty reasonable, however I'm now somewhat confused. I think
>     >> we're talking about different things here. So to disambiguate,
>     >>
>     >> 1. the process is to override errors so to not automatically reject, but force a
>     >> manual review, and *not* to override errors to allow an automatic approval, right?
>     >
>     > What you have implemented thus far is perfect. What Daniel is wanting to do is
>     > further reduce the number of manual reviews for apps where the app was already
>     > manually reviewed, so it doesn't have to be next time. What I have discussed
>     > allows for the overrides to allow warnings and errors that would normally prompt
>     > a manual review to be ignored. I also discussed that we could have logic around
>     > this so that sometimes errors could get a pass, but other times not and if not,
>     > there is extra information that is available for display to the reviewer.
>     >
>     I reread your previous email and see where the confusion may lie. The click
>     reviewers team is used to "manually reviewing" any apps that have warnings
>     and/or errors. This "manual review" does typically include an in depth review.
>     Meanwhile, the click-reviewers-tools outputs errors and warnings and a subset of
>     those errors have the 'MANUAL REVIEW' string (and now the '"manual_review":
>     True' json) which indicates that the reviewer should perform an in depth review
>     of the app. As such, the term "manual review" is a bit overloaded in this
>     conversation.
> 
>     Daniel is hoping to have an override mechanism for any errors and warnings so
>     that your server logic becomes:
> 
>     1. If there are non-overridden errors in the automated review results and none
>        of them are marked as requiring manual review, the app is automatically
>        rejected
>     2. If there are non-overridden errors in the automated review results and some
>        of them are marked as requiring manual review, the app is left in the review
>        queue waiting for a reviewer to pick it up and start a review (just like it
>        is now)
>     3. If there are errors and warnings in the automated review results and all of
>        them are overridden, the app is automatically approved
>     4. If there are no errors and no warnings in the automated review results, the
>        app is automatically approved.
> 
>     Note-- you said only 'errors' in portions of your email (which is captured
>     above) which got me thinking: what does the server do when an app has no errors
>     but there are warnings?
> 
> 
> This sounds good.  I presume the click-reviewers-tools will be marking checks
> with overrides somehow in the check results json so that the server can figure
> out what to do.
> 
Right, this is what I was getting at when I said "I envision the scripts could
add the applicable override json to its json output and return "pass" when
overrides are applied (assuming no other warning or errors)".

> Regarding warnings, if there are warnings but no errors, the app is left in the
> review queue for a reviewer to pick it up (ie, manual review is implied)
>  
Ok, cool.


-- 
Jamie Strandboge                 http://www.ubuntu.com/

Attachment: signature.asc
Description: OpenPGP digital signature