ubuntu-phone team mailing list archive

[James Tait <james.tait@xxxxxxxxxxxxx>]

On 9 April 2015 at 13:38, Martin Albisetti <argentina@xxxxxxxxx> wrote:

On Thu, Apr 9, 2015 at 5:50 AM, Niklas Wenzel
> <nikwen.developer@xxxxxxxxx> wrote:
>
> Thanks for bringing this up!
> I completely agree that the current sorting is naive at best and it is
> not intended to be the way we sort search results at all, just a
> convenient way to get us bootstrapped.
>
>
> > To me it seems that we agree we should improve the sorting algorithm.

​Agreed. In fact, I had lunch with an ex-colleague the week before last and
we were discussing exactly this.​

I
> > acknowledge the problem of having no way to deal with unfair reviews but
> I
> > believe this should overall be separated from the sorting problem, even
> > though a better sorting mechanism might relativise some bad reviews.
> >
> > Until now, we've had two suggestions: the Bayesian average and the
> solution
> > mentioned by Selene, where each app would get a 3 star rating by default
> > which would be adjusted later when people start reviewing the
> application.
> > In my opinion these are pretty good solutions, with each of them having
> > their individual advantages or disadvantages.
> > The Bayesian average tries to predict what would happen if plenty of
> people
> > were asked to rate the app but apps without bad reviews only still rank
> > better than those without any reviews.
> > Selene's method solves the latter, but depending on the implementation it
> > might be disadvantageous for apps with few reviews. It would be nice to
> see
> > a code example here. Selene, is your music player open source?
>
> There are a lot of good points about people gaming the system and
> problems with different approaches. Luckily, this is not a new problem
> at all. In fact, it's not even new to Ubuntu!  Matthew Paul Thomas
> spent some time thinking about this years ago and put together a great
> wiki page[1].
> Which points to a great post on how to avoid the most common problems:
> http://www.evanmiller.org/how-not-to-sort-by-average-rating.html
>
​That post only deals with +1/-1 style reviews, but Selene’s approach to
normalisation might still work with it.​

We also plan to add in other metrics over time that are harder to
> game: # of downloads normalised by download date, retention (how many
> people keep it installed?), usage (opt-in feature to let the store
> know how often the app is used), quality (number of crashes) and
> others. To try and avoid loops where apps get more downloads because
> they started off higher on the list, climbing up higher and making it
> harder for new or less well-known apps to surface, we'll likely
> introduce a small amount of serendipity into the search results,
> occasionally returning something high on the list that might not
> normally make it.
>
One metric everyone has ​so far ignored is relevance. If I’ve searched for
“Flickr Scope”, I expect Flickr Scope to be the top result, even if its 3.0
rating is lower than Flickr Uploader’s 4.0 rating. Not a valid criterion
when the user is only browsing, obviously.

Multiple languages also come into play. If I’m searching in British English
for Metro, I’m more likely interested in the Metro newspaper than the Paris
Metro timetable.

Another is something akin to Github’s Pulse - how active is the project? A
recently-updated project could be given preference over one that hasn’t
seen any developer attention for a while.

So there’s more to ordering the results than just ratings. As Martin said,
this is going to be something that will need constant tweaking to get
reasonable results most of the time. Unfortunately, it’s not an exact
science, so sometimes we will get results that will leave everyone
scratching their heads. The key is to minimise those occurrences, and
respond appropriately when they become problematic.

Cheers,

JT
​
-- 
James Tait, BSc. | https://launchpad.net/~jamestait/
Software Engineer, Canonical Online Services
Ubuntu - Linux for human beings | www.ubuntu.com