launchpad-dev team mailing list archive

[Benji York <benji.york@xxxxxxxxxxxxx>]

On Fri, Mar 25, 2011 at 4:17 PM, Robert Collins
<robertc@xxxxxxxxxxxxxxxxx> wrote:
> On Sat, Mar 26, 2011 at 9:04 AM, Benji York <benji.york@xxxxxxxxxxxxx> wrote:
>> On Thu, Mar 24, 2011 at 10:11 PM, Robert Collins
>> <robert.collins@xxxxxxxxxxxxx> wrote:
>>> This is about optimising a particular part of the current webservice;
>>> its still useful in an expand/filter world I think, because some
>>> collections are massive.
>>
>> If I'm understanding the various moving parts correctly, the
>> filter/expand mechanism would remove the need for this optimization.
>> Not that we shouldn't do it for the short term gains (about which I have
>> no opinion one way or the other).
>>
>> How would filter/expand help in this scenario?  It will first return the
>> membership of whatever set you specify and the client would then batch
>> using that information.  There isn't any need to do batch calculations
>> on the server side, avoiding the expensive surrogate key calculations.
>
> the filter/expand system still runs into trouble in very large
> batches; we don't know how large 'very large' is because we haven't
> written it yet. A trivial example of where very large is too large:
> ubuntu/allbugsever

Indeed, we can't be sure because we haven't built it yet, but I'm
hopeful that we'll find a way.

For example, 1 million bug URIs is 47 megabytes.  Gzipped they take up
2.5 meg.  If that's any indication, we might be able to avoid batching
set membership altogether.
-- 
Benji York