dulwich-users team mailing list archive

[Dave Borowitz <dborowitz@xxxxxxxxxx>]

On Tue, Mar 1, 2011 at 13:57, Dave Borowitz <dborowitz@xxxxxxxxxx> wrote:

> On Tue, Mar 1, 2011 at 13:48, David Blewett <david@xxxxxxxxxxxxxxxx>wrote:
>
>> On Tue, Mar 1, 2011 at 4:39 PM, Dave Borowitz <dborowitz@xxxxxxxxxx>
>> wrote:
>> > On Tue, Mar 1, 2011 at 13:27, David Blewett <david@xxxxxxxxxxxxxxxx>
>> wrote:
>> >> Indeed, that was the problem. I looked around and couldn't find a way
>> >> to disable this behavior in Git,
>> >
>> > Last I checked it was hard-coded over a certain (also hard-coded)
>> threshold,
>> > unfortuantely.
>>
>> I found this in the man page for git-config, but I'm not sure it's
>> what we're after:
>>
>>       http.postBuffer
>>           Maximum size in bytes of the buffer used by smart HTTP
>> transports
>>           when POSTing data to the remote system. For requests larger than
>>           this buffer size, HTTP/1.1 and Transfer-Encoding: chunked is
>> used
>>           to avoid creating a massive pack file locally. Default is 1 MiB,
>>           which is sufficient for most requests.
>>
>> I couldn't figure out where to set that, or what units it was using etc.
>
>
> It's a config variable that you can set with git config [--global]
> http.postBuffer xxx (in bytes). But that's not what you want. That refers to
> the maximum amount of pack data that the client is willing to buffer in
> memory (as opposed to writing directly to the wire). For example, if the
> client is uploading a 400MB pack of the Linux kernel repo to the server, and
> it's generating it on the fly, it doesn't want to have to buffer all that in
> memory. But that's never gzipped, since the pack is already highly
> compressed.
>
> The gzipping only happens (I think) when an upload-pack request exceeds the
> smaller LARGE_PACKET_MAX. An upload-pack request is not pack data, but
> rather a list of SHAs that the client has/wants.
>
> For reference, see:
>
> http://git.kernel.org/?p=git/git.git;a=blob;f=remote-curl.c;h=d0fb0a044a85ec6169c4623c7c2ab2143dbb5784;hb=HEAD#l422
> (Note that use_gzip is set to 1 by default and is only turned off if the
> size is under the threshold.)
>
>
>>  >> but I couldn't find any such
>> >> in a cursory look last night.
>> >
>> > I haven't even done that :). I would not be opposed to rolling our own,
>> both
>> > because it doesn't seem very complicated and because Jelmer doesn't like
>> > adding required third-party dependencies.
>>
>> Yeah, I think I could get something to work this week/weekend maybe.
>>
>
Awesome! If, while you're at it, you managed to get rid of
_LengthLimitedFile, at least for the gzipped case, that would be cool too.


> Thanks,
>>
>> David Blewett
>>
>
>