dulwich-users team mailing list archive

[Gary van der Merwe <garyvdm@xxxxxxxxx>]

On Wed, Dec 18, 2013 at 1:36 PM, anatoly techtonik <techtonik@xxxxxxxxx> wrote:
> I try to clone my repository from GitHub with hg-git and it fails.
> Dulwich 1.9.4, Hg-Git 0.5.0, Windows.
> Please, CC.
<snip>
>   File "dulwich\protocol.pyc", line 104, in read_pkt_line
> ValueError: invalid literal for int() with base 16: 'logi'

I ran into this issue with my python 3 porting. I was getting this
error occurring randomly in the test suit when run with python3. In
python 3, as the bufsize is 0, the read method my return short when
doing read(size); If this happens, then the next call to read_pkt_line
will read data from the previous pkt when trying to read the size.

My solution was to wrap the reads of subprocess stdout with a
io.BufferedReader [1], while leaving the subprocess.Popen(bufsize=0)
so that the stdin remains unbuffered.

In python 3, subprocess.Popen(bufsize=0).stderr is a io.FileIO object.
It's documentation on the read method is very clear [2]:
    "Fewer than size bytes may be returned if the operating system
call returns fewer than size bytes."

In python 2, subprocess.Popen(bufsize=0).stderr is a file object. It's
documentation is a little less clear [3]:
   "Note that this method may call the underlying C function fread()
more than once in an effort to acquire as close to size bytes as
possible."

Not sure exactly what this means, but I read this as: "We will try get
you all the bytes you requested, but we don't guarantee it."


Maybe the solution is to use io.BufferedReader in both python 2 &
python 3. (It would obviously be good to try reproduce the error, as
well as to write a test for it first.)

It might also be good to assert that the len(return value from
read(size)) == size in read_pkt_line. This will let us know for sure
that my diagnoses is correct if we have people run into this in the
future. I will submit a patch for this soon.

Hope this helps,

Gary


[1] https://github.com/garyvdm/dulwich/commit/51afcd1ff3bc0a33601f4a60e2d51d0991b2a093#diff-28cd0777376826bef5605561210673e2L599
[2] https://docs.python.org/3.4/library/io.html#io.RawIOBase.read
[3] https://docs.python.org/2/library/stdtypes.html#file.read