acmeattic-devel team mailing list archive

[krishnan parthasarathi <krishnan.parthasarathi@xxxxxxxxx>]

>
>>>
>>>    - Programming tools: Eg. I vote for tools like Protocol buffers,
>>>    Python Twisted framework, pycrypto library (More on the Wiki [1])
>>>
>>>  I think we should try to reduce the dependencies on the s/w if we are to
>>> make it cross-platform. So, I propose that we should integrate pycryto code
>>> inside our software. Twisted should be available on most platforms (I
>>> think), so we don't have to worry
>>>
>> Twisted is available for all platforms (Linux, Windows, OS X). pyCrypto is
>> also available for all platforms.
>> However, I am not completely convinced of the integration. I would really
>> love to hear from the corporate guys on this. (Lets blow the Vuvuzela to
>> call them!)
>> We could definitely build binaries with the dependencies, but I would like
>> to keep our codebase small. Integration is a release plan right? Does it
>> affect our development?
>>
>
> Integrating pycrypto would probably not increase the codebase much. We will
> be stripping it down to bare necessities, but Suren or at least some one
> else should probably comment on this. Only we seem to be the ones actively
> discussing things.
>
The main disadvantage of 'integrating' pycrypto into our codebase is that we
must maintain that piece of code as well. We must keep track of pycrypto's
development cycle to do this. Our dependency of pycrypto is like any other
s/w dependency. This needs to be resolved in packaging of software. eg
tarball, .deb, .rpm, .egg, etc


>
>>
>>
>>> about that. It would also be quite a challenge to integrate that into our
>>> s/w. I am still a bit ambivalent about Protocol buffers. I don't think we
>>> really need it. It is meant to be usable across programming languages, etc.
>>> We don't really need it. I think that a simpler way to exchange messages
>>> between server/client is to use pickling [1]. We can define a message
>>> structure that contains all the fields we require, and pickle it. We can
>>> wrap the pickled structure in TCP and send it.
>>>
>> I know how Pickle and cPickle works. One of the most important
>> disadvantage of Picke and similar serialization techniques is backward
>> compatibility. Suppose we update the object contents to add another field,
>> all of the servers and clients have to be updated with the new version -
>> otherwise Pickle crashes [1]. This is an important advantage of Protocol
>> buffers.
>>
>
> The pickle documentation page does not seem to say when it will crash. Can
> you be more specific? Unpickling an older version object will simply give an
> object having a different set of attributes than the version that the newer
> software is expecting - this need not crash anything right? This can be
> handled in code itself keeping backwards compatibility in mind.
>
The messages are internal to our software and we can ensure we don't have
our server and client software advance at different speeds in their message
format compatibility.

>
>
>> However, the protocol buffers implementation in Python is pretty slow. I
>> would advocate a serialization format such as ProtoBuf, XML, JSON, Thrift,
>> etc.
>>
>>    - Protobuf - Really good C++ version. Python version is really slow.
>>    We can hope for Google to improve this. Maybe Satya can tell us more about
>>    the status of this.
>>    - XML - too bulky and slow.
>>    - JSON - Less bulkier than XML, but still uses ASCII encoding
>>    - Thrift - uses binary encoding (similar to protobufs). Released out
>>    of Facebook. New and under development.
>>
>> Assuming pickling is really a bad choice (which I am not sure of), I think
> we should concentrate on performance in alternative ways. XML and JSON we
> should definitely avoid. Can someone run a speed comparison between thrift
> and protobuf?
>
I did the following test. I pickled a class with two members. I tried
unpickling it onto a different class and received an AttributeError
execption (not a crash). These exceptions can be handled when we have
backward compatibility issues. Now that this is cleared, I think we can go
ahead with pickling as our message serialization format.


cheers,
krishnan