mugle-dev team mailing list archive

[Matt Giuca <matt.giuca@xxxxxxxxx>]

So there are two major problems with our KVP API:

   1. Users are not able to serialize their own data types, and are instead
   forced to use built-in types like String and Vector.
   2. If the user builds their code with a different version of GWT than we
   did, and that version has different serialization IDs for certain types, the
   RPC will fail (bug #788918 <https://bugs.launchpad.net/mugle/+bug/788918>
   ).

A minor problem is also the warnings you get when compiling student code,
due to being unable to serialize certain types.

It was well-intentioned, but I think our KVP API has failed, and we need a
more robust one. Unfortunately this will require changes to student code.

Note that issue #2 can potentially affect the other APIs as well, but
probably won't, since the other APIs use only primitive types and String,
which are unlikely to change serialization IDs. (The problem we had today
was because java.util.Vector's serialization ID changed.)

So there are a couple of approaches we can take here (not necessarily
mutually exclusive).

*The problem with KVP serialization*

It is important to note that the current KVP serialization is quite
unnecessarily roundabout, given that the server is really just taking
whatever the client gives it, shoving it in the database, and getting it out
again. Currently, this is what happens:

   1. The client takes an arbitrary object and serializes it using
   com.google.gwt.user.client.rpc.IsSerializable.
   2. The client sends the stream to the server.
   3. The server deserializes the stream, turning it into a Java object in
   memory.
   4. The server serializes the object using java.io.Serializable.
   5. The server sends the stream to the database for storage as a blob.

Retrieving the data does this same process in reverse. What should be
apparent is that the server is deserializing and immediately reserializing,
yet it doesn't need to ever inspect the data. It is effectively treating the
data as a dumb blob, yet it is unpacking and repacking. Most (but not all)
of our nightmares happen during steps 3 and 4 -- the fact that the client
and server have different versions is only an issue because the server
deserializes.

A much better approach would be:

   1. The client takes an arbitrary object and serializes it using *some
   serialization framework*.
   2. The client sends the stream to the server.
   3. The server sends the stream to the database for storage as a blob.

The question is what serialization framework to use. Using GWT-RPC seems a
bit silly since it is no longer a client-server communication, it is just
storage. I thought the original plan was to use java.io.Serializable on the
server. But I'm still not sure if that works or not. And then there's JSON.
I'll get to that later.

Finally, note that our current requirement that KVP objects be in
java.io.Serializable is pretty misleading, since we are currently using
GWT-RPC instead.

*Break away from GWT-RPC*

Rather than basing our API on GWT-RPC, we can use a simpler, more standard
serialization format (preferably JSON). This is what I wanted to do in the
first place, but we decided to use GWT-RPC to simplify the interface for
students. However, in hindsight, the interface would actually not need to
change (ignoring the KVP which I'll get to later). This would just mean that
we change the code in the Async classes we give to students. They have the
same Async interface, but internally, they produce JSON on the client side,
and send it to the server, which reads JSON, and sends JSON back.

The KVP service is a bit special, since it currently allows arbitrary data
types, but is broken. Switching to JSON would not break it further, as we
could still perform arbitrary serialization on the client side
*before*putting it into JSON. In fact, that might improve things
somewhat, and even
potentially allow user data types to be serialised, and also fix bug
#788918<https://bugs.launchpad.net/mugle/+bug/788918>,
since there would be no deserialization on the server at all. I will address
this later, since I still don't think that's good enough, but just a note
that JSON shouldn't make this situation worse.

The benefits of this would be:

   - The actual network service would be a proper RESTful API. We could use
   GET and POST appropriately, whereas right now we only do POSTs even when
   receiving data.
   - Easier to debug. You can manually put in requests and see the results
   in a human-readable way.
   - Possibility of allowing non-GWT games on the platform. GWT would no
   longer be necessary; it would just be a recommended framework. Essentially,
   MUGLE would become a generic JavaScript platform with a REST API, and we
   would supply a GWT binding to that API to allow you to write your JavaScript
   code in GWT.
   - Absolutely fixes bug
#788918<https://bugs.launchpad.net/mugle/+bug/788918>-- no versioning
errors, since we completely control the serialization
   format.
   - Source-compatible with existing code. We would still offer the same
   Async API.

The downsides:

   - Not binary-compatible with existing code. All projects on MUGLE would
   need to recompile, because there is client code in the student bundle that
   talks GWT-RPC. Thus it would be better to do this before we put lots more
   games up.
   - Need to change the KVP serialization format to something else, since we
   won't be using GWT-RPC (but that's something we should do anyway).

So I think this would be a good thing to do right away.

*Fixing the KVP value serialization for arbitrary student classes*

Now, the matter of serializing arbitrary student data.

So firstly, I would like to have another go at using java.io.Serializable on
the client. I think we tried to use it on the server before but not the
client. But from what I have read, that may not work. I don't think GWT's
client implements this properly. Still, we should see exactly how it fails.
If it works (that is, we can take an arbitrary class which implements
Serializable, and turn it into a binary blob and back, without causing
warnings), then we can just use that, without any changes to the API. After
applying the changes suggested above, KeyValueServiceAsync will no longer be
a special GWT class; it will be our class and we will be able to put code in
it. Therefore, we will put code in it to serialize on the client, then send
that serialized blob in the form of a JSON string. Note that I am
*not*(yet) talking about encoding student data as a JSON string (since
that is
not possible in the general case without reflection) -- I am talking about
serializing to a binary blob and sending that blob wrapped in a JSON string
as a transport format. Again, this is fine, because while the server
*will*be decoding the JSON objects used by the API, we
*do not* want it to be decoding the blobs of the object data (steps 3 and 4
above), we just want it to store it in the database, and retrieve it again.

Another possibility is that we use some other reflection framework. A quick
google shows that there is someone claiming to have browser-side reflection
working in GWT (GWT Reflection <http://gwtreflection.sourceforge.net/>). If
this works, then it means we can write *our own* custom serialization
mechanism which takes any Object and converts it into a String or byte[] --
for debugging purposes, a JSON format would be handy (but confusing -- our
requests would have a JSON object encoded as a string inside another JSON
object!). We would have special cases for primitive types, arrays, and other
important types such as String, Vector and HashMap, and for all other
classes we would just recursively serialize all of the fields as a JSON
object. One tricky point is reconstructing the objects -- we would either
need to store a lot of dynamic type information in our serialized format, or
ask the user to supply the Class object to convert it back to.

But if it doesn't work, we will need to do something more drastic. We can
change the put/get interface so that rather than taking a
java.io.Serializable, we take a
com.google.gwt.json.client.JSONValue<http://google-web-toolkit.googlecode.com/svn/javadoc/latest/index.html?com/google/gwt/json/client/JSONValue.html>.
This type can store arbitrary data by recursive arrays and dictionaries, and
primitive strings, booleans and numbers, and can be easily serialized and
deserialized. However, it would require work on the part of the students to
set it up.

We should probably then supply some other useful helper methods. We could
have putLong, putDouble, putBoolean, putString and equivalent getters, which
just take ordinary Java values and turn them into JSON primitives. That
would allow most students to fix up their code just by calling the
appropriate put/get methods. The only hard cases would be where students are
currently serializing Vectors; they would need to manually turn them into
JSONArrays and back (we can't do this automatically, since we don't know
what the types of the objects inside the Vector are, unless we had a
putVectorLong, putVectorDouble, etc, which would be awful). So this would be
a bit of work for students. Still, it may be worth it, but I'm hoping one of
the other two methods works.

The last option is we just change put and get to take a good ol' String.
Students can handle serialization themselves. In most cases, this will be
trivial: Integer.toString/parseInt and the other primitives will work fine.
Strings will not need to change. Vectors, students can just do something
simple like insert commas between them to put them in, and call split(",")
to get them out again. But that is certainly not as rich an API as we
originally promised.

Thoughts on all of this?