launchpad-dev team mailing list archive

[Martin Pool <mbp@xxxxxxxxxxxxx>]

We had a chat about what would be desirable in a Twisted-based async
web client api.

Background:
 * https://dev.launchpad.net/LEP/WebservicePerformance

We can get inspiration from existing Python remote-object-like libraries:
 * http://www.lothar.com/tech/papers/PyCon-2003/pb-pycon/pb.html
 * http://twistedmatrix.com/documents/10.0.0/api/twisted.protocols.amp.html

Some things:

 * Application startup should not require an http roundtrip (assuming
we have a reasonably current wadl and auth token - true at the moment
in launchpadlib).
 * If the client can reasonably be expected to know the URL of an
object ('/bzr', or /bugs/1) then it should just use this, without
doing a redundant call merely to be able to traverse through that
object.
 * Anything that does a remote call should be explicit and return a Deferred.
 * Therefore, traversing object links so they need to be fetched
separately ought to be something the client explictly asks for
 * Nothing should implicitly do more than one remote call: for
instance, after putting an object to the server, we should not
automatically read it back in a separate http call.  (It's fine of
course to build higher-level operations that do multiple calls, but
there should be a clear one-to-one layer.)
 * Explicit is better than implicit: for net operations, and for
modifications of remote operations.
 * Objects that do remote calls should be very obvious.  (Perhaps put
them all onto a LaunchpadServer object, passing objects as
parameters.)
 * Proxy objects are just value objects, with little magic.  (For
instance, we don't unwrap _link fields into magic pointers.)
 * Rather than mutating object fields then saving them, we might have
an explicit update operation, which can return a Deferred for its
completion.
 * For collections, we will probably deliver them into a consumer
object.  This can probably opt-in to see batch boundaries but
otherwise it will get all pages.  It will get gotObjects() that gets
the whole batch, and the default can then split them and deliver them
one by one.
 * Getting a coherent list across multiple calls is hard; the best
solution is probably to make the server fast enough that it can return
large lists, and secondarily to have higher-level client code that
will heal tearing between batch pages by matching up unique ids.
 * If Launchpad adds the ws.expand parameter, we can have a way to
pass it.  Rather than this pre-populating attributes that would
otherwise be lazy, it will instead add attributes to the passive
objects.  For instance, if you don't ask for expansion you will get
task.bug_link; if you do expand that you will also have task.bug which
is a local reference to another real passive object.
 * We could possibly provide a synchronous layer on top of this.
 * This seems to be able to accommodate the
<https://dev.launchpad.net/LEP/WebservicePerformance> approach of
first giving you a huge set of URLs, then later doing separate batched
calls to get the details.
 * Although the client code clearly shows when it's doing network io,
it doesn't need to expose exactly what HTTP method is being used,
which is an irrelevant detail.  (Case in point, some conceptually-read
methods might use POST to send large parameters.)

So, sketchy, using inlineCallbacks:

  me = yield lp.get_object(lp.me)
  # gives you back a fully-populated passive object
  print me.full_name

  class ConsumeAndPrintBugs(object):
     def objectReceived(self, bug):
        print bug.title

  done = yield lp.get_collection(me.assigned_bugs_link, ConsumeAndPrintBugs())

  me = yield me.update(full_name='Martin "poolie" Pool')

  urls = yield me.callRemote('getArchiveSubscriptionURLs')

  # make a url without fetching anything
  bzr_url = lp.get_object_link(lp.Products, 'bzr')

  # now get it

  class PrintTaskTitles(object):
     def objectReceived(self, task):
        # need another roundtrip to get the bug
        d = lp.get_object(task.bug)
        d.addCallback(self.print_bug)
     def print_bug(self, bug):
        print bug.title

  bzr = lp.get_object(bzr)
  lp.call_collection_method(bzr, 'searchTasks', PrintTaskTitles())

  # we can have a convenience method that folds up all the collection
objects into one list, if you don't care to get them individually;
this gives a Deferred producing a list
  all_bugs = yield lp.call_collection_method_to_list(bzr, 'searchTasks')
  # time passes...
  print len(all_bugs)  # should be correct; and you now have all the data

-- 
Martin