ubuntu-appstore-developers team mailing list archive

[Sidnei da Silva <sidnei.da.silva@xxxxxxxxxxxxx>]

Here's a summary of the decisions we have made so far and the implementation
status.

As Martin probably mentioned, we're repurposing a good chunk of the
infrastructure used by Ubuntu One for HTTP Uploads and Downloads.

This service is relatively and intentionally simple and has no knowledge
about
the other parts of the Click Packages infrastructure.

The core of this service is a custom Django/Twisted hybrid that efficiently
streams down files from a storage backend without tying up a thread, but
apart
from that is pretty plain Django. [DONE]

For Ubuntu One, the storage backend used is Amazon S3, and files are
historically stored deflated, which means they can be served with gzip or
deflate compression pretty trivially.

For the Click Packages Upload/Download service (a.k.a. Click Up/Down), we've
made the decision to not compress those files and instead store them as-is.
Our
understanding is that the files will already be compressed with zip or
similar,
so compressing them for storage and transfer would yield pretty minimal
savings. Additionally, we're switching the storage backend to OpenStack's
Swift. We wrote a fairly minimal Swift/Keystone wrapper around our core
library
that talks to S3, such that we can trivially switch between S3 and Swift if
needed. [DONE]

The download service will support Range requests, so resumable downloads
should
be possible to implement on the client side without much effort. [DONE]

Public and Private downloads will be served by different subdomains,
public.apps.ubuntu.com and private.apps.ubuntu.com respectively. Since all
packages will be stored in the same place, there will be some state kept in
Click Up/Down to prevent private packages from being fetched via the public
endpoint. [IN PROGRESS]

Public/Free downloads will be heavily cached at various levels, and will
reuse
pretty much the same setup as Ubuntu One public downloads. This includes a
caching service in front of the application server, using consistent hashing
such that the files are not cached on more than one caching server, and
another
caching service behind the application server, for caching downloads from
the
storage backend. [DONE]

At the moment our plan for authenticated downloads is to use the SSO API for
OAuth validation, which means downloads will require a valid SSO OAuth
token. [DONE]

The validation of access to restricted downloads is still to be defined and
may
end up involving an API call to the Software Center service. [IN PROGRESS]

For uploads, the flow involves a developer registering the package with the
Software Center service, and subsequently providing the actual packages to
the
Click Up/Down service via a POST request. [IN PROGRESS]

Since the Software Center service and the Click Up/Down service are
completely
separate and don't share any data, we need a way to restrict uploads to
authenticated users of the Software Center service. We've settled on using
HMAC-SHA256 signed requests for this, with a shared private key between
Software Center and Click Up/Down. All POST fields but the packages
themselves
will be part of the HMAC digest and thus validated upon reception by the
Click
Up/Down service. [IN PROGRESS]

After being received by the Click Up/Down service, the packages will
default to
being in an 'Unpublished' state. [DONE]

Upon (automated) review in the Software Center service, an API call will be
made from the Software Center service to flip the state to 'Published', at
which point the package will be immediately available for download. This API
call will be HMAC-signed too. [IN PROGRESS]

HTTPS will be required for all requests, both for uploads and downloads.
HTTP
requests will be unconditionally redirected to HTTPS. [DONE]

We do not plan to encrypt internal requests (eg, between the frontend and
the
caching servers, and between caching servers and the application server) for
the time being, though this could be revisited if necessary to ensure
end-to-end integrity of the package. [DONE]

Traffic between the backend caching server and Swift should use HTTPS. [IN
PROGRESS]

We do not plan to employ any encryption or validation of the package when
stored in Swift and in the disk cache of caching servers, but this could
also
be revisited if necessary to ensure the integrity of the package. [DONE]

During uploads from Click Up/Down to Swift we provide an MD5 checksum to
Swift,
which as part of it's API ensures that the upload request fails if the
received file does not match the provided MD5 checksum. [DONE]

A Juju stack comprising the whole infrastructure, including the Postgres
database for keeping Published/Unpublished state and the mapping between
filenames and Swift object ids will be created. [DONE]

This infrastructure will be deployed to Canonical's private OpenStack
infrastructure (a.k.a. 'prodstack'). [IN PROGRESS]

We expect a staging deployment of this to be completed in the next few
days. A
demo environment of the items marked as DONE can be provided upon request.
It
takes around 40min to deploy it from scratch into Canonical's private
OpenStack
infrastructure for developers (a.k.a. 'canonistack').

Feel free to ask any questions and suggest improvements where you see fit.
I'm particularly interested in feedback about security and integrity issues.

--
Sidnei

Make the most of Ubuntu with Ubuntu One
http://one.ubuntu.com