u1db-discuss team mailing list archive

[Eric Casteleijn <eric.casteleijn@xxxxxxxxxxxxx>]

Hi Jason,

I thought it might be instructive (for me ;) to try and see to what 
extent those views could be implemented with the current state of the 
art of u1db. Note that I probably will get stuck somewhere, as u1db
indexes are much much simpler than couchdb views.

> Each file in Dmedia has a corresponding doc in CouchDB. This is an
> example doc with the essential schema that drives all the automation
> behaviors (the "make file management go away" aspect of Dmedia):
>
> {
>      "_id": "WEFB4GYQBWOEGCUA5AZEXOKWHCB4IUSYZ2TGJBJUPFWTWCNL",
>      "type": "dmedia/file",
>      "time": 1338560151.467609,
>      "atime": 1342088785,
>      "bytes": 26566829,
>      "origin": "user",
>      "stored": {
>          "3KX6WX6YAUQIQ7M2Y3BMPSAI": {
>              "copies": 1,
>              "mtime": 1338564292,
>              "verified": 1338571742
>          },
>          "PSRJGX2O2N77XLBEBI2TNDNU": {
>              "copies": 1,
>              "mtime": 1338560150,
>              "verified": 1338564292
>          },
>          "YE4SYZAVTFEBDI2JZQBUTYQY": {
>              "copies": 1,
>              "mtime": 1338560150
>          }
>      }
> }
>
> This particular file is stored in 3 different "stores" (let's say hard
> drives, but a store could also be something like UbuntuOne or S3).
> Each of these stores has a durability confidence of 1 copy, and the
> total durability for this file is 3 copies. And the copy in the last
> store hasn't yet been verified (full content-hash verification).
>
> The "stored" dictionary was the hardest thing to get right. Hopefully
> And here's the 4 view functions, respectively, that drive these 4 tasks:
>
> file_stored = """
> function(doc) {
>      if (doc.type == 'dmedia/file') {
>          var key;
>          for (key in doc.stored) {
>              emit(key, null);
>          }
>      }
> }
> """

This one's not currently supported but could easily be if a 'keys' index 
mapping were to be added, and then the index would be:

doc.type, keys(doc.stored)


> file_verified = """
> function(doc) {
>      if (doc.type == 'dmedia/file') {
>          var key;
>          for (key in doc.stored) {
>              emit([key, doc.stored[key].verified], null);
>          }
>      }
> }
> """

this one is harder, as we don't have a way to peek into variable parts 
of the document, and it's not immediately obvious to me how that could 
be added without adding a great deal of complexity, so this might just 
be a use case that u1db is too simple for. (Restructuring the document 
to contain a list of the keys somewhere would work, but that's assuming 
you can and want to do that.)

> file_fragile = """
> function(doc) {
>      if (doc.type == 'dmedia/file' && doc.origin == 'user') {
>          var copies = 0;
>          var key;
>          for (key in doc.stored) {
>              copies += doc.stored[key].copies;
>          }
>          if (copies < 3) {
>              emit(copies, null);
>          }
>      }
> }
> """

Same for this one. We intentionally didn't support any kind of real 
logic in the views, to reduce complexity, both for view authors, and for 
the indexing engine.

> file_reclaimable = """
> function(doc) {
>      if (doc.type == 'dmedia/file' && doc.origin == 'user') {
>          var copies = 0;
>          var key;
>          for (key in doc.stored) {
>              copies += doc.stored[key].copies;
>          }
>          if (copies >= 3) {
>              for (key in doc.stored) {
>                  if (copies - doc.stored[key].copies >= 3) {
>                      emit([key, doc.atime], null);
>                  }
>              }
>          }
>      }
> }
> """

And same for this one.

--
eric casteleijn
https://launchpad.net/~thisfred