novacut-community team mailing list archive

[Jason Gerard DeRose <jderose@xxxxxxxxxxx>]

Each file in Dmedia has a corresponding doc in CouchDB. This is an
example doc with the essential schema that drives all the automation
behaviors (the "make file management go away" aspect of Dmedia):

{
    "_id": "WEFB4GYQBWOEGCUA5AZEXOKWHCB4IUSYZ2TGJBJUPFWTWCNL",
    "type": "dmedia/file",
    "time": 1338560151.467609,
    "atime": 1342088785,
    "bytes": 26566829,
    "origin": "user",
    "stored": {
        "3KX6WX6YAUQIQ7M2Y3BMPSAI": {
            "copies": 1,
            "mtime": 1338564292,
            "verified": 1338571742
        },
        "PSRJGX2O2N77XLBEBI2TNDNU": {
            "copies": 1,
            "mtime": 1338560150,
            "verified": 1338564292
        },
        "YE4SYZAVTFEBDI2JZQBUTYQY": {
            "copies": 1,
            "mtime": 1338560150
        }
    }
}

This particular file is stored in 3 different "stores" (let's say hard
drives, but a store could also be something like UbuntuOne or S3).
Each of these stores has a durability confidence of 1 copy, and the
total durability for this file is 3 copies. And the copy in the last
store hasn't yet been verified (full content-hash verification).

The "stored" dictionary was the hardest thing to get right. Hopefully
it seems like the obvious design choice now, but for me anyway, it
took a full-time year of blood, sweat, and tears to come up with it.

It has a power and a clarity that I don't think you could match with
SQL/tables/columns. It's especially nice to work with from Python, for
example:

What is the total durability?

>>> sum(v['copies'] for v in doc['stored'].values())

Delete from a particular store:

>>> del doc['stored'][store_id]

It's not quite as elegant to work with from JavaScript, but that's
also just a reflection of how well designed the Python `dict` and
`list` types are, and how awesome Python iteration is.

But the (still) amazing thing to me is that it only takes 4 relatively
simple view functions to drive all the complex Dmedia file automation
behavior. Dmedia does 4 critical housekeeping tasks:

1) At a high-frequency it checks that a file is actually still in a
particular store, and that the file has the expected mtime and the
correct file size. Dmedia always works from the assumption that it
shouldn't particularly trust the metadata, so it constantly re-samples
reality. And if it's been too long since a particular store has been
"connected" to Dmedia (think external USB HDD), Dmedia will
automatically update all those files to have "copies": 0, meaning, the
files might be there, but we're not going to count on them.

2) At a lower frequency it does full content-hash verification,
starting with the files that have gone the longest without
verification (starting first with files that have *never* been
verified). This is expensive so we can't do it as often, but this is
the only way to know without any doubt that the exact bytes are stored
with perfect integrity.

3) It monitors the database for files with insufficient durability,
and tries to fix the problem when it finds this, starting with the
files with the lowest durability. Dmedia treats irreplaceable user
created files specially. So we're talking about files with "origin":
"user", and Dmedia tries to maintain at least a durability of 3 copies
for these files.

4) It monitors hard drives for low free space, and when space is
needed, it figures out which files can be safely reclaimed, and
deletes them, starting with the least likely to be used (the oldest
atime). This one always sounds scary, but for pro video especially,
your hard drives are constantly filling up, and you often need to free
space to make room for a new working set of files. This is crazy error
prone for the user to do manually (not to mention a stressful waste of
time). It's far lower risk for Dmedia to do this automatically.

And here's the 4 view functions, respectively, that drive these 4 tasks:

file_stored = """
function(doc) {
    if (doc.type == 'dmedia/file') {
        var key;
        for (key in doc.stored) {
            emit(key, null);
        }
    }
}
"""

file_verified = """
function(doc) {
    if (doc.type == 'dmedia/file') {
        var key;
        for (key in doc.stored) {
            emit([key, doc.stored[key].verified], null);
        }
    }
}
"""

file_fragile = """
function(doc) {
    if (doc.type == 'dmedia/file' && doc.origin == 'user') {
        var copies = 0;
        var key;
        for (key in doc.stored) {
            copies += doc.stored[key].copies;
        }
        if (copies < 3) {
            emit(copies, null);
        }
    }
}
"""

file_reclaimable = """
function(doc) {
    if (doc.type == 'dmedia/file' && doc.origin == 'user') {
        var copies = 0;
        var key;
        for (key in doc.stored) {
            copies += doc.stored[key].copies;
        }
        if (copies >= 3) {
            for (key in doc.stored) {
                if (copies - doc.stored[key].copies >= 3) {
                    emit([key, doc.atime], null);
                }
            }
        }
    }
}
"""

If you want to explore these deeper and think about why they work, I
recommend looking at the unit test for file_reclaimable, as it has the
most complex and subtle behavior:

http://bazaar.launchpad.net/~dmedia/dmedia/trunk/view/head:/dmedia/tests/test_views.py#L336