← Back to team overview

drizzle-discuss team mailing list archive

Re: Toru Maesaka: End of Year Progress on BlitzDB

 

Hi Mark!

IIRC tc(hdb|bdb)optimize() can be used for repairing tables but I
wasn't 100% sure. So, I did a little digging around (I only remembered
this from a coffee conversation with the author) and found your
answer. The best-effort way to repair a TC database file is as
follows:

(1) Open the database file without the lock option. Meaning, supply
HDBONOLCK or BDBONOLCK to the open function of the appropriate
database data structure (TCHDB or TCBDB).

(2) Run tchdboptmize() or tcbdboptimize() depending on the database
data structure.

Alternatively, you could attempt to repair a TC database from the
command line. TC provides a utility program called tchmgr (for hash
databases) and tcbmgr (for b+tree databases) which allows you to run
optimize a on database file. So if you wanted to repair a TC hash
database, you would do the following:

    $ tchmgr optimize -nl /path/to/database_file

You didn't find this in the documentation because it was only noted in
TC's FAQ on the Japanese documentation...

    - http://1978th.net/tokyocabinet/spex-ja.html

Also, you didn't find this in the BlitzDB source because I haven't
implemented it yet (as mentioned in the previous email). Although! I
now have a concrete idea on how to implement REPAIR TABLE so I totally
benefited from this email conversation. Thanks.

> The reason that I asked about this is because someone I know used TC
> and loved the performance. Some time later they realized it wasn't
> crash safe. I think any persistence layer should write NOT CRASH SAFE
> on the project overview page with a link to the semantics

I agree with your point and I too believe that it should be stated in
the project site. I'll ping Mikio about this when I see him next
(hopefully 4th of Jan, JST). I'll also blog about this since this info
isn't available in English...

Cheers,
Toru

On Sat, Jan 2, 2010 at 2:19 AM, MARK CALLAGHAN <mdcallag@xxxxxxxxx> wrote:
> On Fri, Jan 1, 2010 at 8:03 AM, Toru Maesaka <dev@xxxxxxxxx> wrote:
>> G'day Mark,
>>
>>> Is the persistent hash table in Tokyo Cabinet crash safe? I ask because of
>>> text in http://1978th.net/tokyocabinet/spex-en.html. I did not find a
>>> definition of 'broken' nor mention of commands to fix a broken database.
>>
>> The honest answer to this is, "NO, it's not crash safe". If one is
>> lucky, she would be able to repair the table(s) using REPAIR TABLE but
>> this is not guaranteed (best-effort only). IIRC, TC has an internal
>> flag that represents whether the table was left open. The repair
>> function will try it's best to repair the database file in that state
>> (though I haven't implemented this yet). So, BlitzDB is not much safer
>> than MyISAM.
>>
>> My answer to this question is the same as Tokyo Tyrant's philosophy.
>> Reduce unit durability in return for higher throughput. Users are
>> recommended to replicate to gain HA. This isn't a good answer for
>> those that doesn't have the luxury of running multiple physical
>> servers but this is my policy at the moment...
>>
>> HOWEVER, I'm always open to ideas and patches to make this engine
>> better. So if you have any cool ideas (or know of anyone), I would
>> love to talk about it :)
>
> What is done for repair? I cannot find references to 'repair' in Tokyo
> Cabinet and I am not sure where to search in the BlitzDB code.
>
> The reason that I asked about this is because someone I know used TC
> and loved the performance. Some time later they realized it wasn't
> crash safe. I think any persistence layer should write NOT CRASH SAFE
> on the project overview page with a link to the semantics. But TC
> isn't alone in making it easy to miss this potential problem --
> MyISAM, MySQL slave replication state and MongoDB do the same.
>



-- 
Toru Maesaka <dev@xxxxxxxxx>



References