maria-developers team mailing list archive

[Jonas Oreland <jonaso@xxxxxxxxxx>]

Hi again,

1) we have not done column level encryption at all.

2) keys are managed in a separate module (which can be overridden by
plugin, which we do for testing)
3) keys have 32 bit version
4) different key versions can exists simultaneously in database

5) for innodb we encrypt both datafiles and logfiles
6) encryption of innodb datafiles works roughly like this:
- pages are encrypted with aes ctr
- page 0 of each encrypted datafile contains IV
- all pages except page 0 is encrypted
- if compression is used, pages are encrypted after compression
- pages are encrypted/decrypted by calls added to innodb-buffer-pool module
- encrypted pages are "tagged" with which key-version encrypted it
- different pages in database can be encrypted with different key version
- when key-server module reports that a new key version is available, a
background task will re-encrypt database with new key-version (key rotation)
- key rotation is performed with configurable number of threads, that will
perform configurable amount of IOPS.
one can also configure how frequently pages shall be key-rotated (i.e max
key "age")

/Jonas



On Tue, Jun 17, 2014 at 6:40 PM, Elmar Eperiesi-Beck <elmar@xxxxxxxxxxxxxxxx
> wrote:

> Hi,
> I agree with you. If we want to know, what Google has developed as
> encryption feature, we will have to wait for your source code to be
> published.
>
> In the meantime, you can find our concept for the encryption on our
> website: http://bit.ly/1slJyuI
> Feedback (negative and positive) from all of you is welcome - and needed!
>
> Best Regards
> Elmar
>
>
> Am 17.06.2014 um 12:50 schrieb Jonas Oreland <jonaso@xxxxxxxxxx>:
>
> Hi again,
>
> > by "interfaces" I was looking for the Maria DB place/ function / hook...
> where you are enhancing the MariaDB Code.
>
> I'm not sure how to convey this in a digestible form, attaching diffstats
> below. Not sure if it's helps :-(
>
> There are many aspects of it.
> And each of the sub-projects (innodb data, innodb log, maria, tempfiles,
> binlog) has "interesting" details.
>
> /Jonas
>
>
> storage/innodb has this diffstat:
>  CMakeLists.txt       |    2
>  btr/btr0cur.cc       |    9
>  buf/buf0buf.cc       |  213 +++++
>  buf/buf0checksum.cc  |    8
>  buf/buf0dblwr.cc     |   40 -
>  buf/buf0flu.cc       |    6
>  buf/buf0rea.cc       |    7
>  dict/dict0load.cc    |    8
>  fil/fil0crypt.cc     | 1986
> +++++++++++++++++++++++++++++++++++++++++++++++++++
>  fil/fil0fil.cc       |  280 ++++++-
>  fsp/fsp0fsp.cc       |   36
>  handler/ha_innodb.cc |  110 ++
>  handler/i_s.cc       |  292 +++++++
>  handler/i_s.h        |    1
>  include/buf0buf.h    |   60 +
>  include/buf0buf.ic   |   29
>  include/fil0fil.h    |  266 ++++++
>  include/fsp0fsp.h    |    9
>  include/log0crypt.h  |   85 ++
>  include/log0log.h    |   21
>  include/log0recv.h   |    5
>  include/mtr0log.ic   |    2
>  include/mtr0mtr.h    |    8
>  include/srv0srv.h    |    8
>  log/log0crypt.cc     |  256 ++++++
>  log/log0log.cc       |   93 ++
>  log/log0recv.cc      |   35
>  mtr/mtr0log.cc       |    4
>  row/row0import.cc    |    3
>  srv/srv0srv.cc       |   14
>  srv/srv0start.cc     |   29
>  31 files changed, 3853 insertions(+), 72 deletions(-)
>
> storage/maria has this diffstat:
>  CMakeLists.txt                            |   12
>  ha_maria.cc                               |   12
>  ma_bitmap.c                               |   63 ++--
>  ma_blockrec.c                             |  222 ++++++++------
>  ma_blockrec.h                             |   26 +
>  ma_check.c                                |   49 +--
>  ma_checkpoint.c                           |    4
>  ma_close.c                                |    2
>  ma_create.c                               |   56 +++
>  ma_crypt.c                                |  464
> ++++++++++++++++++++++++++++++
>  ma_crypt.h                                |   26 +
>  ma_delete.c                               |    2
>  ma_key_recover.c                          |   10
>  ma_loghandler.c                           |   63 +---
>  ma_open.c                                 |   48 ++-
>  ma_pagecache.c                            |  154 ++++++---
>  ma_pagecache.h                            |   34 +-
>  ma_pagecrc.c                              |  118 ++++---
>  ma_static.c                               |    1
>  ma_write.c                                |   24 -
>  maria_def.h                               |   81 ++---
>  unittest/ma_pagecache_consist.c           |   28 -
>  unittest/ma_pagecache_rwconsist.c         |   27 -
>  unittest/ma_pagecache_rwconsist2.c        |   27 -
>  unittest/ma_pagecache_single.c            |   27 -
>  unittest/ma_test_loghandler_pagecache-t.c |   29 -
>  26 files changed, 1102 insertions(+), 507 deletions(-)
>
> A noticeable difference between innodb and maria is that we didn't
> implement encryption of the log for maria,
> as we only added support for temporary tables. For maria we also only
> added encryption support for BLOCK format
> but added all the features to this format so that it was usable for all
> temp-table scenarios. maria also doesn't have
> key-rotation feature like innodb has.
>
> I couldn't (as) easily extract diffstats for binlog and tempfile
> encryption.
> You have to wait for the code to get published...
>
>
> On Tue, Jun 17, 2014 at 7:29 AM, Elmar Eperiesi-Beck <
> elmar@xxxxxxxxxxxxxxxx> wrote:
>
>> Hi,
>> by "interfaces" I was looking for the Maria DB place/ function / hook...
>> where you are enhancing the MariaDB Code.
>> This would help me to understand what you are trying to do.
>>
>> Elmar
>>
>> Am 17.06.2014 um 07:02 schrieb Jonas Oreland <jonaso@xxxxxxxxxx>:
>>
>> Hi again,
>>
>> > What is the type of license of your code?
>>
>> I asked internally about license, and it seems like we releasing dual
>> gpl2/apache licensed code.
>>
>> > I would like to know, which interfaces from maria-DB you are using.
>>
>> I don't 100% understand the question.
>> We didn't write any actual encryption code, but used the one provided in
>> openssl.
>> Other than that, we didn't really "use interfaces", but rather
>> added/modified functionality/interfaces here and there.
>>
>> Can you be more specific ?
>>
>> /Jonas
>>
>>
>>
>> On Sat, Jun 7, 2014 at 11:20 PM, Elmar Eperiesi-Beck <
>> elmar@xxxxxxxxxxxxxxxx> wrote:
>>
>>> Hi!
>>> We (eperi) would be glad to do a joined work with Google.
>>> Our solution works with MS-SQL, Oracle and other DBs and we are
>>> currently porting it to MariaDB - and - as Monty said - its never to late
>>> to put some sources together and make the best for the open source
>>> community.
>>>
>>> What is the type of license of your code?
>>>
>>> Jonas, I am looking forward to connect to you directly.
>>>
>>> Regards
>>> Elmar
>>>
>>> Hi!
>>>
>>> > Hi Jonas,
>>> > (same Jonas we know from NDBCLUSTER? :-) Good to see you again)
>>> >
>>> > On 6 Jun 2014, at 02:31, Jonas Oreland <jonaso@xxxxxxxxxx> wrote:
>>> >
>>> >> Hi there,
>>> >> I read this blog post
>>> >>
>>> http://monty-says.blogspot.com/2014/05/for-your-eyes-only-or-adding-better.html
>>> >> and wanted to inform you that we at Google has developed
>>> on-disk/block-level encryption for Innodb, aria (as used by temporary
>>> tables), binlogs and temp-files.
>>> >> The code is not yet published, but we expect it to be within a few
>>> weeks or so.
>>> >> We (of course?) think that it would be better if you instead of
>>> developing new code
>>> >> spent the time testing/reviewing ours.
>>>
>>> We are out course happy to do this!
>>>
>>> >> I'm happy to answer questions on the topic,
>>> >> and will let you know once we've published it.
>>>
>>> The main question I have about the Innodb encryption is if it based on
>>> the compression code we did for fusion-io?
>>> The idea we had on our side was that by using the new compression hooks
>>> we could add encryption with very little changes to the Innodb code.
>>> Looking forward to when you are ready to publish the code so we can
>>> discuss your changes in detail.
>>>
>>> > This is great news!
>>> >
>>> > From what I gather, from Monty's blog post (and a 1:1 we had some time
>>> back), this is something done by a partner/external company that has a
>>> mostly OSS solution, that we should integrate into 10.1
>>>
>>> Yes,  that's correct. It I would have known that Google was working on
>>> encryption I would have included them in my discussions with eperi.
>>> Fortunately it's not yet too late to do this.
>>> I am sure eperi would like to work on the Google code as a base!
>>>
>>> > That said, Google's release of something that works for InnoDB, Aria,
>>> binlogs, temp files (and presumably not too hard to add for MyISAM) is
>>> something we should definitely review and target for 10.1
>>>
>>> Yes!
>>>
>>> Regards,
>>> Monty
>>>
>>>
>>>
>>
>
>