maria-developers team mailing list archive

[Roberto Spadim <roberto@xxxxxxxxxxxxx>]

2014-06-17 14:03 GMT-03:00 Jonas Oreland <jonaso@xxxxxxxxxx>:

> Hi again,
>
> 1) we have not done column level encryption at all.
>
nice, did you check that have an idea at mariadb atlassian mdevs, maybe
this could help?
https://mariadb.atlassian.net/browse/MDEV-4912


>
> 2) keys are managed in a separate module (which can be overridden by
> plugin, which we do for testing)
> 3) keys have 32 bit version
> 4) different key versions can exists simultaneously in database
>
nice


>
> 5) for innodb we encrypt both datafiles and logfiles
> 6) encryption of innodb datafiles works roughly like this:
> - pages are encrypted with aes ctr
> - page 0 of each encrypted datafile contains IV
> - all pages except page 0 is encrypted
> - if compression is used, pages are encrypted after compression
> - pages are encrypted/decrypted by calls added to innodb-buffer-pool module
> - encrypted pages are "tagged" with which key-version encrypted it
> - different pages in database can be encrypted with different key version
> - when key-server module reports that a new key version is available, a
> background task will re-encrypt database with new key-version (key rotation)
> - key rotation is performed with configurable number of threads, that will
> perform configurable amount of IOPS.
> one can also configure how frequently pages shall be key-rotated (i.e max
> key "age")
>

nice something like a i/o wrapper between filesystem calls and innodb,
right?




>
> /Jonas
>
>
>
> On Tue, Jun 17, 2014 at 6:40 PM, Elmar Eperiesi-Beck <
> elmar@xxxxxxxxxxxxxxxx> wrote:
>
>> Hi,
>> I agree with you. If we want to know, what Google has developed as
>> encryption feature, we will have to wait for your source code to be
>> published.
>>
>> In the meantime, you can find our concept for the encryption on our
>> website: http://bit.ly/1slJyuI
>> Feedback (negative and positive) from all of you is welcome - and needed!
>>
>> Best Regards
>> Elmar
>>
>>
>> Am 17.06.2014 um 12:50 schrieb Jonas Oreland <jonaso@xxxxxxxxxx>:
>>
>> Hi again,
>>
>> > by "interfaces" I was looking for the Maria DB place/ function /
>> hook... where you are enhancing the MariaDB Code.
>>
>> I'm not sure how to convey this in a digestible form, attaching diffstats
>> below. Not sure if it's helps :-(
>>
>> There are many aspects of it.
>> And each of the sub-projects (innodb data, innodb log, maria, tempfiles,
>> binlog) has "interesting" details.
>>
>> /Jonas
>>
>>
>> storage/innodb has this diffstat:
>>  CMakeLists.txt       |    2
>>  btr/btr0cur.cc       |    9
>>  buf/buf0buf.cc       |  213 +++++
>>  buf/buf0checksum.cc  |    8
>>  buf/buf0dblwr.cc     |   40 -
>>  buf/buf0flu.cc       |    6
>>  buf/buf0rea.cc       |    7
>>  dict/dict0load.cc    |    8
>>  fil/fil0crypt.cc     | 1986
>> +++++++++++++++++++++++++++++++++++++++++++++++++++
>>  fil/fil0fil.cc       |  280 ++++++-
>>  fsp/fsp0fsp.cc       |   36
>>  handler/ha_innodb.cc |  110 ++
>>  handler/i_s.cc       |  292 +++++++
>>  handler/i_s.h        |    1
>>  include/buf0buf.h    |   60 +
>>  include/buf0buf.ic   |   29
>>  include/fil0fil.h    |  266 ++++++
>>  include/fsp0fsp.h    |    9
>>  include/log0crypt.h  |   85 ++
>>  include/log0log.h    |   21
>>  include/log0recv.h   |    5
>>  include/mtr0log.ic   |    2
>>  include/mtr0mtr.h    |    8
>>  include/srv0srv.h    |    8
>>  log/log0crypt.cc     |  256 ++++++
>>  log/log0log.cc       |   93 ++
>>  log/log0recv.cc      |   35
>>  mtr/mtr0log.cc       |    4
>>  row/row0import.cc    |    3
>>  srv/srv0srv.cc       |   14
>>  srv/srv0start.cc     |   29
>>  31 files changed, 3853 insertions(+), 72 deletions(-)
>>
>> storage/maria has this diffstat:
>>  CMakeLists.txt                            |   12
>>  ha_maria.cc                               |   12
>>  ma_bitmap.c                               |   63 ++--
>>  ma_blockrec.c                             |  222 ++++++++------
>>  ma_blockrec.h                             |   26 +
>>  ma_check.c                                |   49 +--
>>  ma_checkpoint.c                           |    4
>>  ma_close.c                                |    2
>>  ma_create.c                               |   56 +++
>>  ma_crypt.c                                |  464
>> ++++++++++++++++++++++++++++++
>>  ma_crypt.h                                |   26 +
>>  ma_delete.c                               |    2
>>  ma_key_recover.c                          |   10
>>  ma_loghandler.c                           |   63 +---
>>  ma_open.c                                 |   48 ++-
>>  ma_pagecache.c                            |  154 ++++++---
>>  ma_pagecache.h                            |   34 +-
>>  ma_pagecrc.c                              |  118 ++++---
>>  ma_static.c                               |    1
>>  ma_write.c                                |   24 -
>>  maria_def.h                               |   81 ++---
>>  unittest/ma_pagecache_consist.c           |   28 -
>>  unittest/ma_pagecache_rwconsist.c         |   27 -
>>  unittest/ma_pagecache_rwconsist2.c        |   27 -
>>  unittest/ma_pagecache_single.c            |   27 -
>>  unittest/ma_test_loghandler_pagecache-t.c |   29 -
>>  26 files changed, 1102 insertions(+), 507 deletions(-)
>>
>> A noticeable difference between innodb and maria is that we didn't
>> implement encryption of the log for maria,
>> as we only added support for temporary tables. For maria we also only
>> added encryption support for BLOCK format
>> but added all the features to this format so that it was usable for all
>> temp-table scenarios. maria also doesn't have
>> key-rotation feature like innodb has.
>>
>> I couldn't (as) easily extract diffstats for binlog and tempfile
>> encryption.
>> You have to wait for the code to get published...
>>
>>
>> On Tue, Jun 17, 2014 at 7:29 AM, Elmar Eperiesi-Beck <
>> elmar@xxxxxxxxxxxxxxxx> wrote:
>>
>>> Hi,
>>> by "interfaces" I was looking for the Maria DB place/ function / hook...
>>> where you are enhancing the MariaDB Code.
>>> This would help me to understand what you are trying to do.
>>>
>>> Elmar
>>>
>>> Am 17.06.2014 um 07:02 schrieb Jonas Oreland <jonaso@xxxxxxxxxx>:
>>>
>>> Hi again,
>>>
>>> > What is the type of license of your code?
>>>
>>> I asked internally about license, and it seems like we releasing dual
>>> gpl2/apache licensed code.
>>>
>>> > I would like to know, which interfaces from maria-DB you are using.
>>>
>>> I don't 100% understand the question.
>>> We didn't write any actual encryption code, but used the one provided in
>>> openssl.
>>> Other than that, we didn't really "use interfaces", but rather
>>> added/modified functionality/interfaces here and there.
>>>
>>> Can you be more specific ?
>>>
>>> /Jonas
>>>
>>>
>>>
>>> On Sat, Jun 7, 2014 at 11:20 PM, Elmar Eperiesi-Beck <
>>> elmar@xxxxxxxxxxxxxxxx> wrote:
>>>
>>>> Hi!
>>>> We (eperi) would be glad to do a joined work with Google.
>>>> Our solution works with MS-SQL, Oracle and other DBs and we are
>>>> currently porting it to MariaDB - and - as Monty said - its never to late
>>>> to put some sources together and make the best for the open source
>>>> community.
>>>>
>>>> What is the type of license of your code?
>>>>
>>>> Jonas, I am looking forward to connect to you directly.
>>>>
>>>> Regards
>>>> Elmar
>>>>
>>>> Hi!
>>>>
>>>> > Hi Jonas,
>>>> > (same Jonas we know from NDBCLUSTER? :-) Good to see you again)
>>>> >
>>>> > On 6 Jun 2014, at 02:31, Jonas Oreland <jonaso@xxxxxxxxxx> wrote:
>>>> >
>>>> >> Hi there,
>>>> >> I read this blog post
>>>> >>
>>>> http://monty-says.blogspot.com/2014/05/for-your-eyes-only-or-adding-better.html
>>>> >> and wanted to inform you that we at Google has developed
>>>> on-disk/block-level encryption for Innodb, aria (as used by temporary
>>>> tables), binlogs and temp-files.
>>>> >> The code is not yet published, but we expect it to be within a few
>>>> weeks or so.
>>>> >> We (of course?) think that it would be better if you instead of
>>>> developing new code
>>>> >> spent the time testing/reviewing ours.
>>>>
>>>> We are out course happy to do this!
>>>>
>>>> >> I'm happy to answer questions on the topic,
>>>> >> and will let you know once we've published it.
>>>>
>>>> The main question I have about the Innodb encryption is if it based on
>>>> the compression code we did for fusion-io?
>>>> The idea we had on our side was that by using the new compression hooks
>>>> we could add encryption with very little changes to the Innodb code.
>>>> Looking forward to when you are ready to publish the code so we can
>>>> discuss your changes in detail.
>>>>
>>>> > This is great news!
>>>> >
>>>> > From what I gather, from Monty's blog post (and a 1:1 we had some
>>>> time back), this is something done by a partner/external company that has a
>>>> mostly OSS solution, that we should integrate into 10.1
>>>>
>>>> Yes,  that's correct. It I would have known that Google was working on
>>>> encryption I would have included them in my discussions with eperi.
>>>> Fortunately it's not yet too late to do this.
>>>> I am sure eperi would like to work on the Google code as a base!
>>>>
>>>> > That said, Google's release of something that works for InnoDB, Aria,
>>>> binlogs, temp files (and presumably not too hard to add for MyISAM) is
>>>> something we should definitely review and target for 10.1
>>>>
>>>> Yes!
>>>>
>>>> Regards,
>>>> Monty
>>>>
>>>>
>>>>
>>>
>>
>>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~maria-developers
> Post to     : maria-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~maria-developers
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Roberto Spadim
SPAEmpresarial
Eng. Automação e Controle