← Back to team overview

fuel-dev team mailing list archive

Re: filtering out passwords from diagnostic snapshot

 

On 11/20/2013 11:48 AM, Aleksey Kasatkin wrote:
Bogdan,
Is it more time-consuming than simple grep?
Is the indexing done in real-time (at network transfer speed) ?

I believe, we should look deeper into its docs and make a research, I have a poor experience with Elasticsearch, the only usecase I was involved in was Openstack logs filtering task (Logstash logger -> Elasticsearch search engine -> Kibana UI). Elasticsearch perfomance was good for PoC, cannot say anything about product loads yet. AFAIK, it does support map reduce in real time, as well as indexing and searching RT engines.

Aleksey Kasatkin

S. Software Developer | Mirantis, Inc. | http://www.mirantis.com
cell: +380938330852 | skype: alexeyk_ru


On Wed, Nov 20, 2013 at 10:46 AM, Bogdan Dobrelya <bdobrelia@xxxxxxxxxxxx <mailto:bdobrelia@xxxxxxxxxxxx>> wrote:

    On 11/20/2013 10:32 AM, Vladimir Kozhukalov wrote:
    Thank you, Bogdan.

    Is "password sanity checks" built-in feature in Elasticsearch?
    I've not managed to find anything about this feature. What
    exactly do you mean taking about "password sanity checks"? How
    can index help if the password looks like "admin" or something
    like this?
    Elasticsearch provides an API
    http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-search.html
    and should be queried by nailgun for "password sanity checks" as
    well. Almost the same, as simple grepping could do, but much more
    flexible for big deployments.




    On Tue, Nov 19, 2013 at 3:04 PM, Bogdan Dobrelya
    <bdobrelia@xxxxxxxxxxxx <mailto:bdobrelia@xxxxxxxxxxxx>> wrote:

        On 11/19/2013 12:18 PM, Vladimir Kozhukalov wrote:
        The issue is that when we make diagnostic snapshot we get
        files as they are. Those files like /etc/astute.yaml contain
        plain text passwords which are strongly desirable  to be
        filtered out from wherever they appear.

        There are two major approaches here.

        First is to use bare filtering such as sed. We have set of
        passwords taken from database and we can find those pieces
        of plain text throughout snapshot files and substitute them
        with something. The problem here is that passwords can look
        like "1" or "admin", so we are enforced to filter out all
        such occurrences. To avoid this problem we need to check
        passwords for their strength. Strong passwords like
        "Ainei0oh" can be found and substituted being sure that they
        are actual passwords and not meaningful strings.

        Second, you have data about where and how passwords appear.
        Those data are something like set of regular expressions
        /(foo:\s+)(PASSWORD)(bar)$/ with file names. The problem
        here is that we need somehow to gather those data and they
        eventually could turn out to be invalid so we are likely to
        skip one of the occurrences.

        Let's have a discussion about it and make a decision.

-- Vladimir Kozhukalov


        I believe we should consider all configuration files in
        snapshot as documents and use any document based indexing
        systems, f.e. Elasticsearch, to index it for every word
        inside, and to run /password sanity checks/ against it. If
        none matches was found for password given, we consider it OK,
        otherwise, it have to be changed and verified again...

-- Best regards,
        Bogdan Dobrelya,
        Researcher TechLead, Mirantis, Inc.
        +38 (066) 051 07 53  <tel:%2B38%20%28066%29%20051%2007%2053>
        Skypebogdando_at_yahoo.com  <http://bogdando_at_yahoo.com>
        38, Lenina ave.
        Kharkov, Ukraine
        www.mirantis.com  <http://www.mirantis.com>
        www.mirantis.ru  <http://www.mirantis.ru>
        bdobrelia@xxxxxxxxxxxx  <mailto:bdobrelia@xxxxxxxxxxxx>




-- Vladimir Kozhukalov


-- Best regards,
    Bogdan Dobrelya,
    Researcher TechLead, Mirantis, Inc.
    +38 (066) 051 07 53  <tel:%2B38%20%28066%29%20051%2007%2053>
    Skypebogdando_at_yahoo.com  <http://bogdando_at_yahoo.com>
    38, Lenina ave.
    Kharkov, Ukraine
    www.mirantis.com  <http://www.mirantis.com>
    www.mirantis.ru  <http://www.mirantis.ru>
    bdobrelia@xxxxxxxxxxxx  <mailto:bdobrelia@xxxxxxxxxxxx>


    --
    Mailing list: https://launchpad.net/~fuel-dev
    <https://launchpad.net/%7Efuel-dev>
    Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
    <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
    Unsubscribe : https://launchpad.net/~fuel-dev
    <https://launchpad.net/%7Efuel-dev>
    More help   : https://help.launchpad.net/ListHelp




--
Best regards,
Bogdan Dobrelya,
Researcher TechLead, Mirantis, Inc.
+38 (066) 051 07 53
Skype bogdando_at_yahoo.com
38, Lenina ave.
Kharkov, Ukraine
www.mirantis.com
www.mirantis.ru
bdobrelia@xxxxxxxxxxxx


Follow ups

References