fuel-dev team mailing list archive

[Vladimir Kozhukalov <vkozhukalov@xxxxxxxxxxxx>]

The problem is not about which tool to use for "grepping". The problem is
about what to grep. Elasticsearch can help us to find occurrences, but we
at least need to know what it looks like. Sensitive data can occur in a way
like "not even secret bla bla admin". And we need to know that it is
actually "top secret" information and we need to store those mappings
somewhere. Are we sure that we can support those mappings in actual state?
If we enforce user to have strong passwords, we'll be able to grep out them
using sed, grep, Elasticsearch or something just because something like
"iLuapa4f" will never be the same as something meaningful. We just can get
those passwords from database and filter them out from log files.

I still can not understand how Elasticsearch can help here except
increasing performance.



On Wed, Nov 20, 2013 at 2:10 PM, Bogdan Dobrelya <bdobrelia@xxxxxxxxxxxx>wrote:

>  On 11/20/2013 11:48 AM, Aleksey Kasatkin wrote:
>
>  Bogdan,
>  Is it more time-consuming than simple grep?
>  Is the indexing done in real-time (at network transfer speed) ?
>
>  I believe, we should look deeper into its docs and make a research, I
> have a poor experience with Elasticsearch, the only usecase I was involved
> in was Openstack logs filtering task (Logstash logger -> Elasticsearch
> search engine -> Kibana UI). Elasticsearch perfomance was good for PoC,
> cannot say anything about product loads yet. AFAIK, it does support map
> reduce in real time, as well as indexing and searching RT engines.
>
>
>  Aleksey Kasatkin
>
> S. Software Developer | Mirantis, Inc. | http://www.mirantis.com
> cell: +380938330852 | skype: alexeyk_ru
>
>
> On Wed, Nov 20, 2013 at 10:46 AM, Bogdan Dobrelya <bdobrelia@xxxxxxxxxxxx>wrote:
>
>>  On 11/20/2013 10:32 AM, Vladimir Kozhukalov wrote:
>>
>>  Thank you, Bogdan.
>>
>>  Is "password sanity checks" built-in feature in Elasticsearch? I've not
>> managed to find anything about this feature. What exactly do you mean
>> taking about "password sanity checks"? How can index help if the password
>> looks like "admin" or something like this?
>>
>>  Elasticsearch provides an API
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-search.htmland should be queried by nailgun for "password sanity checks" as well.
>> Almost the same, as simple grepping could do, but much more flexible for
>> big deployments.
>>
>>
>>
>>
>> On Tue, Nov 19, 2013 at 3:04 PM, Bogdan Dobrelya <bdobrelia@xxxxxxxxxxxx>wrote:
>>
>>>   On 11/19/2013 12:18 PM, Vladimir Kozhukalov wrote:
>>>
>>> The issue is that when we make diagnostic snapshot we get files as they
>>> are. Those files like /etc/astute.yaml contain plain text passwords which
>>> are strongly desirable  to be filtered out from wherever they appear.
>>>
>>>  There are two major approaches here.
>>>
>>>  First is to use bare filtering such as sed. We have set of passwords
>>> taken from database and we can find those pieces of plain text throughout
>>> snapshot files and substitute them with something. The problem here is that
>>> passwords can look like "1" or "admin", so we are enforced to filter out
>>> all such occurrences. To avoid this problem we need to check passwords for
>>> their strength. Strong passwords like "Ainei0oh" can be found and
>>> substituted being sure that they are actual passwords and not meaningful
>>> strings.
>>>
>>>  Second, you have data about where and how passwords appear. Those data
>>> are something like set of regular expressions /(foo:\s+)(PASSWORD)(bar)$/
>>> with file names. The problem here is that we need somehow to gather those
>>> data and they eventually could turn out to be invalid so we are likely to
>>> skip one of the occurrences.
>>>
>>>  Let's have a discussion about it and make a decision.
>>>
>>>  --
>>> Vladimir Kozhukalov
>>>
>>>
>>>   I believe we should consider all configuration files in snapshot as
>>> documents and use any document based indexing systems, f.e. Elasticsearch,
>>> to index it for every word inside, and to run *password sanity checks*against it. If none matches was found for password given, we consider it
>>> OK, otherwise, it have to be changed and verified again...
>>>
>>> --
>>> Best regards,
>>> Bogdan Dobrelya,
>>> Researcher TechLead, Mirantis, Inc.+38 (066) 051 07 53
>>> Skype bogdando_at_yahoo.com
>>> 38, Lenina ave.
>>> Kharkov, Ukrainewww.mirantis.comwww.mirantis.rubdobrelia@xxxxxxxxxxxx
>>>
>>>
>>
>>
>>  --
>> Vladimir Kozhukalov
>>
>>
>>
>> --
>> Best regards,
>> Bogdan Dobrelya,
>> Researcher TechLead, Mirantis, Inc.+38 (066) 051 07 53
>> Skype bogdando_at_yahoo.com
>> 38, Lenina ave.
>> Kharkov, Ukrainewww.mirantis.comwww.mirantis.rubdobrelia@xxxxxxxxxxxx
>>
>>
>> --
>> Mailing list: https://launchpad.net/~fuel-dev
>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~fuel-dev
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Best regards,
> Bogdan Dobrelya,
> Researcher TechLead, Mirantis, Inc.
> +38 (066) 051 07 53
> Skype bogdando_at_yahoo.com
> 38, Lenina ave.
> Kharkov, Ukrainewww.mirantis.comwww.mirantis.rubdobrelia@xxxxxxxxxxxx
>
>


-- 
Vladimir Kozhukalov