fuel-dev team mailing list archive

[Andrey Danin <adanin@xxxxxxxxxxxx>]

Also we thought about some kind of back-connected shell which allows
customer to grant us full access to the master node instantly.


On Thu, Feb 27, 2014 at 8:28 PM, Sergey Yudin <syudin@xxxxxxxxxxxx> wrote:

> 2 Pavel Vaylov:
> >Sergey, as I informed we have a solution for it, Andrey Ryabinin can
> confirm (we bought subscription to some file sharing service). But the
> solution is require to many manual steps from the Operation engineer.
> Could you please clarify about that file sharing service? Is it suitable
> for big files and private data? Maybe any documentation?
>
> 2 Matthew Mosesohn:
> >Any log/snapshot submission system (like anonymous FTP upload) should accept
> anonymous uploads (but not anonymous read/list) for user ease of use.
> +1 for ftp, simple enoguht and suitable for our purpose. If there is no
> solution exist yet, lets use ftp.
>
> >I believe anything extra beyond our current logging implementation should
> involve manual command workarounds for data gathering.
> Agreed, manual actions may be required, but even in that case i'm not sure
> if it's desirable to involve customer into this process. It will be
> easier if our manual workarounds will be shipped to the customer in a black
> box, and will be executed as "some fuel diagnostic tool".
>
> And even if we will not take in according any custom manual workarounds we
> still have to upgrade troubleshooting tool to be able to support customers
> with old installation.
>
> 2 Vladimir Kozhukalov:
> >Besides, our current solution has some problems with the ability to
> gather diagnostic data from a number of nodes in parallel. At the moment it
> is just one thread which runs one command on one remote node at each moment.
> I've solved this issue in mine tool.
>
> >We also thought a bit about how to filter out sensitive data from
> diagnostic snapshot
> Well, it's countable number of objects which is sensetive for private
> information, it only openstack configs in our case, i suppose. It is
> possible to consider all of them, parse and cut passwords. It is not
> implemented, but it seems possible.
>
>
> >Anyway, please describe as much clearly as you can how you feel you can
> integrate parts of your troubleshooting tool into Fuel diagnostic snapshot
> tool.
> It is just standalone python script right now, which is designed to be run
> against fuel environment and generate single tarball with report. I'm not
> familiar with fuel architerture enought to propose any solution on how to
> integrate it into fuel, but it seems not so hard to be able via fuel UI
> upload this tool for update, run it, and provide reports.
>
>
> On Thu, Feb 27, 2014 at 6:01 PM, Vladimir Kozhukalov <
> vkozhukalov@xxxxxxxxxxxx> wrote:
>
>> Sergey,
>>
>> Great initiative. If you have thoughts about how we can improve Fuel
>> "diagnostic snapshot" feature, let's discuss it in our maling list
>> fuel-dev@xxxxxxxxxxxxxxxxxxx or on our weekly IRC meeting
>> https://wiki.openstack.org/wiki/Meetings/Fuel.
>>
>> Adding some commands to save their output is simple. One can add whatever
>> they need into /etc/nailgun/settings.yaml file. Upgrading this tool on the
>> fly is much more complicated improvement. Besides, our current solution has
>> some problems with the ability to gather diagnostic data from a number of
>> nodes in parallel. At the moment it is just one thread which runs one
>> command on one remote node at each moment.
>>
>> We also thought a bit about how to filter out sensitive data from
>> diagnostic snapshot, but we've not managed to find a suitable solution. At
>> the moment all passwords are stored in a database as a plain text. So we
>> tried to find all passwords wherever they appear and substitute them with a
>> meaningless string. Because passwords maybe something like "1" or "admin",
>> it is not so simple to filter them out without breaking other useful data.
>>
>> We also discussed the initiative which is about create shell wrapper for
>> arbitrary command which would be able to save stdout and stderr into files
>> or publish them on a paste service.
>>
>> Anyway, please describe as much clearly as you can how you feel you can
>> integrate parts of your troubleshooting tool into Fuel diagnostic snapshot
>> tool.
>>
>>
>> Vladimir Kozhukalov
>>
>>
>> On Thu, Feb 27, 2014 at 4:43 PM, Matthew Mosesohn <mmosesohn@xxxxxxxxxxxx
>> > wrote:
>>
>>> Any log/snapshot submission system (like anonymous FTP upload) should
>>> accept anonymous uploads (but not anonymous read/list) for user ease
>>> of use.
>>>
>>> I believe anything extra beyond our current logging implementation
>>> should involve manual command workarounds for data gathering. For
>>> example, debugging glance-api with logs redirected for 5 minutes:
>>> service openstack-glance-api stop
>>> mkdir /root/glance-capture-logs
>>> timeout 5m glance-api -v -d --log-file
>>> /root/glance-capture-logs/glance-api.log
>>> service openstack-glance-api start
>>>
>>> Then collector will pull all files out of /root, for example..
>>>
>>>
>>>
>>> On Thu, Feb 27, 2014 at 4:36 PM, Pavel Vaylov <pvaylov@xxxxxxxxxxxx>
>>> wrote:
>>> > 1. Absolutely need a way to upgrade/fix log collection process, not
>>> only
>>> > snapshot tool. For example this
>>> https://bugs.launchpad.net/fuel/+bug/1284867
>>> > and probably new issues with logging.
>>> >
>>> > 2. Sergey, as I informed we have a solution for it, Andrey Ryabinin can
>>> > confirm (we bought subscription to some file sharing service). But the
>>> > solution is require to many manual steps from the Operation engineer.
>>> >
>>> > 3. Will test the tool on my Lab.
>>> >
>>> >
>>> >
>>> >
>>> > On Thu, Feb 27, 2014 at 2:19 PM, Sergey Yudin <syudin@xxxxxxxxxxxx>
>>> wrote:
>>> >>
>>> >> Hi, guys.
>>> >> Being part of TET team, i noticed that most difficult and long part of
>>> >> troubleshotting is getting debug information from the customer, and i
>>> want
>>> >> improve this.
>>> >>
>>> >> I wanna discuss fuel snapshot tool, and related blueprint on
>>> launchpad.
>>> >>
>>> https://blueprints.launchpad.net/fuel/+spec/fuel-snapshot-additional-info
>>> >>
>>> >>
>>> >> From my point of view the tools not only has lack of information, as
>>> >> mentioned in the blueprint, but also has lack of some base features
>>> which
>>> >> will make it usefull.
>>> >>
>>> >> 1. We need ability to upgrade this tool on fly, so if customer will
>>> face
>>> >> some issue support team will be able to send updated snapshot tool,
>>> or even
>>> >> fix exist tool for getting required information, and not rely on tool
>>> which
>>> >> was written years ago.
>>> >>
>>> >> 2. We need to think of storage for snapshots, which may be huge, so
>>> will
>>> >> not fit onto email, and also may contain private information, so will
>>> not
>>> >> fit for public storage.
>>> >>
>>> >> 3. There is tool written by me for gathering information from customer
>>> >> installation,
>>> >>
>>> >>
>>> https://docs.google.com/a/mirantis.com/document/d/1KyVuJcKy6lln4rI1d2kEZS63NU1ykswz0ib4MoM5bog/edit#
>>> >> https://github.com/tsipa/troubleshooting
>>> >>  maybe we should use it. I'm agree to participate in integration it
>>> into
>>> >> fuel snapshot tool.
>>> >>
>>> >> Maybe i missed something and guys from support team has something
>>> more to
>>> >> say.
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Best regards
>>> >
>>> > Pavel Vaylov
>>> > Operations Engineer
>>> > Mirantis Service Desk
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> Groups
>>> > "fuel-core-team" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> an
>>> > email to fuel-core-team+unsubscribe@xxxxxxxxxxxx.
>>> > For more options, visit
>>> > https://groups.google.com/a/mirantis.com/groups/opt_out.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "fuel-core-team" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to fuel-core-team+unsubscribe@xxxxxxxxxxxx.
>>> For more options, visit
>>> https://groups.google.com/a/mirantis.com/groups/opt_out.
>>>
>>
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "fuel-core-team" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to fuel-core-team+unsubscribe@xxxxxxxxxxxx.
> For more options, visit
> https://groups.google.com/a/mirantis.com/groups/opt_out.
>



-- 
Andrey Danin
adanin@xxxxxxxxxxxx
skype: gcon.monolake