openstack team mailing list archive

[Steven Hardy <shardy@xxxxxxxxxx>]

On Wed, Jul 03, 2013 at 04:21:38PM +0200, Michaël Van de Borne wrote:
> Hello Steven,
> I'm mikemowgli from IRC. As requested, here are the logs.
> 
> 
> 1. First, here's a stack trace I*get in my shell periodically (once
> per minute approximately), but not in the logs: *
> http://pastebin.com/kPswnGNL
> (this might not be related to cloudwatch as I got this permanently)

This is due to https://bugs.launchpad.net/heat/+bug/1196618

Which is now fixed in heat master, and it is unrelated to the heat-watch
authentication failure

> 2. Then, here is the error I get when I perform a heat-watch
> command. The logs of engine and cloudwatch are in attachment. In
> order to minimize their size, I launched and killed the daemons for
> this single heat-watch command.
> 
> It seems that my AWS creds are accepted, but that the user does have
> enough permissions. However, in keystone, the heat user is admin of
> the service tenant. The config files of engine, cloudwatch and boto
> (2.9.0) are also in attachment.

As confirmed on IRC (repeated here for the benefit of others), the reason
is that your user associated with the ec2 keypair has the "heat_stack_user"
role, which is intended to be reserved for users created as part of a heat
stack (where the credentials are deployed inside an implicitly untrusted
instance).

The clues are (from api-cloudwatch.log)

2013-07-03 16:12:13.358 27398 INFO heat.api.aws.ec2token [-] AWS
authentication successful.

which proves the ec2 keypair and resulting signed request via boto is OK

Then we get "User is not authorized to perform action" response, but with
no corresponding log message in the engine.log - so it's the policy.json
rules which limit access to in-instance "heat_stack_user"s which is
preventing access, here's the rule in the /etc/heat/policy.json:

    "deny_stack_user": "not role:heat_stack_user",

    ...

    "cloudwatch:DeleteAlarms": "rule:deny_stack_user",
    "cloudwatch:DescribeAlarmHistory": "rule:deny_stack_user",
    "cloudwatch:DescribeAlarms": "rule:deny_stack_user",
    "cloudwatch:DescribeAlarmsForMetric": "rule:deny_stack_user",
    "cloudwatch:DisableAlarmActions": "rule:deny_stack_user",
    "cloudwatch:EnableAlarmActions": "rule:deny_stack_user",
    "cloudwatch:GetMetricStatistics": "rule:deny_stack_user",
    "cloudwatch:ListMetrics": "rule:deny_stack_user",
    "cloudwatch:PutMetricAlarm": "rule:deny_stack_user",
    "cloudwatch:PutMetricData": "",
    "cloudwatch:SetAlarmState": "rule:deny_stack_user"

As you can see, the policy only allows users with the heat_stack_user role
to access the PutMetricData action, which is the only action required by
the in-instance user (for the cfn-push-stats agent)

So no "real" users should ever have the "heat_stack_user" role, it should
be reserved for use inside heat.

Steve