ugh... I had an example file called "nova.conf" in /root
Apparently nova-manage looks for that file first. Case closed. I
wish I could get the last week back.
Sam
On Mon, Mar 11, 2013 at 2:37 PM, Miller, Mark M (EB SW Cloud - R&D -
Corvallis) <mark.m.miller@xxxxxx <mailto:mark.m.miller@xxxxxx>> wrote:
I ran into a similar problem with the Grizzly-3 Keystone release.
I had to specify keystone.conf as the config-file with
keystone-manage/ db_sync command otherwise it would not use the
mysql statement in the keystone.conf file.
Mark
*From:*openstack-bounces+mark.m.miller=hp.com@xxxxxxxxxxxxxxxxxxx
<mailto:hp.com@xxxxxxxxxxxxxxxxxxx>
[mailto:openstack-bounces+mark.m.miller
<mailto:openstack-bounces%2Bmark.m.miller>=hp.com@xxxxxxxxxxxxxxxxxxx
<mailto:hp.com@xxxxxxxxxxxxxxxxxxx>] *On Behalf Of *Samuel Winchenbach
*Sent:* Monday, March 11, 2013 11:18 AM
*To:* Sylvain Bauza
*Cc:* openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
*Subject:* Re: [Openstack] Incredibly odd mysql permission error
OK!!!! Someone on the IRC channel got me closer, but we have no
idea why this would happen:
this works: "root@test1:~# nova-manage
--config-file=/etc/nova/nova.conf service list"
Why would I have to specify the config file though? It is in the
standard place.
Thanks,
Sam
On Mon, Mar 11, 2013 at 2:01 PM, Samuel Winchenbach
<swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>> wrote:
For completeness here the routing table, and ip listing for both
test1 and test2. Doubt this will help much:
http://paste2.org/p/3117125
On Mon, Mar 11, 2013 at 1:52 PM, Samuel Winchenbach
<swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>> wrote:
#1 - No change
#2 - All of grants are in the ip/mask form such as:
'nova'@'10.21.0.0/255.255.0.0 <http://10.21.0.0/255.255.0.0>' I
have also tried adding 'nova'@'test1' and 'nova'@'10.21.0.1'. No
change.
#3 - I changed the SQL connection string over to IP instead of
hostname. No change. I didn't restart nova-api because it isn't
running. If I understand correctly nova-manage communicated
directly with the db, bypassing nova-api. This would appear true
seeing "nova-manage service list" works correctly on test2.
:(
Thanks for the help!
Sam
On Mon, Mar 11, 2013 at 12:24 PM, Sylvain Bauza
<sylvain.bauza@xxxxxxxxxxxx <mailto:sylvain.bauza@xxxxxxxxxxxx>>
wrote:
When looking at MySQL 5.1 refman
(http://dev.mysql.com/doc/refman/5.1/en/access-denied.html ), I
would suggest to follow the procedure :
1. 'mysqladmin flush-hosts'
2. replace DNS entries in mysql.user table by IP addresses instead
3. modify /etc/nova/nova.conf with IP address of HA Mysql instead
(and restart nova-api !)
I wouldn't bet on it, but I would say this is due to some name
resolution which is incorrect.
-Sylvain
Le 11/03/2013 17:00, Sylvain Bauza a écrit :
Ok, lemme try to summarize.
You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1
thanks to Pacemaker.
This setup is relying on two hosts, test1 (10.21.0.1) and
test2 (10.21.0.2).
Your nova.conf is pointing to mysql://10.21.1.1
<http://10.21.1.1> which is the VIP.
Are you sure your my.cnf is actually the same in between both
DRBD nodes ? (I would recommend to symlink it to a physical
file hosted on the DRBD device).
One thing is hurting me : you told me that nova is also
pacemake'd. If so, why can I still see my_ip=10.21.0.2 (test2)
? It should be pointing to nova-ha (assuming 10.21.2.4 as per
/etc/hosts).
Also, as per my understanding of Pacemaker, DRBD partition is
setup by default on test2, correct ?
Sorry, as per my first reading, I can't see anything obvious.
That said, I'm not sure this is a Nova bug, as the tcpdump
trace is seeing a correct MySQL connection attempt. But maybe
I'm wrong ?
Anyway, are you sure you only have *one* MySQL engine running
(either on test1 or test2) and nova-manage trying to access
this right one ?
Perms look good to me. As it a test setup, you could try to
unleash the grants by deleting them and allowing nova@'%' to
see if it's a basic dns mapping issue.
-Sylvain
Le 11/03/2013 16:09, Samuel Winchenbach a écrit :
I
enabled general_log in /etc/mysql/my.cnf Here are the
results of connecting from "test1", "test2" and using the
client:
http://paste2.org/p/3115525
I purposefully used the real password in case there is a
problem with it.
I changed before submitting post.
here is a raw packet TCP dump (tcpdump -w rawdump port
3306) of an attempted "nova-manage service list" from test1:
https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump
I looked at it with wireshark and couldn't see anything
that jumped out at me as incorrect. I have not yet tried
to recreate the salted password.
Here is my pacemaker configuration for mysql. I stripped
out openstack services, rabbitmq and others for clarity.
All resources are currently disabled (other than MySQL):
http://paste2.org/p/3115685
Please don't yell at me for having STONITH disabled :P
This is a testing cluster and I am working on getting
routed to the IPMI interface.
/etc/hosts:
http://paste2.org/p/3115713
/etc/nova/nova.conf:
http://paste2.org/p/3115739
If there is anything else I can provide you, please let me
know! I have pulled out most of my hair at this point!
Sam
On Mon, Mar 11, 2013 at 10:11 AM, Sylvain Bauza
<sylvain.bauza@xxxxxxxxxxxx
<mailto:sylvain.bauza@xxxxxxxxxxxx>> wrote:
So as to reproduce the nova-manage SQL command, I would
recommand to tcpdump -A port 3306 on the host and get the
SQL trace on what's failing.
Could you please explain further what is your HA config ?
Are you using pacemaker/heartbeat or any VIP ?
-Sylvain
Le 11/03/2013 14:23, Samuel Winchenbach a écrit :
Does anyone think this could be an openstack bug? I
just want to check before submitting a bug report.
Sam
On Fri, Mar 8, 2013 at 4:02 PM, Jay Pipes
<jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>> wrote:
Sorry, I really can't think of anything :(
On 03/08/2013 03:52 PM, Samuel Winchenbach wrote:
> I dropped those users and no change.
>
> I also set up general logging in mysql but it really
doesn't provide any
> additional information. Any idea for a next step I
could take?
>
> I am almost at the point of taking a tcpdump and
trying to recreate the
> salted password. :/
>
> Thanks for the help
>
> Sam
>
>
>
>
> On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes
<jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
> <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx>>> wrote:
>
> I'm stumped :( Looks like everything is set up
correctly to me. What is
> interested is that your nova user access works from
test2, but there is
> no nova@test2 user in the mysql.user table. What
about doing a DROP USER
> nova@test1; FLUSH PRIVILEGES; and then see if that
fixes things... since
> the nova@10.21.0.0/255.255.0.0
<http://nova@10.21.0.0/255.255.0.0>
<http://nova@10.21.0.0/255.255.0.0>
> user is clearly working for the access
> from test2.
>
> Also, I'd recommend highly removing the nova@% user.
>
> Best,
> -jay
>
> On 03/08/2013 03:09 PM, Samuel Winchenbach wrote:
> >
> > http://paste2.org/p/3085807
> >
> >
> > On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes
<jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
> <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>
> > <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx>>>> wrote:
> >
> > Please paste the results of SELECT User, Host,
Password FROM
> mysql.user
> > when running as root...
> >
> > Thanks!
> > -jay
> >
> > On 03/08/2013 02:25 PM, Samuel Winchenbach
wrote:
> > > Here are my grants. I don't know if
this helps, but I did
> verify that
> > > the password was identical for each grant:
> > http://paste2.org/p/3085361
> > >
> > >
> > > On Fri, Mar 8, 2013 at 2:17 PM, Samuel
Winchenbach
> > <swinchen@xxxxxxxxx
<mailto:swinchen@xxxxxxxxx> <mailto:swinchen@xxxxxxxxx
<mailto:swinchen@xxxxxxxxx>>
> <mailto:swinchen@xxxxxxxxx
<mailto:swinchen@xxxxxxxxx> <mailto:swinchen@xxxxxxxxx
<mailto:swinchen@xxxxxxxxx>>>
> > > <mailto:swinchen@xxxxxxxxx
<mailto:swinchen@xxxxxxxxx> <mailto:swinchen@xxxxxxxxx
<mailto:swinchen@xxxxxxxxx>>
> <mailto:swinchen@xxxxxxxxx
<mailto:swinchen@xxxxxxxxx> <mailto:swinchen@xxxxxxxxx
<mailto:swinchen@xxxxxxxxx>>>>> wrote:
> > >
> > > root@test1:/var/log# mysql -hmysql-ha -unova
> > > -p********************************
-e"SELECT User, Host,
> Password
> > > FROM mysql.user;"
> > > ERROR 1142 (42000) at line 1: SELECT
command denied to user
> > > 'nova'@'test1' for table 'user'
> > >
> > >
> > > On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes
> <jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
<mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>
> > <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx>>>
> > > <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx>>
> <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx>>>>> wrote:
> > >
> > > What does this show?
> > >
> > > mysql -hmysql-ha -unova -p<PASS>
-e"SELECT User, Host,
> > Password FROM
> > > mysql.user"
> > >
> > > -jay
> > >
> > > On 03/08/2013 01:46 PM, Samuel
Winchenbach wrote:
> > > > Sorry, that must have been a copy
and paste error.
> Here
> > is what I
> > > > actually ran:
> > > >
> > > > http://paste2.org/p/3084996
> > > >
> > > >
> > > > On Fri, Mar 8, 2013 at 12:40 PM, Jay
Pipes
> > <jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx>>
> <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx>>>
> > > <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx>
> <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx>>
<mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
> <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx>>>>
> > > > <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx>
> <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx>>
<mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
> <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>>
> > <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx>>
> <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
<mailto:jaypipes@xxxxxxxxx>>>>>> wrote:
> > > >
> > > > On 03/08/2013 12:19 PM, Samuel
Winchenbach wrote:
> > > > > Hi All,
> > > > >
> > > > > I have two nodes (test1 and test2)
that I am
> trying to
> > > set up in a
> > > > > highly available configuration.
> > > > >
> > > > > During the setup process I tried
running
> "nova-manage
> > > service list" on
> > > > > both nodes. It worked fine on
test2, but
> fails on
> > > test1 even
> > > > though I
> > > > > can connect to the database with
the mysql
> client from
> > > test1.
> > > > >
> > > > > Here is a screen capture that
shows the setup on
> > the two
> > > nodes are
> > > > > basically identical:
> http://paste2.org/p/3084223
> > > >
> > > > In the above paste you are doing:
> > > >
> > > > mysql -unova - hmysql-ha -u root nova
> > > > -p********************************
> > > >
> > > > Note you are supplying 2 -u
arguments, and mysql
> > will take
> > > the second
> > > > (root).
> > > >
> > > > -jay
> > > >
> > > >
_______________________________________________
> > > > Mailing list:
https://launchpad.net/~openstack
<https://launchpad.net/%7Eopenstack>
> > > > Post to :
openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> <mailto:openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
> > <mailto:openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> <mailto:openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>
> > > <mailto:openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> <mailto:openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
> > <mailto:openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> <mailto:openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>>
> > > >
<mailto:openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> <mailto:openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
> > <mailto:openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> <mailto:openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>
> > > <mailto:openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> <mailto:openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
> > <mailto:openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> <mailto:openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>>>
> > > > Unsubscribe :
https://launchpad.net/~openstack
<https://launchpad.net/%7Eopenstack>
> > > > More help :
https://help.launchpad.net/ListHelp
> > > >
> > > >
> > >
> > >
> > >
> >
> >
>
>
_______________________________________________
Mailing list:https://launchpad.net/~openstack <https://launchpad.net/%7Eopenstack>
Post to :openstack@xxxxxxxxxxxxxxxxxxx <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
Unsubscribe :https://launchpad.net/~openstack <https://launchpad.net/%7Eopenstack>
More help :https://help.launchpad.net/ListHelp
_______________________________________________
Mailing list: https://launchpad.net/~openstack
<https://launchpad.net/%7Eopenstack>
Post to : openstack@xxxxxxxxxxxxxxxxxxx
<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
Unsubscribe : https://launchpad.net/~openstack
<https://launchpad.net/%7Eopenstack>
More help : https://help.launchpad.net/ListHelp
