← Back to team overview

openstack team mailing list archive

  1. openstack team
  2. Mailing list archive
  3. Message #21770

Re: Incredibly odd mysql permission error

  Thread Previous • Date Previous • • Thread Next 
I ran into a similar problem with the Grizzly-3 Keystone release. I had to specify keystone.conf as the config-file with keystone-manage/ db_sync command otherwise it would not use the mysql statement in the keystone.conf file.

Mark

From: openstack-bounces+mark.m.miller=hp.com@xxxxxxxxxxxxxxxxxxx [mailto:openstack-bounces+mark.m.miller=hp.com@xxxxxxxxxxxxxxxxxxx] On Behalf Of Samuel Winchenbach
Sent: Monday, March 11, 2013 11:18 AM
To: Sylvain Bauza
Cc: openstack@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Openstack] Incredibly odd mysql permission error

OK!!!!  Someone on the IRC channel got me closer, but we have no idea why this would happen:

this works: "root@test1:~# nova-manage --config-file=/etc/nova/nova.conf service list"

Why would I have to specify the config file though?  It is in the standard place.

Thanks,
Sam



On Mon, Mar 11, 2013 at 2:01 PM, Samuel Winchenbach <swinchen@xxxxxxxxx<mailto:swinchen@xxxxxxxxx>> wrote:
For completeness here the routing table, and ip listing for both test1 and test2.   Doubt this will help much:  http://paste2.org/p/3117125

On Mon, Mar 11, 2013 at 1:52 PM, Samuel Winchenbach <swinchen@xxxxxxxxx<mailto:swinchen@xxxxxxxxx>> wrote:
#1 - No change
#2 - All of grants are in the ip/mask form such as: 'nova'@'10.21.0.0/255.255.0.0<http://10.21.0.0/255.255.0.0>'  I have also tried adding 'nova'@'test1' and 'nova'@'10.21.0.1'.  No change.
#3 - I changed the SQL connection string over to IP instead of hostname.  No change.  I didn't restart nova-api because it isn't running.   If I understand correctly nova-manage communicated directly with the db, bypassing nova-api.   This would appear true seeing "nova-manage service list" works correctly on test2.


:(

Thanks for the help!
Sam

On Mon, Mar 11, 2013 at 12:24 PM, Sylvain Bauza <sylvain.bauza@xxxxxxxxxxxx<mailto:sylvain.bauza@xxxxxxxxxxxx>> wrote:
When looking at MySQL 5.1 refman (http://dev.mysql.com/doc/refman/5.1/en/access-denied.html ), I would suggest to follow the procedure :
 1. 'mysqladmin flush-hosts'
 2. replace DNS entries in mysql.user table by IP addresses instead
 3. modify /etc/nova/nova.conf with IP address of HA Mysql instead (and restart nova-api !)

I wouldn't bet on it, but I would say this is due to some name resolution which is incorrect.

-Sylvain


Le 11/03/2013 17:00, Sylvain Bauza a écrit :
Ok, lemme try to summarize.
You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1 thanks to Pacemaker.
This setup is relying on two hosts, test1 (10.21.0.1) and test2 (10.21.0.2).
Your nova.conf is pointing to mysql://10.21.1.1<http://10.21.1.1> which is the VIP.

Are you sure your my.cnf is actually the same in between both DRBD nodes ? (I would recommend to symlink it to a physical file hosted on the DRBD device).

One thing is hurting me : you told me that nova is also pacemake'd. If so, why can I still see my_ip=10.21.0.2 (test2) ? It should be pointing to nova-ha (assuming 10.21.2.4 as per /etc/hosts).

Also, as per my understanding of Pacemaker, DRBD partition is setup by default on test2, correct ?


Sorry, as per my first reading, I can't see anything obvious. That said, I'm not sure this is a Nova bug, as the tcpdump trace is seeing a correct MySQL connection attempt. But maybe I'm wrong ?

Anyway, are you sure you only have *one* MySQL engine running (either on test1 or test2) and nova-manage trying to access this right one ?

Perms look good to me. As it a test setup, you could try to unleash the grants by deleting them and allowing nova@'%' to see if it's a basic dns mapping issue.

-Sylvain



Le 11/03/2013 16:09, Samuel Winchenbach a écrit :
I
enabled general_log in /etc/mysql/my.cnf  Here are the results of connecting from "test1", "test2" and using the client:
http://paste2.org/p/3115525
I purposefully used the real password in case there is a problem with it.
 I changed before submitting post.

here is a raw packet TCP dump (tcpdump -w rawdump port 3306) of an attempted "nova-manage service list" from test1:
https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump

I looked at it with wireshark and couldn't see anything that jumped out at me as incorrect.  I have not yet tried to recreate the salted password.


Here is my pacemaker configuration for mysql.  I stripped out openstack services, rabbitmq and others for clarity.  All resources are currently disabled (other than MySQL):
http://paste2.org/p/3115685


Please don't yell at me for having STONITH disabled :P  This is a testing cluster and I am working on getting routed to the IPMI interface.

/etc/hosts:
http://paste2.org/p/3115713
/etc/nova/nova.conf:
http://paste2.org/p/3115739


If there is anything else I can provide you, please let me know!  I have pulled out most of my hair at this point!

Sam



On Mon, Mar 11, 2013 at 10:11 AM, Sylvain Bauza <sylvain.bauza@xxxxxxxxxxxx<mailto:sylvain.bauza@xxxxxxxxxxxx>> wrote:
So as to reproduce the nova-manage SQL command, I would recommand to tcpdump -A port 3306 on the host and get the SQL trace on what's failing.

Could you please explain further what is your HA config ? Are you using pacemaker/heartbeat or any VIP ?

-Sylvain

Le 11/03/2013 14:23, Samuel Winchenbach a écrit :
Does anyone think this could be an openstack bug?  I just want to check before submitting a bug report.

Sam

On Fri, Mar 8, 2013 at 4:02 PM, Jay Pipes <jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>> wrote:
Sorry, I really can't think of anything :(

On 03/08/2013 03:52 PM, Samuel Winchenbach wrote:
> I dropped those users and no change.
>
> I also set up general logging in mysql but it really doesn't provide any
> additional information.  Any idea for a next step I could take?
>
> I am almost at the point of taking a tcpdump and trying to recreate the
> salted password.  :/
>
> Thanks for the help
>
> Sam
>
>
>
>
> On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes <jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>
> <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>>> wrote:
>
>     I'm stumped :( Looks like everything is set up correctly to me. What is
>     interested is that your nova user access works from test2, but there is
>     no nova@test2 user in the mysql.user table. What about doing a DROP USER
>     nova@test1; FLUSH PRIVILEGES; and then see if that fixes things... since
>     the nova@10.21.0.0/255.255.0.0<http://nova@10.21.0.0/255.255.0.0> <http://nova@10.21.0.0/255.255.0.0>
>     user is clearly working for the access
>     from test2.
>
>     Also, I'd recommend highly removing the nova@% user.
>
>     Best,
>     -jay
>
>     On 03/08/2013 03:09 PM, Samuel Winchenbach wrote:
>     >
>     > http://paste2.org/p/3085807
>     >
>     >
>     > On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes <jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>
>     <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>>
>     > <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>>>> wrote:
>     >
>     >     Please paste the results of SELECT User, Host, Password FROM
>     mysql.user
>     >     when running as root...
>     >
>     >     Thanks!
>     >     -jay
>     >
>     >     On 03/08/2013 02:25 PM, Samuel Winchenbach wrote:
>     >     > Here are my grants.  I don't know if this helps, but I did
>     verify that
>     >     > the password was identical for each grant:
>     >      http://paste2.org/p/3085361
>     >     >
>     >     >
>     >     > On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach
>     >     <swinchen@xxxxxxxxx<mailto:swinchen@xxxxxxxxx> <mailto:swinchen@xxxxxxxxx<mailto:swinchen@xxxxxxxxx>>
>     <mailto:swinchen@xxxxxxxxx<mailto:swinchen@xxxxxxxxx> <mailto:swinchen@xxxxxxxxx<mailto:swinchen@xxxxxxxxx>>>
>     >     > <mailto:swinchen@xxxxxxxxx<mailto:swinchen@xxxxxxxxx> <mailto:swinchen@xxxxxxxxx<mailto:swinchen@xxxxxxxxx>>
>     <mailto:swinchen@xxxxxxxxx<mailto:swinchen@xxxxxxxxx> <mailto:swinchen@xxxxxxxxx<mailto:swinchen@xxxxxxxxx>>>>> wrote:
>     >     >
>     >     >     root@test1:/var/log# mysql -hmysql-ha -unova
>     >     >     -p******************************** -e"SELECT User, Host,
>     Password
>     >     >     FROM mysql.user;"
>     >     >     ERROR 1142 (42000) at line 1: SELECT command denied to user
>     >     >     'nova'@'test1' for table 'user'
>     >     >
>     >     >
>     >     >     On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes
>     <jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>>
>     >     <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>>>
>     >     >     <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>>
>     <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>>>>> wrote:
>     >     >
>     >     >         What does this show?
>     >     >
>     >     >         mysql -hmysql-ha -unova -p<PASS> -e"SELECT User, Host,
>     >     Password FROM
>     >     >         mysql.user"
>     >     >
>     >     >         -jay
>     >     >
>     >     >         On 03/08/2013 01:46 PM, Samuel Winchenbach wrote:
>     >     >         > Sorry, that must have been a copy and paste error.
>      Here
>     >     is what I
>     >     >         > actually ran:
>     >     >         >
>     >     >         > http://paste2.org/p/3084996
>     >     >         >
>     >     >         >
>     >     >         > On Fri, Mar 8, 2013 at 12:40 PM, Jay Pipes
>     >     <jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>>
>     <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>>>
>     >     >         <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>
>     <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>> <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>
>     <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>>>>
>     >     >         > <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>
>     <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>> <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>
>     <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>>>
>     >     <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>>
>     <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx<mailto:jaypipes@xxxxxxxxx>>>>>> wrote:
>     >     >         >
>     >     >         >     On 03/08/2013 12:19 PM, Samuel Winchenbach wrote:
>     >     >         >     > Hi All,
>     >     >         >     >
>     >     >         >     > I have two nodes (test1 and test2) that I am
>     trying to
>     >     >         set up in a
>     >     >         >     > highly available configuration.
>     >     >         >     >
>     >     >         >     > During the setup process I tried running
>     "nova-manage
>     >     >         service list" on
>     >     >         >     > both nodes.   It worked fine on test2, but
>     fails on
>     >     >         test1 even
>     >     >         >     though I
>     >     >         >     > can connect to the database with the mysql
>     client from
>     >     >         test1.
>     >     >         >     >
>     >     >         >     > Here is a screen capture that shows the setup on
>     >     the two
>     >     >         nodes are
>     >     >         >     > basically identical:
>      http://paste2.org/p/3084223
>     >     >         >
>     >     >         >     In the above paste you are doing:
>     >     >         >
>     >     >         >     mysql -unova -       hmysql-ha -u  root      nova
>     >     >         >     -p********************************
>     >     >         >
>     >     >         >     Note you are supplying 2 -u arguments, and mysql
>     >     will take
>     >     >         the second
>     >     >         >     (root).
>     >     >         >
>     >     >         >     -jay
>     >     >         >
>     >     >         >     _______________________________________________
>     >     >         >     Mailing list: https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
>     >     >         >     Post to     : openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
>     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>
>     >     >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
>     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>>
>     >     >         >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
>     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>
>     >     >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
>     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>>>
>     >     >         >     Unsubscribe : https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
>     >     >         >     More help   : https://help.launchpad.net/ListHelp
>     >     >         >
>     >     >         >
>     >     >
>     >     >
>     >     >
>     >
>     >
>
>



_______________________________________________

Mailing list: https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>

Post to     : openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>

Unsubscribe : https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>

More help   : https://help.launchpad.net/ListHelp


_______________________________________________
Mailing list: https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
Post to     : openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>
Unsubscribe : https://launchpad.net/~openstack<https://launchpad.net/%7Eopenstack>
More help   : https://help.launchpad.net/ListHelp

Follow ups

References

  Thread Previous • Date Previous • • Thread Next