← Back to team overview

openstack team mailing list archive

Re: Incredibly odd mysql permission error

 

When looking at MySQL 5.1 refman (http://dev.mysql.com/doc/refman/5.1/en/access-denied.html ), I would suggest to follow the procedure :
 1. 'mysqladmin flush-hosts'
 2. replace DNS entries in mysql.user table by IP addresses instead
3. modify /etc/nova/nova.conf with IP address of HA Mysql instead (and restart nova-api !)

I wouldn't bet on it, but I would say this is due to some name resolution which is incorrect.

-Sylvain


Le 11/03/2013 17:00, Sylvain Bauza a écrit :
Ok, lemme try to summarize.
You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1 thanks to Pacemaker. This setup is relying on two hosts, test1 (10.21.0.1) and test2 (10.21.0.2).
Your nova.conf is pointing to mysql://10.21.1.1 which is the VIP.

Are you sure your my.cnf is actually the same in between both DRBD nodes ? (I would recommend to symlink it to a physical file hosted on the DRBD device).

One thing is hurting me : you told me that nova is also pacemake'd. If so, why can I still see my_ip=10.21.0.2 (test2) ? It should be pointing to nova-ha (assuming 10.21.2.4 as per /etc/hosts).

Also, as per my understanding of Pacemaker, DRBD partition is setup by default on test2, correct ?


Sorry, as per my first reading, I can't see anything obvious. That said, I'm not sure this is a Nova bug, as the tcpdump trace is seeing a correct MySQL connection attempt. But maybe I'm wrong ?

Anyway, are you sure you only have *one* MySQL engine running (either on test1 or test2) and nova-manage trying to access this right one ?

Perms look good to me. As it a test setup, you could try to unleash the grants by deleting them and allowing nova@'%' to see if it's a basic dns mapping issue.

-Sylvain



Le 11/03/2013 16:09, Samuel Winchenbach a écrit :
I
enabled general_log in /etc/mysql/my.cnf Here are the results of connecting from "test1", "test2" and using the client:
http://paste2.org/p/3115525
I purposefully used the real password in case there is a problem with it.
 I changed before submitting post.

here is a raw packet TCP dump (tcpdump -w rawdump port 3306) of an attempted "nova-manage service list" from test1:
https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump
I looked at it with wireshark and couldn't see anything that jumped out at me as incorrect. I have not yet tried to recreate the salted password.


Here is my pacemaker configuration for mysql. I stripped out openstack services, rabbitmq and others for clarity. All resources are currently disabled (other than MySQL):
http://paste2.org/p/3115685

Please don't yell at me for having STONITH disabled :P This is a testing cluster and I am working on getting routed to the IPMI interface.

/etc/hosts:
http://paste2.org/p/3115713
/etc/nova/nova.conf:
http://paste2.org/p/3115739

If there is anything else I can provide you, please let me know! I have pulled out most of my hair at this point!

Sam




On Mon, Mar 11, 2013 at 10:11 AM, Sylvain Bauza <sylvain.bauza@xxxxxxxxxxxx <mailto:sylvain.bauza@xxxxxxxxxxxx>> wrote:

    So as to reproduce the nova-manage SQL command, I would recommand
    to tcpdump -A port 3306 on the host and get the SQL trace on
    what's failing.

    Could you please explain further what is your HA config ? Are you
    using pacemaker/heartbeat or any VIP ?

    -Sylvain

    Le 11/03/2013 14:23, Samuel Winchenbach a écrit :
    Does anyone think this could be an openstack bug?  I just want
    to check before submitting a bug report.

    Sam


    On Fri, Mar 8, 2013 at 4:02 PM, Jay Pipes <jaypipes@xxxxxxxxx
    <mailto:jaypipes@xxxxxxxxx>> wrote:

        Sorry, I really can't think of anything :(

        On 03/08/2013 03:52 PM, Samuel Winchenbach wrote:
        > I dropped those users and no change.
        >
        > I also set up general logging in mysql but it really
        doesn't provide any
        > additional information.  Any idea for a next step I could
        take?
        >
        > I am almost at the point of taking a tcpdump and trying to
        recreate the
        > salted password.  :/
        >
        > Thanks for the help
        >
        > Sam
        >
        >
        >
        >
        > On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes
        <jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
        > <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>>
        wrote:
        >
        >     I'm stumped :( Looks like everything is set up
        correctly to me. What is
        >     interested is that your nova user access works from
        test2, but there is
        >     no nova@test2 user in the mysql.user table. What about
        doing a DROP USER
        >     nova@test1; FLUSH PRIVILEGES; and then see if that
        fixes things... since
        >     the nova@10.21.0.0/255.255.0.0
        <http://nova@10.21.0.0/255.255.0.0>
        <http://nova@10.21.0.0/255.255.0.0>
        >     user is clearly working for the access
        >     from test2.
        >
        >     Also, I'd recommend highly removing the nova@% user.
        >
        >     Best,
        >     -jay
        >
        >     On 03/08/2013 03:09 PM, Samuel Winchenbach wrote:
        >     >
        >     > http://paste2.org/p/3085807
        >     >
        >     >
        >     > On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes
        <jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
        >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>
        >     > <mailto:jaypipes@xxxxxxxxx
        <mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
        <mailto:jaypipes@xxxxxxxxx>>>> wrote:
        >     >
        >     >     Please paste the results of SELECT User, Host,
        Password FROM
        >     mysql.user
        >     >     when running as root...
        >     >
        >     >     Thanks!
        >     >     -jay
        >     >
        >     >     On 03/08/2013 02:25 PM, Samuel Winchenbach wrote:
        >     >     > Here are my grants.  I don't know if this
        helps, but I did
        >     verify that
        >     >     > the password was identical for each grant:
        >     > http://paste2.org/p/3085361
        >     >     >
        >     >     >
        >     >     > On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach
        >     >     <swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>
        <mailto:swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>>
        >     <mailto:swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>
        <mailto:swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>>>
        >     >     > <mailto:swinchen@xxxxxxxxx
        <mailto:swinchen@xxxxxxxxx> <mailto:swinchen@xxxxxxxxx
        <mailto:swinchen@xxxxxxxxx>>
        >     <mailto:swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>
        <mailto:swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>>>>>
        wrote:
        >     >     >
        >     >     > root@test1:/var/log# mysql -hmysql-ha -unova
        >     >     > -p******************************** -e"SELECT
        User, Host,
        >     Password
        >     >     >     FROM mysql.user;"
        >     >     >     ERROR 1142 (42000) at line 1: SELECT
        command denied to user
        >     >     > 'nova'@'test1' for table 'user'
        >     >     >
        >     >     >
        >     >     >     On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes
        >     <jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
        <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>
        >     >     <mailto:jaypipes@xxxxxxxxx
        <mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
        <mailto:jaypipes@xxxxxxxxx>>>
        >     >     > <mailto:jaypipes@xxxxxxxxx
        <mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
        <mailto:jaypipes@xxxxxxxxx>>
        >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
        <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>>>>
        wrote:
        >     >     >
        >     >     >         What does this show?
        >     >     >
        >     >     >         mysql -hmysql-ha -unova -p<PASS>
        -e"SELECT User, Host,
        >     >     Password FROM
        >     >     > mysql.user"
        >     >     >
        >     >     >         -jay
        >     >     >
        >     >     >         On 03/08/2013 01:46 PM, Samuel
        Winchenbach wrote:
        >     >     >         > Sorry, that must have been a copy
        and paste error.
        >      Here
        >     >     is what I
        >     >     >         > actually ran:
        >     >     >         >
        >     >     >         > http://paste2.org/p/3084996
        >     >     >         >
        >     >     >         >
        >     >     >         > On Fri, Mar 8, 2013 at 12:40 PM, Jay
        Pipes
        >     >     <jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
        <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>
        >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
        <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>>
        >     >     > <mailto:jaypipes@xxxxxxxxx
        <mailto:jaypipes@xxxxxxxxx>
        >     <mailto:jaypipes@xxxxxxxxx
        <mailto:jaypipes@xxxxxxxxx>> <mailto:jaypipes@xxxxxxxxx
        <mailto:jaypipes@xxxxxxxxx>
        >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>>>
        >     >     >         > <mailto:jaypipes@xxxxxxxxx
        <mailto:jaypipes@xxxxxxxxx>
        >     <mailto:jaypipes@xxxxxxxxx
        <mailto:jaypipes@xxxxxxxxx>> <mailto:jaypipes@xxxxxxxxx
        <mailto:jaypipes@xxxxxxxxx>
        >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>>
        >     >     <mailto:jaypipes@xxxxxxxxx
        <mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
        <mailto:jaypipes@xxxxxxxxx>>
        >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
        <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>>>>>
        wrote:
        >     >     >         >
        >     >     >         > On 03/08/2013 12:19 PM, Samuel
        Winchenbach wrote:
        >     >     >         > > Hi All,
        >     >     >         > >
        >     >     >         > > I have two nodes (test1 and test2)
        that I am
        >     trying to
        >     >     >         set up in a
        >     >     >         > > highly available configuration.
        >     >     >         > >
        >     >     >         > > During the setup process I tried
        running
        >     "nova-manage
        >     >     >         service list" on
        >     >     >         > > both nodes.   It worked fine on
        test2, but
        >     fails on
        >     >     >         test1 even
        >     >     >         > though I
        >     >     >         > > can connect to the database with
        the mysql
        >     client from
        >     >     >         test1.
        >     >     >         > >
        >     >     >         > > Here is a screen capture that
        shows the setup on
        >     >     the two
        >     >     >         nodes are
        >     >     >         > > basically identical:
        > http://paste2.org/p/3084223
        >     >     >         >
        >     >     >         > In the above paste you are doing:
        >     >     >         >
        >     >     >         > mysql -unova -       hmysql-ha -u
         root      nova
        >     >     >         > -p********************************
        >     >     >         >
        >     >     >         > Note you are supplying 2 -u
        arguments, and mysql
        >     >     will take
        >     >     >         the second
        >     >     >         > (root).
        >     >     >         >
        >     >     >         > -jay
        >     >     >         >
        >     >     >         >
        _______________________________________________
        >     >     >         > Mailing list:
        https://launchpad.net/~openstack
        <https://launchpad.net/%7Eopenstack>
        >     >     >         > Post to     :
        openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
        >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
        >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
        >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>
        >     >     > <mailto:openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
        >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
        >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
        >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>>
        >     >     >         >
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
        >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
        >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
        >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>
        >     >     > <mailto:openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
        >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
        >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
        >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>>>
        >     >     >         > Unsubscribe :
        https://launchpad.net/~openstack
        <https://launchpad.net/%7Eopenstack>
        >     >     >         > More help   :
        https://help.launchpad.net/ListHelp
        >     >     >         >
        >     >     >         >
        >     >     >
        >     >     >
        >     >     >
        >     >
        >     >
        >
        >




    _______________________________________________
    Mailing list:https://launchpad.net/~openstack  <https://launchpad.net/%7Eopenstack>
    Post to     :openstack@xxxxxxxxxxxxxxxxxxx  <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
    Unsubscribe :https://launchpad.net/~openstack  <https://launchpad.net/%7Eopenstack>
    More help   :https://help.launchpad.net/ListHelp


    _______________________________________________
    Mailing list: https://launchpad.net/~openstack
    <https://launchpad.net/%7Eopenstack>
    Post to     : openstack@xxxxxxxxxxxxxxxxxxx
    <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
    Unsubscribe : https://launchpad.net/~openstack
    <https://launchpad.net/%7Eopenstack>
    More help   : https://help.launchpad.net/ListHelp





Follow ups

References