openstack team mailing list archive

[Sylvain Bauza <sylvain.bauza@xxxxxxxxxxxx>]

Ok, lemme try to summarize.
You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1 thanks to 
Pacemaker.
This setup is relying on two hosts, test1 (10.21.0.1) and test2 (10.21.0.2).
Your nova.conf is pointing to mysql://10.21.1.1 which is the VIP.

Are you sure your my.cnf is actually the same in between both DRBD nodes 
? (I would recommend to symlink it to a physical file hosted on the DRBD 
device).

One thing is hurting me : you told me that nova is also pacemake'd. If 
so, why can I still see my_ip=10.21.0.2 (test2) ? It should be pointing 
to nova-ha (assuming 10.21.2.4 as per /etc/hosts).

Also, as per my understanding of Pacemaker, DRBD partition is setup by 
default on test2, correct ?


Sorry, as per my first reading, I can't see anything obvious. That said, 
I'm not sure this is a Nova bug, as the tcpdump trace is seeing a 
correct MySQL connection attempt. But maybe I'm wrong ?

Anyway, are you sure you only have *one* MySQL engine running (either on 
test1 or test2) and nova-manage trying to access this right one ?

Perms look good to me. As it a test setup, you could try to unleash the 
grants by deleting them and allowing nova@'%' to see if it's a basic dns 
mapping issue.

-Sylvain



Le 11/03/2013 16:09, Samuel Winchenbach a écrit :
> I
> enabled general_log in /etc/mysql/my.cnf  Here are the results of 
> connecting from "test1", "test2" and using the client:
> http://paste2.org/p/3115525
> I purposefully used the real password in case there is a problem with it.
>  I changed before submitting post.
>
> here is a raw packet TCP dump (tcpdump -w rawdump port 3306) of an 
> attempted "nova-manage service list" from test1:
> https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump
> I looked at it with wireshark and couldn't see anything that jumped 
> out at me as incorrect.  I have not yet tried to recreate the salted 
> password.
>
>
> Here is my pacemaker configuration for mysql.  I stripped out 
> openstack services, rabbitmq and others for clarity.  All resources 
> are currently disabled (other than MySQL):
> http://paste2.org/p/3115685
>
> Please don't yell at me for having STONITH disabled :P  This is a 
> testing cluster and I am working on getting routed to the IPMI interface.
>
> /etc/hosts:
> http://paste2.org/p/3115713
> /etc/nova/nova.conf:
> http://paste2.org/p/3115739
>
> If there is anything else I can provide you, please let me know!  I 
> have pulled out most of my hair at this point!
>
> Sam
>
>
>
>
> On Mon, Mar 11, 2013 at 10:11 AM, Sylvain Bauza 
> <sylvain.bauza@xxxxxxxxxxxx <mailto:sylvain.bauza@xxxxxxxxxxxx>> wrote:
>
>     So as to reproduce the nova-manage SQL command, I would recommand
>     to tcpdump -A port 3306 on the host and get the SQL trace on
>     what's failing.
>
>     Could you please explain further what is your HA config ? Are you
>     using pacemaker/heartbeat or any VIP ?
>
>     -Sylvain
>
>     Le 11/03/2013 14:23, Samuel Winchenbach a écrit :
> >     Does anyone think this could be an openstack bug?  I just want to
> >     check before submitting a bug report.
> >
> >     Sam
> >
> >
> >     On Fri, Mar 8, 2013 at 4:02 PM, Jay Pipes <jaypipes@xxxxxxxxx
> >     <mailto:jaypipes@xxxxxxxxx>> wrote:
> >
> >         Sorry, I really can't think of anything :(
> >
> >         On 03/08/2013 03:52 PM, Samuel Winchenbach wrote:
> >         > I dropped those users and no change.
> >         >
> >         > I also set up general logging in mysql but it really
> >         doesn't provide any
> >         > additional information.  Any idea for a next step I could take?
> >         >
> >         > I am almost at the point of taking a tcpdump and trying to
> >         recreate the
> >         > salted password.  :/
> >         >
> >         > Thanks for the help
> >         >
> >         > Sam
> >         >
> >         >
> >         >
> >         >
> >         > On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes
> >         <jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
> >         > <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>> wrote:
> >         >
> >         >     I'm stumped :( Looks like everything is set up
> >         correctly to me. What is
> >         >     interested is that your nova user access works from
> >         test2, but there is
> >         >     no nova@test2 user in the mysql.user table. What about
> >         doing a DROP USER
> >         >     nova@test1; FLUSH PRIVILEGES; and then see if that
> >         fixes things... since
> >         >     the nova@10.21.0.0/255.255.0.0
> >         <http://nova@10.21.0.0/255.255.0.0>
> >         <http://nova@10.21.0.0/255.255.0.0>
> >         >     user is clearly working for the access
> >         >     from test2.
> >         >
> >         >     Also, I'd recommend highly removing the nova@% user.
> >         >
> >         >     Best,
> >         >     -jay
> >         >
> >         >     On 03/08/2013 03:09 PM, Samuel Winchenbach wrote:
> >         >     >
> >         >     > http://paste2.org/p/3085807
> >         >     >
> >         >     >
> >         >     > On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes
> >         <jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
> >         >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>
> >         >     > <mailto:jaypipes@xxxxxxxxx
> >         <mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
> >         <mailto:jaypipes@xxxxxxxxx>>>> wrote:
> >         >     >
> >         >     >     Please paste the results of SELECT User, Host,
> >         Password FROM
> >         >     mysql.user
> >         >     >     when running as root...
> >         >     >
> >         >     >     Thanks!
> >         >     >     -jay
> >         >     >
> >         >     >     On 03/08/2013 02:25 PM, Samuel Winchenbach wrote:
> >         >     >     > Here are my grants.  I don't know if this
> >         helps, but I did
> >         >     verify that
> >         >     >     > the password was identical for each grant:
> >         >     > http://paste2.org/p/3085361
> >         >     >     >
> >         >     >     >
> >         >     >     > On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach
> >         >     >     <swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>
> >         <mailto:swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>>
> >         >     <mailto:swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>
> >         <mailto:swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>>>
> >         >     >     > <mailto:swinchen@xxxxxxxxx
> >         <mailto:swinchen@xxxxxxxxx> <mailto:swinchen@xxxxxxxxx
> >         <mailto:swinchen@xxxxxxxxx>>
> >         >     <mailto:swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>
> >         <mailto:swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>>>>> wrote:
> >         >     >     >
> >         >     >     > root@test1:/var/log# mysql -hmysql-ha -unova
> >         >     >     > -p******************************** -e"SELECT
> >         User, Host,
> >         >     Password
> >         >     >     >     FROM mysql.user;"
> >         >     >     >     ERROR 1142 (42000) at line 1: SELECT
> >         command denied to user
> >         >     >     >     'nova'@'test1' for table 'user'
> >         >     >     >
> >         >     >     >
> >         >     >     >     On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes
> >         >     <jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
> >         <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>
> >         >     >     <mailto:jaypipes@xxxxxxxxx
> >         <mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
> >         <mailto:jaypipes@xxxxxxxxx>>>
> >         >     >     >     <mailto:jaypipes@xxxxxxxxx
> >         <mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
> >         <mailto:jaypipes@xxxxxxxxx>>
> >         >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
> >         <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>>>> wrote:
> >         >     >     >
> >         >     >     >         What does this show?
> >         >     >     >
> >         >     >     >         mysql -hmysql-ha -unova -p<PASS>
> >         -e"SELECT User, Host,
> >         >     >     Password FROM
> >         >     >     >         mysql.user"
> >         >     >     >
> >         >     >     >         -jay
> >         >     >     >
> >         >     >     >         On 03/08/2013 01:46 PM, Samuel
> >         Winchenbach wrote:
> >         >     >     >         > Sorry, that must have been a copy and
> >         paste error.
> >         >      Here
> >         >     >     is what I
> >         >     >     >         > actually ran:
> >         >     >     >         >
> >         >     >     >         > http://paste2.org/p/3084996
> >         >     >     >         >
> >         >     >     >         >
> >         >     >     >         > On Fri, Mar 8, 2013 at 12:40 PM, Jay
> >         Pipes
> >         >     >     <jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
> >         <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>
> >         >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
> >         <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>>
> >         >     >     >         <mailto:jaypipes@xxxxxxxxx
> >         <mailto:jaypipes@xxxxxxxxx>
> >         >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>
> >         <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
> >         >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>>>
> >         >     >     >         > <mailto:jaypipes@xxxxxxxxx
> >         <mailto:jaypipes@xxxxxxxxx>
> >         >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>
> >         <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
> >         >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>>
> >         >     >     <mailto:jaypipes@xxxxxxxxx
> >         <mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
> >         <mailto:jaypipes@xxxxxxxxx>>
> >         >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
> >         <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>>>>>
> >         wrote:
> >         >     >     >         >
> >         >     >     >         >     On 03/08/2013 12:19 PM, Samuel
> >         Winchenbach wrote:
> >         >     >     >         > > Hi All,
> >         >     >     >         > >
> >         >     >     >         > > I have two nodes (test1 and test2)
> >         that I am
> >         >     trying to
> >         >     >     >         set up in a
> >         >     >     >         > > highly available configuration.
> >         >     >     >         > >
> >         >     >     >         > > During the setup process I tried
> >         running
> >         >     "nova-manage
> >         >     >     >         service list" on
> >         >     >     >         > > both nodes.   It worked fine on
> >         test2, but
> >         >     fails on
> >         >     >     >         test1 even
> >         >     >     >         > though I
> >         >     >     >         > > can connect to the database with
> >         the mysql
> >         >     client from
> >         >     >     >         test1.
> >         >     >     >         > >
> >         >     >     >         > > Here is a screen capture that shows
> >         the setup on
> >         >     >     the two
> >         >     >     >         nodes are
> >         >     >     >         > > basically identical:
> >         > http://paste2.org/p/3084223
> >         >     >     >         >
> >         >     >     >         >     In the above paste you are doing:
> >         >     >     >         >
> >         >     >     >         > mysql -unova -       hmysql-ha -u
> >          root    nova
> >         >     >     >         > -p********************************
> >         >     >     >         >
> >         >     >     >         > Note you are supplying 2 -u
> >         arguments, and mysql
> >         >     >     will take
> >         >     >     >         the second
> >         >     >     >         > (root).
> >         >     >     >         >
> >         >     >     >         > -jay
> >         >     >     >         >
> >         >     >     >         >
> >         _______________________________________________
> >         >     >     >         > Mailing list:
> >         https://launchpad.net/~openstack
> >         <https://launchpad.net/%7Eopenstack>
> >         >     >     >         > Post to     :
> >         openstack@xxxxxxxxxxxxxxxxxxx
> >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> >         >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
> >         >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> >         >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>
> >         >     >     >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> >         >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
> >         >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> >         >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>>
> >         >     >     >         > <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> >         >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
> >         >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> >         >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>
> >         >     >     >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> >         >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
> >         >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> >         >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>>>
> >         >     >     >         > Unsubscribe :
> >         https://launchpad.net/~openstack
> >         <https://launchpad.net/%7Eopenstack>
> >         >     >     >         > More help   :
> >         https://help.launchpad.net/ListHelp
> >         >     >     >         >
> >         >     >     >         >
> >         >     >     >
> >         >     >     >
> >         >     >     >
> >         >     >
> >         >     >
> >         >
> >         >
> >
> >
> >
> >
> >     _______________________________________________
> >     Mailing list:https://launchpad.net/~openstack  <https://launchpad.net/%7Eopenstack>
> >     Post to     :openstack@xxxxxxxxxxxxxxxxxxx  <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> >     Unsubscribe :https://launchpad.net/~openstack  <https://launchpad.net/%7Eopenstack>
> >     More help   :https://help.launchpad.net/ListHelp
>
>
>     _______________________________________________
>     Mailing list: https://launchpad.net/~openstack
>     <https://launchpad.net/%7Eopenstack>
>     Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>     Unsubscribe : https://launchpad.net/~openstack
>     <https://launchpad.net/%7Eopenstack>
>     More help   : https://help.launchpad.net/ListHelp