openstack team mailing list archive

[Samuel Winchenbach <swinchen@xxxxxxxxx>]

ugh...  I had an example file called "nova.conf" in /root

Apparently nova-manage looks for that file first.  Case closed.  I wish I
could get the last week back.

Sam


On Mon, Mar 11, 2013 at 2:37 PM, Miller, Mark M (EB SW Cloud - R&D -
Corvallis) <mark.m.miller@xxxxxx> wrote:

>  I ran into a similar problem with the Grizzly-3 Keystone release. I had
> to specify keystone.conf as the config-file with keystone-manage/ db_sync
> command otherwise it would not use the mysql statement in the keystone.conf
> file.****
>
> ** **
>
> Mark****
>
> ** **
>
> *From:* openstack-bounces+mark.m.miller=hp.com@xxxxxxxxxxxxxxxxxxx[mailto:
> openstack-bounces+mark.m.miller=hp.com@xxxxxxxxxxxxxxxxxxx] *On Behalf Of
> *Samuel Winchenbach
> *Sent:* Monday, March 11, 2013 11:18 AM
> *To:* Sylvain Bauza
> *Cc:* openstack@xxxxxxxxxxxxxxxxxxx
> *Subject:* Re: [Openstack] Incredibly odd mysql permission error****
>
> ** **
>
> OK!!!!  Someone on the IRC channel got me closer, but we have no idea why
> this would happen:****
>
> ** **
>
> this works: "root@test1:~# nova-manage --config-file=/etc/nova/nova.conf
> service list"****
>
> ** **
>
> Why would I have to specify the config file though?  It is in the standard
> place.****
>
> ** **
>
> Thanks,****
>
> Sam****
>
> ** **
>
> ** **
>
> ** **
>
> On Mon, Mar 11, 2013 at 2:01 PM, Samuel Winchenbach <swinchen@xxxxxxxxx>
> wrote:****
>
> For completeness here the routing table, and ip listing for both test1 and
> test2.   Doubt this will help much:  http://paste2.org/p/3117125****
>
> ** **
>
> On Mon, Mar 11, 2013 at 1:52 PM, Samuel Winchenbach <swinchen@xxxxxxxxx>
> wrote:****
>
> #1 - No change****
>
> #2 - All of grants are in the ip/mask form such as: 'nova'@'
> 10.21.0.0/255.255.0.0'  I have also tried adding 'nova'@'test1' and
> 'nova'@'10.21.0.1'.  No change.****
>
> #3 - I changed the SQL connection string over to IP instead of hostname.
>  No change.  I didn't restart nova-api because it isn't running.   If I
> understand correctly nova-manage communicated directly with the db,
> bypassing nova-api.   This would appear true seeing "nova-manage service
> list" works correctly on test2.****
>
> ** **
>
> ** **
>
> :(****
>
> ** **
>
> Thanks for the help!****
>
> Sam****
>
> ** **
>
> On Mon, Mar 11, 2013 at 12:24 PM, Sylvain Bauza <
> sylvain.bauza@xxxxxxxxxxxx> wrote:****
>
> When looking at MySQL 5.1 refman (
> http://dev.mysql.com/doc/refman/5.1/en/access-denied.html ), I would
> suggest to follow the procedure :
>  1. 'mysqladmin flush-hosts'
>  2. replace DNS entries in mysql.user table by IP addresses instead
>  3. modify /etc/nova/nova.conf with IP address of HA Mysql instead (and
> restart nova-api !)
>
> I wouldn't bet on it, but I would say this is due to some name resolution
> which is incorrect.
>
> -Sylvain
>
>
> Le 11/03/2013 17:00, Sylvain Bauza a écrit :****
>
>  Ok, lemme try to summarize.
> You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1 thanks to
> Pacemaker.
> This setup is relying on two hosts, test1 (10.21.0.1) and test2
> (10.21.0.2).
> Your nova.conf is pointing to mysql://10.21.1.1 which is the VIP.
>
> Are you sure your my.cnf is actually the same in between both DRBD nodes ?
> (I would recommend to symlink it to a physical file hosted on the DRBD
> device).
>
> One thing is hurting me : you told me that nova is also pacemake'd. If so,
> why can I still see my_ip=10.21.0.2 (test2) ? It should be pointing to
> nova-ha (assuming 10.21.2.4 as per /etc/hosts).
>
> Also, as per my understanding of Pacemaker, DRBD partition is setup by
> default on test2, correct ?
>
>
> Sorry, as per my first reading, I can't see anything obvious. That said,
> I'm not sure this is a Nova bug, as the tcpdump trace is seeing a correct
> MySQL connection attempt. But maybe I'm wrong ?
>
> Anyway, are you sure you only have *one* MySQL engine running (either on
> test1 or test2) and nova-manage trying to access this right one ?
>
> Perms look good to me. As it a test setup, you could try to unleash the
> grants by deleting them and allowing nova@'%' to see if it's a basic dns
> mapping issue.
>
> -Sylvain
>
>
>
> Le 11/03/2013 16:09, Samuel Winchenbach a écrit :****
>
>  I ****
>
> enabled general_log in /etc/mysql/my.cnf  Here are the results of
> connecting from "test1", "test2" and using the client:****
>
> http://paste2.org/p/3115525****
>
> I purposefully used the real password in case there is a problem with it.
> ****
>
>  I changed before submitting post.****
>
> ** **
>
> here is a raw packet TCP dump (tcpdump -w rawdump port 3306) of an
> attempted "nova-manage service list" from test1:  ****
>
> https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump****
>
>  ****
>
> I looked at it with wireshark and couldn't see anything that jumped out at
> me as incorrect.  I have not yet tried to recreate the salted password.***
> *
>
> ** **
>
> ** **
>
> Here is my pacemaker configuration for mysql.  I stripped out openstack
> services, rabbitmq and others for clarity.  All resources are currently
> disabled (other than MySQL): ****
>
> http://paste2.org/p/3115685 ****
>
>  ****
>
> ** **
>
> Please don't yell at me for having STONITH disabled :P  This is a testing
> cluster and I am working on getting routed to the IPMI interface.****
>
> ** **
>
> /etc/hosts: ****
>
> http://paste2.org/p/3115713 ****
>
> /etc/nova/nova.conf: ****
>
> http://paste2.org/p/3115739 ****
>
>  ****
>
> ** **
>
> If there is anything else I can provide you, please let me know!  I have
> pulled out most of my hair at this point!****
>
> ** **
>
> Sam****
>
> ** **
>
> ** **
>
> ** **
>
> On Mon, Mar 11, 2013 at 10:11 AM, Sylvain Bauza <
> sylvain.bauza@xxxxxxxxxxxx> wrote:****
>
> So as to reproduce the nova-manage SQL command, I would recommand to
> tcpdump -A port 3306 on the host and get the SQL trace on what's failing.
>
> Could you please explain further what is your HA config ? Are you using
> pacemaker/heartbeat or any VIP ?
>
> -Sylvain
>
> Le 11/03/2013 14:23, Samuel Winchenbach a écrit :****
>
>  Does anyone think this could be an openstack bug?  I just want to check
> before submitting a bug report.****
>
> ** **
>
> Sam****
>
> ** **
>
> On Fri, Mar 8, 2013 at 4:02 PM, Jay Pipes <jaypipes@xxxxxxxxx> wrote:****
>
> Sorry, I really can't think of anything :(****
>
>
> On 03/08/2013 03:52 PM, Samuel Winchenbach wrote:
> > I dropped those users and no change.
> >
> > I also set up general logging in mysql but it really doesn't provide any
> > additional information.  Any idea for a next step I could take?
> >
> > I am almost at the point of taking a tcpdump and trying to recreate the
> > salted password.  :/
> >
> > Thanks for the help
> >
> > Sam
> >
> >
> >
> >
> > On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes <jaypipes@xxxxxxxxx****
>
> > <mailto:jaypipes@xxxxxxxxx>> wrote:
> >
> >     I'm stumped :( Looks like everything is set up correctly to me. What
> is
> >     interested is that your nova user access works from test2, but there
> is
> >     no nova@test2 user in the mysql.user table. What about doing a DROP
> USER
> >     nova@test1; FLUSH PRIVILEGES; and then see if that fixes things...
> since****
>
> >     the nova@10.21.0.0/255.255.0.0 <http://nova@10.21.0.0/255.255.0.0>**
> **
>
> >     user is clearly working for the access
> >     from test2.
> >
> >     Also, I'd recommend highly removing the nova@% user.
> >
> >     Best,
> >     -jay
> >
> >     On 03/08/2013 03:09 PM, Samuel Winchenbach wrote:
> >     >
> >     > http://paste2.org/p/3085807
> >     >
> >     >
> >     > On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes <jaypipes@xxxxxxxxx
> >     <mailto:jaypipes@xxxxxxxxx>****
>
> >     > <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>> wrote:
> >     >****
>
> >     >     Please paste the results of SELECT User, Host, Password FROM
> >     mysql.user
> >     >     when running as root...
> >     >
> >     >     Thanks!
> >     >     -jay
> >     >
> >     >     On 03/08/2013 02:25 PM, Samuel Winchenbach wrote:
> >     >     > Here are my grants.  I don't know if this helps, but I did
> >     verify that
> >     >     > the password was identical for each grant:
> >     >      http://paste2.org/p/3085361
> >     >     >
> >     >     >
> >     >     > On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach
> >     >     <swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>
> >     <mailto:swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>>
> >     >     > <mailto:swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>
> >     <mailto:swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>>>> wrote:
> >     >     >
> >     >     >     root@test1:/var/log# mysql -hmysql-ha -unova
> >     >     >     -p******************************** -e"SELECT User, Host,
> >     Password
> >     >     >     FROM mysql.user;"
> >     >     >     ERROR 1142 (42000) at line 1: SELECT command denied to
> user
> >     >     >     'nova'@'test1' for table 'user'
> >     >     >
> >     >     >
> >     >     >     On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes
> >     <jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
> >     >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>****
>
> >     >     >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
> >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>>> wrote:
> >     >     >
> >     >     >         What does this show?
> >     >     >
> >     >     >         mysql -hmysql-ha -unova -p<PASS> -e"SELECT User,
> Host,
> >     >     Password FROM
> >     >     >         mysql.user"
> >     >     >
> >     >     >         -jay
> >     >     >
> >     >     >         On 03/08/2013 01:46 PM, Samuel Winchenbach wrote:
> >     >     >         > Sorry, that must have been a copy and paste error.
> >      Here
> >     >     is what I
> >     >     >         > actually ran:
> >     >     >         >
> >     >     >         > http://paste2.org/p/3084996
> >     >     >         >
> >     >     >         >
> >     >     >         > On Fri, Mar 8, 2013 at 12:40 PM, Jay Pipes
> >     >     <jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
> >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>
> >     >     >         <mailto:jaypipes@xxxxxxxxx
> >     <mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
> >     <mailto:jaypipes@xxxxxxxxx>>>
> >     >     >         > <mailto:jaypipes@xxxxxxxxx
> >     <mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
> >     <mailto:jaypipes@xxxxxxxxx>>
> >     >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
> >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>>>> wrote:
> >     >     >         >
> >     >     >         >     On 03/08/2013 12:19 PM, Samuel Winchenbach
> wrote:
> >     >     >         >     > Hi All,
> >     >     >         >     >
> >     >     >         >     > I have two nodes (test1 and test2) that I am
> >     trying to
> >     >     >         set up in a
> >     >     >         >     > highly available configuration.
> >     >     >         >     >
> >     >     >         >     > During the setup process I tried running
> >     "nova-manage
> >     >     >         service list" on
> >     >     >         >     > both nodes.   It worked fine on test2, but
> >     fails on
> >     >     >         test1 even
> >     >     >         >     though I
> >     >     >         >     > can connect to the database with the mysql
> >     client from
> >     >     >         test1.
> >     >     >         >     >
> >     >     >         >     > Here is a screen capture that shows the
> setup on
> >     >     the two
> >     >     >         nodes are
> >     >     >         >     > basically identical:
> >      http://paste2.org/p/3084223
> >     >     >         >
> >     >     >         >     In the above paste you are doing:
> >     >     >         >
> >     >     >         >     mysql -unova -       hmysql-ha -u  root
>  nova
> >     >     >         >     -p********************************
> >     >     >         >
> >     >     >         >     Note you are supplying 2 -u arguments, and
> mysql
> >     >     will take
> >     >     >         the second
> >     >     >         >     (root).
> >     >     >         >
> >     >     >         >     -jay
> >     >     >         >
> >     >     >         >     _______________________________________________
> >     >     >         >     Mailing list: https://launchpad.net/~openstack
> >     >     >         >     Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
> >     >     >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>
> >     >     >         >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
> >     >     >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
> >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>>
> >     >     >         >     Unsubscribe : https://launchpad.net/~openstack
> >     >     >         >     More help   :
> https://help.launchpad.net/ListHelp
> >     >     >         >
> >     >     >         >
> >     >     >
> >     >     >
> >     >     >
> >     >
> >     >
> >
> >****
>
> ** **
>
> ** **
>
> _______________________________________________****
>
> Mailing list: https://launchpad.net/~openstack****
>
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx****
>
> Unsubscribe : https://launchpad.net/~openstack****
>
> More help   : https://help.launchpad.net/ListHelp****
>
>  ** **
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp****
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>