openstack team mailing list archive

[Samuel Winchenbach <swinchen@xxxxxxxxx>]

OK!!!!  Someone on the IRC channel got me closer, but we have no idea why
this would happen:

this works: "root@test1:~# nova-manage --config-file=/etc/nova/nova.conf
service list"

Why would I have to specify the config file though?  It is in the standard
place.

Thanks,
Sam




On Mon, Mar 11, 2013 at 2:01 PM, Samuel Winchenbach <swinchen@xxxxxxxxx>wrote:

> For completeness here the routing table, and ip listing for both test1 and
> test2.   Doubt this will help much:  http://paste2.org/p/3117125
>
>
> On Mon, Mar 11, 2013 at 1:52 PM, Samuel Winchenbach <swinchen@xxxxxxxxx>wrote:
>
>> #1 - No change
>> #2 - All of grants are in the ip/mask form such as: 'nova'@'
>> 10.21.0.0/255.255.0.0'  I have also tried adding 'nova'@'test1' and
>> 'nova'@'10.21.0.1'.  No change.
>> #3 - I changed the SQL connection string over to IP instead of hostname.
>>  No change.  I didn't restart nova-api because it isn't running.   If I
>> understand correctly nova-manage communicated directly with the db,
>> bypassing nova-api.   This would appear true seeing "nova-manage service
>> list" works correctly on test2.
>>
>>
>> :(
>>
>> Thanks for the help!
>> Sam
>>
>>
>> On Mon, Mar 11, 2013 at 12:24 PM, Sylvain Bauza <
>> sylvain.bauza@xxxxxxxxxxxx> wrote:
>>
>>>  When looking at MySQL 5.1 refman (
>>> http://dev.mysql.com/doc/refman/5.1/en/access-denied.html ), I would
>>> suggest to follow the procedure :
>>>  1. 'mysqladmin flush-hosts'
>>>  2. replace DNS entries in mysql.user table by IP addresses instead
>>>  3. modify /etc/nova/nova.conf with IP address of HA Mysql instead (and
>>> restart nova-api !)
>>>
>>> I wouldn't bet on it, but I would say this is due to some name
>>> resolution which is incorrect.
>>>
>>> -Sylvain
>>>
>>>
>>> Le 11/03/2013 17:00, Sylvain Bauza a écrit :
>>>
>>> Ok, lemme try to summarize.
>>> You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1 thanks to
>>> Pacemaker.
>>> This setup is relying on two hosts, test1 (10.21.0.1) and test2
>>> (10.21.0.2).
>>> Your nova.conf is pointing to mysql://10.21.1.1 which is the VIP.
>>>
>>> Are you sure your my.cnf is actually the same in between both DRBD nodes
>>> ? (I would recommend to symlink it to a physical file hosted on the DRBD
>>> device).
>>>
>>> One thing is hurting me : you told me that nova is also pacemake'd. If
>>> so, why can I still see my_ip=10.21.0.2 (test2) ? It should be pointing to
>>> nova-ha (assuming 10.21.2.4 as per /etc/hosts).
>>>
>>> Also, as per my understanding of Pacemaker, DRBD partition is setup by
>>> default on test2, correct ?
>>>
>>>
>>> Sorry, as per my first reading, I can't see anything obvious. That said,
>>> I'm not sure this is a Nova bug, as the tcpdump trace is seeing a correct
>>> MySQL connection attempt. But maybe I'm wrong ?
>>>
>>> Anyway, are you sure you only have *one* MySQL engine running (either on
>>> test1 or test2) and nova-manage trying to access this right one ?
>>>
>>> Perms look good to me. As it a test setup, you could try to unleash the
>>> grants by deleting them and allowing nova@'%' to see if it's a basic
>>> dns mapping issue.
>>>
>>> -Sylvain
>>>
>>>
>>>
>>> Le 11/03/2013 16:09, Samuel Winchenbach a écrit :
>>>
>>> I
>>> enabled general_log in /etc/mysql/my.cnf  Here are the results of
>>> connecting from "test1", "test2" and using the client:
>>>  http://paste2.org/p/3115525
>>>  I purposefully used the real password in case there is a problem with
>>> it.
>>>  I changed before submitting post.
>>>
>>>   here is a raw packet TCP dump (tcpdump -w rawdump port 3306) of an
>>> attempted "nova-manage service list" from test1:
>>>  https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump
>>>
>>>  I looked at it with wireshark and couldn't see anything that jumped
>>> out at me as incorrect.  I have not yet tried to recreate the salted
>>> password.
>>>
>>>
>>>   Here is my pacemaker configuration for mysql.  I stripped out
>>> openstack services, rabbitmq and others for clarity.  All resources are
>>> currently disabled (other than MySQL):
>>> http://paste2.org/p/3115685
>>>
>>>
>>>   Please don't yell at me for having STONITH disabled :P  This is a
>>> testing cluster and I am working on getting routed to the IPMI interface.
>>>
>>>   /etc/hosts:
>>> http://paste2.org/p/3115713
>>>   /etc/nova/nova.conf:
>>> http://paste2.org/p/3115739
>>>
>>>
>>>   If there is anything else I can provide you, please let me know!  I
>>> have pulled out most of my hair at this point!
>>>
>>>   Sam
>>>
>>>
>>>
>>>
>>> On Mon, Mar 11, 2013 at 10:11 AM, Sylvain Bauza <
>>> sylvain.bauza@xxxxxxxxxxxx> wrote:
>>>
>>>>  So as to reproduce the nova-manage SQL command, I would recommand to
>>>> tcpdump -A port 3306 on the host and get the SQL trace on what's failing.
>>>>
>>>> Could you please explain further what is your HA config ? Are you using
>>>> pacemaker/heartbeat or any VIP ?
>>>>
>>>> -Sylvain
>>>>
>>>> Le 11/03/2013 14:23, Samuel Winchenbach a écrit :
>>>>
>>>>  Does anyone think this could be an openstack bug?  I just want to
>>>> check before submitting a bug report.
>>>>
>>>>  Sam
>>>>
>>>>
>>>> On Fri, Mar 8, 2013 at 4:02 PM, Jay Pipes <jaypipes@xxxxxxxxx> wrote:
>>>>
>>>>> Sorry, I really can't think of anything :(
>>>>>
>>>>> On 03/08/2013 03:52 PM, Samuel Winchenbach wrote:
>>>>> > I dropped those users and no change.
>>>>> >
>>>>> > I also set up general logging in mysql but it really doesn't provide
>>>>> any
>>>>> > additional information.  Any idea for a next step I could take?
>>>>> >
>>>>> > I am almost at the point of taking a tcpdump and trying to recreate
>>>>> the
>>>>> > salted password.  :/
>>>>> >
>>>>> > Thanks for the help
>>>>> >
>>>>> > Sam
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes <jaypipes@xxxxxxxxx
>>>>>  > <mailto:jaypipes@xxxxxxxxx>> wrote:
>>>>> >
>>>>> >     I'm stumped :( Looks like everything is set up correctly to me.
>>>>> What is
>>>>> >     interested is that your nova user access works from test2, but
>>>>> there is
>>>>> >     no nova@test2 user in the mysql.user table. What about doing a
>>>>> DROP USER
>>>>> >     nova@test1; FLUSH PRIVILEGES; and then see if that fixes
>>>>> things... since
>>>>>  >     the nova@10.21.0.0/255.255.0.0 <
>>>>> http://nova@10.21.0.0/255.255.0.0>
>>>>> >     user is clearly working for the access
>>>>> >     from test2.
>>>>> >
>>>>> >     Also, I'd recommend highly removing the nova@% user.
>>>>> >
>>>>> >     Best,
>>>>> >     -jay
>>>>> >
>>>>> >     On 03/08/2013 03:09 PM, Samuel Winchenbach wrote:
>>>>> >     >
>>>>> >     > http://paste2.org/p/3085807
>>>>> >     >
>>>>> >     >
>>>>> >     > On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes <jaypipes@xxxxxxxxx
>>>>> >     <mailto:jaypipes@xxxxxxxxx>
>>>>>  >     > <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>>
>>>>> wrote:
>>>>> >     >
>>>>>  >     >     Please paste the results of SELECT User, Host, Password
>>>>> FROM
>>>>> >     mysql.user
>>>>> >     >     when running as root...
>>>>> >     >
>>>>> >     >     Thanks!
>>>>> >     >     -jay
>>>>> >     >
>>>>> >     >     On 03/08/2013 02:25 PM, Samuel Winchenbach wrote:
>>>>> >     >     > Here are my grants.  I don't know if this helps, but I
>>>>> did
>>>>> >     verify that
>>>>> >     >     > the password was identical for each grant:
>>>>> >     >      http://paste2.org/p/3085361
>>>>> >     >     >
>>>>> >     >     >
>>>>> >     >     > On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach
>>>>> >     >     <swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>
>>>>> >     <mailto:swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>>
>>>>> >     >     > <mailto:swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>
>>>>> >     <mailto:swinchen@xxxxxxxxx <mailto:swinchen@xxxxxxxxx>>>> wrote:
>>>>> >     >     >
>>>>> >     >     >     root@test1:/var/log# mysql -hmysql-ha -unova
>>>>> >     >     >     -p******************************** -e"SELECT User,
>>>>> Host,
>>>>> >     Password
>>>>> >     >     >     FROM mysql.user;"
>>>>> >     >     >     ERROR 1142 (42000) at line 1: SELECT command denied
>>>>> to user
>>>>> >     >     >     'nova'@'test1' for table 'user'
>>>>> >     >     >
>>>>> >     >     >
>>>>> >     >     >     On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes
>>>>> >     <jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
>>>>> >     >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>
>>>>>   >     >     >     <mailto:jaypipes@xxxxxxxxx <mailto:
>>>>> jaypipes@xxxxxxxxx>
>>>>> >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>>> wrote:
>>>>> >     >     >
>>>>> >     >     >         What does this show?
>>>>> >     >     >
>>>>> >     >     >         mysql -hmysql-ha -unova -p<PASS> -e"SELECT User,
>>>>> Host,
>>>>> >     >     Password FROM
>>>>> >     >     >         mysql.user"
>>>>> >     >     >
>>>>> >     >     >         -jay
>>>>> >     >     >
>>>>> >     >     >         On 03/08/2013 01:46 PM, Samuel Winchenbach wrote:
>>>>> >     >     >         > Sorry, that must have been a copy and paste
>>>>> error.
>>>>> >      Here
>>>>> >     >     is what I
>>>>> >     >     >         > actually ran:
>>>>> >     >     >         >
>>>>> >     >     >         > http://paste2.org/p/3084996
>>>>> >     >     >         >
>>>>> >     >     >         >
>>>>> >     >     >         > On Fri, Mar 8, 2013 at 12:40 PM, Jay Pipes
>>>>> >     >     <jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
>>>>> >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>
>>>>> >     >     >         <mailto:jaypipes@xxxxxxxxx
>>>>> >     <mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
>>>>> >     <mailto:jaypipes@xxxxxxxxx>>>
>>>>> >     >     >         > <mailto:jaypipes@xxxxxxxxx
>>>>> >     <mailto:jaypipes@xxxxxxxxx> <mailto:jaypipes@xxxxxxxxx
>>>>> >     <mailto:jaypipes@xxxxxxxxx>>
>>>>> >     >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>
>>>>> >     <mailto:jaypipes@xxxxxxxxx <mailto:jaypipes@xxxxxxxxx>>>>>
>>>>> wrote:
>>>>> >     >     >         >
>>>>> >     >     >         >     On 03/08/2013 12:19 PM, Samuel Winchenbach
>>>>> wrote:
>>>>> >     >     >         >     > Hi All,
>>>>> >     >     >         >     >
>>>>> >     >     >         >     > I have two nodes (test1 and test2) that
>>>>> I am
>>>>> >     trying to
>>>>> >     >     >         set up in a
>>>>> >     >     >         >     > highly available configuration.
>>>>> >     >     >         >     >
>>>>> >     >     >         >     > During the setup process I tried running
>>>>> >     "nova-manage
>>>>> >     >     >         service list" on
>>>>> >     >     >         >     > both nodes.   It worked fine on test2,
>>>>> but
>>>>> >     fails on
>>>>> >     >     >         test1 even
>>>>> >     >     >         >     though I
>>>>> >     >     >         >     > can connect to the database with the
>>>>> mysql
>>>>> >     client from
>>>>> >     >     >         test1.
>>>>> >     >     >         >     >
>>>>> >     >     >         >     > Here is a screen capture that shows the
>>>>> setup on
>>>>> >     >     the two
>>>>> >     >     >         nodes are
>>>>> >     >     >         >     > basically identical:
>>>>> >      http://paste2.org/p/3084223
>>>>> >     >     >         >
>>>>> >     >     >         >     In the above paste you are doing:
>>>>> >     >     >         >
>>>>> >     >     >         >     mysql -unova -       hmysql-ha -u  root
>>>>>    nova
>>>>> >     >     >         >     -p********************************
>>>>> >     >     >         >
>>>>> >     >     >         >     Note you are supplying 2 -u arguments, and
>>>>> mysql
>>>>> >     >     will take
>>>>> >     >     >         the second
>>>>> >     >     >         >     (root).
>>>>> >     >     >         >
>>>>> >     >     >         >     -jay
>>>>> >     >     >         >
>>>>> >     >     >         >
>>>>> _______________________________________________
>>>>> >     >     >         >     Mailing list:
>>>>> https://launchpad.net/~openstack
>>>>> >     >     >         >     Post to     :
>>>>> openstack@xxxxxxxxxxxxxxxxxxx
>>>>> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>>>>> >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
>>>>> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
>>>>> >     >     >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx
>>>>> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>>>>> >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
>>>>> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>
>>>>> >     >     >         >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
>>>>> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>>>>> >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
>>>>> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
>>>>> >     >     >         <mailto:openstack@xxxxxxxxxxxxxxxxxxx
>>>>> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
>>>>> >     >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx
>>>>> >     <mailto:openstack@xxxxxxxxxxxxxxxxxxx>>>>
>>>>> >     >     >         >     Unsubscribe :
>>>>> https://launchpad.net/~openstack
>>>>> >     >     >         >     More help   :
>>>>> https://help.launchpad.net/ListHelp
>>>>> >     >     >         >
>>>>> >     >     >         >
>>>>> >     >     >
>>>>> >     >     >
>>>>> >     >     >
>>>>> >     >
>>>>> >     >
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~openstack
>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~openstack
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~openstack
>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~openstack
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>>
>>>
>>
>