maas-devel team mailing list archive

[Michael Hughes <itismike@xxxxxxxxx>]

Funny - I think I was on to something a week ago when I added this to the wiki:
https://wiki.ubuntu.com/ServerTeam/MAAS/Juju?action=recall&rev=10#Generating_SSH_keypair_for_Juju

It annoys me that some heavy-handed editor reverted my contributions
rather than confirming or enhancing them.

On Thu, Apr 19, 2012 at 3:53 PM, Diogo Matsubara
<matsubara@xxxxxxxxxxxxx> wrote:
> On Thu, Apr 19, 2012 at 5:31 PM, Michael Hughes <itismike@xxxxxxxxx> wrote:
>> On Thu, Apr 19, 2012 at 3:02 PM, Diogo Matsubara
>> <matsubara@xxxxxxxxxxxxx> wrote:
>>> On Thu, Apr 19, 2012 at 4:40 PM, Michael Hughes <itismike@xxxxxxxxx> wrote:
>>>> Okay I'll hang tight on the reinstall. Yes, I added an ssh key.
>>>> [edit - I just re-read your comment a little closer. You want me to
>>>> add an ssh key for the ADMIN user? As in root?]
>>>>
>>>
>>> For the user created by `sudo maas createsuperuser`. Are you logging
>>> into MAAS using that user?
>>
>> I am logging into the web interface as that user, yes. The ssh key I
>> generated was done within my normal user account (not root, not
>> 'superuser'... does superuser even has login privileges via ssh?)
>
> This is correct. We just found a bug in the documentation. An
> important part of the step after generating your ssh-key is to add
> that key in your user's MAAS preferences page. The same page used to
> get the API key you used on your .juju/environment.yaml
>
> So what I think is happening:
>
> - node boots
> - node talks to maas server and identifies itself
> - maas server sees the user associated with that node
> - node ask for the public ssh key for that user
> - It can't find the key
>  maas.log: MAASAPINotFound: No registered public keys
>  access.log: 172.23.1.161 - - [19/Apr/2012:11:38:32 -0500] "GET
> /MAAS/metadata//2012-03-01/meta-data/public-keys HTTP/1.1" 404 249 "-"
> "Python-urllib/2.7"
> - it continues booting but the nodes become inacessible to that user
> since there's no user/pass set and auth should happen through pubkey
> authentication
>
> If you have registered your ssh key into the MAAS interface, then it's
> probably another bug.
>
>>
>>>
>>>> Here's select parts of my history:
>>>>
>>>> sudo aptitude [updates were applied]
>>>> sudo maas createsuperuser
>>>> sudo apt-get install maas-dhcp
>>>> sudo vi /etc/network/interfaces
>>>> sudo ifup eth1
>>>> sudo maas-import-isos
>>>> sudo mount /var/lib/maas/ephemeral/precise/server/amd64/20120328/disk.img /mnt/
>>>> sudo su -
>>>> chroot /mnt
>>>> apt-get update
>>>> apt-get install cloud-init
>>>> exit
>>>> exit
>>>> sudo umount /mnt
>>>> sudo apt-get install juju
>>>> mkdir .juju
>>>> vi .juju/environments.yaml
>>>> ssh-keygen
>>>> juju bootstrap
>>>>  - [encountered errors here, so tried adding PPAs]
>>>> sudo apt-get install python-software-properties
>>>> sudo add-apt-repository ppa:juju/pkgs
>>>> sudo apt-get update
>>>> sudo aptitude [updates were applied]
>>>> juju bootstrap
>>>>  - [encountered errors here]
>>>> cat .juju/environments.yaml
>>>> juju --verbose bootstrap
>>>> juju bootstrap
>>>> sudo cat /var/log/maas/maas.log
>>>> sudo cat /var/log/apache2/access.log
>>>> sudo cat /var/log/apache2/error.log
>>>>
>>>> I just created a 4th node and 5th nodes and each said "Commissioning"
>>>> in the GUI before I booted them. They booted normally, generating a
>>>
>>> How are you creating the nodes? Are you creating a new VM and then
>>> adding them through the MAAS UI? Or you create the VM, pxe boot it and
>>> let MAAS enlist them automatically?
>>
>> I haven't figured out how to set it to auto-create nodes yet, so I'm
>> just entering the MAC addresses into the MAAS web interface. Full
>> automation is the end-goal though :)
>>
>>>
>>>> SSH host key and reported:
>>>> cloud-init boot finished at Thu... Up 23.35 seconds.
>>>> Followed shortly by:
>>>> Skipping profile in /etc/aparmor.d/disable: usr.sbin.rsyslogd
>>>> landscape-client is not configured...
>>>>
>>>
>>> Can you ssh into the instance?
>>
>> No:
>> $ ssh 172.23.1.5
>> Permission denied (publickey).
>> $
>>
>> I couldn't ssh into it anyway could I? I don't know the
>> username/password that the nodes use.
>
> Yes, you can only ssh into the node using public key authentication. I
> don't think there's a user/pass set
>
>>
>>> What does ssh -vvv <ip-of-a-node> reports?
>>
>> http://pastebin.ubuntu.com/937427/
>>
>>>
>>>> The Nodes screen indicated these new nodes were "Commissioning" until
>>>> I clicked home and back into Nodes. Then they indicated "Ready." I did
>>>> not see the word "Declared."
>>>
>>> Can you run:
>>>
>>> sudo maas shell
>>> from maasserver.models import Node
>>> [node.owner for node in Node.objects.all()]
>>>
>>> and paste back the result?
>>>
>>
>> $ sudo maas shell
>> Python 2.7.3 (default, Apr 10 2012, 12:29:04)
>> [GCC 4.6.3] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> (InteractiveConsole)
>>>>> from maasserver.models import Node
>>>>> [node.owner for node in Node.objects.all()]
>> [<User: mike>, <User: mike>, <User: mike>, <User: mike>, <User: mike>]
>>>>>
>>
>>
>>> Thanks,
>>>
>>> Diogo
>>>>
>>>>
>>>>
>>>> On Thu, Apr 19, 2012 at 2:04 PM, Diogo Matsubara
>>>> <matsubara@xxxxxxxxxxxxx> wrote:
>>>>> On Thu, Apr 19, 2012 at 3:48 PM, Michael Hughes <itismike@xxxxxxxxx> wrote:
>>>>>> maas.log ==> http://pastebin.ubuntu.com/937067/
>>>>>> access.log ==> http://pastebin.ubuntu.com/937084/
>>>>>> error.log ==> http://pastebin.ubuntu.com/937268/
>>>>>>
>>>>>> I don't think I'm doing anything special. Maybe I'll try reinstalling
>>>>>> the base OS again and start from the beginning.
>>>>>>
>>>>>
>>>>> Please don't reinstall. I think I've run into the same bug as you.
>>>>>
>>>>> Did you add a ssh key for the admin user?
>>>>> Could you check in any node edit page if the node has a owner assigned?
>>>>> You said that they are skipping the commissioning state and going
>>>>> directly from Declared to Ready? Is this still happening?
>>>>>
>>>>> Thank you and sorry for all this inconvenience.
>>>>>
>>>>>> On Thu, Apr 19, 2012 at 11:31 AM, Diogo Matsubara
>>>>>> <matsubara@xxxxxxxxxxxxx> wrote:
>>>>>>> On Thu, Apr 19, 2012 at 1:22 PM, Michael Hughes <itismike@xxxxxxxxx> wrote:
>>>>>>>> Thanks Diogo! I added the reference to the Ephemeral bug to the troubleshooter.
>>>>>>>>
>>>>>>>
>>>>>>> Thank you!
>>>>>>>
>>>>>>>> Verbose provides a few more lines at the bottom:
>>>>>>>> ~$ juju --verbose bootstrap
>>>>>>>> 2012-04-19 10:43:15,254 DEBUG Initializing juju bootstrap runtime
>>>>>>>> ...
>>>>>>>> Traceback (most recent call last):
>>>>>>>> Failure: juju.errors.ProviderInteractionError: Unexpected TimeoutError
>>>>>>>> interacting with provider: User timeout caused connection failure.
>>>>>>>> 2012-04-19 10:43:45,297 ERROR Traceback (most recent call last):
>>>>>>>> Failure: juju.errors.ProviderInteractionError: Unexpected TimeoutError
>>>>>>>> interacting with provider: User timeout caused connection failure.
>>>>>>>>
>>>>>>>> Unexpected TimeoutError interacting with provider: User timeout caused
>>>>>>>> connection failure.
>>>>>>>> 2012-04-19 10:43:45,299 ERROR Unexpected TimeoutError interacting with
>>>>>>>> provider: User timeout caused connection failure.
>>>>>>>> ~$
>>>>>>>
>>>>>>> When you bootstrap, do you see anything in the maas.log? How about
>>>>>>> apache's access.log and error.log?
>>>>>>> /var/log/maas/maas.log
>>>>>>> /var/log/apache2/access.log
>>>>>>> /var/log/apache2/error.log
>>>>>>>
>>>>>>> Could you paste those logs on pastebin.ubuntu.com?
>>>>>>> I'm asking the juju developers what causes this kinds of errors and
>>>>>>> will get back to you.
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> On Thu, Apr 19, 2012 at 9:45 AM, Diogo Matsubara
>>>>>>>> <matsubara@xxxxxxxxxxxxx> wrote:
>>>>>>>>> Oh, forgot to say, feel free to update
>>>>>>>>> https://wiki.ubuntu.com/ServerTeam/MAAS/Troubleshooting with any
>>>>>>>>> further debugging tips you have.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> Diogo
>>>>>>>>>
>>>>>>>>> On Thu, Apr 19, 2012 at 11:37 AM, Diogo Matsubara
>>>>>>>>> <matsubara@xxxxxxxxxxxxx> wrote:
>>>>>>>>>> Hi MIchael,
>>>>>>>>>>
>>>>>>>>>> On Thu, Apr 19, 2012 at 10:32 AM, Michael Hughes <itismike@xxxxxxxxx> wrote:
>>>>>>>>>>> Hi Jeroen. Thanks for the idea but I don't believe I've done anything
>>>>>>>>>>> fancy in my setup that would require manipulating the database by
>>>>>>>>>>> hand. My MAAS server has two NICs: one public which connects to the
>>>>>>>>>>> Internet and one private. Each node has a single NIC which is
>>>>>>>>>>> connected to that private NIC via a switch. Isn't this a pretty
>>>>>>>>>>> standard setup for a group of machines with a dedicated DHCP server
>>>>>>>>>>> like MAAS employs?
>>>>>>>>>>>
>>>>>>>>>>> Further, while I was not able to reconfigure the gateway address for
>>>>>>>>>>> the nodes by running dpkg-reconfigure maas-dhcp, I performed a full
>>>>>>>>>>> reinstall (using the steps below) the proper gateway settings
>>>>>>>>>>> (172.x.x.x) stuck just fine. Nodes now boot and appear to install a
>>>>>>>>>>> base system.
>>>>>>>>>>>
>>>>>>>>>>> There is still no 'Accept and Commission' button in the GUI for me,
>>>>>>>>>>> but the nodes now report as "Ready" rather than "Commissioning."  My
>>>>>>>>>>> next challenge is that Juju isn't bootstrapping but since I've made
>>>>>>>>>>> some progress with MAAS I figured I'd spell out what has worked for me
>>>>>>>>>>> to this point:
>>>>>>>>>>>
>>>>>>>>>>>  • Environment: 64-bit Virtualbox running on Windows 7 host
>>>>>>>>>>>  • 4 64-bit guest VMs consisting of:
>>>>>>>>>>>     - 1 MAAS server with two NICs - one bridged to the Internet, one
>>>>>>>>>>> Internal Network (intnet)
>>>>>>>>>>>     - 3 nodes each with one NIC set to Internal Network (intnet)
>>>>>>>>>>>  • Installed yesterday's daily build of precise-server-amd64.iso
>>>>>>>>>>> (18-Apr-2012 06:40) onto the MAAS server following this wiki:
>>>>>>>>>>> https://wiki.ubuntu.com/ServerTeam/MAAS
>>>>>>>>>>>  • Overrode the default gateway address for nodes and plugged in my
>>>>>>>>>>> private IP address (172.x.x.x) [BTW, the language at this step is
>>>>>>>>>>> worded to suggest leaving the guessed IP address as-is, which was
>>>>>>>>>>> incorrect for my vanilla(?) setup. Perhaps this needs tweaking. Just
>>>>>>>>>>> remove the language and describe the gateway a little better ("path to
>>>>>>>>>>> your MAAS server" rather than "path to the Internet",) and let the
>>>>>>>>>>> admin figure out which IP is appropriate)
>>>>>>>>>>>  • Copied down the MAAS management URL when provided
>>>>>>>>>>>  • Bring the system up-to-date with sudo apt-get update and sudo apt-get install
>>>>>>>>>>>  • Edit /etc/network/interfaces to add an IP address for my internal
>>>>>>>>>>> network; ifup eth1
>>>>>>>>>>>  • Return to the wiki to complete the createsuperuser and import-isos steps
>>>>>>>>>>>  • Follow the steps in comment #2 of this bug to bring ephemerals
>>>>>>>>>>> current: https://bugs.launchpad.net/ubuntu/+source/maas/+bug/981845
>>>>>>>>>>>  • Return to the wiki to add nodes
>>>>>>>>>>>
>>>>>>>>>>> At this point, I started the nodes and the nodes boot well past the
>>>>>>>>>>> previous stoppage. An auto-login processes, SSH keys are generated,
>>>>>>>>>>> and the tty output on the nodes stops with 'landscape-client is not
>>>>>>>>>>> configured, please run landscape-config.'
>>>>>>>>>>>
>>>>>>>>>>> The GUI reports that all nodes are "Ready," though none are ready for
>>>>>>>>>>> deployment:
>>>>>>>>>>> "3 nodes in this MAAS
>>>>>>>>>>> 0 nodes reserved for named deployment.
>>>>>>>>>>> 0 retired nodes not represented."
>>>>>>>>>>
>>>>>>>>>> Ready means they're ready for deployment. I think the message 0 nodes
>>>>>>>>>> reserved for named deployment is confusing you here. (This has been
>>>>>>>>>> fixed recently: https://bugs.launchpad.net/maas/+bug/979902)
>>>>>>>>>> Named deployment is when you want to deploy to a specific node, using
>>>>>>>>>> its hostname as a constraint.
>>>>>>>>>> Something like `juju deploy --constraints
>>>>>>>>>> "maas-name=node-00e081ddd11b.local" mysql`
>>>>>>>>>> More info about juju constraints can be found here:
>>>>>>>>>> https://juju.ubuntu.com/docs/constraints.html
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Clicking "Start node" in the node editing screen shows "Node Started"
>>>>>>>>>>> in the notification area, yet no changes are observed on the node's
>>>>>>>>>>> tty.
>>>>>>>>>>
>>>>>>>>>> Did you configure virsh as the power type for your nodes? I think that
>>>>>>>>>> needs to be set before you can start a node.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  • Continue to the juju wiki. However, since I'm using 12.04 I did not
>>>>>>>>>>> add the PPA as indicated in the 'Before you get going' section.
>>>>>>>>>>>
>>>>>>>>>>> Running juju on the command-line of the MAAS server returns this error:
>>>>>>>>>>> ==
>>>>>>>>>>> $ juju bootstrap
>>>>>>>>>>> 2012-04-18 18:23:50,052 INFO Bootstrapping environment 'maas' (origin:
>>>>>>>>>>> distro type: maas)...
>>>>>>>>>>> Unhandled Error
>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>  File "/usr/lib/python2.7/dist-packages/juju/control/__init__.py",
>>>>>>>>>>> line 188, in main
>>>>>>>>>>>    options.command(options)
>>>>>>>>>>>  File "/usr/lib/python2.7/dist-packages/juju/control/command.py",
>>>>>>>>>>> line 44, in __call__
>>>>>>>>>>>    reactor.run()
>>>>>>>>>>>  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py",
>>>>>>>>>>> line 1169, in run
>>>>>>>>>>>    self.mainLoop()
>>>>>>>>>>>  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py",
>>>>>>>>>>> line 1178, in mainLoop
>>>>>>>>>>>    self.runUntilCurrent()
>>>>>>>>>>> --- <exception caught here> ---
>>>>>>>>>>>  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py",
>>>>>>>>>>> line 800, in runUntilCurrent
>>>>>>>>>>>    call.func(*call.args, **call.kw)
>>>>>>>>>>>  File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py",
>>>>>>>>>>> line 362, in resolveAddress
>>>>>>>>>>>    self._setRealAddress(self.addr[0])
>>>>>>>>>>>  File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py",
>>>>>>>>>>> line 369, in _setRealAddress
>>>>>>>>>>>    self.doConnect()
>>>>>>>>>>>  File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py",
>>>>>>>>>>> line 395, in doConnect
>>>>>>>>>>>    connectResult = self.socket.connect_ex(self.realAddress)
>>>>>>>>>>>  File "/usr/lib/python2.7/socket.py", line 224, in meth
>>>>>>>>>>>    return getattr(self._sock,name)(*args)
>>>>>>>>>>> exceptions.TypeError: an integer is required
>>>>>>>>>>> Unexpected TimeoutError interacting with provider: User timeout caused
>>>>>>>>>>> connection failure.
>>>>>>>>>>> 2012-04-18 18:24:20,090 ERROR Unexpected TimeoutError interacting with
>>>>>>>>>>> provider: User timeout caused connection failure.
>>>>>>>>>>> $
>>>>>>>>>>> ==
>>>>>>>>>>>
>>>>>>>>>>> I've attempted both with and without the juju PPA but it still
>>>>>>>>>>> produces the same results.
>>>>>>>>>>
>>>>>>>>>> Does juju --verbose bootstrap gives you any more info?
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I hope this feedback is helpful and am excited to continue working on
>>>>>>>>>>> this in anticipation of the 12.04 release!
>>>>>>>>>>
>>>>>>>>>> It is very useful, please keep it coming! :-)
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Diogo
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -Mike
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Apr 18, 2012 at 11:25 PM, Jeroen Vermeulen <jtv@xxxxxxxxxxxxx> wrote:
>>>>>>>>>>>> On 2012-04-19 01:30, Michael Hughes wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Maybe I'm using am incompatible base OS. What are others testing MAAS
>>>>>>>>>>>>> with?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> In the development team we all run 12.04.
>>>>>>>>>>>>
>>>>>>>>>>>> Looks like you to change the maas_url setting to adapt to your setup. This
>>>>>>>>>>>> setting tells nodes where they can reach the MAAS service.  The URL's
>>>>>>>>>>>> hostname part defaults to the IP address of whatever interface your server
>>>>>>>>>>>> uses for its default route, but it sounds like that's not going to be right
>>>>>>>>>>>> for your network.
>>>>>>>>>>>>
>>>>>>>>>>>> I don't think we have the UI for such settings accessible yet.  But you can
>>>>>>>>>>>> change the setting directly in the database:
>>>>>>>>>>>>
>>>>>>>>>>>>    INSERT INTO maasserver_config (name, value)
>>>>>>>>>>>>    VALUES ('maas_url', 'http://<ip>/MAAS/');
>>>>>>>>>>>>
>>>>>>>>>>>> The <ip> is the address by which the nodes can reach the server.  The
>>>>>>>>>>>> “MAAS/” path is as what you'd use to get to the MAAS user interface (we use
>>>>>>>>>>>> just “/” on development setups).
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Jeroen
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Mailing list: https://launchpad.net/~maas-devel
>>>>>>>>>>> Post to     : maas-devel@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>>> Unsubscribe : https://launchpad.net/~maas-devel
>>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Diogo M. Matsubara
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Diogo M. Matsubara
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Diogo M. Matsubara
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Diogo M. Matsubara
>>>
>>>
>>>
>>> --
>>> Diogo M. Matsubara
>
>
>
> --
> Diogo M. Matsubara