maas-devel team mailing list archive

[Diogo Matsubara <matsubara@xxxxxxxxxxxxx>]

On Thu, Apr 19, 2012 at 6:16 PM, Michael Hughes <itismike@xxxxxxxxx> wrote:
> No prob. Just wish the editor would have provided a reason for wiping
> out my additions.
>
> It appears to be erroring out the same way. I pasted the entire key
> into the GUI. Was I supposed to trim off the leading "ssh-rsa A" or
> the trailing " user_a@cssm-maasserver"?
>

No need to remove the leading ssh and trailing user.

Did you delete the node and added it again?

In any case I think there's a bug here, where the MAAS server allows
nodes to be initialized that won't be accessible by anyone. I'll
confirm with the devels and log a bug about it.

Thanks!

Diogo


> On Thu, Apr 19, 2012 at 4:14 PM, Diogo Matsubara
> <matsubara@xxxxxxxxxxxxx> wrote:
>> On Thu, Apr 19, 2012 at 6:09 PM, Michael Hughes <itismike@xxxxxxxxx> wrote:
>>> Funny - I think I was on to something a week ago when I added this to the wiki:
>>> https://wiki.ubuntu.com/ServerTeam/MAAS/Juju?action=recall&rev=10#Generating_SSH_keypair_for_Juju
>>>
>>> It annoys me that some heavy-handed editor reverted my contributions
>>> rather than confirming or enhancing them.
>>>
>>
>> Sorry about that. I just updated the instructions to include a step to
>> add the ssh-key to the MAAS user account.
>>
>> After adding to your account, did it work?
>>
>>> On Thu, Apr 19, 2012 at 3:53 PM, Diogo Matsubara
>>> <matsubara@xxxxxxxxxxxxx> wrote:
>>>> On Thu, Apr 19, 2012 at 5:31 PM, Michael Hughes <itismike@xxxxxxxxx> wrote:
>>>>> On Thu, Apr 19, 2012 at 3:02 PM, Diogo Matsubara
>>>>> <matsubara@xxxxxxxxxxxxx> wrote:
>>>>>> On Thu, Apr 19, 2012 at 4:40 PM, Michael Hughes <itismike@xxxxxxxxx> wrote:
>>>>>>> Okay I'll hang tight on the reinstall. Yes, I added an ssh key.
>>>>>>> [edit - I just re-read your comment a little closer. You want me to
>>>>>>> add an ssh key for the ADMIN user? As in root?]
>>>>>>>
>>>>>>
>>>>>> For the user created by `sudo maas createsuperuser`. Are you logging
>>>>>> into MAAS using that user?
>>>>>
>>>>> I am logging into the web interface as that user, yes. The ssh key I
>>>>> generated was done within my normal user account (not root, not
>>>>> 'superuser'... does superuser even has login privileges via ssh?)
>>>>
>>>> This is correct. We just found a bug in the documentation. An
>>>> important part of the step after generating your ssh-key is to add
>>>> that key in your user's MAAS preferences page. The same page used to
>>>> get the API key you used on your .juju/environment.yaml
>>>>
>>>> So what I think is happening:
>>>>
>>>> - node boots
>>>> - node talks to maas server and identifies itself
>>>> - maas server sees the user associated with that node
>>>> - node ask for the public ssh key for that user
>>>> - It can't find the key
>>>>  maas.log: MAASAPINotFound: No registered public keys
>>>>  access.log: 172.23.1.161 - - [19/Apr/2012:11:38:32 -0500] "GET
>>>> /MAAS/metadata//2012-03-01/meta-data/public-keys HTTP/1.1" 404 249 "-"
>>>> "Python-urllib/2.7"
>>>> - it continues booting but the nodes become inacessible to that user
>>>> since there's no user/pass set and auth should happen through pubkey
>>>> authentication
>>>>
>>>> If you have registered your ssh key into the MAAS interface, then it's
>>>> probably another bug.
>>>>
>>>>>
>>>>>>
>>>>>>> Here's select parts of my history:
>>>>>>>
>>>>>>> sudo aptitude [updates were applied]
>>>>>>> sudo maas createsuperuser
>>>>>>> sudo apt-get install maas-dhcp
>>>>>>> sudo vi /etc/network/interfaces
>>>>>>> sudo ifup eth1
>>>>>>> sudo maas-import-isos
>>>>>>> sudo mount /var/lib/maas/ephemeral/precise/server/amd64/20120328/disk.img /mnt/
>>>>>>> sudo su -
>>>>>>> chroot /mnt
>>>>>>> apt-get update
>>>>>>> apt-get install cloud-init
>>>>>>> exit
>>>>>>> exit
>>>>>>> sudo umount /mnt
>>>>>>> sudo apt-get install juju
>>>>>>> mkdir .juju
>>>>>>> vi .juju/environments.yaml
>>>>>>> ssh-keygen
>>>>>>> juju bootstrap
>>>>>>>  - [encountered errors here, so tried adding PPAs]
>>>>>>> sudo apt-get install python-software-properties
>>>>>>> sudo add-apt-repository ppa:juju/pkgs
>>>>>>> sudo apt-get update
>>>>>>> sudo aptitude [updates were applied]
>>>>>>> juju bootstrap
>>>>>>>  - [encountered errors here]
>>>>>>> cat .juju/environments.yaml
>>>>>>> juju --verbose bootstrap
>>>>>>> juju bootstrap
>>>>>>> sudo cat /var/log/maas/maas.log
>>>>>>> sudo cat /var/log/apache2/access.log
>>>>>>> sudo cat /var/log/apache2/error.log
>>>>>>>
>>>>>>> I just created a 4th node and 5th nodes and each said "Commissioning"
>>>>>>> in the GUI before I booted them. They booted normally, generating a
>>>>>>
>>>>>> How are you creating the nodes? Are you creating a new VM and then
>>>>>> adding them through the MAAS UI? Or you create the VM, pxe boot it and
>>>>>> let MAAS enlist them automatically?
>>>>>
>>>>> I haven't figured out how to set it to auto-create nodes yet, so I'm
>>>>> just entering the MAC addresses into the MAAS web interface. Full
>>>>> automation is the end-goal though :)
>>>>>
>>>>>>
>>>>>>> SSH host key and reported:
>>>>>>> cloud-init boot finished at Thu... Up 23.35 seconds.
>>>>>>> Followed shortly by:
>>>>>>> Skipping profile in /etc/aparmor.d/disable: usr.sbin.rsyslogd
>>>>>>> landscape-client is not configured...
>>>>>>>
>>>>>>
>>>>>> Can you ssh into the instance?
>>>>>
>>>>> No:
>>>>> $ ssh 172.23.1.5
>>>>> Permission denied (publickey).
>>>>> $
>>>>>
>>>>> I couldn't ssh into it anyway could I? I don't know the
>>>>> username/password that the nodes use.
>>>>
>>>> Yes, you can only ssh into the node using public key authentication. I
>>>> don't think there's a user/pass set
>>>>
>>>>>
>>>>>> What does ssh -vvv <ip-of-a-node> reports?
>>>>>
>>>>> http://pastebin.ubuntu.com/937427/
>>>>>
>>>>>>
>>>>>>> The Nodes screen indicated these new nodes were "Commissioning" until
>>>>>>> I clicked home and back into Nodes. Then they indicated "Ready." I did
>>>>>>> not see the word "Declared."
>>>>>>
>>>>>> Can you run:
>>>>>>
>>>>>> sudo maas shell
>>>>>> from maasserver.models import Node
>>>>>> [node.owner for node in Node.objects.all()]
>>>>>>
>>>>>> and paste back the result?
>>>>>>
>>>>>
>>>>> $ sudo maas shell
>>>>> Python 2.7.3 (default, Apr 10 2012, 12:29:04)
>>>>> [GCC 4.6.3] on linux2
>>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>> (InteractiveConsole)
>>>>>>>> from maasserver.models import Node
>>>>>>>> [node.owner for node in Node.objects.all()]
>>>>> [<User: mike>, <User: mike>, <User: mike>, <User: mike>, <User: mike>]
>>>>>>>>
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Diogo
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Apr 19, 2012 at 2:04 PM, Diogo Matsubara
>>>>>>> <matsubara@xxxxxxxxxxxxx> wrote:
>>>>>>>> On Thu, Apr 19, 2012 at 3:48 PM, Michael Hughes <itismike@xxxxxxxxx> wrote:
>>>>>>>>> maas.log ==> http://pastebin.ubuntu.com/937067/
>>>>>>>>> access.log ==> http://pastebin.ubuntu.com/937084/
>>>>>>>>> error.log ==> http://pastebin.ubuntu.com/937268/
>>>>>>>>>
>>>>>>>>> I don't think I'm doing anything special. Maybe I'll try reinstalling
>>>>>>>>> the base OS again and start from the beginning.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Please don't reinstall. I think I've run into the same bug as you.
>>>>>>>>
>>>>>>>> Did you add a ssh key for the admin user?
>>>>>>>> Could you check in any node edit page if the node has a owner assigned?
>>>>>>>> You said that they are skipping the commissioning state and going
>>>>>>>> directly from Declared to Ready? Is this still happening?
>>>>>>>>
>>>>>>>> Thank you and sorry for all this inconvenience.
>>>>>>>>
>>>>>>>>> On Thu, Apr 19, 2012 at 11:31 AM, Diogo Matsubara
>>>>>>>>> <matsubara@xxxxxxxxxxxxx> wrote:
>>>>>>>>>> On Thu, Apr 19, 2012 at 1:22 PM, Michael Hughes <itismike@xxxxxxxxx> wrote:
>>>>>>>>>>> Thanks Diogo! I added the reference to the Ephemeral bug to the troubleshooter.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thank you!
>>>>>>>>>>
>>>>>>>>>>> Verbose provides a few more lines at the bottom:
>>>>>>>>>>> ~$ juju --verbose bootstrap
>>>>>>>>>>> 2012-04-19 10:43:15,254 DEBUG Initializing juju bootstrap runtime
>>>>>>>>>>> ...
>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>> Failure: juju.errors.ProviderInteractionError: Unexpected TimeoutError
>>>>>>>>>>> interacting with provider: User timeout caused connection failure.
>>>>>>>>>>> 2012-04-19 10:43:45,297 ERROR Traceback (most recent call last):
>>>>>>>>>>> Failure: juju.errors.ProviderInteractionError: Unexpected TimeoutError
>>>>>>>>>>> interacting with provider: User timeout caused connection failure.
>>>>>>>>>>>
>>>>>>>>>>> Unexpected TimeoutError interacting with provider: User timeout caused
>>>>>>>>>>> connection failure.
>>>>>>>>>>> 2012-04-19 10:43:45,299 ERROR Unexpected TimeoutError interacting with
>>>>>>>>>>> provider: User timeout caused connection failure.
>>>>>>>>>>> ~$
>>>>>>>>>>
>>>>>>>>>> When you bootstrap, do you see anything in the maas.log? How about
>>>>>>>>>> apache's access.log and error.log?
>>>>>>>>>> /var/log/maas/maas.log
>>>>>>>>>> /var/log/apache2/access.log
>>>>>>>>>> /var/log/apache2/error.log
>>>>>>>>>>
>>>>>>>>>> Could you paste those logs on pastebin.ubuntu.com?
>>>>>>>>>> I'm asking the juju developers what causes this kinds of errors and
>>>>>>>>>> will get back to you.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Mike
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Apr 19, 2012 at 9:45 AM, Diogo Matsubara
>>>>>>>>>>> <matsubara@xxxxxxxxxxxxx> wrote:
>>>>>>>>>>>> Oh, forgot to say, feel free to update
>>>>>>>>>>>> https://wiki.ubuntu.com/ServerTeam/MAAS/Troubleshooting with any
>>>>>>>>>>>> further debugging tips you have.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> Diogo
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Apr 19, 2012 at 11:37 AM, Diogo Matsubara
>>>>>>>>>>>> <matsubara@xxxxxxxxxxxxx> wrote:
>>>>>>>>>>>>> Hi MIchael,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Apr 19, 2012 at 10:32 AM, Michael Hughes <itismike@xxxxxxxxx> wrote:
>>>>>>>>>>>>>> Hi Jeroen. Thanks for the idea but I don't believe I've done anything
>>>>>>>>>>>>>> fancy in my setup that would require manipulating the database by
>>>>>>>>>>>>>> hand. My MAAS server has two NICs: one public which connects to the
>>>>>>>>>>>>>> Internet and one private. Each node has a single NIC which is
>>>>>>>>>>>>>> connected to that private NIC via a switch. Isn't this a pretty
>>>>>>>>>>>>>> standard setup for a group of machines with a dedicated DHCP server
>>>>>>>>>>>>>> like MAAS employs?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Further, while I was not able to reconfigure the gateway address for
>>>>>>>>>>>>>> the nodes by running dpkg-reconfigure maas-dhcp, I performed a full
>>>>>>>>>>>>>> reinstall (using the steps below) the proper gateway settings
>>>>>>>>>>>>>> (172.x.x.x) stuck just fine. Nodes now boot and appear to install a
>>>>>>>>>>>>>> base system.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> There is still no 'Accept and Commission' button in the GUI for me,
>>>>>>>>>>>>>> but the nodes now report as "Ready" rather than "Commissioning."  My
>>>>>>>>>>>>>> next challenge is that Juju isn't bootstrapping but since I've made
>>>>>>>>>>>>>> some progress with MAAS I figured I'd spell out what has worked for me
>>>>>>>>>>>>>> to this point:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  • Environment: 64-bit Virtualbox running on Windows 7 host
>>>>>>>>>>>>>>  • 4 64-bit guest VMs consisting of:
>>>>>>>>>>>>>>     - 1 MAAS server with two NICs - one bridged to the Internet, one
>>>>>>>>>>>>>> Internal Network (intnet)
>>>>>>>>>>>>>>     - 3 nodes each with one NIC set to Internal Network (intnet)
>>>>>>>>>>>>>>  • Installed yesterday's daily build of precise-server-amd64.iso
>>>>>>>>>>>>>> (18-Apr-2012 06:40) onto the MAAS server following this wiki:
>>>>>>>>>>>>>> https://wiki.ubuntu.com/ServerTeam/MAAS
>>>>>>>>>>>>>>  • Overrode the default gateway address for nodes and plugged in my
>>>>>>>>>>>>>> private IP address (172.x.x.x) [BTW, the language at this step is
>>>>>>>>>>>>>> worded to suggest leaving the guessed IP address as-is, which was
>>>>>>>>>>>>>> incorrect for my vanilla(?) setup. Perhaps this needs tweaking. Just
>>>>>>>>>>>>>> remove the language and describe the gateway a little better ("path to
>>>>>>>>>>>>>> your MAAS server" rather than "path to the Internet",) and let the
>>>>>>>>>>>>>> admin figure out which IP is appropriate)
>>>>>>>>>>>>>>  • Copied down the MAAS management URL when provided
>>>>>>>>>>>>>>  • Bring the system up-to-date with sudo apt-get update and sudo apt-get install
>>>>>>>>>>>>>>  • Edit /etc/network/interfaces to add an IP address for my internal
>>>>>>>>>>>>>> network; ifup eth1
>>>>>>>>>>>>>>  • Return to the wiki to complete the createsuperuser and import-isos steps
>>>>>>>>>>>>>>  • Follow the steps in comment #2 of this bug to bring ephemerals
>>>>>>>>>>>>>> current: https://bugs.launchpad.net/ubuntu/+source/maas/+bug/981845
>>>>>>>>>>>>>>  • Return to the wiki to add nodes
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> At this point, I started the nodes and the nodes boot well past the
>>>>>>>>>>>>>> previous stoppage. An auto-login processes, SSH keys are generated,
>>>>>>>>>>>>>> and the tty output on the nodes stops with 'landscape-client is not
>>>>>>>>>>>>>> configured, please run landscape-config.'
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The GUI reports that all nodes are "Ready," though none are ready for
>>>>>>>>>>>>>> deployment:
>>>>>>>>>>>>>> "3 nodes in this MAAS
>>>>>>>>>>>>>> 0 nodes reserved for named deployment.
>>>>>>>>>>>>>> 0 retired nodes not represented."
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ready means they're ready for deployment. I think the message 0 nodes
>>>>>>>>>>>>> reserved for named deployment is confusing you here. (This has been
>>>>>>>>>>>>> fixed recently: https://bugs.launchpad.net/maas/+bug/979902)
>>>>>>>>>>>>> Named deployment is when you want to deploy to a specific node, using
>>>>>>>>>>>>> its hostname as a constraint.
>>>>>>>>>>>>> Something like `juju deploy --constraints
>>>>>>>>>>>>> "maas-name=node-00e081ddd11b.local" mysql`
>>>>>>>>>>>>> More info about juju constraints can be found here:
>>>>>>>>>>>>> https://juju.ubuntu.com/docs/constraints.html
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Clicking "Start node" in the node editing screen shows "Node Started"
>>>>>>>>>>>>>> in the notification area, yet no changes are observed on the node's
>>>>>>>>>>>>>> tty.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Did you configure virsh as the power type for your nodes? I think that
>>>>>>>>>>>>> needs to be set before you can start a node.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  • Continue to the juju wiki. However, since I'm using 12.04 I did not
>>>>>>>>>>>>>> add the PPA as indicated in the 'Before you get going' section.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Running juju on the command-line of the MAAS server returns this error:
>>>>>>>>>>>>>> ==
>>>>>>>>>>>>>> $ juju bootstrap
>>>>>>>>>>>>>> 2012-04-18 18:23:50,052 INFO Bootstrapping environment 'maas' (origin:
>>>>>>>>>>>>>> distro type: maas)...
>>>>>>>>>>>>>> Unhandled Error
>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>  File "/usr/lib/python2.7/dist-packages/juju/control/__init__.py",
>>>>>>>>>>>>>> line 188, in main
>>>>>>>>>>>>>>    options.command(options)
>>>>>>>>>>>>>>  File "/usr/lib/python2.7/dist-packages/juju/control/command.py",
>>>>>>>>>>>>>> line 44, in __call__
>>>>>>>>>>>>>>    reactor.run()
>>>>>>>>>>>>>>  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py",
>>>>>>>>>>>>>> line 1169, in run
>>>>>>>>>>>>>>    self.mainLoop()
>>>>>>>>>>>>>>  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py",
>>>>>>>>>>>>>> line 1178, in mainLoop
>>>>>>>>>>>>>>    self.runUntilCurrent()
>>>>>>>>>>>>>> --- <exception caught here> ---
>>>>>>>>>>>>>>  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py",
>>>>>>>>>>>>>> line 800, in runUntilCurrent
>>>>>>>>>>>>>>    call.func(*call.args, **call.kw)
>>>>>>>>>>>>>>  File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py",
>>>>>>>>>>>>>> line 362, in resolveAddress
>>>>>>>>>>>>>>    self._setRealAddress(self.addr[0])
>>>>>>>>>>>>>>  File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py",
>>>>>>>>>>>>>> line 369, in _setRealAddress
>>>>>>>>>>>>>>    self.doConnect()
>>>>>>>>>>>>>>  File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py",
>>>>>>>>>>>>>> line 395, in doConnect
>>>>>>>>>>>>>>    connectResult = self.socket.connect_ex(self.realAddress)
>>>>>>>>>>>>>>  File "/usr/lib/python2.7/socket.py", line 224, in meth
>>>>>>>>>>>>>>    return getattr(self._sock,name)(*args)
>>>>>>>>>>>>>> exceptions.TypeError: an integer is required
>>>>>>>>>>>>>> Unexpected TimeoutError interacting with provider: User timeout caused
>>>>>>>>>>>>>> connection failure.
>>>>>>>>>>>>>> 2012-04-18 18:24:20,090 ERROR Unexpected TimeoutError interacting with
>>>>>>>>>>>>>> provider: User timeout caused connection failure.
>>>>>>>>>>>>>> $
>>>>>>>>>>>>>> ==
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've attempted both with and without the juju PPA but it still
>>>>>>>>>>>>>> produces the same results.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Does juju --verbose bootstrap gives you any more info?
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I hope this feedback is helpful and am excited to continue working on
>>>>>>>>>>>>>> this in anticipation of the 12.04 release!
>>>>>>>>>>>>>
>>>>>>>>>>>>> It is very useful, please keep it coming! :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Diogo
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Mike
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Apr 18, 2012 at 11:25 PM, Jeroen Vermeulen <jtv@xxxxxxxxxxxxx> wrote:
>>>>>>>>>>>>>>> On 2012-04-19 01:30, Michael Hughes wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Maybe I'm using am incompatible base OS. What are others testing MAAS
>>>>>>>>>>>>>>>> with?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In the development team we all run 12.04.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Looks like you to change the maas_url setting to adapt to your setup. This
>>>>>>>>>>>>>>> setting tells nodes where they can reach the MAAS service.  The URL's
>>>>>>>>>>>>>>> hostname part defaults to the IP address of whatever interface your server
>>>>>>>>>>>>>>> uses for its default route, but it sounds like that's not going to be right
>>>>>>>>>>>>>>> for your network.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I don't think we have the UI for such settings accessible yet.  But you can
>>>>>>>>>>>>>>> change the setting directly in the database:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    INSERT INTO maasserver_config (name, value)
>>>>>>>>>>>>>>>    VALUES ('maas_url', 'http://<ip>/MAAS/');
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The <ip> is the address by which the nodes can reach the server.  The
>>>>>>>>>>>>>>> “MAAS/” path is as what you'd use to get to the MAAS user interface (we use
>>>>>>>>>>>>>>> just “/” on development setups).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Jeroen
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Mailing list: https://launchpad.net/~maas-devel
>>>>>>>>>>>>>> Post to     : maas-devel@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>>>>>> Unsubscribe : https://launchpad.net/~maas-devel
>>>>>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Diogo M. Matsubara
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Diogo M. Matsubara
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Diogo M. Matsubara
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Diogo M. Matsubara
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Diogo M. Matsubara
>>>>
>>>>
>>>>
>>>> --
>>>> Diogo M. Matsubara
>>
>>
>>
>> --
>> Diogo M. Matsubara



-- 
Diogo M. Matsubara