maas-builds team mailing list archive

[Mike Pontillo <mike.pontillo@xxxxxxxxxxxxx>]

   Well, the benefit is that we don't have engineers and managers spending
cycles looking at the log and wondering if there was a timeout due to
network latency! (though it would be interesting to see how effective the
caching on the proxy is at mitigating this, as well - maybe it IS a non
issue.)

   I understand about always testing with the latest images, but I think we
can live with the small delay introduced by the mirroring. (should be
measured in minutes, but if it's measured in hours+, that means our users
are probably hurting too! and it would be nice to be able to measure that
separately.)

   Agree on the need for a promotion job, but that seems like a larger task
since you'd have to verify the image on a wide variety of hardware, no?

Regards,
Mike

On Thu, Jul 9, 2015 at 5:15 PM, Blake Rouse <blake.rouse@xxxxxxxxxxxxx>
wrote:

> Also a proxy is already used so those files should be cached in the test
> environment if they haven't changed.
> On Jul 9, 2015 8:14 PM, "Blake Rouse" <blake.rouse@xxxxxxxxxxxxx> wrote:
>
>> I really don't see a benefit, it would only make the import faster. We
>> never had this issue before, something in the code has affected it.
>>
>> Also the CI tests that the images work on deployed nodes. If a mirror is
>> used then that test would be delayed until the mirror is updated as well.
>> Keeping the CI in sync with maas.ubuntu.com will make sure not only that
>> the mirror is always working but also the images in the mirror actually
>> work and deploy.
>>
>> Now I do think there needs to be a CI job that takes new images from the
>> daily stream test them and then promote them to release stream.
>> On Jul 9, 2015 8:09 PM, "Mike Pontillo" <mike.pontillo@xxxxxxxxxxxxx>
>> wrote:
>>
>>> I seriously think it would be better if we created a separate job for
>>> mirroring the images, and the CI could just use the downloaded mirror.
>>>
>>> Mike
>>>
>>> On Thu, Jul 9, 2015 at 5:07 PM, Blake Rouse <blake.rouse@xxxxxxxxxxxxx>
>>> wrote:
>>>
>>>> Haha. We don't need another, we got this one. :-)
>>>> On Jul 9, 2015 8:02 PM, "Mike Pontillo" <mike.pontillo@xxxxxxxxxxxxx>
>>>> wrote:
>>>>
>>>>> Then we need another CI to test that! ;-)
>>>>>
>>>>> On Thu, Jul 9, 2015 at 4:53 PM, Blake Rouse <blake.rouse@xxxxxxxxxxxxx
>>>>> > wrote:
>>>>>
>>>>>> I disagree as using the maas.ubuntu.com images makes sure that
>>>>>> service is working as everyone else relies on it.
>>>>>> On Jul 9, 2015 7:37 PM, "Mike Pontillo" <mike.pontillo@xxxxxxxxxxxxx>
>>>>>> wrote:
>>>>>>
>>>>>>> Maybe we should change the CI to use a local mirror of the images,
>>>>>>> to make the CI faster and eliminate side effects from network latency.
>>>>>>>
>>>>>>> Mike
>>>>>>>
>>>>>>> On Tue, Jul 7, 2015 at 7:38 AM, Andres Rodriguez <
>>>>>>> andres.rodriguez@xxxxxxxxxxxxx> wrote:
>>>>>>>
>>>>>>>> Yeah, so then the only issues we need to figure out are:
>>>>>>>>
>>>>>>>> 1. Why is the region trying to access clusterd.conf
>>>>>>>> 2. Why the tests failing to pass? Is it not importing for another
>>>>>>>> issue or is it just taking longer than expected, since judging by the logs,
>>>>>>>> both the cluster and region are up.
>>>>>>>>
>>>>>>>> On Tue, Jul 7, 2015 at 9:33 AM, Gavin Panella <
>>>>>>>> gavin.panella@xxxxxxxxxxxxx> wrote:
>>>>>>>>
>>>>>>>>> On 7 July 2015 at 13:47, Andres Rodriguez
>>>>>>>>> <andres.rodriguez@xxxxxxxxxxxxx> wrote:
>>>>>>>>> > Hi Gavin,
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> > http://162.213.35.104:8080/job/maas-utopic-trunk/576/console
>>>>>>>>> >> >
>>>>>>>>> >> >
>>>>>>>>> http://162.213.35.104:8080/job/maas-utopic-trunk/576/artifact/results/artifacts/maas-logs/var/log/maas/regiond.log
>>>>>>>>> >> >
>>>>>>>>> >> >
>>>>>>>>> http://162.213.35.104:8080/job/maas-utopic-trunk/576/artifact/results/artifacts/maas-logs/var/log/maas/clusterd.log
>>>>>>>>> >> >
>>>>>>>>> >> > So, the last issue seems liek the test does not wait long
>>>>>>>>> enough for
>>>>>>>>> >> > the images to be present, however, that's caused for other
>>>>>>>>> things (see
>>>>>>>>> >> > regiond.log):
>>>>>>>>> >> >
>>>>>>>>> >> > 1. OSError: [Errno 13] Permission denied:
>>>>>>>>> '/etc/maas/regiond.conf'
>>>>>>>>> >> > 2. django.db.utils.OperationalError: fe_sendauth: no password
>>>>>>>>> supplied
>>>>>>>>> >> > 3. exceptions.OSError: [Errno 13] Permission denied:
>>>>>>>>> >> > '/etc/maas/clusterd.conf'
>>>>>>>>> >> >
>>>>>>>>> >> > So, by the looks of it, it seems that after the installation
>>>>>>>>> goes
>>>>>>>>> >> > through the regiond.conf is not with the correct permissions.
>>>>>>>>> >> > Packaging itself does not configure nor changes the
>>>>>>>>> permissions. So
>>>>>>>>> >> > questions are:
>>>>>>>>> >> > 1. Why are the daemons not creating the configs with correct
>>>>>>>>> >> > permissions?
>>>>>>>>> >>
>>>>>>>>> >> I'm investigating this first. This was working, for the region.
>>>>>>>>> At the
>>>>>>>>> >> time I had to make several changes in trunk and in packaging to
>>>>>>>>> get this
>>>>>>>>> >> working. I haven't yet figured out what has made it regress.
>>>>>>>>> >>
>>>>>>>>> >
>>>>>>>>> > After some investigation, the permissions issue is only seen for
>>>>>>>>> a
>>>>>>>>> > while, but it gets to the point that it is correct and the
>>>>>>>>> daemons can
>>>>>>>>> > read it just fine.
>>>>>>>>>
>>>>>>>>> I've figured out a fix for this.
>>>>>>>>>
>>>>>>>>> >
>>>>>>>>> >>
>>>>>>>>> >> > 2. What causes the permissions to be changed correctly later
>>>>>>>>> on?
>>>>>>>>> >> >
>>>>>>>>> >> > Then, after the daemon successfully loads regiond.conf, no
>>>>>>>>> password is
>>>>>>>>> >> > supplied to the DB:
>>>>>>>>> >> > 3. So, why does the daemon seem not to have the DB password
>>>>>>>>> if it was
>>>>>>>>> >> > supposed to be set?
>>>>>>>>> >>
>>>>>>>>> >> This is probably a symptom of #1.
>>>>>>>>>
>>>>>>>>> The database password is configured by
>>>>>>>>> maas-region-controller.postinst,
>>>>>>>>> but is needed by maas-region-controller-min which is started
>>>>>>>>> before the
>>>>>>>>> database has been configured. The errors do eventually go away.
>>>>>>>>>
>>>>>>>>> >>
>>>>>>>>> >> >
>>>>>>>>> >> > Last, but not least, the regiond is trying to access
>>>>>>>>> cluster.conf:
>>>>>>>>> >> > 4. Why is the region daemon trying to access cluster.conf ?
>>>>>>>>> This
>>>>>>>>> >> > should not be happening at all.
>>>>>>>>> >>
>>>>>>>>> >> This looks like a bug, but I'm not 100% sure about that.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > yup. IIRC, i mentioned this before. For whatever reason the
>>>>>>>>> region is
>>>>>>>>> > trying to access this.
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> >
>>>>>>>>> >> > Gavin, can you please look at the above and fix asap?
>>>>>>>>> >> >
>>>>>>>>> >> > Thanks.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > Andres Rodriguez
>>>>>>>>> > Engineering Manager, MAAS & OIL DevOps
>>>>>>>>> > Canonical USA, Inc.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Andres Rodriguez
>>>>>>>> Engineering Manager, MAAS & OIL DevOps
>>>>>>>> Canonical USA, Inc.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Mailing list: https://launchpad.net/~maas-builds
>>>>>>>> Post to     : maas-builds@xxxxxxxxxxxxxxxxxxx
>>>>>>>> Unsubscribe : https://launchpad.net/~maas-builds
>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Mailing list: https://launchpad.net/~maas-builds
>>>>>>> Post to     : maas-builds@xxxxxxxxxxxxxxxxxxx
>>>>>>> Unsubscribe : https://launchpad.net/~maas-builds
>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>
>>>>>>>
>>>>>
>>>