libravatar-fans team mailing list archive

[Lars Kruse <lists@xxxxxxxxxxxxx>]

Hello,

sadly I realized only now, that I forgot to send the protocol of the last
meeting (2019-01-27).

But here it is:
* attached is the full log
* the summary is in the wiki:
  https://wiki.libravatar.org/shutdown-coordination/?updated#index2h2

See you for the next meeting in about two hours *today*!
(#libravatar channel on freenode)

Cheers,
Lars

20:01 <sumpfralle2> It is Sunday again! :)
20:01 <clime> yes!
20:04 <sumpfralle2> So I guess, we can start.
20:04 <sumpfralle2> Does someone have news to share about the last two weeks?
20:05 <clime> i didn't do much last weeks
20:05 <clime> except the discussion about the domain ownership on mailing list
20:05 <sumpfralle2> Yes, that was a good start.
20:06 <sumpfralle2> We are waiting for results regarding the sposorship request by gandi, correct?
20:06 <sumpfralle2> n
20:06 <clime> basically, I would like the libravatar group to own the domain
20:06 <clime> yes
20:06 <clime> until then i would probably wait with the new service official start
20:06 <sumpfralle2> Sounds reasonable.
20:07 <clime> good things take time :)
20:07 <sumpfralle2> They are worth it :)
20:07 <sumpfralle2> The only other topic is the need for a "plan", or?
20:08 <clime> sumpfralle2: do you know if i have permissions to edit the wiki?
20:08 <sumpfralle2> everyone has
20:08 <@fmarier> So is the domain transfer the main blocker to the migration now?
20:08 <clime> ok, cool
20:08 <sumpfralle2> you request a temporary login after clicking at "edit"
20:08 <clime> well, i would say so
20:08 <@fmarier> dev@xxxxxxxxxxxxxx is the admin account on the wiki
20:08 <sumpfralle2> good to know - I will add this to the wiki
20:09 <sumpfralle2> blocker: the fine-grained plan for the migration to the new instance is the other one, I guess
20:13 <clime> personally, i would just switch the domain to new ip and when it's done on dns side, i would immediatelly setup the ssl certs
20:13 <clime> i will also need if fmarier dumps me the latest data by the export script shortly before that.
20:14 <clime> there might some more optimal/fancy way to do it but this would work as well.
20:14 <@fmarier> I think ofalk has the access he needs to export the current data
20:14 <@fmarier> He's done it before without my help.
20:15 <sumpfralle2> I think, we should plan it in a way that would allow us to switch back to the old instance, if something interesting fails.
20:15 <clime> ok, cool
20:15 <sumpfralle2> But I can also live with the daring mode of just doing it and expecting a day of issues.
20:15 <sumpfralle2> (probably to be fixed by you and ofalk)
20:16 <clime> yup
20:17 <sumpfralle2> Should we decide here, which approach we take?
20:17 <clime> why not
20:18 <sumpfralle2> good
20:18 <sumpfralle2> So what amount of problems could we expect?
20:18 <clime> i can look at if it is possible to setup the ssl certs in advance, you mentioned some option to do that
20:18 <clime> through dns record...that would be interesting
20:19 <@fmarier> You could just copy the existing cert from the current server.
20:19 <clime> oh ye, sounds good
20:19 <clime> is it let's encrypt?
20:19 <sumpfralle2> It uses the dns-01 challenge of letsencrypt. I do not really like this approach, since you need to store an API key with write permissions for the domain data on the host.
20:19 <sumpfralle2> Thus the copy is the best approach, I guess.
20:20 <clime> ok, i see
20:20 <@fmarier> Yes, it's letsencrypt
20:20 <clime> ok
20:20 <@fmarier> It uses certbot and the webroot plugin I believe.
20:21 <clime> sure
20:21 <@fmarier> With a clever hack for the mirrors.
20:21 <sumpfralle2> Personally I prefer dehydrated for its ease of administration. Just an opinion.
20:21 <sumpfralle2> fmarier: could you elaborate the hack a bit? :)
20:22 <clime> fmarier: are these mirrors by community?
20:22 <@fmarier> https://feeding.cloud.geek.nz/posts/proxy-acme-challenges-to-single-machine/
20:23 <@fmarier> The config for the main server also involved a small amount of cleverness: https://feeding.cloud.geek.nz/posts/redirecting-entire-site-except-certbot-webroot/
20:24 <@fmarier> Yes, the mirrors can be run by anybody.
20:24 <@fmarier> Currently there's only one though.
20:24 <@fmarier> The basic idea is that mirrors proxy the letsencrypt verification ping to the main server.
20:24 <@fmarier> The main server gets a cert for seccdn.libravatar.org and then scp's it over to the mirrors.
20:24 <sumpfralle2> But they need access to the private certificate data, or?
20:24 <sumpfralle2> ok
20:25 <sumpfralle2> This model trusts the mirrors quite a lot :)
20:26 <@fmarier> Yes, each mirror has the same private cert for seccdn.libravatar.org
20:26 <@fmarier> But that domain is not security-critical since it's only serving static images.
20:26 <sumpfralle2> ah - ok. I forgt.
20:26 <sumpfralle2> forgot
20:27 <clime> fmarier: about the mirror, the ivatar won't probably be compatible with libravatar software with respect to the sync script
20:28 <sumpfralle2> The certificate sync script?
20:28 <clime> i think we could fix it along the way
20:28 <clime> i mean for the static images
20:28 <sumpfralle2> due to rsync, yes
20:28 <clime> ivatar stores them in db, instead of filesystem
20:29 <clime> i don't know who runs the mirror but maybe we could communicate it with them.
20:31 <@fmarier> I thought the plan was to do away with the mirrors entirely and just have things served from a single server?
20:32 <sumpfralle2> I also have a vague memory of this kind.
20:33 <clime> yup, well, we can do it that way
20:33 <clime> i mean, the mirroring is not probably needed (for now) unless the load won't get super high :)
20:34 <sumpfralle2> It is also about community building - but this specific feature is tricky now. So probably we have to skip it.
20:34 <clime> yeah, i just wanted to say that we can make that work again if want to at some point
20:35 <sumpfralle2> Regarding the migration plan: I think I would either reduce the TTL of the relevant domain entries to a low value (for quick back-and-forth changes in case of problems) or I would configure a proxy setup on the original host that redirects requests to the new instance.
20:35 <sumpfralle2> Both approaches would allow to go back instantly, if problems arise.
20:35 <@fmarier> It doesn't look to me like the new architecture is designed for distributed mirrors like the old one.
20:36 <clime> sumpfralle2: oh, okay, i like the redirect to the new instance
20:36 <sumpfralle2> fmarier: I have the same feeling. More modern in some regard - but with drawbacks.
20:36 <clime> or the proxy rather but that would need some work on fmarier side i think
20:37 <sumpfralle2> I could do that, if we want it.
20:37 <clime> fmarier: i don't think it is impossible to dump images from db
20:37 <clime> and rsync them then
20:37 <sumpfralle2> But the timely synchronization would not be efficient/quick.
20:37 <clime> but ye, we could then also look at db replication for it.
20:37 <sumpfralle2> But yes: doable.
20:38 <@fmarier> To be honest if the main server is big enough, the lack of mirrors shouldn't be a problem.
20:38 <clime> true
20:39 <clime> well, it has 8GB ram and 4 procs
20:39 <sumpfralle2> plenty, I guess
20:39 <sumpfralle2> Regarding the proxy configuration: we can also do it on the new instance. Then we would just change the DNS entries to the new one and later replace the proxy setup with direct access to ivatar.
20:40 <sumpfralle2> (talking about days - not weeks - for the proxy operation)
20:40 <@fmarier> Are you talking about the letsencrypt proxy?
20:40 <clime> ye, well, personally, i would prefer just a switch and fix if something doesn't work but maybe it's just me
20:41 <clime> i expect everything to work :)
20:41 <clime> but probably i am naive
20:41 <@fmarier> That's only needed because of the mirrors. Once they're out of the way, the server can just get all of the certs without any proxying.
20:41 <clime> ofalk should be around...
20:42 <@fmarier> What timezone are clime and ofalk in?
20:42 <clime> i think we are both in UTC+1
20:42 <clime> +-1
20:42 <clime> hour
20:42 <sumpfralle2> Regarding emergency switching: just a matter of taste. I like these things. But it is not necessary.
20:42 <clime> right
20:43 <@fmarier> I would suggest still having a written plan for the steps you will follow during the switch, even if you don't plan on switching back to the old server.
20:43 <clime> ok
20:43 <@fmarier> And starting early if it might involve some live-patching :)
20:44 <clime> alright, i agree
20:44 <@fmarier> Having a written plan has made this whole process almost stress-free for the 3-4 times I did a server migration.
20:44 <clime> i will put up some plan on mailing list then but it will be super simple and waiting for improvevemnt if any
20:45 <@fmarier> You can start from https://feeding.cloud.geek.nz/posts/server-migration-plan/ if you don
20:45 <sumpfralle2> A conservative approach could include disabling write access to both instances for the duration of the change. But this may not be trivial. Thus maybe just leave it out.
20:45 <@fmarier> t want to start from scratch.
20:46 <clime> i think disabling write access would be nice for the old instance if it is doable
20:46 <@fmarier> Some of the stuff won't apply (mirrors), but most of it will, I suspect.
20:46 <@fmarier> Essentially, that plan gives you a zero-downtime migration no matter how long it takes to bring the new server up.
20:46 <@fmarier> The only thing that's down is the application server. The images continue to get served by the mirrors.
20:47 <@fmarier> So sites that use Libravatar don't notice anything.
20:47 <clime> right ok
20:47 <@fmarier> It also handles the fact that DNS servers take a long time to figure out that the server has moved.
20:47 <@fmarier> It's surprising how many of them ignore TTLs.
20:48 <clime> fmarier: ok, thank you
20:50 <@fmarier> If you want, one way we could do the migration is that I would prepare the old server, disabling mirror sync'ing, put the application server in maintenance / static mode.
20:50 <clime> sounds good to me
20:50 <@fmarier> Then you guys wake up and get the new server ready.
20:51 <clime> that sounds doable :)
20:51 <@fmarier> When I wake up, we flip the switch and have the new server proxy all traffic for the DNS stragglers.
20:51 <@fmarier> (I'm on Pacific time, currently UTC-8.)
20:51 <clime> oh okay
20:52 <sumpfralle2> clime: in case you can plan the time of the migration (i.e. tell it to the mailing list in advance), then maybe people will want to follow it here on the irc channel and maybe jump in if something interesting happens?
20:52 <clime> sumpfralle2: yes, that is a good idea
20:52 <sumpfralle2> (not really necessary - but it would be nice)
20:52 <@fmarier> If we do it like this, it means that the application server will be down for up to 24 hours, but the image-serving will never be interrupted.
20:53 <clime> no, for sure, i'll be on irc here
20:53 <clime> fmarier: right
20:53 <sumpfralle2> So it sounds like we have a plan :)
20:53 <clime> great!
20:54 <sumpfralle2> So we meet again two weeks and maybe at that moment the world already enjoys a fancy new libravatar implementation ...
20:54 <sumpfralle2> "in two weeks"
20:55 <@fmarier> Thinking about it, transferring the domain doesn't have to block the migration. If I'm involved as above, then I can do the DNS changes.
20:55 <clime> ye, well, if we can sort out the domain thing next week, i guess the week after we could do the transition or maybe have one more meeting and do it e.g. on Monday after it.
20:55 <@fmarier> We just have to transfer the domain before May. That's when it needs to be renewed.
20:55 <clime> fmarier: yup, that's probably true
20:56 <clime> but i would probably do the migration in 14 days anyway
20:56 <sumpfralle2> Good.
20:57 <clime> 11.2. i would say
20:57 <clime> as "preliminary" date
20:57 <clime> if you agree
20:58 <@fmarier> On a Monday instead of during the weekend?
20:58 <clime> i mean we could probably do it earlier but i would like to have time for little bit more testing
20:58 <clime> well, right, maybe weekend would be better
20:59 <clime> so then probably 9.2. ?
21:00 <clime> personally i wouldn't mind Monday but i don't know about ofalk
21:00 <@fmarier> https://framadate.org/6M4LoPdqn3VNHoun
21:00 <@fmarier> Put your availability / preferences on here. Hopefully ofalk can do the same too.
21:02 <clime> well, 11 Feb is kind of missing for me that would e.g. be preferred
21:02 <@fmarier> I can't really do Mondays, but I'll it.
21:02 <clime> alright
21:03 <clime> thank you
21:03 <@fmarier> Sadly, they're not sorted anymore, but I've added all of the Mondays.
21:03 <@fmarier> Turns out 18 Feb is a public holiday for me, so I could do that.
21:04 <clime> right cool
21:05 <@fmarier> I will have to drop off soon. Anything else on the agenda?
21:05 <clime> not here
21:06 <clime> fmarier: just wanted to say that i you are going to be super welcome if you stay involved in libravatar
21:07 <@fmarier> thanks :)