libravatar-fans team mailing list archive

[Lars Kruse <lists@xxxxxxxxxxxxx>]

Hello,

"it" is approaching quickly: tomorrow is the planned day of migration from the
old libravatar implementation to the new one.

*** Cheers for everyone working towards this huge jump! ***


Lot of details about the specific steps were discussed during the IRC meeting
of last Sunday.
*shame on me again*: I forgot to send and summarize that IRC meeting. See the
attached log, if you want to go through the details. We mainly discussed the
migration details.

Today there will be the final meeting before the migration.
It will start at 19:00 UTC in #libravatar on freenode.


Cheers,
Lars

PS: Sadly today I will not manage to take part in that meeting. Have fun!

20:02 --> clime (~clime@xxxxxxxxxxxxxxxxxxxxxxxxxxxx) hat den Channel #libravatar betreten
20:02 <clime> hey, sry i am late
20:02 <nipos> hello
20:02 <falko> salut!
20:02 <falko> don't worry, nothing happened yet.
20:03 <clime> falko, nipos hi!
20:03 <falko> o/ clime nipos
20:04 <clime> i am opening some merge requests on gitlab for you
20:04 <falko> clime, shall i be worried or happy? ;-)
20:04 <clime> you can have a look
20:05 <clime> well...
20:05 <clime> both i guess:)
20:05 <falko> clime, ack!
20:05 <falko> clime, have you been at devconf? i didn't see you there.
20:05 <clime> falko: this issue is kind of worrying: https://git.linux-kernel.at/oliver/ivatar/issues/35
20:05 <clime> ye, i was there
20:06 <clime> i had one lightning talk btw
20:06 <sumpfralle3> good evening!
20:06 <falko> omg.... it's starting to get too big. :-{
20:06 <sumpfralle3> :)
20:06 <falko> o/ sumpfralle3!
20:06 <clime> sumpfralle3: hey!
20:07 <sumpfralle3> We are getting closer to the switch - I hope, everyone is a bit scared! :)
20:07 <falko> sumpfralle3, a bit??? you have no idea.....
20:07 <clime> haha
20:08 <clime> ofalk will be hotfixing stuff next week :)
20:08 <nipos> I'm not scared.I'm sure everything will work perfect and I'm happy that we can finally do the switch
20:08 <falko> in order to test the export on the live system, i'd need rights there. it's in the pad on riseup.
20:09 <falko> clime, i will, probably. :-)
20:09 <sumpfralle3> "rights on the live system": you have only limited access at the moment?
20:09 <clime> falko: will you be able to do the export and just dump me the data on the new production serv?
20:09 <falko> clime, prod server is ready?
20:10 <clime> i can do it too if you show me how to do the export
20:10 <clime> falko: well, it is not yet switched to the new domain
20:10 <falko> sumpfralle3, yes i have no sudo there. at least i believe. i tried a few minutes ago.
20:10 <clime> so httpd and postfix contain still references libravatar.fedorainfracloud.org
20:10 <clime> but apart from that it is ready (imho)
20:11 <clime> i will do this last bit during the migration
20:11 <falko> the script is pretty easy. i'll run it and copy over the data and run the import. would be good, if you, clime, would take care about apache + ssl.
20:11 <clime> sure
20:12 <clime> i will need to get the current ssl certs from old prod
20:12 <clime> so maybe fmarier can copy me it to the new prod
20:12 <falko> clime, as soon as i have sudo on the old srv, i'll provide you with the ssl certs.
20:12 <clime> i will ask him
20:12 <clime> ok, great!
20:12 <clime> thanks!
20:13 <sumpfralle3> regarding the DNS zone: it did not move to gandi, yet - right?
20:13 <clime> no to the shared account yet
20:13 <clime> *not
20:13 <sumpfralle3> ok
20:13 <clime> so I have the transfer code from fmarier but i am waiting so that he can do the move on dns
20:13 <falko> how is taking care about this? i cannot remember out of my head :-(
20:14 <sumpfralle3> I think, we planned that fmarier will do the changes for now.
20:14 <sumpfralle3> So the DNS zone will move after the migration.
20:14 <falko> ack. so clime is handling dns as well.
20:14 <clime> i knows the stuff about right ttls values etc better than me
20:15 <clime> *he knows sry :)
20:15 <falko> np.
20:15 <sumpfralle3> Is there still something open with regard to the mail setup?
20:15 <sumpfralle3> (I remember something with "tls" and something about "mailgun")
20:15 <falko> ack, if fmarier is more experienced with dns stuff, it's perfectly fine this way.
20:16 <clime> i have switched sending emails to postfix on the new server
20:16 <sumpfralle3> cool
20:16 <falko> sumpfralle3, i used mailgun, but the new code can run w/ or w/o mailgun.
20:16 <falko> the reason for me to use mg was that I wasn't sure if it's going to be deployed in some cloud instance with bad IP reputation.
20:16 <clime> it works except for using ssl certs for libravatar.fp.o and except this minor issue: https://git.linux-kernel.at/oliver/ivatar/issues/37
20:17 <clime> ye, right
20:17 <clime> so i am not sure what's the reputation of fedora cloud but i guess people are monitoring unusual traffic there
20:17 <clime> so i think we should be in "gray" zone
20:18 <falko> i hope so. if not, we can still switch to mg (mailgun).
20:18 <falko> but for the moment it's probably better to not use it, in order to have better control.
20:18 <clime> yes, we can do that as a fallback
20:18  * falko will have a look at issue 37 tomorrow.
20:19 <clime> well, i mainly wouldn't like to overload your account there ;)
20:19 <@fmarier> falko: you don't have sudo on the old server?
20:20 <falko> o/ fmarier
20:20 <falko> fmarier, sudo asks for password, but i don't think i set a password there....
20:20 <@fmarier> it would be your user password
20:21 <@fmarier> do you have one set?
20:21 <falko> fmarier, i know it should be my user pw, but no, i don't have a user pw set. i always only logged in with ssh key. pw auth is even disabled.
20:23 <@fmarier> ok, I'll set a password for you and put it in a text file in your homedir. Please change it quickly for a good password and then delete the file
20:23 <falko> great! thx.
20:24 <sumpfralle3> another topic: do we have a bit of monitoring metrics being collected on the new host? Something like munin?
20:24 <falko> good point, sumpfralle3!
20:25 <falko> clime, do weß
20:25 <clime> there is some internal monitoring provided by apache itself
20:25 <falko> ?
20:25 <@fmarier> falko: done
20:25 <clime> there is also monitoring as a part of fedora infra but that is just "server is down/up" at the moment, nothing else
20:25 <clime> though it is nagios so probably any check can be added.
20:26 <clime> anyway we can setup munin there
20:26 <clime> that's probably a good idea.
20:26 <falko> fmarier, thx: uid=0(root) gid=0(root) groups=0(root)
20:26 <nipos> Maybe Netdata would make sense.It gives detailed real time statistics about cpu load,ram usage,traffic and much more
20:27 <falko> clime, pcp? or do love pcp as much as i do... ?
20:27 <clime> i don't know netdata but it looks interesting
20:27 <sumpfralle3> If we decide for munin, then I can do it (and add a libravatar-specific plugin).
20:28 <sumpfralle3> But if anyone else in doing something else, I have no problems with not doing it :)
20:28 <nipos> You can try it out at https://t1.tchncs.de for example
20:28 <clime> ye, okay, i can setup munin, but if you guys prefer sometihng else, you can do it :)
20:29 <nipos> Well,I'm personally a fan of Netdata but Munin is ok for me,too
20:29 <clime> nipos: that looks certainly nicer than what munin generates imho
20:29 <clime> :)
20:29 <falko> but we can still do munin (or whatever) for libravatar-specific metrics?!
20:30 <clime> well, sure
20:30 <nipos> Sure,that would make sense
20:32 <@fmarier> Can we run throught the migration plan at a high level to make sure that everybody is on the same page as to who is doing what?
20:32 <@fmarier> https://feeding.cloud.geek.nz/posts/server-migration-plan/
20:32 <falko> sure, fmarier!
20:32 <@fmarier> So the first step (preparing the DNS) is done. I've just taken care of that.
20:33 <clime> cool!
20:33 <falko> yay!
20:33 <@fmarier> "Preparing the new server" could happen now since all you need for that is to copy the TLS certs onto the new server.
20:33 <@fmarier> And do an preliminary export of the data.
20:33 <@fmarier> clime: do you have an account on the old server?
20:34 <clime> no, falko has though
20:34 <@fmarier> ok, let's get you one as well.
20:34 <@fmarier> What's the address of the new server again?
20:34 <clime> fmarier: ye, i would like if falko could fetch the data from the old server
20:34 <clime> if he is willing to
20:35 <falko> sure, np.
20:35 <nipos> libravatar.fedorainfracloud.org is the new server
20:35 <@fmarier> I think that makes sense since he wrote the export script :)
20:35 <clime> i don't mind having account there as well of course
20:35 <@fmarier> clime: I can't login to the new server. can you confirm my username?
20:35 <clime> fmarier: you need to login as root
20:36 <@fmarier> ok, I'm not sure why ssh pubkey I used then
20:36 <@fmarier> Is there any way to look that up?
20:37 <nipos> You can check this in your Fedora account
20:37 <@fmarier> ok, cool, I'm in.
20:38 <@fmarier> clime: Is clime@xxxxxxxxxxxxxxxxxxxxxx the key you want to use for the old server?
20:39 --> clime1 (~clime@xxxxxxxxxxxxxxxxxxxxxxxxxx) hat den Channel #libravatar betreten
20:39 <clime1> sry lagged
20:39 <clime1> fmarier: email in the pub key is francois@akranes(foss)
20:39 <@fmarier> clime1: Is clime@xxxxxxxxxxxxxxxxxxxxxx the key you want to use for the old server?
20:40 <clime1> fmarier: yes, please
20:40 <@fmarier> ok, I'll create a clime account on there with that key...
20:40 <clime1> thanks!
20:40 <-- clime (~clime@xxxxxxxxxxxxxxxxxxxxxxxxxxxx) hat den IRC verlassen (Ping timeout: 245 seconds)
20:41 <falko> clime, see ~root/cert.tbz for the certificates.
20:41 <falko> on the new prod machine.
20:42 <clime1> ye ok cool
20:42 <clime1> should i apply them now that will probably spawn some ssl error?
20:42 <@fmarier> clime1: try logging into grindavik.libravatar.org and then try sudo there
20:42 <clime1> ok
20:42 <@fmarier> your password is in a textfile in your homedir
20:43 <@fmarier> please change it and then delete the file
20:43 <clime1> ok
20:44 <clime1> well, i am getting permission denined stangely
20:44 <clime1>  ssh clime@xxxxxxxxxxxxxxxxxxxxxxxx
20:44 <clime1> should work no?
20:44 <@fmarier> yes, let me check
20:44 <falko> clime, you can set the ssl certs at any time, yes, if you connect to the machine, it will give errors, but nobody should be connecting to it
20:45 <clime1> okay
20:47 <@fmarier> clime1: try again
20:48 <falko> switching accounts/machines. don't wonder
20:48 <-- falko (~falko@xxxxxxxxxxxxxxxxxxxxxxxxxxxxx) verließ #libravatar
20:48 <ofalk> back here :-)
20:49 <clime1> fmarier: working, thanks!
20:49 <@fmarier> clime1: \o/
20:49 <@fmarier> don
20:49 <clime1> so i have changed the certs https://libravatar.fedorainfracloud.org/
20:49 <@fmarier> t forget to change your password
20:49 <clime1> sure doing that now
20:49 <@fmarier> and delete the textfile
20:50 <ofalk> fmarier, we have 7768 users in the db, right?
20:50 <clime1> fmarier: done
20:50 <@fmarier> let me check
20:51 <@fmarier> ofalk: yup
20:51 <@fmarier> so back to the migration plan
20:52 <@fmarier> ofalk and clime now have everything they need to work on "Preparing the new server"
20:52 <ofalk> perfect. so the export works as designed and takes about ~ 5 minutes.
20:52 <clime1> cool!
20:52 <clime1> so, if you can give me fresh data somewhere, that would be nice
20:52 <@fmarier> I will take care of "Preparing the old server" on the 17th in the evening (Pacific time).
20:54 <@fmarier> I will also do the first three steps of "Migrating servers".
20:54 <clime1> great
20:55 <ofalk> sounds good!
20:55 <@fmarier> ofalk: how does your export script work? does it connect to postgres?
20:55 <@fmarier> does it need Django/Apache to be running?
20:55 <clime1> i will return the certs back to libravatar.fp.o for the time being if you don't mind
20:55 <clime1> cause i would like the site easily accessible for testing
20:55 <ofalk> fmarier, no need to have it running, it just needs everything in place + postgresql running; it works pretty much the same way your exportaccount.py works. it's a re-write of that with a loop around it (simplified said)
20:55 <ofalk> clime1, ack!
20:55 <@fmarier> ofalk: ok and what port does it connect to for postgres?
20:56 <clime1> kdone
20:56 <@fmarier> i.e. does it connect to pgbouncer or postgres?
20:56 <ofalk> whatever is configured in the settings.py.
20:59 <ofalk> it really uses the whole django-framework.
20:59 <ofalk> fmarier, /tmp/exportaccounts.py on the old machine.
20:59 <ofalk> i'm pretty sure you'll recognize the code :-)
20:59 <@fmarier> it does look familiar :)
20:59 <@fmarier> ok i guess I'll have to keep pgbouncer running otherwise your script won't be able to connect to it.
21:00 <ofalk> fmarier, you know your setup better!
21:01 <ofalk> clime, a current export of the prod-data is being uploaded on the new prod server to /tmp/ - should finish in ~ 10 minutes.
21:01 <clime1> cool
21:01 <clime1> thx
21:02 <@fmarier> so slight correction, I will take care of the first two steps of "Migrating servers".
21:02 <@fmarier> We will replace the rest of that section with ofalk running the export script and then importing it on the new server.
21:03 <@fmarier> ofalk and clime will do that on the 18th while I'm sleeping.
21:03 <clime1> okay
21:03 <@fmarier> Also I will do "Disable mirror sync" at the same time as the first two steps of "Migrating servers"
21:04 <@fmarier> When I wake up on the 18th, we can do "Testing the main site" together.
21:04 <ofalk> yep. we'll do that. fmarier, what about twitter and identica? did you hand-over the data to these accounts to someone?
21:04 <ofalk> fmarier, it would be awesome if you could do tests!
21:05 <clime1> fmarier: sounds good
21:05 <@fmarier> Those are now registered to social@xxxxxxxxxxxxxx.
21:05 <@fmarier> So anybody on that alias can reset the password.
21:06 <@fmarier> But maybe we should keep it the way it is until the migration is over so that I can easily post there about the migration.
21:06 <clime1> +1
21:06 <ofalk> +1
21:06 <ofalk> seems social goes anyway to only you fmarier and /me
21:07 <clime1> could go to me as well or maybe other guys too?
21:07 <clime1> i would maybe like to tweet sometime something about libravatar :)
21:07 <@fmarier> Finally, I will take care of the "Post-migration steps" on the 18th once we're done and in the week or two after that.
21:09 <@fmarier> One thing that might be helpful is to have a quick meeting at the usual time in a week. Just to confirm that we're all still good to go. What do you think?
21:09 <@fmarier> IRC meeting, that is.
21:09 <clime1> ok, thanks
21:09 <clime1> sure
21:09 <ofalk> clime, i'd appreciate it, if you could tweet about libravatar. but let's move this to after 18th. everybody can still tweet with @-ing libravatar.
21:09 <ofalk> fmarier, completely on your side!
21:10 <nipos> I agree that we should have an additional meeting
21:10 <clime1> so what day?
21:11 <clime1> next sunday or earlier?
21:11 <nipos> Next sunday sounds good
21:11 <sumpfralle3> Sunday sounds good to me.
21:11 <sumpfralle3> clime1: I will add your mail to social
21:11 <ofalk> 2019-02-17, 19 UTC. noted in the wiki.
21:11 <clime1> sumpfralle3: thank you!
21:12 <@fmarier> great
21:12 <ofalk> sumpfralle3, can you do the minutes again?
21:13 <sumpfralle3> yes
21:13 <sumpfralle3> So we are finished?
21:13 <ofalk> sumpfralle3, phenomenal
21:13 <clime1> ofalk: can you look at this bug https://git.linux-kernel.at/oliver/ivatar/issues/35
21:14 <clime1> when you have time
21:14 <clime1> i think we are finished.
21:15 <ofalk> clime, it uses the library underneath, not direct access. therefore you requests go to the real production instance :-)
21:15 <ofalk> anyway, will go over the open issues and pulls beginning of next week.