A tale of two [mattermost servers, container types, databases]

It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair…

Okay, okay. I feel compelled to deliver an explanation after trashing a provincial-level server. It started off simply enough, two projects. 1) Move the TRU Mattermost server out of the BC OpenEd docker server to its own server node, add encrypted traffic. 2) On the BC OpenEd docker server: upgrade os, upgrade docker, migrate BC OpenEd Mattermost from the preview container version to a multi-container production version, update to current version of Mattermost and add encrypted traffic.

Seemed simple.

So I tackled number 1 first (I know, everybody does that). With the care of a dynamite technician I did a test migration with the usual steps, dump and import database, move data files and adjust config settings. All checked out. Did the real migration (from the list of commands I’d carefully saved), worked a treat. Migration is easy apparently.

So then I tackled number 2. Many more moving pieces here (maybe not so much moving as swirling in an ethereal way).

First, upgraded the server OS, everything went fine.

Second, upgraded docker. Not so fine, several of the containers entered an endless restart loop indicating that something was off. In particular, the mattermost docker can was complaining about not being able to find WeaveWorks network drivers. Grant Potter and I did some digging and couldn’t figure out what the issue was. We decided to pull some unnecessary components. Namely docker cloud which uses WeaveWorks to underpin the tangled web it weaves. (Sorry for that, couldn’t help it.) Now we had more problems than we started with.

Third, Grant suggested a scorched earth approach which I heartedly agreed with. Since the early days our docker-fu has grown, and sometimes starting afresh is the way to go, Grant figures maybe blue belt. Also, based on my first migration I assumed (anyone spot the problem) that it would be a simple matter of dumping theh original database, copying the files and adjusting the config.
What I failed to account for was two things (well, actually I thought about them, but there wasn’t anything to be done at that point): the BCOpenEd Mattermost was still in a Preview container, while the TRU instance was in a set of production containers. (The latter more ameniable to things like backups and migration, the former surprisingly resilient).

Four, so, I built what I was reasonably sure would be a good safety net. Backed up the docker environment directory, the container file system for the mattermost container along with database dump, file and config backups, before scorching the earth.

Five, fired up docker, installed a multi-can production version of Mattermost and then proceeded to re-importing the database. Wow, nice page of errors, I wonder what…oh, I’m importing a mysql database dump into a postgresql database. Turns out they are different. All sorts of philosophical questions drifted across my mind, the most poigniant being, “why would they build their single container preview on top of mysql and their production version on top of postgresql?”. Well, I know why from a which database is probably better point of view, but from a trying it out and shifting to production point of view it made no sense.

Six, well then, I’ll just convert the database from mysql to postgresql and proceed from there. That just rolls off the tongue doesn’t it. Pro solutions to doing this seem to start at about $600 USD, but there was a nice open source utility called pgloader that the internetions seemed to like. Easy, peasy, connect to the db in mysql and the utility will convert and push it in to postgres. That means I need to have both database servers running, rather not do that in my local environment, I know, I’ll build a docker container and do it there. (Sawing and hammering noises in the background). Linux container, install mysql, postgresql and pgloader. Run the command, errors. Look at blog and forum posts, blah, issues doing this inside a docker can. Over to a linux box I’m running on an old computer at home. Rinse and repeat, ah ha, the conversion worked.

Seven, remember now, that not only was I converting databases, but the old container was a version 3.x preview can and the new version was a 4.4 production set of containers. Imported the database and nothing, didn’t recognize any of it.

Eight, somewhat despondant I outlined the situation to Grant along with some options, some of them very time consuming, like copying and pasting from one database to another. His thoughts were to just take the hit (the hit being the loss of our chat history, well, it’s not lost, it can be accessed via the database if we really need to) and move on in to the new environment.

So, we find ourselves here. If anyone needs anything from the history, contact me at twelch@tru.ca and I’ll pull it out of the old database for you.

Both apologies and welcome to the new docker environment which is easily backed up and updated. It sort of has that same feeling as when the carpet cleaner has finished up, things smell fresh and the salsa stains are gone.

Cheers