News → Server migration post mortem

Server migration post mortem

We have successfully moved Redstoner to a new, more powerful server. We're talking about this beast with 8x 3.40GHz, 32GB memory, SSDs, and Gigabit network. We also have IPv6 support now.

The server does have a monthly fee which is generously donated to us by one of our players who prefers to stay anonymous. Thank you very much sponsor!
You can help them out with covering the costs by donating, all donations will go to our sponsor.

We had about 32GB of Minecraft files (mostly world data) that had to be moved at an upload speed of ~20Mbit/s. The transfer duration was estimated to be ~4 hours and we set for 5 hours during which the server would be offline. We already prepared everything for the move in the previous days, such as installing all dependencies and setting up the website, raging about utf8mb4, etc. We had a temporary server set up (as usual) and shut down the Minecraft server.

We used scp to copy the files. 32GB is a lot of data and we thought about compressing it first. There's a compression option:

Compression is desirable on modem lines and other slow connections, but will only slow down things on fast networks.

Well then, no compression; Minecraft world files are compressed anyways!

We quickly noticed this wasn't going to work out at all, maybe the large amount of files caused it to slow down by a factor of 10 compared to our estimations. The transfer was aborted and we created a tar.bz2 of our files. This took a couple minutes, but it was only 4.7GB and transfer went much faster.

Once transferred, we began extracting on the new server. This process took a while but eventually completed. Then we noticed the Minecraft data had different file sizes on the old and new servers and started investigating. This is very common due to different data structure & blocks in the underlying file systems. The odd thing was that du --bytes still reported a different size. Some time was wasted until we figured that we had different versions of du and that only the size of directories varied – not the actual files.

For the database (website, PlotMe, LoginSecurity, LogBlock, …) data we started by creating mysqldumps of each database and transferred them to the server, which only took a few minutes.

All data was imported in no time, except LogBlock. LogBlock is special...
The server was importing like crazy and processed 20,000 - 50,000 rows per second. However, with about 325,000,000 rows in total, this took about 2 hours, which was longer than we expected.

However, even that eventually completed and we did some checks to ensure the Minecraft server and all its plugins were operating as expected. Luckily this was the case. We also ensured our backups, scripts, and cron jobs were running flawlessly.

The TS3 server was moved without any problems, and we switched the domain to our new IP address. The temp server was shut off and replaced with a message of the new server IP while DNS was propagating.

At this point the Minecraft server, website, and TS3 server were fully operational again.
The total time of downtime for the Minecraft server was 6.5 hours (1.5 more than planned).
Website and TS3 were only down for a few minutes each.

You should be able to join the server as usual, if you didn't notice anything, that's great! If you feel something is odd, please report it to us.

I would like to thank @PanFritz again for previous hosting and our new server sponsor for donating this rad new machine! Consider donating!


thx Jomo ;D

rsync > scp xD

Please Log in to post a reply.