We've just finished moving every message in every user's message center — all 6 billion of them — to a brand new set of database servers, and we managed to do it without anyone noticing. We didn't find much published material about the best way to do this, so we wanted to share a few details with you. It does get a little bit technical though so I won't be offended if you skip over some parts!
The data that powers message center was stored on one of our main sharded MySQL database clusters. That same cluster powers a lot of different features of the site and we recognised a lot of traffic was caused by this relatively simple but large set of data.
To free up capacity on that cluster, and help keep the site quick as we continue to grow, we decided to move the message center data to its own servers.
Some fun figures:~6 billion rows of data in 13 tables across 16 shardsAverage 5k, daily peak of 10k queries per second for this data across the clusterAverage 100k, daily