Shop Forum More Submit  Join Login
About Varied / Hobbyist Senior Member Ben RogersMale/United States Recent Activity
Deviant for 6 Years
Core Member 'til Hell freezes over
Statistics 5 Deviations 29 Comments 29,511 Pageviews

Newest Deviations

Breadlands duck by Nodren Breadlands duck :iconnodren:Nodren 1 2 stars by Nodren stars :iconnodren:Nodren 0 1 Yin Yang by Nodren Yin Yang :iconnodren:Nodren 1 0

Favourites

Endeavor Flyover, Los Angeles by xraystyle Endeavor Flyover, Los Angeles :iconxraystyle:xraystyle 1,667 356 Light and Shadow by porbital Light and Shadow :iconporbital:porbital 2,910 263 Grasses of Pew Tor by Alex37 Grasses of Pew Tor :iconalex37:Alex37 1,927 450 underappreciated. by theycome underappreciated. :icontheycome:theycome 128 44 the lake. by theycome the lake. :icontheycome:theycome 115 59 Vivid by Dani-Lefrancois Vivid :icondani-lefrancois:Dani-Lefrancois 279 91 Maligne Lake 2 by GoTcHaSuCcA Maligne Lake 2 :icongotchasucca:GoTcHaSuCcA 275 45 Loki, Prince of Asgard by alicexz Loki, Prince of Asgard :iconalicexz:alicexz 15,065 543 Flight of the Fireflies (Animation) by ebonrouge Flight of the Fireflies (Animation) :iconebonrouge:ebonrouge 6 11 Endless Possibility by Thomas-Koidhis Endless Possibility :iconthomas-koidhis:Thomas-Koidhis 5,476 690 Sunset in Mauritius by Nightline Sunset in Mauritius :iconnightline:Nightline 323 41 breathless. by theycome breathless. :icontheycome:theycome 713 363 Paradise II by IsacGoulart Paradise II :iconisacgoulart:IsacGoulart 2,472 218 Bay of Fires Final by MarkKenworthy Bay of Fires Final :iconmarkkenworthy:MarkKenworthy 178 20 New York: Brooklyn Bridge. by inbrainstorm New York: Brooklyn Bridge. :iconinbrainstorm:inbrainstorm 451 44
Journal
Moving 6 Billion Messages Without Being Noticed
We've just finished moving every message in every user's message center — all 6 billion of them — to a brand new set of database servers, and we managed to do it without anyone noticing. We didn't find much published material about the best way to do this, so we wanted to share a few details with you. It does get a little bit technical though so I won't be offended if you skip over some parts!
The data that powers message center was stored on one of our main sharded MySQL database clusters. That same cluster powers a lot of different features of the site and we recognised a lot of traffic was caused by this relatively simple but large set of data.
To free up capacity on that cluster, and help keep the site quick as we continue to grow, we decided to move the message center data to its own servers.
Some fun figures:~6 billion rows of data in 13 tables across 16 shardsAverage 5k, daily peak of 10k queries per second for this data across the clusterAverage 100k, daily
:icondt:dt
:icondt:dt 22 50

Watchers

Activity


deviantID

Nodren
Ben Rogers
Artist | Hobbyist | Varied
United States

Comments


Add a Comment:
 
:iconphoenixleo:
phoenixleo Featured By Owner Nov 30, 2014
Happy Birthday! :iconballoonplz:
Reply
:iconpeppermintsoda:
PeppermintSoda Featured By Owner Nov 30, 2014  Student
Happy birthday! :cake: :party: :la:
Reply
:iconcatluvr2:
catluvr2 Featured By Owner Dec 1, 2013  Hobbyist General Artist
Happy birthday in -1 days! :la:

... wait. :stinkeye:
Reply
:iconphoenixleo:
phoenixleo Featured By Owner Nov 30, 2013
Happy Birthday! :iconballoonplz:
Reply
:iconkingabe:
kinGabe Featured By Owner Nov 29, 2013  Student Artist
:iconballoonsplz::iconbirthdplz::iconballoonsplz2:
Reply
:iconlintu47:
Lintu47 Featured By Owner Dec 22, 2012  Hobbyist Photographer
Happy Holidays! :santa:
:iconbradut2::iconfur12:
Reply
:iconmiloticscale:
MiloticScale Featured By Owner Dec 6, 2012
:huggle: Good luck with your future endeavors.
Reply
:iconphoenixleo:
phoenixleo Featured By Owner Dec 5, 2012
:salute:
Reply
:iconprincejose:
PrinceJose Featured By Owner Dec 3, 2012
:hug: :bye:
Reply
:iconarichy:
Arichy Featured By Owner Nov 30, 2012  Hobbyist General Artist
Happy Birthday :la:
Reply
Add a Comment: