i love how simple it is now to go from not knowing something at all to getting into the depths why someone designed something the way it is.
i was on my flight and opened twitter not realising my flight wifi was down. twitter still loaded the cached tweets. it got me thinking.
twitter has ~250M DAU and no amount of horizontal scaling would be enough to read all the tweets from their followed account it, sort it pased on params and build the feed everytime someone comes in to scroll
few messages with gpt and turns out this is a common concept in scaling systems. a hybrid system of fan out on write + fan out on read.
what i described earlier is fan out on read, where you read the db everytime someone comes in. that however doesn't scale beyond a limit even to get the partial data for a tweet from an extremely fast db like cassandra(btw twitter started with sql, moved to cassandra and now has build custom dbs)
what's fan out on write?
everytime i write a tweet, it would push this to a queue. workers will then pull it and get all the followers for the creator. now each follower has a timeline cache and so it will be appended to it based on the parameters you think important such as time of posting, value etc.
now what happens to people like
@elonmusk
posts something. they have millions and millions of followers and it doesn't make sense to do fan out on write. we'd call them mega influencers.
for mega influencers, you can still use fan out on read but because you have a cache already and you know the cursor position of the user i.e. which tweet the user is viewing, you can append it below the current view-able tweets after you have read from the db.
ofc the system in place would be much more complex than this but its so interesting how you make these choices and then find bottlenecks in them and then make choices again until you find a bottleneck which is minute enough to ignore.