We want to make sure every Bukalapak user received the message they need. So, we develop a notification platform. It’s similar to Facebook notification, personalized notification for every user. This is the onsite notification.
- Support multiple types of notification
- User personalized notification
- Accommodate 10 million Bukalapak users
- Independent service
Architecture: bit by bit
MySQL. Simple reason, most of our engineers had the most experience in using MySQL. This will alleviate the scaling pain in the long run. We considered other bleeding edge databases like MongoDB, Cassandra, and RethinkDB, but for the sake of stability and maintainability, we chose to be conservative this time.
Only 1 table for this service. These are the fields.
|1||id||SERIAL||your standard row identification|
|2||user_id||INT||reference to user, indexed|
|3||type||VARCHAR(10)||to differentiate kind of notifications, indexed|
|5||read_at||DATETIME||time of read, indexed|
|6||created_at||DATETIME||standard timestamp, indexed, just in case|
We chose to as denormalize as possible to improve performance and easier future maintenance. We considered splitting into 2 tables to reduce redundancy but joining table reducing performance by quite a number. If in the future we want to scale horizontally by parting the table, one table is easier to split. We used
id field to accommodate more than billions of notifications. Using standard `INT` will make us reach the ceiling in a short time. In fact, we have reached 1/3 road to that 2 billion integer ceiling after 4 months of the release. We need new SSD.
data field will be JSON serialized string. The rest of the field are indexed except the data field as we swore not to query based on the data field.
We want the notification platform to be independent in order to reduce the load of our main monolithic service and make teams move swifter. Yes, this is a step into micro-services architecture.
On to the application, we used Rails because this starts as an experimental feature, we want to develop the app rapidly. Secondly, we tried other lower level language like Go, but after some calculation, we believe choosing Go will not give us a significant performance boost. After all, this application is an API only application with no templating, no authentication, and no complex logic, simple insertion and simple read based on
user_id. But, of course, if we need more performance and efficiency, we would happy to try other options. Rust, maybe, since we have rust prophet in our engineering team.
To make other services could insert notification, we use RabbitMQ as the queue service. Thus, we only need to implement worker(background application) that listen to the queue. We decided not to make API for inserting notification because by using queue service, we are not worried to be overwhelmed by tremendous amount of request. If the request is greater than our throughput, the rest of the notification will safely be stored in the queue and will be consumed eventually. Like every good things in this life, it’s best consumed with moderation.
While on the other hand if the same case happens with API, either we need to scale up the application server or the performance will degrade. Though we will implement insertion API for other services that need synchronize request.
All API endpoint got less than 20 ms in average response time. Moreover, the most used endpoint, the one that counts the unread notification, is averaging in 3 ms. We clearly did not expect to be this fast, since ruby is only “fast enough”! The service store about 40 GiB worth of notifications every month and there is no sign of slowing the performance. The only time that slows down the response time is when we insert a lot of(millions) notifications in short time. Phew.
This notification platform is only available in desktop, for now. The mobile web, Android, and iOS will be released in the near future. Let us know if you have any insight!