Recently, we got an opportunity to reduce the infrastructure cost of a medium-sized e-commerce project. In this blog we discuss how we reduced the total infrastructure cost by 10%.
Changes to MongoDB instances
Depending on the requirements, modern web applications use different third-party services. For example, it's easy and cost effective to subscribe to a GeoIP lookup service than building and maintaining one. Some third-party services get very expensive as the usage increases but people don't look for alternatives due to legacy reasons.
In our case, our client had been paying more than $5,000/month for a third-party MongoDB service. This service charges based on the storage used and we had years of data in it. This data is consumed by a machine learning system to fight fraudulent purchases and users. We had a look at both the ML system and the data in MongoDB and found we actually didn't need all the data in the database. The system never read data older than 30-60 days in some of the biggest mongo collections.
Since we were already using nomad as our scheduler, we wrote a periodic nomad job that runs every week to delete unnecessary data. The nomad job syncs both primary and secondary MongoDB instances to release the free space back to OS. This helped reduce monthly bill to $630/month.
Changes to MongoDB service provider
Then we looked at the MongoDB service provider. It was configured years back when the application was built. There are other vendors who provided the same service for a much cheaper price. We switched our MongoDB to mLab and now the database runs in a $180/month dedicated cluster. With WiredTiger's compression enabled, we don't use as much storage we used to use before.
Making use of Auto Scaling
Auto Scaling can be a powerful tool when it comes to reducing costs. We had been running around 15 large EC2 instances. This was inefficient due to following two reasons.
- It cannot cope up when the traffic increases beyond its limit.
- Resources are underused when traffic is less.
Auto Scaling solves both the issues. For web servers, we switched to smaller instances and used Target Tracking Scaling Policy to keep the average aggregate CPU utilization at 70%.
Background job workers made use of a nomad job we built. It periodically calculated the number of required instances based on the count of pending jobs and the job's queue priority. This number was pushed to CloudWatch as a metric and the Auto Scaling group scaled based on that. This approach was effective in boosting performance and reducing cost.
Buying reserved instances
AWS has a feature to reserve instances for services like EC2, RDS, etc.. It's often preferable to buy reserved instances than running the application using on-demand instances. We evaluated reserved instance utilization using the reporting tool and bought the required reserved instances.
Looking for cost-effective solutions
Sometimes, different solutions to the same problem can have different costs. For example, we had been facing small DDoS attack regularly and we had to rate-limit requests based on IP and other parameters. Since we had been using Cloudflare, we could have used their rate-limiting feature. Performance wise, it was the best solution but they charge based on the number of good requests. It would be expensive for us since it's a high-traffic application. We looked for other solutions and solved the problem using Rack::Attack. We wrote a blog about it sometime back. The solution presented in the blog was effective in mitigating the DDoS attack we faced and didn't cost us anything significant.
Requesting custom pricing
If you are a comparatively larger customer of a third-party service, it's more likely that you don't have to pay the published price. Instead, we could request for custom pricing. Many companies will be happy to give 20% to 50% price discounts if we can commit to a minimum spending in the year. We tried negotiating a new contract for an expensive third-party service and got the deal with 40% discount compared to their published minimum price.
Running an infrastructure can be both technically and economically challenging. But if we can look between the lines and if we are willing to update existing systems, we would be amazed in terms of how much money we can save every month.