---
title: "Three case studies of debugging redis running out of memory"
description: "Three case studies of debugging redis running out of memory"
canonical_url: "https://www.bigbinary.com/blog/debugging-redis-memory-issue"
markdown_url: "https://www.bigbinary.com/blog/debugging-redis-memory-issue.md"
---
# Three case studies of debugging redis running out of memory
Three case studies of debugging redis running out of memory
- Author: Unnikrishnan KP
- Published: September 12, 2022
- Categories: Ruby, Rails
In this blog, we will discuss three separate case studies of Redis running out
of memory. All three case studies have videos demonstrating how the debugging
was done.
All three videos were prepared for my team members to show how to go about
debugging. The videos are being presented "as it was recorded".
## First Case Study
When a job fails in [Sidekiq](https://sidekiq.org/), Sidekiq puts that job in
[RetrySet](https://github.com/mperham/sidekiq/wiki/API#retries) and retries that
job until the job succeeds or the job reaches the maximum number of retries. By
default the maximum number of retries is 25. If a job fails 25 times, then that
job is moved to the [DeadSet](https://github.com/mperham/sidekiq/wiki/API#dead).
By default, Sidekiq will store up to 10,000 jobs in the deadset.
We had a situation where Redis was running out of memory. Here is how the
debugging was done.
### How to inspect the deadset
```ruby
ds = Sidekiq::DeadSet.new
ds.each do |job|
puts "Job #{job['jid']}: #{job['class']} failed at #{job['failed_at']}"
end
```
Running the following to view the latest entry to the dataset:
```ruby
ds.first
ds.count
```
To see the memory usage following commands were executed in the Redis console.
```
> memory usage dead
30042467
> type dead
zset
```
As discussed in the video large amount of payload was being sent. This is not
the right way to send data to the worker. Ideally, some sort of `id` should be
sent to the worker and the worker should be able to get the necessary data from
the database based on the received `id`.
#### References
1. [How to increase the number of jobs in the Sidekiq deadset or disable deadset](https://github.com/mperham/sidekiq/discussions/5011)
2. [Maximum number of job retries in Sidekiq](https://github.com/mperham/sidekiq/blob/main/lib/sidekiq/job_retry.rb#L71)
3. [Maximum number of jobs in Sidekiq Deadset](https://github.com/mperham/sidekiq/blob/a89d84509c569a78882e24e0e28913a22c9311f5/lib/sidekiq.rb#L38)
## Second case study
In this case, the Redis instance of
[neetochat](https://www.neeto.com/neetochat/) was running out of memory. The
Redis instance had 50MB capacity, but we were getting the following error.
```
ERROR: heartbeat: OOM command not allowed when used memory > 'maxmemory'.
```
We were pushing too many geo info records to Redis and that caused the memory to
fill up. Here is the video capturing the debugging session.
The following are the commands executed while debugging.
```
> ping
PONG
> info
> info memory
> info keyspace
> keys *failed*
> keys *process*
> keys *geocoder*
> get getocoder:http://ipinfo.io/41.174.30.55/geo?
```
## Third Case Study
In this case, the authentication service of [Neeto](https://www.neeto.com/) was
failing because of memory exhaustion.
Here the number of keys was limited, but the payload data was huge and all that
payload data was hogging the memory. Here is the video capturing the debugging
session.
The following are the commands executed while debugging.
```
> ping
> info keyspace
db0:keys=106, expires=86,avg_ttl=1233332728573
> key * (to see all the keys)
```
The last command listed all 106 keys. Next, we needed to find how much memory
each of these keys are using. For that, the following commands were executed.
```
> memory usage organizations/subdomains/bigbinary/neeto_app_links
736 bytes
> memory usage failed
10316224 (10MB)
> memory usage dead
29871174 (29MB)
```
## Links
- [Human page](https://www.bigbinary.com/blog/debugging-redis-memory-issue)