How my server got infected with a crypto mining malware and how I fixed it

I was working on a side project recently where I had faced an issue when running a PostgreSQL database. The database server was getting shut down randomly for no apparent reason. I had deployed my Rails application along with its dependencies, like Redis and PostgreSQL in one of my EC2 instances in AWS.

PostgreSQL was running on the machine at the default port of 5432. Ports 443 and 80 were open to everyone, for handling HTTP/S traffic. Port 22 was also open to everyone so that anyone with their public SSH keys added in the authorized_keys file in the remote server, or having access to the private key file of the server could log into the machine remotely.

For development I needed to access this remote database locally, so I edited the pg_hba.conf file and opened PostgreSQL to the network. I added a new rule to open port 5432 so that anyone could connect to the PostgreSQL instance remotely if they had all the credentials. If you notice in the below screenshot, you will see all the ports that are open to the public network. This was all working great for me, until one fine day it wasn’t.

I realized that something was wrong when I couldn’t connect to the PostgreSQL instance remotely one day. The response I was getting was the standard is PostgreSQL running? error.

psql: could not connect to server: No such file or directory
Connection refused Is the server running on host ${hostname}
and accepting TCP/IP connections on port 5432?

I was still able to SSH into the VM so I tried to restart PostgreSQL. After some investigation I figured out that PostgreSQL was back up momentarily when I do systemctl restart postgresql, but it goes down again.

Inspecting the processes with htop I was able to see that all the CPU cores were at 100% usage. Something didn’t feel right. Sorting the processes based on the percentage of CPU and memory used, I came across two peculiar processes - kdevtmpfsi and kinsing. A quick Google search showed that this was a crypto mining malware that spreads by exploiting flaws in resources that are exposed to the public. Killing the process was of no use since the malware also adds a cron job to replicate itself so that it can’t be stopped.

Removing the malware

I found all files in the system with kdevtmpfsi and kinsing in their names using the unix find command and deleted them. The malware’s files was inside the /tmp directory.

find / -name kdevtmpfsi*
find / -name kinsing*

Then I checked if there were any cron jobs running in the machine with the crontab command. There were some jobs running which were there to reload the malware script even if you delete it. I deleted the jobs related to kdevtmpfsi and kinsing. Another information I learnt was that in Unix, each user will have their own crontab which can run jobs as that particular user.

crontab -l  #To list all running cron jobs
crontab -e #To delete running jobs

Things to pay attention to

I made all the passwords stronger, especially for the resources that were being exposed to the public. One of the lessons I learnt was that you can always be more secure, and that you should never compromise on your passwords. The passwords that I had set for my users were weak, with just a dictionary word, a digit and a special character - something like the format of himalaya7!

Instead of opening the required ports to the public network, I exposed them to only the IP addresses that I needed to access it from.

Notice how the ports for SSH and PostgreSQL are only exposed to the required IP addresses now.

I moved the application database to a managed PostgreSQL service rather than running it in a VM by myself. This also means that I need not worry about the performance or uptime as all of this will be taken care of by AWS themselves.

For extra security, I also set up a reverse proxy so that no one can ping my deployed URL and get the IP address of the VM where the application is running.

Securing your deployments is as important as any other step when deploying your application and needs to be a priority right from when you are designing the architecture of your application. Taking care of such small details during development will facilitate you in writing good code and follow the right patterns from the start.