Is My VPS Hacked? How to Detect, Contain and Rebuild a Compromised Server

Your provider sent an abuse notice about outbound SSH scanning. Or your load average is pinned at 8 with no traffic. Or you noticed an SSH key in authorized_keys that you never added. The question is always the same: is this server compromised, and what do I do in the next thirty minutes?

This is a practical incident response guide for a single Linux VPS. It covers how to triage quickly, which indicators actually mean something, how to contain damage without destroying evidence, and how to decide between cleaning and rebuilding. It also covers the uncomfortable part most guides skip: on a compromised machine, the tools you use to investigate may be lying to you.

Direct answer

If you have real evidence of unauthorized code execution — an unknown process mining or scanning, a modified system binary, a webshell in your document root, an added SSH key or a new UID 0 account — treat the box as fully compromised and rebuild it from scratch on a new instance. Do not "clean" it.

The correct order of operations is:

1. Contain the network (block outbound and inbound at the firewall or provider level, don't power off yet).
2. Capture what you need: a disk snapshot if available, plus a quick evidence dump.
3. Investigate enough to find the entry point.
4. Rebuild on a fresh instance, restore only data you can validate.
5. Rotate every credential the server could see.

Skipping step 3 means you will get compromised again through the same hole. Skipping step 5 means the attacker keeps access to your other systems.

Fast triage: what to run in the first ten minutes

Run these read-only checks over SSH. They won't fix anything, but they tell you whether you have an incident or a misconfiguration.

What is using the CPU and network?

ps auxf --sort=-%cpu | head -30
ss -tunap | grep -v 127.0.0.1

Cryptominers are the most common outcome of an opportunistic compromise. Look for high-CPU processes with random names, processes running from /tmp, /dev/shm, /var/tmp, or a deleted binary. Check that last case explicitly:

ls -l /proc/*/exe 2>/dev/null | grep deleted

A running process whose executable has been unlinked is a strong signal. Legitimate software does this only briefly during upgrades.

Who logged in?

last -Fa | head -40
lastb | head -20            # failed logins, if btmp exists
journalctl -u ssh -u sshd --since "7 days ago" | grep -i "Accepted"

"Accepted publickey" or "Accepted password" from an IP you don't recognize is the clearest evidence you will get. Note that accepted password logins are impossible if you followed a keys-only SSH configuration — which is exactly why that setting matters.

Persistence in the obvious places:

for u in $(cut -d: -f1 /etc/passwd); do
  echo "== $u"; crontab -u "$u" -l 2>/dev/null
done
ls -la /etc/cron.* /etc/cron.d/ /var/spool/cron/
systemctl list-unit-files --state=enabled | tail -40
systemctl list-timers --all
ls -la /root/.ssh/ /home/*/.ssh/
cat /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys 2>/dev/null
awk -F: '$3==0 {print}' /etc/passwd
grep -rn "" /etc/ld.so.preload 2>/dev/null

Also check user-level systemd units (~/.config/systemd/user/), shell profiles (~/.bashrc, ~/.profile, /etc/profile.d/), and authorized_keys files for service accounts that should not have any.

Files changed recently:

find /etc /usr/bin /usr/sbin /usr/local -type f -newermt "-14 days" -ls 2>/dev/null

On Debian/Ubuntu, verify package file integrity with debsums -c (install debsums); on RHEL-family systems use rpm -Va. Both produce noise — config files legitimately change — but an altered /usr/bin/ssh, /bin/ls or /usr/sbin/sshd is not noise.

The indicators that actually matter

Not every anomaly is a breach. Sorting real signals from noise saves hours.

Strong indicators of compromise

• Successful SSH login from an unexpected IP, or a public key you cannot account for
• A new account, or an existing account with UID 0 / added to sudo/wheel
• Modified system binaries reported by debsums/rpm -Va
• Outbound connections to mining pools, IRC, or scanning traffic on ports 22/23/2323/5900
• A process running from a world-writable directory, or with a deleted executable
• PHP/JS files with eval(base64_decode(...)) patterns in a web root
• /etc/ld.so.preload containing anything, or unexplained entries in /etc/passwd

Weak or misleading indicators

• Thousands of failed SSH logins. That is ambient internet noise on every public IP, not a breach.
• High CPU that matches your own workload or a cron job.
• rkhunter / chkrootkit warnings. These tools generate frequent false positives on modern distros and miss most current threats. Use them as a hint, never as proof of either compromise or cleanliness.
• Slow disk or high load with no unusual processes — that may simply be a noisy or oversold node, not an intrusion.

Why on-box tools can lie

If an attacker installed a kernel module or an LD_PRELOAD library rootkit, ps, ss, ls and find can be patched to hide specific PIDs, ports and paths. Two practical cross-checks:

• Compare ss -tunap output with the firewall's own view. If your nftables counters show outbound traffic your process list can't explain, something is hidden. Adding a logged drop rule for unexpected outbound ports makes this visible; see this nftables baseline ruleset for the structure.
• Do the real inspection offline. Boot the VPS into your provider's rescue/recovery environment, or attach the disk to a second instance, then mount the filesystem read-only and inspect from a known-good userland. This is the only inspection you should fully trust.

Containment without destroying evidence

The instinct to reboot or shut down is understandable, and usually wrong. A shutdown loses all volatile state — running processes, open sockets, in-memory payloads that never touched disk — which is often the only place the payload exists. A reboot can also trigger persistence you haven't found yet.

Instead:

1. Cut the network at the edge. Use the provider's firewall/network isolation if available, or on the host set a default-deny policy in both directions with a single allow rule for your management IP on SSH. Removing outbound access is what stops data exfiltration, further spam, and scanning that generates abuse complaints.
2. Snapshot the disk if your provider offers snapshots or images. This is your evidence and your safety net for data recovery.
3. Dump volatile state to a file, then copy it off:

mkdir -p /root/ir && cd /root/ir
date -u > timestamp.txt
ps auxfww > ps.txt
ss -tunap > sockets.txt
lsof -nP > lsof.txt 2>/dev/null
ls -l /proc/*/exe > proc_exe.txt 2>/dev/null
crontab -l -u root > cron_root.txt 2>/dev/null
systemctl list-units --all > units.txt
cp -a /var/log ./varlog 2>/dev/null
tar czf /root/ir.tar.gz -C /root ir

Then scp the archive off the server. If you later decide you need help from someone with forensics experience, this bundle is what they will ask for.

4. Do not log in as root and start editing files. Every write changes timestamps you may want later.

Find the entry point before you rebuild

Rebuilding without root cause analysis just resets the clock. In practice, the entry point on a small VPS is almost always one of five things:

A weak or reused SSH credential. Check journalctl for accepted logins and their source IPs, and whether password authentication was enabled at all.

An exposed service that shouldn't have been public. Redis, Memcached, Elasticsearch, MongoDB, Docker's API on 2375, a database bound to 0.0.0.0, or a .env file served by the web server. Check what was listening: ss -ltnp output plus your web server's access logs for requests to /.env, /.git/config, /wp-content/..., /vendor/phpunit/....

A vulnerable application. Outdated WordPress plugins, Laravel/Node dependencies, or a self-hosted panel. Search web logs around the first suspicious timestamp for long POST bodies, unusual user agents, or requests to files that shouldn't exist.

A container escape or, far more often, a container misconfiguration. A container run with --privileged, with the Docker socket mounted, or with the host filesystem bind-mounted is effectively root on the host.

A compromised developer machine or CI token. If the attacker logged in with a valid key from an unusual IP and nothing else looks vulnerable, the leak may not be on the server at all.

Correlate the earliest indicator's timestamp — the file creation time of the malicious binary, the cron entry, or the added key — with your access logs from a few minutes before. That window usually contains the request that got them in.

This is also where log minimization creates real tension. Aggressive retention limits are good for privacy but bad for incident analysis; if you follow a log minimization strategy, decide deliberately how many days of web and auth logs you want to keep, and consider shipping them off-box so a compromise can't erase them.

Rebuild: the only reliable cleanup

Cleaning a compromised system means proving the absence of persistence across the kernel, initramfs, systemd units, cron, package hooks, shell profiles, PAM, web application code and your own data. On a machine you can redeploy in a minute, that effort is never justified.

A workable rebuild sequence:

1. Provision a new instance with a fresh IP. Do not reinstall over the old disk until you have finished with it; keep the snapshot.
2. Harden before exposing anything: keys-only SSH, default-deny firewall, automatic security updates, no services bound to public interfaces that don't need to be.
3. Restore data selectively. Databases from a dump, uploads and documents by file type. Never restore application code, cron jobs, systemd units, or authorized_keys from the compromised system — reinstall code from your own version control or upstream packages.
4. Scan restored web content for webshells: look for recently modified PHP/JS files, files in upload directories with executable extensions, and obfuscated eval chains.
5. Prefer a pre-compromise backup if you have one and can identify the compromise date. This is where tested, versioned encrypted off-site backups pay for themselves: with proper retention you can restore a snapshot from before the intrusion instead of guessing which files are clean.
6. Rotate everything the old server could read: SSH keys, API tokens, database passwords, SMTP credentials, TLS private keys (reissue and revoke), OAuth secrets, cloud API keys, and any wallet or seed material that ever touched that disk. Assume anything on that filesystem or in that process memory is public now.
7. Notify if user data was in scope. Depending on jurisdiction and data type, this may be a legal obligation, not a choice.
8. Handle the abuse ticket honestly. Tell the provider you were compromised, that the instance is isolated, and that you are rebuilding. Providers deal with this daily; a clear response resolves tickets far faster than silence.

Common mistakes

• Rebooting first. Destroys volatile evidence and may not remove persistence.
• Deleting the malicious binary and calling it done. The cron job or systemd timer that downloads it again is still there.
• Trusting rkhunter output as an all-clear. A clean scan on a compromised host means nothing.
• Restoring the whole filesystem from a post-compromise backup. You just restored the backdoor.
• Reusing the same SSH key you had on the compromised box — if the private key lived there, it is burned.
• Not rotating credentials for third-party services. The server's value to an attacker is often the keys it held, not its CPU.
• Assuming a fresh IP fixes reputation. If the box sent spam, mail deliverability from the new IP starts from an unknown reputation, which is a separate problem entirely.

FAQ

Can I just remove the miner and keep running? You can, and it often appears to work for a while. But you have not established how they got in or what else they installed, and any credential on that host should now be considered leaked. For a single-purpose VPS, rebuilding is faster and more defensible than proving the box is clean.

How do I know when the compromise started? Use the earliest reliable timestamp among: file creation time of attacker artifacts, the first accepted SSH login from an unknown IP, the first anomalous request in web logs, and the point where CPU or outbound traffic changed. Treat attacker-controlled timestamps with suspicion — touch can set mtime to anything, though ctime is harder to forge.

Does full disk encryption protect me here? No. LUKS protects data at rest against offline access to the disk; it does nothing against an attacker who has code execution on a running system where the volume is already unlocked. The honest scope of LUKS on a VPS is worth understanding before relying on it as a security control.

Does a No-KYC or crypto-paid VPS change the response? Not technically. Privacy in billing has no effect on server security — the same hardening, patching and response steps apply. What it does change is your operational hygiene: if you use anonymous hosting, a compromise that exposes your credentials, browser sessions or personal files can undo the separation you set up, which is another reason to rebuild rather than clean.

Conclusion

Treat "is my VPS hacked?" as a triage problem, not a search for a magic scanner. Look for unauthorized execution, unexplained persistence and unexpected network activity. If you find any of them, isolate the network first, capture volatile state and a snapshot, spend enough time in the logs to identify the entry point, then rebuild on a fresh instance and rotate every secret the old host could reach.

The infrastructure part is cheap — spinning up a clean VPS takes a minute. The expensive parts are root cause analysis and credential rotation, and those are exactly the steps people skip. Do them once properly, and the same attacker doesn't come back through the same door.