Running a Mail Server on a VPS: PTR, SPF, DKIM, DMARC and the Deliverability Reality

Installing Postfix and Dovecot is the easy part of self-hosting email. The hard part is convincing Gmail, Outlook and a few thousand spam filters that your brand-new VPS IP is not a spam cannon. This article covers what actually determines whether your mail lands in the inbox, in what order to set it up, and where self-hosting genuinely fails.

The direct answer

You can run a production-grade mail server on a single VPS, but only if three things are true before you write a line of config:

1. Outbound TCP port 25 is open on your provider's network. Many hosts block it by default for new accounts. If it's blocked, you can still receive mail and use a relay for sending, but you cannot deliver directly.
2. You can set a PTR (reverse DNS) record for the IPv4 address — and for IPv6 if you send over IPv6. Without forward-confirmed reverse DNS, Outlook and many corporate filters will reject or junk your mail regardless of SPF and DKIM.
3. The IP has no significant blocklist history. Recycled IPs from churned VPS customers are common, and reputation is attached to the address and sometimes the surrounding /24.

Check all three before buying an annual plan. Ask the provider directly about port 25 policy and rDNS; do not assume. Everything else — Postfix tuning, spam filtering, TLS — is under your control and well documented.

Software choices, briefly

There's no single right stack, but the options fall into three groups:

• Assembled by hand: Postfix (MTA) + Dovecot (IMAP) + Rspamd (filtering/DKIM signing) + OpenDKIM or Rspamd's signing module. Maximum control, most learning, most maintenance. Reasonable on a 2 GB VPS.
• Integrated distributions: Mailcow (Docker-based), Mail-in-a-Box, iRedMail. These wire up MTA, IMAP, webmail, filtering and DKIM keys for you. Mailcow realistically wants 6 GB+ RAM; Mail-in-a-Box is lighter but opinionated about owning the whole server and its DNS.
• Single-binary servers: Stalwart, Maddy. Modern, much simpler to configure, smaller ecosystems and fewer battle-tested deployment guides.

For a personal or small-team domain, Postfix + Dovecot + Rspamd or a Mailcow instance both work. Pick based on how much RAM you have and how much you want to hand-hold.

The DNS records that actually matter

Deliverability is mostly a DNS exercise. Set these up in this order.

A/AAAA and PTR

Give the server a hostname you control, e.g. mx1.example.com, pointing to its addresses. Then set the PTR for the IP to that same hostname. The forward and reverse must agree — this is "forward-confirmed reverse DNS" and it's the single most common reason self-hosted mail fails at Microsoft. Your Postfix myhostname and HELO/EHLO name must match the PTR exactly.

If you send over IPv6, either set IPv6 rDNS too or disable IPv6 for outbound SMTP (smtp_address_preference = ipv4). Gmail is notably strict about IPv6 senders lacking valid rDNS.

MX

example.com.  IN MX 10 mx1.example.com.

MX must point at a hostname, never an IP, and that hostname should not be a CNAME. A single MX is fine; a secondary MX only helps if it actually queues for you, and a badly configured backup MX is a spam magnet.

SPF

example.com.  IN TXT "v=spf1 mx -all"

Use mx if your MX also sends, or ip4:/ip6: literals if you prefer explicitness. Add include: mechanisms for any third party that sends as your domain (newsletter tools, monitoring, transactional relays). Stay within the 10-DNS-lookup limit.

-all (hard fail) is the correct end state. ~all (soft fail) is a reasonable place to sit for the first week while you find services you forgot about.

DKIM

Generate a 2048-bit RSA key, publish the public key at selector._domainkey.example.com, and sign all outbound mail. Rspamd, OpenDKIM and every integrated stack handle this. Keep the private key readable only by the signing user, and include it in your backup plan — losing it means every archived signed message becomes unverifiable and you have to rotate.

You can additionally publish an Ed25519 key with a second selector, but RSA remains the interoperable baseline; some verifiers still ignore Ed25519.

DMARC

_dmarc.example.com. IN TXT "v=DMARC1; p=none; rua=mailto:[email protected]; fo=1"

Start at p=none with aggregate reports enabled. Read the reports for two or three weeks, confirm that everything legitimate is passing SPF or DKIM with alignment (the authenticated domain matches the From: domain), then move to p=quarantine and eventually p=reject. Publishing p=reject on day one is how people silently kill their own mail.

Note that DMARC alignment is what large receivers check — passing SPF for your VPS hostname while sending From: your domain does not align. Forwarding breaks SPF alignment, which is exactly why DKIM matters.

MTA-STS, TLS-RPT and DANE

These are optional but cheap credibility signals and real security improvements:

• MTA-STS: publish _mta-sts.example.com TXT plus a policy file over HTTPS at mta-sts.example.com/.well-known/mta-sts.txt. Start with mode: testing, move to enforce. It prevents downgrade attacks against senders that support it (Gmail and Microsoft do).
• TLS-RPT: a TXT record that asks senders to report TLS failures to you. Useful diagnostics.
• DANE/TLSA: only meaningful if your zone is DNSSEC-signed. Widely used in the European mail ecosystem, and required by some German and Dutch providers' policies. Automate the TLSA rollover with your certificate renewal or you will break inbound mail at renewal time.

Sending: reputation, not configuration

Once DNS is correct, delivery depends on behaviour over time.

Warm up gradually. A brand-new IP sending a burst of a few hundred messages looks like a compromised host. Low, steady, human-shaped volume builds reputation faster than a batch send.

Rate-limit yourself. Configure Postfix destination concurrency and rate limits so that a bug or a compromised web app cannot dump thousands of messages before you notice. This one setting has saved more IPs than any DNS record.

Never run an open relay. Restrict relaying to authenticated users only, offer submission on 587 (STARTTLS) and 465 (implicit TLS), require strong passwords or app passwords, and fail2ban the SASL auth log. Auth brute-forcing on 587 is constant background noise on any public IP.

Monitor blocklists. Check Spamhaus ZEN, Barracuda and Spamcop for your IP periodically, and register for Google Postmaster Tools and Microsoft SNDS for your domain and IP. Both give you signal you cannot get any other way. Handle any listing immediately — delistings are usually straightforward if the cause is fixed.

Set up abuse and postmaster addresses that a human reads. abuse@ and postmaster@ on your domain are how you find out you have a problem before a provider disconnects you.

Microsoft (outlook.com, hotmail.com, live.com) is consistently the toughest destination for small self-hosted senders. Even with perfect authentication, mail from unknown low-volume IPs is sometimes junked or deferred. That is not a config bug you can fix; it's reputational, and it improves with recipient engagement over months.

Receiving: filtering without losing real mail

Inbound is more forgiving. A sane baseline:

• Reject non-existent recipients at SMTP time (no backscatter from bouncing later).
• Use Rspamd or SpamAssassin with DNSBL checks, SPF/DKIM/DMARC verification, and Bayesian training from user actions.
• Skip aggressive greylisting on a personal server; it delays password resets and confuses users more than it helps now that DNSBLs and content filtering are good.
• Enable opportunistic TLS for inbound and outbound (smtpd_tls_security_level = may, smtp_tls_security_level = may) plus DANE/MTA-STS where available. Requiring TLS for all inbound mail will lose legitimate messages.
• Cap message size and connection rates so one sender can't exhaust the disk or CPU.

Common mistakes

• PTR mismatch or missing entirely. Fix this first; nothing else compensates.
• HELO name that doesn't resolve. Postfix defaults to the system hostname, which may be localhost or a template name.
• Publishing p=reject before reading DMARC reports.
• Forgetting outbound IPv6. Mail goes out over v6 with no rDNS, gets rejected, and the admin debugs SPF for a day.
• No backups of /var/vmail, the mail database and DKIM keys. Mailboxes are the one thing on a mail server you truly cannot regenerate. Set up encrypted off-site backups with restic or Borg and actually test a restore before you migrate your primary address.
• Leaving management interfaces exposed. Webmail, Mailcow's admin UI, Dovecot's doveadm and databases should be behind a firewall, a VPN or at minimum strong auth. The same SSH hardening basics apply to every other admin surface on the box.
• Certificate renewal breaking SMTP. Reload Postfix and Dovecot in your ACME deploy hook, and update TLSA records if you use DANE.

Privacy: what self-hosting email does and doesn't give you

Self-hosting removes the provider that scans, indexes and could be compelled to hand over your mailbox. That's a real gain — your archive lives on hardware you control, under whatever jurisdiction you chose.

What it does not do:

• Metadata is still exposed. Every message you send reveals your server's IP and hostname in the Received headers, plus envelope addresses to every relay in the path. Running your own mail server usually makes you more identifiable, not less, because the IP maps to one domain and one operator.
• Content is only as private as the other side. Mail to a Gmail user sits in Google's storage. End-to-end encryption still requires PGP or S/MIME.
• At-rest encryption has limits on a VPS. The hypervisor operator can access running memory, so full-disk encryption on a remote server protects mainly against disk decommissioning and offline seizure, not a live compromise.
• Anonymous mail hosting is not a thing. If you want the service to be unlinkable to your identity, that's an account and payment question — see the Monero payment opsec guide — and it's separate from the mail domain itself, which is public by design.

Choosing where the mailbox physically lives is a genuine decision, though. A full-root KVM VPS in Romania keeps the data inside the EU and GDPR scope while remaining outside the usual US-headquartered mail ecosystem, and IronBalkans' no-KYC crypto billing means the hosting relationship itself doesn't require identity documents. Just confirm port 25 and rDNS handling with any provider before you commit to a mail workload.

Pros and cons of self-hosted email on a VPS

Pros: full control of filtering and retention, no per-mailbox pricing, unlimited aliases and domains, no third party reading or profiling your mail, excellent learning value, and a jurisdiction of your choosing.

Cons: deliverability to Microsoft and some corporate filters is an ongoing effort; you own every outage at 3 a.m.; blocklist and certificate maintenance never stops; losing mailbox data is catastrophic; and a compromised server can burn your domain's reputation quickly.

FAQ

Can I self-host if my provider blocks port 25? Partly. Inbound mail on port 25 and IMAP work fine. For outbound, route through an SMTP relay as a smarthost. You keep control of storage and filtering, at the cost of the relay seeing your outbound metadata and usually requiring an identified account.

Do I need a dedicated IP? Yes, for direct sending. Shared outbound IPs mean shared reputation.

How do I test the setup? swaks for SMTP conversations, dig for every record, openssl s_client -starttls smtp for TLS, a mail-tester-style scoring service for a full report, and a real test to a Gmail and an Outlook account — then read the received message's Authentication-Results header.

Should I run mail and web on the same VPS? It works, but a compromised web application then owns your mail server. If you can afford separation, separate them.

Takeaway

Get port 25, PTR and a clean IP confirmed first; publish A/AAAA, MX, SPF, DKIM and a p=none DMARC record; verify alignment from real test messages; then tighten DMARC, add MTA-STS, rate-limit outbound and back up mailboxes off-site with tested restores. Do those in order and self-hosted email is maintainable. Skip the reverse DNS and reputation work and no amount of Postfix tuning will save you.