Proxy IMAP/POP traffic and SMTP authentication

Motivation

I’m currently moving virtual servers from an old Virtualmin hosting instance to a new one. Because I do not have time for moving all virtual servers at once I need to make sure the commonly used server name for accessing IMAP, POP and SMTP will work for both - users already moved and users still using the old server. Since I did not find an easy to follow How-To for a scenario like this I just want to share here what worked for me.

Current and desired scenarios

That’s the situation before any users got moved - both servers are running the Virtualmin stack (Dovecot, Postfix):

mail.example.com => CNAME for old-server.example.com
new-server.example.com

The idea is to transparently switch to this configuration:

old-server.example.com
mail.example.com => CNAME for new-server.example.com

Still many users need to access their email at old-server.example.com. I don’t want to ask everyone to change the configuration of their email clients from mail.example.com to the new or old server depending on there current status.

Note: old-server.example.com, new-server.example.com and mail.example.com are just examples.

Solution

The solution is to have new-server.example.com act as proxy for old-server.example.com (common email protocols only): users of old-server.example.com are able to authenticate at new-server.example.com even if new-server.example.com does not know them.

Instead new-server.example.com need to forward (proxy) the authentication request to old-server.example.com for unknown users. But it must only forward authentication requests for users or domains which actually exists at old-server.example.com.

Also new-server.example.com need to proxy IMAP/POP data from the user’s email client to old-server.example.com. SMTP traffic does not need to be proxied because users can send email using both servers if the SPF record of their domains are configured accordingly.

Now to the real configuration as it is working at my new-server.example.com - a box running CentOS 8 btw.

Proxying Dovecot authentication and traffic

Dovecot authentication is handled by password databases (passdb). It can use more than one. Good for us: a second one will be used as fallback if the first one failed. So what we need is a passdb which forwards authentication request to old-server.example.com in case it doesn’t find the user locally (at new-server.example.com).

Create a file /etc/dovecot/conf.d/old-server-auth-proxy.conf.ext containing:

# Authentication for passwd-file users. Included from 10-auth.conf.
#
# passwd-like file with specified location.
# <doc/wiki/AuthDatabase.PasswdFile.txt>

passdb {
  driver = passwd-file
  args = username_format=%Ld /etc/dovecot/old-server-proxy.passwd
}

Create the passwd-like file /etc/dovecot/old-server-proxy.passwd which defines what domains are allowed to be proxied:

# Used by conf.d/old-server-auth-proxy.conf.ext
# No restart needed if conten changes.
domain1.tld::::::: nopassword=y proxy=y ssl=any-cert host=old-server.example.com
domain2.tld::::::: nopassword=y proxy=y ssl=any-cert host=<IP of old-server.example.com>

One line per domain (aka Virtual server). It will work for all users of that domain. You may use a user-centric approach instead.
nopassword=y Authentication takes place at old-server.example.com (works for IMAP authentication mechanisms plain only).
proxy=y Enables the proxying.
ssl=any-cert Use SSL port (993) but disable SSL cert verification. I did not get verification working when using the Let’s Encrypt CA. See notes at the Proxy PasswordDatabase documentation about this.
host=old-server.example.com Use hostname or IP of the server the authentication request must be send to.

See the Proxy PasswordDatabase documentation for more options and explanations.

Load old-server-auth-proxy.conf.ext in the Dovecot configuration:

echo "!include old-server-auth-proxy.conf.ext" | sudo tee -a /etc/dovecot/conf.d/10-auth.conf

systemctl restart dovecot

That’s the IMAP/POP part - let’s test it.

At old-server.example.com - no proxying needed

openssl s_client -connect old-server.example.com:993

. login user@domain1.tld <password in plain text>
. list "" "*"
. logout

Should generate something like:

. OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE] Logged in
* LIST (\HasNoChildren \Drafts) "." Drafts
* LIST (\HasNoChildren \Sent) "." Sent
* LIST (\HasNoChildren) "." trash
* LIST (\HasNoChildren) "." spam
* LIST (\HasNoChildren \Trash) "." Trash
* LIST (\HasNoChildren) "." INBOX
. OK List completed.
* BYE Logging out
. OK Logout completed.
closed

Now try the same with new-server.example.com. The same IMAP commands will work regardless if used directly at old-server.example.com or forwarded to it by new-server.

Watch the Dovecot logs at new-server.example.com for entries like:

Jul 30 10:26:07 new-server.example.com auth[46950]: pam_unix(dovecot:auth): check pass; user unknown
Jul 30 10:26:07 new-server.example.com auth[46950]: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=user@domain1.tld rhost=2a02:xxx:yyy:zzz:xxx:yyy:zzz:xxx

Jul 30 10:26:08 new-server.example.com dovecot[1284]: imap-login: proxy(user@domain1.tld): started proxying to 10.0.0.2:993: user=<user@domain1.tld>, method=PLAIN, rip=2a02:xxx:yyy:zzz:xxx:yyy:zzz:xxx, lip=2a01:xxx:yyy:zzz::4, TLS, session=<+AGX+FLINOMqAoEKgwAkgF2gVQTxqs9s>
Jul 30 10:26:08 new-server.example.com dovecot[1284]: imap-login: proxy(user@domain1.tld): disconnecting 2a02:xxx:yyy:zzz:xxx:yyy:zzz:xxx (Disconnected by server(0s idle, in=307, out=761)): user=<user@domain1.tld>, method=PLAIN, rip=2a02:xxx:yyy:zzz:xxx:yyy:zzz:xxx, lip=2a01:xxx:yyy:zzz::4, TLS, session=<+AGX+FLINOMqAoEKgwAkgF2gVQTxqs9s>

As you can see Dovecot first tries to authenticate user@domain1.tld as local user (via PAM). When that fails it tries the next available mechanism which is our newly configured passwd-file based proxy.

Next step will be..

Proxying SMTP authentication

Only authentication requests need to be proxied from new-server.example.com to old-server. Users of old-server.example.com can send email through new-server.example.com if new-server.example.com will be added to the SPF record of the user’s domain as an allowed sending host.

So if the SPF record of domain1.tld was:

"v=spf1 mx -all"

It will become:

"v=spf1 mx a:new-server.example.com -all"

That is still correct after moving domain1.tld from old-server.example.com to new-server.example.com although probably somewhat redundant if the MX(s) of domain1.tld will be the same as new-server.example.com.

Last todo is switching Postfix to authenticate using Dovecot. Dovecot has an answer for that too - search for the # Postfix smtp-auth part in /etc/dovecot/conf.d/10-master.conf and insert/activate there the following configuration there (inside of the service auth {} section):

# Postfix smtp-auth
unix_listener /var/spool/postfix/private/auth {
  mode = 0660
  user = postfix
  group = postfix
}

Postfix in Virtualmin is configured to use Cyrus SASL for authentication. Configure it to use Dovecot instead. At a teminal enter:

postconf smtpd_sasl_auth_enable=yes
postconf smtpd_sasl_path=private/auth
postconf smtpd_sasl_type=dovecot

Now restart both and watch their logs:

systemctl restart dovecot
postfix reload

journalctl -xf -u dovecot -u postfix

Test it using your Email client or a tool like Swaks - Swiss Army Knife SMTP:

swaks --from user@domain1.tld --to someone@else.tld \
  --auth --auth-user user@domain1.tld --auth-password '***' \
  --tls --server new-server.example.com:25

You should see many lines of SMTP protocol speak and between it a successful authentication as well as sending of the email:

...
<~  235 2.7.0 Authentication successful
...
<~  250 2.0.0 Ok: queued as 65C3C2013C6D
 ~> QUIT
<~  221 2.0.0 Bye

The Dovecot log will contain the same imap-login: proxy(user@domain1.tld) entries as seen above.

Questions? Discuss it at the Virtualmin Forum.

Motivation#

Current and desired scenarios#

Solution#

Proxying Dovecot authentication and traffic#

Proxying SMTP authentication#

Motivation

Current and desired scenarios

Solution

Proxying Dovecot authentication and traffic

Proxying SMTP authentication