These are my implementation notes/tutorial for the Rohe domain management project; this only describes the email setup.
Rohe's email service is based on MySQL, Exim, Anubis, Dovecot, Squirrelmail, Clam-AV, Spam Assassin and Procmail (or Sieve).
Rohe's usage of MySQL is pretty standard, and will happen over the TCP port as well as a local socket. The ‘rohe’ database will be populated initially only by an external MySQL administrator, and the various applications will all use their own users, limited to SELECT only on the ROHE database.
CREATE DATABASE rohe;
INSERT INTO mysql.user SET Host = 'localhost', User = '***', Password = PASSWORD('********'), Select_priv = 'N', Insert_priv = 'N', Update_priv = 'N', Delete_priv = 'N', Create_priv = 'N', Drop_priv = 'N', Reload_priv = 'N', Shutdown_priv = 'N', Process_priv = 'N', File_priv = 'N', Grant_priv = 'N', References_priv = 'N', Index_priv = 'N', Alter_priv = 'N';
GRANT Select ON `rohe`.* TO 'anubis'@'localhost';
The initial users will be ‘exim’. You probably should use a front-end tool like phpmyadmin to get the users created if you are not familiar with MySQL, and don't forget that the database will need to be reloaded after the INSERTs, and after the GRANTs.
Next, the minimum rohe schema has to be loaded. This consists of two tables
These tables will be extended later, and more tables will be added. But right now we don't need them.
#
# Table structure for table `domain`
#
CREATE TABLE domain (
domain_id tinyint(4) NOT NULL auto_increment,
fqdn varchar(255) NOT NULL default '',
PRIMARY KEY (domain_id),
UNIQUE KEY domain_id_2 (domain_id),
KEY domain_id (domain_id)
) TYPE=MyISAM COMMENT='Domain names managed by rohe';
#
# Table structure for table `eaddr`
#
CREATE TABLE eaddr (
eaddr_id tinyint(4) NOT NULL auto_increment,
lhs varchar(255) NOT NULL default '',
domain_id tinyint(4) NOT NULL default '0',
default_addr enum('n','y') NOT NULL default 'n',
target enum('Maildir','alias') NOT NULL default 'Maildir',
target_data varchar(255) default NULL,
PRIMARY KEY (eaddr_id,eaddr_id),
UNIQUE KEY email_id_2 (eaddr_id),
KEY email_id (eaddr_id),
KEY emaddr_id (eaddr_id)
) TYPE=MyISAM COMMENT='email addresses';
Some data can be inserted. First, we declare a domain (test.domain)that we are interested in :-
INSERT INTO `domain` (`domain_id`, `fqdn`, `admin_id`) VALUES ('', 'test.domain', NULL);
Then we create a user in it - postmaster, who is the default user and forwards to a real address (you choose …)
INSERT INTO `eaddr` (`eaddr_id`, `lhs`, `domain_id`, `default`, `target`, `target_data`) VALUES ('', 'postmaster', '1', 'y', 'alias', 'test.domain.postmaster@somerealdomain');
This is sufficient to get us going for the first target action …
We're going to be using Exim v4 for email handling, so this needs to be downloaded and configured.
First of all, get the source (I'm using v4.33) and put it somewhere useful, a subdirectory of /usr/local/src isn't a bad option. Then build the source in the default configuration, to check that the basics work. Don't install it though!
cd /usr/local/src tar zxf exim-4.33.tar.gz cd exim-4.33 cp src/EDITME Local/Makefile
You need to edit the Local/Makefile, and specify a series of settings. Here's my site changes …
BIN_DIRECTORY=/usr/local/exim4/bin CONFIGURE_FILE=/usr/local/exim4/etc/exim.conf EXIM_USER=ref:exim SPOOL_DIRECTORY=/var/spool/exim4 #EXIM_MONITOR=eximon.bin LOG_FILE_PATH=/var/log/exim4/%slog
Now run make. This will compile the Exim binary, and you can use this opportunity to fix any errors or dependancies that crop up. Hopefully there aren't any! Once we know we can get the basic Exim built, we can go back to Local/Makefile and set up the more advanced options that we will need.
SUPPORT_MAILDIR=yes # ROUTER_IPLITERAL=yes # ROUTER_MANUALROUTE=yes # ROUTER_QUERYPROGRAM=yes # LOOKUP_DBM=yes # LOOKUP_LSEARCH=yes LOOKUP_MYSQL=yes LOOKUP_INCLUDE=-I /usr/include/mysql LOOKUP_LIBS=-lmysqlclient AUTH_CRAM_MD5=yes HAVE_ICONV=yes SUPPORT_TLS=yes TLS_LIBS=-lssl -lcrypto
make again, and this time you should end up with the most appropriate exim binary for rohe, with only the features we need, and no others. You might need some extra packages at this stage, like OpenSSL. Now you can make install as root to get exim installed.
The default config file might need some editing for a basic test to succeed, especially if you are running another MTA anywhere as production. Check local_interfaces = 192.168.10.158 and daemon_smtp_ports = 25 particularly. Then run the Exim daemon in test mode.
# ../bin/exim -bV Exim version 4.33 #2 built 15-Sep-2004 21:52:52 Copyright (c) University of Cambridge 2004 Berkeley DB: Sleepycat Software: Berkeley DB 3.2.9: (April 7, 2002) Support for: iconv() OpenSSL Lookups: mysql Authenticators: cram_md5 Routers: accept dnslookup redirect Transports: appendfile/maildir autoreply pipe smtp Fixed never_users: 0 Configuration file is /usr/local/exim4/etc/exim.conf
If that looks sensible, run it as a real (foreground) program, # ../bin/exim -d -bd and try to telnet to it from another shell.
$ telnet 192.168.10.158 25 Trying 192.168.10.158... Connected to exim4. Escape character is '^]'. 220 mta0.testserver ESMTP Exim 4.33 Wed, 15 Sep 2004 22:00:03 +1200 quit 221 mta0.testserver closing connection Connection closed by foreign host.
We've got a working Exim program! Now we can go into the config file as and when we need to add new features!
Now we can configure it for these tasks :-
First, Exim has to be configured to talk to MySQL, and given some queries to use. These are best added as a series of macros, before the Main Configuration secrion …
######################################################################
# MACRO DEFINITIONS #
######################################################################
MYQ_SERVER=localhost
MYQ_DATABASE=rohe
MYQ_USER=exim
MYQ_PASSWORD=******
Then, in the main configuration section, before the domainlist specifications, pop in the actual mysql configuration command :-
hide mysql_servers = "MYQ_SERVER/MYQ_DATABASE/MYQ_USER/MYQ_PASSWORD"
The “hide” command means that these parameters will not be available to anyone dumping out the Exim configuration parameters from the command-line – you'll have to read the config file to get them, and we can protect that.
In order to accept incoming email for a user who has an alias listed (i.e. we don't have to do any delivery to the local machine) we first need to allow roheDB-listed domains as valid recipients of mail, then we have to add a router that will be able to look up the redirection.
We're adding all the SQL queries as macros in the same place, and then using them further on down in the rules where appropriate.
The first query tells us how to identify a domain that is allowed to receive email, and sets this up as a new domainlist object, which is included in the default local_domains object.
MYQ_LOCAL_DOMAINS=SELECT DISTINCT fqdn FROM domain NATURAL JOIN eaddr WHERE fqdn='$domain'
domainlist local_domains = @ : +virtual_domains
domainlist virtual_domains = ${lookup mysql{MYQ_LOCAL_DOMAINS}}
Next, we need to get a query that will retrieve the target of an aliased address, and a router that will test for them. The order in which routers are specified is critical - this one goes right after the dnslookup router, which will be handling outgoing mail.
MYQ_ALIAS=SELECT target_data FROM eaddr NATURAL JOIN domain WHERE lhs='$local_part' AND fqdn='$domain' AND target='alias'
### This router allows aliasing to occur in the virtual domains
maildb_alias:
driver = redirect
domains = +virtual_domains
allow_fail
allow_defer
data = ${lookup mysql{MYQ_ALIAS}}
While we're here, let's comment out all the other routers after this one, just leaving in a small stub for the last one to reject the message (which should be rejected by the absense of further routers, actually).
rejecteverythingelse:
driver = redirect
allow_fail
data = :fail: Rejected Message
cannot_route_message = Message Rejected
Lets test our config so far. Run exim -bV to confirm the config file is valid, then execute exim -d -bd in one window, and connect to the mail server in another …
First test is for impossible addresses, which should be rejected :-
$ nc exim4 25
220 mta0.testserver ESMTP Exim 4.33 Thu, 16 Sep 2004 12:11:22 +1200
helo jim
250 mta0.testserver Hello exim4 [192.168.10.158]
mail from:<random@invalid>
250 OK
rcpt to:<otherandom@otherinvalid>
550-Verification failed for <random@invalid>
550-Unrouteable address
550 Sender verify failed
quit
221 mta0.testserver closing connection
Then we test with real, but external addresses.
$ nc exim4 25
220 mta0.testserver ESMTP Exim 4.33 Thu, 16 Sep 2004 12:13:21 +1200
helo jim
250 mta0.testserver Hello exim4 [192.168.10.158]
mail from:<jim@gonzul.net>
250 OK
rcpt to:<jim@gonzul.net>
550 relay not permitted
quit
221 mta0.testserver closing connection
Then we check to see if mail to our sample address, postmaster@test.domain, will be accepted.
$ nc exim4 25
220 mta0.testserver ESMTP Exim 4.33 Thu, 16 Sep 2004 12:17:14 +1200
helo jim
250 mta0.testserver Hello exim4 [192.168.10.158]
mail from:<jim@gonzul.net>
250 OK
rcpt to:<postmaster@test.domain>
250 Accepted data
354 Enter message, ending with "." on a line by itself
Subject: test message
test message
.
250 OK id=1C7k4p-00089B-0P
quit
221 mta0.testserver closing connection
By watching the output from exim in the other window, you will see it accepting the message, expanding the alias, and sending it on to the remote destination. Hooray!
Finally, to be complete, we test for mail to a non-postmaster address in the test domain :-
$ nc exim4 25
220 mta0.testserver ESMTP Exim 4.33 Thu, 16 Sep 2004 12:52:08 +1200
helo jim
250 mta0.testserver Hello exim4 [192.168.10.158]
mail from:<jim@gonzul.net>
250 OK
rcpt to:<jim@test.domain>
550 Rejected Message
quit
221 mta0.testserver closing connection
Now we want to see how to send mail to “anything else” in our test.domain - we've set the DB flag ‘default’ to ‘y’ already, all we need is another router in Exim. This will go just after the maildb_alias router, and will get the email address of the ‘default’ entry for the target domain, if any.
MYQ_DEFAULT=SELECT concat(lhs,'@',fqdn) FROM eaddr NATURAL JOIN domain WHERE fqdn='$domain' AND default_addr='y'
maildb_default:
driver = redirect
domains = +virtual_domains
data = ${lookup mysql{MYQ_DEFAULT}}
Bouncing mail straight back out, as in the previous two routers, is fine for many people, but this system will also accept mail for local delivery. As the mail users will be virtual, they obviously don't have a home directlry or anything like that. Instead, we're going to be storing their mail in Maildir format in /var/mailstore/domain/user/. The router for this, maildb_maildir, must come before maildb_default.
MYQ_MAILDIR=SELECT target FROM eaddr NATURAL JOIN domain WHERE fqdn='$domain' AND lhs='$local_part' AND target='Maildir'
maildb_maildir:
driver = accept
domains = +virtual_domains
condition = ${lookup mysql{MYQ_MAILDIR}}
transport = maildb_maildir_delivery
This router accepts mail that's destined for a Maildir, and hands over to a transport director, which we also need to specify further down the config file :-
maildb_maildir_delivery:
driver = appendfile
maildir_format
user = mail
group = mail
mode = 0660
directory = /var/mailstore/${domain}/${local_part}/Maildir/
We've fixed the ownership and mode for these files to be the ‘mail’ user - not the ‘exim’ user. Also note that exim will automatically create the Maildir if it needs to (and if it has permissions to /var/mailstore). Let's create a suitable mailstore directory …
Now we need to add a user to the database, and declare them as having a Maildir …
INSERT INTO `eaddr` (`eaddr_id`, `lhs`, `domain_id`, `default_addr`, `target`, `target_data`) VALUES ('', 'jim', '1', 'n', 'Maildir', NULL);
And send this user an email, as a test!
$ nc exim4 25
220 mta0.testserver ESMTP Exim 4.33 Thu, 16 Sep 2004 14:14:11 +1200
helo jim
250 mta0.testserver Hello exim4 [192.168.10.158]
mail from:<jim@gonzul.net>
250 OK
rcpt to:<jim@test.domain>
250 Accepted
data
354 Enter message, ending with "." on a line by itself
Subject: test message to Maildir?
This should end up in a maildir ...
.
250 OK id=1C7ln7-0000UK-Kn
quit
221 mta0.testserver closing connection
Watching the exim output, you might see the Maildir being created and populated - but in any case we can have a quick look in the mailstore to see what happened, using tree (or ls -R if you don't have tree)
# tree /var/mailstore
/var/mailstore
`-- test.domain
`-- jim
`-- Maildir
|-- cur
|-- new
| `-- 1095300890.H651886P1886.mta0.testserver
`-- tmp
6 directories, 1 file
At this point we can happily receive mail.
For users with a local mailstore, we'd like to run the message through a filter first (Actually, I'd like to be able to do it for all mail users, but I'm willing to forgo this at the moment - I don't know where to store their filter files, for example). Exim will handle both Sieve and it's own filter language, transparently.
Filter files will be used if specified in target_data for a Maildir user. This is the file name, appended to /var/mailstore/domain/user/.
Actually, this is new territory at the moment, so I'll not implement it just yet. Access mail
We're going to be using the Dovecot IMAP server - the latest version does support MySQL, even though the documentation doesn't admit to it.
The Dovecot IMAP server comes from http://www.dovecot.org and we're looking at version 0.99.11
Grab the source into /usr/local/src/, extract and cd …
./configure --with-ssl=openssl --with-mysql --without-static-userdb --without-passwd --without-passwd-file --without-shadow --with-storages=maildir
This adds the capabilities we want, and removes the ones we don't.
Install prefix ...................... : /usr/local File offsets ........................ : 64bit Building with SSL support ........... : yes (OpenSSL) Building with IPv6 support .......... : yes Building with pop3 server ........... : yes Building with user database modules . : mysql (modules) Building with password lookup modules : mysql (modules)
We start by declaring that we want to use imaps, which means we need some SSL keys. These will need to be signed by a CA - if you don't need to use a public CA (and for this sort of service you shouldn't need to) it's still worthwhile setting up a root CA (and not particularly difficult either -see http://clug.net.nz/index.php/OpenSSL )
openssl genrsa -out dovecot.servername.key 1024
openssl req -new -key dovecot.servername.key -out dovecot.servername.csr
Org Unit: dovecot
Common Name: servername.fqdn
Then send the CSR to your CA, and they will return a certificate file.
We set the options to request IMAP and IMAPS protocols, to listen on the right interface, which SSL files to consult, which UID to access files as (this is a virtual mail system, so everything lives under one uid), where to find the mailboxes, and what authentication mechanisms to use.
protocols = imap imaps
imap_listen = interface
ssl_disable = no
ssl_cert_file = /usr/local/etc/dovecot.servername.crt
ssl_key_file = /usr/local/etc/dovecot.servername.key
disable_plaintext_auth = yes
first_valid_uid = 8
last_valid_uid = 8
first_valid_gid = 8
last_valid_gid = 8
mail_extra_groups = mail
default_mail_env = maildir:/var/mailstore/%d/%u/Maildir
auth = default
auth_mechanisms = plain
auth_userdb = mysql /usr/local/etc/dovecot-mysql.conf
auth_passdb = mysql /usr/local/etc/dovecot-mysql.conf
auth_user = dovecot
Also, create a new system user for dovecot. No shell or possibility of login.
While we're testing, upgrade the error logging a little …
log_path = /dev/stderr
info_log_path = /dev/stderr
verbose_ssl = yes
auth_verbose = yes
auth_debug = yes
We also need to set up the (referenced in dovecot.conf) dovecot-mysql.conf file. We'll use a socket connection rather than a TCP connection, it should be cleaner. Then we set the queries that will be used - the user_query is a bit of a hack, because everything is running under the same user … so I'm just returning static data from a SQL query.
But we also now need to store authemtication data in the database, so we'll create the eauth table and populate it :-
CREATE TABLE eauth (
eauth_id tinyint(4) NOT NULL auto_increment,
eaddr_id tinyint(4) NOT NULL default '0',
pw_plain varchar(255) NOT NULL default '',
imaps enum('y','n') NOT NULL default 'y',
PRIMARY KEY (eauth_id),
UNIQUE KEY emailuser_id_2 (eauth_id),
KEY emailuser_id (eauth_id)
) TYPE=MyISAM;
INSERT INTO `eauth` (`eauth_id`, `eaddr_id`, `pw_plain`, `imaps`) VALUES ('', '9', '*****', 'y');
#db_host = localhost
#db_port = 3306
db_unix_socket = /var/run/mysqld/mysqld.sock
db = rohe
db_user = dovecot
db_passwd = *****
db_client_flags = 0
default_pass_scheme = PLAIN
password_query = SELECT pw_plain FROM eauth NATURAL JOIN eaddr NATURAL JOIN domain WHERE target='Maildir' AND lhs='%n' AND fqdn = '%d'
user_query = SELECT '/var/mailstore/%d/%n' as home, 'maildir:/var/mailstore/%d/%n/Maildir' as mail, 'mail' as system_user, '8' as uid, '8' as gid
Now we can start our Dovecot server from the command-line, and see what happens :-
dovecot -F -c /usr/local/etc/dovecot.conf
Because we have enabled SSL, we can't easily test this server from the command-line, we'll have to pass everything through an SSL-capable client, and a standard mail client program isn't a bad choice. mutt is an excellent choice for this.
Create this simple ~/.muttrc file :-
set spoolfile=imaps://jim\@test.domain@mx0/INBOX
set folder=imaps://jim\@test.domain@mx0/INBOX
and then run up mutt. You should see your SSL certificate being presented for verification - you may accept this o nce or a lways. Then you should be asked for your account password, and finally you'll be presented with the message(s) that you sent during the exim tests!
Now we can delete the extra debug lines from the conf file, and invoke the individual dovecot components from xinetd …
# Dovecot
service imap
{
socket_type = stream
user = root
server = /usr/local/libexec/dovecot/imap-login
wait =no
}
service imaps
{
socket_type = stream
user = root
server = /usr/local/libexec/dovecot/imap-login
server_args = --ssl
wait =no
}
Now we have IMAP working, we can run up webmail. Because this service all runs on the same server, I'm not using SSL for webmail to talk to Dovecot. Build Squirrelmail, then have a look in conf/config.php, or run ./configure …
SMTP Settings
-------------
4. SMTP Server : mx0.servername
5. SMTP Port : 25
6. POP before SMTP : false
7. SMTP Authentication : cram-md5
8. Secure SMTP (TLS) : false
IMAP Settings
--------------
4. IMAP Server : mx0.servername
5. IMAP Port : 143
6. Authentication type : login
7. Secure IMAP (TLS) : false
8. Server software : other
9. Delimiter : detect
Note that at this stage, you are unlikely to be very successful in sending mail from Squirrelmail to anything except yourself, because we haven't provided any config to allow you to send elsewhere …
Check the Squirrelmail implementation by visiting its webpage and logging on.
We've asked Squirrelmail to authenticate in order to send mail (which is good) but we don't need the overhead of TLS, because this is on the same host. We therefore use CRAM-MD5 for the authentication method, as this at least obscures the password for the user.
Exim needs to have the cram authenticator set up ;-
MYQ_AUTH_PW=SELECT pw_plain FROM eauth NATURAL JOIN eaddr NATURAL JOIN domain WHERE concat(lhs,'@',fqdn)='$1'
cram:
driver = cram_md5
public_name = CRAM-MD5
server_advertise_condition = ${if eq{$tls_cipher}{}{${if eq{$sender_host_address}{192.168.10.158}{yes}{no}}}{yes}}
server_secret = ${lookup mysql{MYQ_AUTH_PW}}
server_set_id = $1
Then everything works well … Squirrelmail can send via SMTP! Auth is restricted to just the local server connections, or to TLS users …
Use http://www.net-track.ch/opensource/cmd5/ to generate valid CRAM challenge responses.
Anubis should listen on port 24, and use the same authentication methods as exim .. it needs to provide onwards TLS (patch to 3.9.95 available) and onwards AUTH (which currently needs a config-file per user).