Rachel's Yard

| A New Continuation

The last "guide" was pretty shitty, I know. But, fear not, I will save you.

Step 1: Consider your architecture

When I was writing Dermail, application level and infrastructure level redudancy and failover was considered, and it was a big thing. Of course, being a young programmer, I will make mistakes. Please open an issue on Github or send me an email when you do encouter problems when you are trying out Dermail.

Anyway, architecture. Dermail runs with three major softwares:

  1. node.js
  2. RethinkDB
  3. Redis

Therefore, any one of three components can be redundant.

RethinkDB

RethinkDB provides a dead simple approach for redundancy. Read RethinkDB's official documents on Scaling, sharding and replication, Architecture FAQ, and Failover.

You can literally control shards and replicas in the WebUI, and you decide how safe you want your data to be. In my application, I have three instances running in a cluster, and each instance is running as a VM on three separate hypervisor. Although, they are connected to a shared storage, so there's my single point of failure. When I have more capital, I will consider invest in an SAN with multipath redundancy. But for, this will have to do.

Redis

Come on, just Google on how to setup master-slave redis already.

In my application, all redis instance are running solo, meaning that there's no redundancy for message queue. I hope to solve that problem by introducing some form of permanant storage in case the queue dies catastrophically, then at least the queue can be restored manually.

node.js

All instances are running in a cluster mode (4 instances) by default, therefore redundancy and failover should be taken care of.

However, in my application:

  1. API is running behind two nginx (but without HA setup), with two instances.
  2. MTA has three geographically independent instances
  3. TX has two independent instances
  4. Webmail has only one instance running

The worse case senario would be the API died, and MTA somehow didn't save your incoming mails to be stored in the API, and MTA died as well. Therefore, that mail is lost permanantly. However, that would require all 7 (nginx x2, API x2, MTA x3) instances to die simultaneously to happen.

Step 2: Database (RethinkDB)

Please refers to the official document on how to install RethinkDB on your distro. I'm running Debian for its legendary stability.

If you do decide to run RethinkDB in a cluster, please consider running a proxy RethinkDB instance on your API instance, to alleviate the network and CPU pressure.

Step 3: Message Queue (Redis)

By default, apt-get install redis-server should be all you need. However, you do want to change the redis.conf when eviction does happen. You want to change it to LRU instead of volatile, because of Bull, the message queue implementation that I'm using, does not set expiration. Therefore, LRU would be your best bet.


For now on, assuming that you want to use the domain myemail.com as your domain, and mx-1.myemail.com as your MX server, api.myemail.com as your API endpoint, tx-1.myemail.com as your TX helper endpoint, and web.myemail.com as your Webmail endpoint.

Step 4: SMTP Outbound, TX

This should be pretty straightforward. On your VPS/dedicated server,

  1. Clone the TX repo (Github, git.fm)
  2. npm install
  3. npm install -g forever
  4. forever start cluster.js

This concludes the installation for your TX component. You want to point tx-1.myemail.com to this server's IP, and best to set the rDNS of the IP to tx-1.myemail.com as well.

Step 5: Center of Operation, API

This requires more configration, but it should be straightforward as well.

  1. Clone the API repo (Github, git.fm)

  2. npm install

  3. npm install -g pm2

  4. Install Redis on this server. Or if you prefer, you can put Redis on a separate server

  5. Install RethinkDB on this server. Or if you prefer, you can put RethinkDB on a separate server.

  6. Then, you will need to fill out the config.json:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    {
    "remoteSecret": "SECRET",
    "jwt": "ANOTHER SECRET",
    "gcm_api_key": "GCM KEY",
    "tx": [{
    "priority": 10,
    "hook": "https://TX-ENDPOINT/tx-hook"
    }],
    "s3": {
    "endpoint": "S3-ENDPOINT",
    "key": "S3-KEY",
    "secret": "S3-SECRET",
    "bucket": "S3-BUCKET"
    },
    "rethinkdb": {
    "host": "127.0.0.1",
    "port": 28015,
    "db": "dermail"
    },
    "redisQ": {
    "host": "127.0.0.1",
    "port": 6379
    }
    }

  7. remoteSecret is used to "authenticate" MTA or other components. You will need this again later

  8. jwt is your secret key to encrypt JWT token. Since Webmail is a single page app, it will use JWT for, authentication and authorization. Please, make it safe.

  9. gcm_api_key is your developer API key. You will need it and you will want it for push notification.

  10. tx is your TX endpoint. Change tx to your TX endpoint configured above, or you will not be able to send emails. It is an array of endpoints, so you can have multiple endpoints to avoid single point of failure.

  11. s3 is your S3 information. It can be AWS, it can be on-premise, but you will need a S3 somewhere for your attachments

  12. rethinkdb and redisQ should be straightforward.

And you are almost done.

Setting up the database

Under the API repo, there should be a database.md. The easiest way to do it is to copy the code, and put them into the data explorer of RethinkDB, then you are done with the setup on the database part.

Or, here's aggregated version. This assumes that your database is called dermail:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
r.db('dermail').tableCreate('accounts', {
primaryKey: 'accountId'
})
r.db('dermail').tableCreate('addresses', {
primaryKey: 'addressId'
})
r.db('dermail').tableCreate('attachments', {
primaryKey: 'attachmentId'
})
r.db('dermail').tableCreate('domains', {
primaryKey: 'domainId'
})
r.db('dermail').tableCreate('folders', {
primaryKey: 'folderId'
})
r.db('dermail').tableCreate('pushSubscriptions', {
primaryKey: 'userId'
})
r.db('dermail').tableCreate('messageHeaders', {
primaryKey: 'headerId'
})
r.db('dermail').tablseCreate('messages', {
primaryKey: 'messageId'
})
r.db('dermail').tableCreate('queue', {
primaryKey: 'queueId'
})
r.db('dermail').tableCreate('users', {
primaryKey: 'userId'
})
r.db('dermail').tableCreate('filters', {
primaryKey: 'filterId'
})
r.db('dermail').tableCreate('payload', {
primaryKey: 'endpoint'
})

Then, for the indicies:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
r.db('dermail').table("domains").indexCreate("domain")
r.db('dermail').table("domains").indexCreate("alias", {multi: true})
r.db('dermail').table("users").indexCreate("username")
r.db('dermail').table("accounts").indexCreate("userId")
r.db('dermail').table("folders").indexCreate("accountId")
r.db('dermail').table("messages").indexCreate("accountId")
r.db('dermail').table("queue").indexCreate("userId")
r.db('dermail').table("filters").indexCreate("accountId")
r.db('dermail').table("attachments").indexCreate("checksum")
r.db('dermail').table('messages').indexCreate('folderDate', [ r.row('folderId'), r.row('date')])
r.db('dermail').table('messages').indexCreate('unreadCount', [r.row('folderId'), r.row('isRead')])
r.db('dermail').table('accounts').indexCreate('userAccountMapping', [ r.row('userId'), r.row('accountId')])
r.db('dermail').table('messages').indexCreate('messageAccountMapping', [r.row('messageId'), r.row('accountId')])
r.db('dermail').table('folders').indexCreate('accountFolderMapping', [ r.row('accountId'), r.row('folderId')])
r.db('dermail').table('accounts').indexCreate('accountDomainId', [ r.row('account'), r.row('domainId')])
r.db('dermail').table('addresses').indexCreate('accountDomain', [ r.row('account'), r.row('domain')])
r.db('dermail').table('addresses').indexCreate('accountDomainAccountId', [ r.row('account'), r.row('domain'), r.row('accountId')])
r.db('dermail').table('folders').indexCreate('accountIdInbox', [ r.row('accountId'), r.row('displayName')])

Then, run pm2 start app.json, and your API should be up and running.

Reverse Proxy (nginx)

Please refers to the pro tips.

This concludes the installation for your API component. You want to point api.myemail.com to reverse proxy's IP, and best to set the rDNS of the IP to api.myemail.com as well.

Step 6: Create a first user

Refers to the firstUser.js under usefulScripts/ at the API, change the information accordingly. For the bcrypt hash, refers to the screenshot bcrypt.png. After you have changed the information, run node usefulScripts/firstUser.js to create you very first user. You can't do anything yet, because you don't have everything set up.

Step 7: SMTP Inbound, MTA

This should be pretty straightforward. On your VPS/dedicated server,

  1. Clone the MTA repo (Github, git.fm)

  2. npm install

  3. npm install -g pm2

  4. Install Redis on this server. Or if you prefer, you can put Redis on a separate server

  5. Then, you will need to fill out the config.json:

    1
    2
    3
    4
    5
    6
    7
    8
    {
    "redisQ":{
    "host": "127.0.0.1",
    "port": 6379
    },
    "remoteSecret": "SECRET",
    "apiEndpoint": "https://API"
    }

  6. redisQ should be straightforward.

  7. remoteSecret must match the one in API

  8. API is the API endpoint setup earlier. Change API to your API endpoint configured above, or you will not receive emails. IT MUST BE HTTPS. For example, https://api.myemail.com

Then, run pm2 start app.json, and your MTA should be up and running.

This concludes the installation for your MTA component. You want to point mx-1.myemail.com to this server's IP, and best to set the rDNS of the IP to mx-1.myemail.com as well.

Step 8: Interface, Webmail

Unfortunately, Dermail does not have IMAP/POP3 capability yet. Although they are on the roadmap. For now, the only way to interact with Dermail is either Webmail, or using the API.

This step will setup the Webmail of the Dermail system, and Dermail requests for resources via the API, so it is merely doing rendering.

  1. Clone the Webmail repo (Github, git.fm)

  2. npm install

  3. npm install -g pm2

  4. Then, you will need to fill out the config.json:

    1
    2
    3
    4
    5
    {
    "port": 2001,
    "apiEndpoint": "API",
    "siteURL": "DOMAIN"
    }

  5. API is the API endpoint setup earlier. Change API to your API endpoint configured above. For example, https://api.myemail.com

  6. Default port is 2001. Point your reverse proxy to this. 3.siteURL is where your webmail is accesible. For example, https://web.myemail.com WITHOUT TRAILING SLASHES

Reverse Proxy (nginx)

Come on, just Google this already

This concludes the installation for your Webmail component. You want to point web.myemail.com to reverse proxy's IP, and best to set the rDNS of the IP to web.myemail.com as well.

Step 9: Profit

Your instance of Dermail is now up and running. Send a first email from other email addresses to your Dermail address, assuming that you have setup the MX record correctly, and you should see a new email under Inbox!

If you run into problems, again, open an issue on Github, or send me an email.

Weightless Theme
Rocking Basscss
RSS