Rachel's Yard

| A New Continuation
Tags Redis

The last "guide" was pretty shitty, I know. But, fear not, I will save you.

Step 1: Consider your architecture

When I was writing Dermail, application level and infrastructure level redudancy and failover was considered, and it was a big thing. Of course, being a young programmer, I will make mistakes. Please open an issue on Github or send me an email when you do encouter problems when you are trying out Dermail.

Anyway, architecture. Dermail runs with three major softwares:

  1. node.js
  2. RethinkDB
  3. Redis

Therefore, any one of three components can be redundant.

RethinkDB

RethinkDB provides a dead simple approach for redundancy. Read RethinkDB's official documents on Scaling, sharding and replication, Architecture FAQ, and Failover.

You can literally control shards and replicas in the WebUI, and you decide how safe you want your data to be. In my application, I have three instances running in a cluster, and each instance is running as a VM on three separate hypervisor. Although, they are connected to a shared storage, so there's my single point of failure. When I have more capital, I will consider invest in an SAN with multipath redundancy. But for, this will have to do.

Redis

Come on, just Google on how to setup master-slave redis already.

In my application, all redis instance are running solo, meaning that there's no redundancy for message queue. I hope to solve that problem by introducing some form of permanant storage in case the queue dies catastrophically, then at least the queue can be restored manually.

node.js

All instances are running in a cluster mode (4 instances) by default, therefore redundancy and failover should be taken care of.

However, in my application:

  1. API is running behind two nginx (but without HA setup), with two instances.
  2. MTA has three geographically independent instances
  3. TX has two independent instances
  4. Webmail has only one instance running

The worse case senario would be the API died, and MTA somehow didn't save your incoming mails to be stored in the API, and MTA died as well. Therefore, that mail is lost permanantly. However, that would require all 7 (nginx x2, API x2, MTA x3) instances to die simultaneously to happen.

Step 2: Database (RethinkDB)

Please refers to the official document on how to install RethinkDB on your distro. I'm running Debian for its legendary stability.

If you do decide to run RethinkDB in a cluster, please consider running a proxy RethinkDB instance on your API instance, to alleviate the network and CPU pressure.

Step 3: Message Queue (Redis)

By default, apt-get install redis-server should be all you need. However, you do want to change the redis.conf when eviction does happen. You want to change it to LRU instead of volatile, because of Bull, the message queue implementation that I'm using, does not set expiration. Therefore, LRU would be your best bet.


For now on, assuming that you want to use the domain myemail.com as your domain, and mx-1.myemail.com as your MX server, api.myemail.com as your API endpoint, tx-1.myemail.com as your TX helper endpoint, and web.myemail.com as your Webmail endpoint.

Step 4: SMTP Outbound, TX

This should be pretty straightforward. On your VPS/dedicated server,

  1. Clone the TX repo (Github, git.fm)
  2. npm install
  3. npm install -g forever
  4. forever start cluster.js

This concludes the installation for your TX component. You want to point tx-1.myemail.com to this server's IP, and best to set the rDNS of the IP to tx-1.myemail.com as well.

Step 5: Center of Operation, API

This requires more configration, but it should be straightforward as well.

  1. Clone the API repo (Github, git.fm)

  2. npm install

  3. npm install -g pm2

  4. Install Redis on this server. Or if you prefer, you can put Redis on a separate server

  5. Install RethinkDB on this server. Or if you prefer, you can put RethinkDB on a separate server.

  6. Then, you will need to fill out the config.json:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    {
    "remoteSecret": "SECRET",
    "jwt": "ANOTHER SECRET",
    "gcm_api_key": "GCM KEY",
    "tx": [{
    "priority": 10,
    "hook": "https://TX-ENDPOINT/tx-hook"
    }],
    "s3": {
    "endpoint": "S3-ENDPOINT",
    "key": "S3-KEY",
    "secret": "S3-SECRET",
    "bucket": "S3-BUCKET"
    },
    "rethinkdb": {
    "host": "127.0.0.1",
    "port": 28015,
    "db": "dermail"
    },
    "redisQ": {
    "host": "127.0.0.1",
    "port": 6379
    }
    }

  7. remoteSecret is used to "authenticate" MTA or other components. You will need this again later

  8. jwt is your secret key to encrypt JWT token. Since Webmail is a single page app, it will use JWT for, authentication and authorization. Please, make it safe.

  9. gcm_api_key is your developer API key. You will need it and you will want it for push notification.

  10. tx is your TX endpoint. Change tx to your TX endpoint configured above, or you will not be able to send emails. It is an array of endpoints, so you can have multiple endpoints to avoid single point of failure.

  11. s3 is your S3 information. It can be AWS, it can be on-premise, but you will need a S3 somewhere for your attachments

  12. rethinkdb and redisQ should be straightforward.

And you are almost done.

Setting up the database

Under the API repo, there should be a database.md. The easiest way to do it is to copy the code, and put them into the data explorer of RethinkDB, then you are done with the setup on the database part.

Or, here's aggregated version. This assumes that your database is called dermail:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
r.db('dermail').tableCreate('accounts', {
primaryKey: 'accountId'
})
r.db('dermail').tableCreate('addresses', {
primaryKey: 'addressId'
})
r.db('dermail').tableCreate('attachments', {
primaryKey: 'attachmentId'
})
r.db('dermail').tableCreate('domains', {
primaryKey: 'domainId'
})
r.db('dermail').tableCreate('folders', {
primaryKey: 'folderId'
})
r.db('dermail').tableCreate('pushSubscriptions', {
primaryKey: 'userId'
})
r.db('dermail').tableCreate('messageHeaders', {
primaryKey: 'headerId'
})
r.db('dermail').tablseCreate('messages', {
primaryKey: 'messageId'
})
r.db('dermail').tableCreate('queue', {
primaryKey: 'queueId'
})
r.db('dermail').tableCreate('users', {
primaryKey: 'userId'
})
r.db('dermail').tableCreate('filters', {
primaryKey: 'filterId'
})
r.db('dermail').tableCreate('payload', {
primaryKey: 'endpoint'
})

Then, for the indicies:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
r.db('dermail').table("domains").indexCreate("domain")
r.db('dermail').table("domains").indexCreate("alias", {multi: true})
r.db('dermail').table("users").indexCreate("username")
r.db('dermail').table("accounts").indexCreate("userId")
r.db('dermail').table("folders").indexCreate("accountId")
r.db('dermail').table("messages").indexCreate("accountId")
r.db('dermail').table("queue").indexCreate("userId")
r.db('dermail').table("filters").indexCreate("accountId")
r.db('dermail').table("attachments").indexCreate("checksum")
r.db('dermail').table('messages').indexCreate('folderDate', [ r.row('folderId'), r.row('date')])
r.db('dermail').table('messages').indexCreate('unreadCount', [r.row('folderId'), r.row('isRead')])
r.db('dermail').table('accounts').indexCreate('userAccountMapping', [ r.row('userId'), r.row('accountId')])
r.db('dermail').table('messages').indexCreate('messageAccountMapping', [r.row('messageId'), r.row('accountId')])
r.db('dermail').table('folders').indexCreate('accountFolderMapping', [ r.row('accountId'), r.row('folderId')])
r.db('dermail').table('accounts').indexCreate('accountDomainId', [ r.row('account'), r.row('domainId')])
r.db('dermail').table('addresses').indexCreate('accountDomain', [ r.row('account'), r.row('domain')])
r.db('dermail').table('addresses').indexCreate('accountDomainAccountId', [ r.row('account'), r.row('domain'), r.row('accountId')])
r.db('dermail').table('folders').indexCreate('accountIdInbox', [ r.row('accountId'), r.row('displayName')])

Then, run pm2 start app.json, and your API should be up and running.

Reverse Proxy (nginx)

Please refers to the pro tips.

This concludes the installation for your API component. You want to point api.myemail.com to reverse proxy's IP, and best to set the rDNS of the IP to api.myemail.com as well.

Step 6: Create a first user

Refers to the firstUser.js under usefulScripts/ at the API, change the information accordingly. For the bcrypt hash, refers to the screenshot bcrypt.png. After you have changed the information, run node usefulScripts/firstUser.js to create you very first user. You can't do anything yet, because you don't have everything set up.

Step 7: SMTP Inbound, MTA

This should be pretty straightforward. On your VPS/dedicated server,

  1. Clone the MTA repo (Github, git.fm)

  2. npm install

  3. npm install -g pm2

  4. Install Redis on this server. Or if you prefer, you can put Redis on a separate server

  5. Then, you will need to fill out the config.json:

    1
    2
    3
    4
    5
    6
    7
    8
    {
    "redisQ":{
    "host": "127.0.0.1",
    "port": 6379
    },
    "remoteSecret": "SECRET",
    "apiEndpoint": "https://API"
    }

  6. redisQ should be straightforward.

  7. remoteSecret must match the one in API

  8. API is the API endpoint setup earlier. Change API to your API endpoint configured above, or you will not receive emails. IT MUST BE HTTPS. For example, https://api.myemail.com

Then, run pm2 start app.json, and your MTA should be up and running.

This concludes the installation for your MTA component. You want to point mx-1.myemail.com to this server's IP, and best to set the rDNS of the IP to mx-1.myemail.com as well.

Step 8: Interface, Webmail

Unfortunately, Dermail does not have IMAP/POP3 capability yet. Although they are on the roadmap. For now, the only way to interact with Dermail is either Webmail, or using the API.

This step will setup the Webmail of the Dermail system, and Dermail requests for resources via the API, so it is merely doing rendering.

  1. Clone the Webmail repo (Github, git.fm)

  2. npm install

  3. npm install -g pm2

  4. Then, you will need to fill out the config.json:

    1
    2
    3
    4
    5
    {
    "port": 2001,
    "apiEndpoint": "API",
    "siteURL": "DOMAIN"
    }

  5. API is the API endpoint setup earlier. Change API to your API endpoint configured above. For example, https://api.myemail.com

  6. Default port is 2001. Point your reverse proxy to this. 3.siteURL is where your webmail is accesible. For example, https://web.myemail.com WITHOUT TRAILING SLASHES

Reverse Proxy (nginx)

Come on, just Google this already

This concludes the installation for your Webmail component. You want to point web.myemail.com to reverse proxy's IP, and best to set the rDNS of the IP to web.myemail.com as well.

Step 9: Profit

Your instance of Dermail is now up and running. Send a first email from other email addresses to your Dermail address, assuming that you have setup the MX record correctly, and you should see a new email under Inbox!

If you run into problems, again, open an issue on Github, or send me an email.

Yeah.. Unfortunately, Dermail being the side project, it doesn't have a beautiful or convenient installer. You need to setup Dermail manually. But once it is up, assuming that it is setup correctly, it should be the best thing you ever have.

RethinkDB

The main protagonist is the RethinkDB database. Refer to offcial document on how to install on your distro and configurating.

my setup: My (three) RethinkDB instances are running in a private VLAN, and setup up as a cluster (3 replicas, 3 shreds).

Once you have RethinkDB setup, we will create the tables, and setup the components.

Misc

Source Code: https://github.com/zllovesuki/dermail-misc

This is a very simple Express application that will handle link and images sanitization in the mail body.

Configure config.js accordingly. Yes, this requires access to the database.

It is expected to be behind a reverse proxy, like Nginx. You can put Misc on the same machine as Webmail.

Updated: 04/29/2016 - Misc is no longer needed; it is consolidated into a central api

TX

SMTP outbound is actually the easiest part.

Source Code: https://github.com/zllovesuki/dermail-tx

You will need to:

  1. npm install
  2. Put your SSL private key in "ssl/key" and your SSL chain in "ssl/chain", because TX will receive request via HTTP
  3. Change the domain name in app.js. Somehow I hardcoded that, oops.
  4. node cluster.js or forever start cluster.js after the last component

You might want to point a subdomain to this machine. For example, tx-1.email.com, you will need this later.

RX

This is the MDA for Dermail, and this too requires SSL and a domain.

Source Code: https://github.com/zllovesuki/dermail-rx

Configure config.js accordingly. rethinkdb is pretty straightforward. However, basePort is for clustering. Say, 8 instances, then RX will spawn and listen at 8000, 8001, 8002... 8007.

It is expected to be behind a reverse proxy, like Nginx. You can put RX on the same machine as Webmail.

Again,

  1. npm install
  2. node cluster.js or forever start cluster.js after the last component
  3. A subdomain is expected as well. For example. rx-1.email.com, You are gonna need it.

Updated: 04/29/2016 - RX is no longer needed; it is consolidated into a central api

API

After some consideration, why not just combine some elements of Dermail into one central place?

As of 04/29/2016, Misc, RX, and the backend of Webmail are moved into API.

Source Code: https://github.com/zllovesuki/dermail-api

Then, you will need a config.json for sensitive information:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{
"remoteSecret": "SECRET",
"jwt": "ANOTHER SECRET",
"gcm_api_key": "GCM KEY",
"tx": [{
"priority": 10,
"hook": "https://TX-ENDPOINT/tx-hook"
}],
"s3": {
"endpoint": "S3-ENDPOINT",
"key": "S3-KEY",
"secret": "S3-SECRET",
"bucket": "S3-BUCKET"
},
"rethinkdb": {
"host": "127.0.0.1",
"port": 28015,
"db": "dermail"
},
"redisQ": {
"host": "127.0.0.1",
"port": 6379
}
}

What? Redis? Yes, you need Redis for API. This is the queue for outbound SMTP.

  1. remoteSecret is used to "authenticate" MTA or other components. You will need this again later
  2. jwt is your secret key to encrypt JWT token. Since Webmail is a single page app, it will use JWT for, authentication and authorization. Please, make it safe.
  3. gcm_api_key is your developer API key. You will need it and you will want it for push notification.
  4. tx is your TX endpoint. Change tx to your TX endpoint configured above, or you will not be able to send emails. It is an array of endpoints, so you can have multiple endpoints to avoid single point of failure.
  5. s3 is your S3 information. It can be AWS, it can be on-premise, but you will need a S3 somewhere for your attachments
  6. rethinkdb and redisQ should be straightforward.
  7. Change the S3 URL in api/safe.js, or your attachment links will be incorrect. Sorry about the hardcoded parameters. This is no longer needed.
  8. You then need to generate the app.js for the single-page app. Run npm prod. If you run into problems, most likely it is because of vueify and uglify. Install those two globally.

It is expected to be behind a reverse proxy, like Nginx. If WebSocket doesn't work, make sure that you have:

1
2
3
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;

MTA

Trust me, this is pretty straightforward as well, but it requires a bit more.

Redis

Install Redis as usual on your distro, make sure it only listens on 127.0.0.1

named

By default, your machine will have a nameserver of 8.8.8.8 or some other crap. However, since the MTA uses Spamhaus Zen to detect bad senders, you will need to run your own resolver. MTA will enforce nameserver 127.0.0.1 on startup

Actual MTA

Source Code: https://github.com/zllovesuki/dermail-mta

Configure config.js accordingly. redisQ usually stays the same. Then, you will need a config.json for API:

1
2
3
4
5
6
7
8
{
"redisQ":{
"host": "127.0.0.1",
"port": 6379
},
"remoteSecret": "SECRET",
"apiEndpoint": "https://API"
}

  1. redisQ should be straightforward.
  2. remoteSecret must match the one in API
  3. API is the API endpoint setup earlier. Change API to your API endpoint configured above, or you will not receive emails. IT MUST BE HTTPS

Then,

  1. npm install
  2. Put your SSL private key in "ssl/key" and your SSL chain in "ssl/chain", because MTA will have STARTSSL support
  3. node clusterMTA.js or forever start clusterMTA.js after the last component
  4. node clusterWorker.js or forever start clusterWorker.js after the last component

You will point your domain's MX record to this machine

Webmail

Yeah! The final piece of the puzzles!

Source Code: https://github.com/zllovesuki/dermail-webmail

As usual, do npm install.

Then, you will need a config.json:

1
2
3
4
5
{
"port": 2001,
"apiEndpoint": "API",
"siteURL": "DOMAIN"
}

  1. API is the API endpoint setup earlier. Change API to your API endpoint configured above, or you will not receive emails. IT MUST BE HTTPS
  2. Default port is 2001. Point your reverse proxy to this.
  3. siteURL is where your webmail is accesible. For example, https://dermail.net WITHOUT TRAILING SLASHES
  4. Change the S3 parameters in src/lib/st.js, or your attachment links will be incorrect. Sorry about the hardcoded parameters. This is no longer needed.
  5. Change public/manifest.json for your GCM push notification.
  6. You then need to generate the app.js for the single-page app. Run npm prod. If you run into problems, most likely it is because of vueify and uglify. Install those two globally.

It is expected to be behind a reverse proxy, like Nginx.

DB Scheme (or lack thereof)

Use database.md in the API component to setup the database.

Start the Engine

Now your Dermail should be up and running. For now, you will need to mannually add the first account:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
var config = require(__dirname + "/config.js");
var r = require('rethinkdb');
var reset = false; // DO NOT SET TRUE TO PRODUCTION ENVIRONMENT
r.connect(config.rethinkdb, function(err, conn) {
if (reset) {
r.table('users').delete().run(conn).then(function(deleted) {
r.table('domains').delete().run(conn).then(function(deleted) {
r.table('accounts').delete().run(conn).then(function(deleted) {
r.table('folders').delete().run(conn).then(function(deleted) {
r.table('addresses').delete().run(conn).then(function(deleted) {
actual(conn);
});
});
});
});
});
} else {
actual(conn);
}
});

function handleError(err) {
if (err) console.error(err);
}
var account = 'me';
var domain = 'domain.com'
var fN = 'John';
var lN = 'Doe';

function actual(conn) {
r.table('users').insert({
username: 'JohnDoe',
password: '', // bcrypt salt
firstName: fN,
lastName: lN
}).getField('generated_keys').do(function(keys) {
return keys(0);
}).run(conn).then(function(userId) {
r.table('domains').insert({
userId: userId,
domain: domain,
alias: []
}).getField('generated_keys').do(function(keys) {
return keys(0);
}).run(conn).then(function(domainId) {
r.table('accounts').insert({
userId: userId,
domainId: domainId,
account: account
}).getField('generated_keys').do(function(keys) {
return keys(0);
}).run(conn).then(function(accountId) {
r.table('folders').insert({
accountId: accountId,
parent: null,
displayName: 'Inbox',
description: 'Main Inbox',
mutable: false
}).getField('generated_keys').do(function(keys) {
return keys(0);
}).run(conn).then(function(folderId) {
r.table('addresses').insert({
account: account,
domain: domain,
friendlyName: fN + ' ' + lN,
internalOwner: userId
}).getField('generated_keys').do(function(keys) {
return keys(0);
}).run(conn).then(function(addressId) {
console.log('Account ID: ' + accountId);
console.log('User ID: ' + userId);
console.log('Domain ID: ' + domainId);
console.log('Folder ID: ' + folderId);
console.log('Address ID: ' + addressId);
conn.close();
});
})
})
})
})
}

If there is a problem, please email me at private@sdapi.net

Apr 25 2016

So the SPAM categorization didn't really fly as my computer science is much limited... But, I can do a filter instead!

One of my email addresses is plagued with SPAM that looks like this: SPAM Folder

SPAM

Jesus... Fear not, filter is here to save the day: Criteria

Results

Actions

I hope to add more functionalities later down the road, but I'm too busy recently. :/

Weightless Theme
Rocking Basscss
RSS