Rachel's Yard

| A New Continuation
Feb 27 2016

As part of the operation of "De-Clouding" everything, one of the core component would be moving away from Gmail. However, email ain't something simple. I've heard the tale of sys admin where they switched from SunMail to Postfix/Dovecot, and amazed by "Human Readable" configuration file.

Well of course, Postfix, Dovecot, whatever, are too complicated to negotiate with... Therefore, time to...

(Yes, I know about iRedMail/MailInABox)

Meme

And here we are, Dermail.

Technologies:

  1. RethinkDB
  2. NodeJS
  3. Redis
  4. Vue.js

Awesomeness

  1. RethinkDB can do eqJoin and MapReduce, also secondary indexes with complex index or compound index. It is just powerful
  2. Oh, have I mentioned how you can shard and replicate in the WebUI? RethinkDB AdminUI

When I was doing architecture planning, I was struggling to find a suitable database schema, or database software for that matter. NoSQL was definitely the way to go, and yet I still want Join and Indexes and what not. And... RethinkDB is here to save the day.

Seriously, check it out. It's awesome. Try it in your next project.

RethinkDB


Speaking of query language (when you are talking about SQL), usually you think of this:

SELECT * FROM mails WHERE user='UUID' ORDER BY date DESC

So on and so fort. RethinkDB, on the other hand, it's NoSQL, and they have their own language, ReQL.

r.db('dermail').table('mails').filter({user: 'UUID'}).orderBy(r.desc('date'))

Isn't that intutative? Instead thinking how to instruct the program to fetch the data, you think of how the data "flow":

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
r
.table('messages')
.getAll(messageId)
.eqJoin('messageId',
r
.table('messageHeaders'), {
index: 'messageId'
}
)
.without({
right: ['to', 'from', 'date'],
left: 'text'
}) // no annoying override in headers and text in mail
.map(function(doc) {
return doc.merge(function() {
return {
'left': {
'headers': doc('right').without('accountId')
}
};
})
})
.zip() // Noice
.pluck('messageId', 'date', 'to', 'from', 'folderId', 'accountId', 'subject', 'html', 'attachments', 'isRead', 'isStar')
// Save some bandwidth and processsing
.map(function(doc) {
return doc.merge(function() {
return {
'to': doc('to').concatMap(function(to) { // It's like a subquery
return [r.table('addresses').get(to).without('accountId', 'addressId', 'internalOwner')]
}),
'from': doc('from').concatMap(function(from) { // It's like a subquery
return [r.table('addresses').get(from).without('accountId', 'addressId', 'internalOwner')]
}),
'attachments': r
.table('attachments')
.getAll(messageId, {
index: 'messageId'
}
)
.pluck('checksum', 'generatedFileName') // Actually, we only need the ID and filename
.coerceTo('array')
}
})
})

That's how you join 4 tables together and put them into its own "fields" (headers, from, to, and attachments). Very nice.

Webmail


Originally, I want to do IMAP. But,

For one, RFC 3501 is complicated.

For two, IMAP looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
telnet: > open imapserver.example.com imap
telnet: Trying 192.0.2.2...
telnet: Connected to imapserver.example.com.
telnet: Escape character is '^]'.
server: * OK Dovecot ready.
client: a1 LOGIN MyUsername MyPassword
server: a1 OK Logged in.
client: a2 LIST "" "*"
server: * LIST (\HasNoChildren) "." "INBOX"
server: a2 OK List completed.
client: a3 EXAMINE INBOX
server: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
server: * OK [PERMANENTFLAGS ()] Read-only mailbox.
server: * 1 EXISTS
server: * 1 RECENT
server: * OK [UNSEEN 1] First unseen.
server: * OK [UIDVALIDITY 1257842737] UIDs valid
server: * OK [UIDNEXT 2] Predicted next UID
server: a3 OK [READ-ONLY] Select completed.
client: a4 FETCH 1 BODY[]
server: * 1 FETCH (BODY[] {405}
server: Return-Path: sender@example.com
server: Received: from client.example.com ([192.0.2.1])
server: by mx1.example.com with ESMTP
server: id <20040120203404.CCCC18555.mx1.example.com@client.example.com>
server: for <recipient@example.com>; Tue, 20 Jan 2004 22:34:24 +0200
server: From: sender@example.com
server: Subject: Test message
server: To: recipient@example.com
server: Message-Id: <20040120203404.CCCC18555.mx1.example.com@client.example.com>
server:
server: This is a test message.
server: )
server: a4 OK Fetch completed.
client: a5 LOGOUT
server: * BYE Logging out
server: a5 OK Logout completed.

For three, there is no good IMAP implementation in Nodejs, yet. I spent 3 days researching existing implementation in Nodejs and trying to figure something, but... I will come back to it.

For now, I will settle on Webmail. It is using Vue.js, and it is so awesome.

login

accounts

folders

inbox

compose

mail

Architecture


What's good of a system if it fails easily?

Architecture

Receving

The system is using mailin.io for SMTP inbound. By default, it will post to a Webhook. However, if the webhook is unreachable for some reasons, then the mail is forever lost.

Updated Mar. 3 2016 Mailin is not modern enough (ES6 Promise/Bluebird/Q or whatever Promise you have). Therefore, I wrote my own version of Mailin. The code is hugely influenced by Mailin, many thanks.

Nope

Therefore, a minor modification of mailin is needed. A redis queue is used so that, when a mailin instance receives a mail, it is sent the queue, then a worker picks up the queue, and the worker sends the mail to the API, then it is stored to the database. mx-smtp-X sends a HTTP request to API component, and check if the recipient(s) is valid. Valid -> put to queue; invalid -> rejects the mail.

Then the workers pull mails from the queue, post the actual mail to the API component, and a separate worker will uploads attachments to S3 or S3-compatible storage.

Q.E.D the receving part.

Sending

I love this the sending architecture. tx-X are powered by NodeMailer, which can use any transports that it supports. Therefore, in the eye of tx component, the underlying transports are abstracted away, and the tx component will just post the mail body to any of the tx-X server, and the mail will be handled without tx knowning which transport is using.

Right now sending mail is only via Webmail. However, adding a SMTP is easy. Adding another component, and throw the mail into the queue, and tx will pick it up. Perfect.

Of course, retry is in place:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
servers.sort(function(a,b) {return (a.priority > b.priority) ? 1 : ((b.priority > a.priority) ? -1 : 0);} );

var send = function(servers, data) {
if (servers.length === 0) {
var errorMsg = 'No more outbound servers available.'
// Push a failed delivery message
Return;
}
var server = servers.shift();
var hook = server.hook;
return request
.post(hook)
.timeout(10000)
.send(data)
.set('Accept', 'application/json')
.end(function(err, res){
if (err || res.body.error !== null) {
// Push a retry message
return send(servers, data);
}
// Push a success message
return done();
});
}

send(servers, data);

Of course, using direct transport usually causes problem, especially when dealing with Microsoft. *cough* *cough*. Make sure that your IP is not blacklisted by Microsoft, and make sure that your SPF record is valid.

Weightless Theme
Rocking Basscss
RSS