Skip to content

Home

Jump to bottom
Mario Dimitrov edited this page Jan 6, 2016 · 8 revisions

Table of Contents

Project Overview

Purpose

RobustMail is a mail delivery service, that allows abstraction between multiple mail delivery providers and improves the reliability of message delivery by employing a failover mechanism in case some of the underlying services is unavailable.

Restrictions and Assumptions

  • Messages will be sent by the instance owner or its trusted associates to conduct their relevant communication. Identification of spamming behavior is out of service scope.
  • Message delivery is important - successful delivery is favored over latency, losing 200 ms or more of queued messages does not fulfill the acceptance criteria.
  • Non-repudiation is out of scope - the service makes best effort to suggest recipient address correction and deliver the message to the proper mail provider and assumes that it will handle it properly, not when or whether the recipients have opened it.

How to run it yourself

  • Install the project requirements with pip install -r requirements.txt
  • Export your relevant api keys and login data as per settings.py (RabbitMQ, Redis, Mailgun, SendGrid)
  • Test it with nosetests
  • Run the application with python RobustMail.py
  • Try the REST API

    Send a message

    /api/send-message Accepts POST
    Checks whether the message contains all required information, checks the email addresses according to /api/validate-email and tries to send the message In case of failure only the first fail message is returned
    Accepted fields
    - sender- *(required)* The sender of the email
          - name (optional)
          - email *(required)*
    - subject - The subject of the email
    - recipients[] - *(required)* array of recipients
         A recipient consists of:
             - name (optional)
             - email (required)
    - cc [] (optional) array of carbon copy recipients
             A recipient consists of:
             - name (optional)
             - email (required)
    - bcc [] (optional) array of bcc recipients
             A recipient consists of:
             - name (optional)
             - email (required)
    - text - *(semi-required)* if body is NOT set, this is required. This is the text representation of the message. You may choose to have both.
    - body - *(semi-required)* if text is NOT set, this is required. This is the html representation of the message. You may choose to have both.
    Example responses
    { "success": true, "message" : "Message accepted for processing!" }
    { "success" :false, "message" : "Text or body is required" }
    { "success" :false, "message" : "Sender email invalid" }

    Check an email

    /api/validate-email Accepts POST
    Checks the email address syntactically and MX record-wise, does not guarantee that the address actually exists on the server (Provided in the front-end)
    - email *(required)* The email address to be checked
    Example responses
    { "success": true, "message" : "Syntax and domain accepted" }
    { "success" :false, "message" : "Address domain not found" }
    { "success" :false, "message" : "Address syntax invalid" }

Architecture

Overview

The service is accessible through an API or simple Front-end. Upon receiving a message via the REST service, it is validated and if accepted enters the RabbitMQ powered Celery task queue which passed it to the MessageService. It takes care of managing message providers and which one to process the message. This simple design allows retrying providers in case some of them are unavailable and also improves the reliability in case the service itself is restarted, because of the RabbitMQ nature. Syntax-wise and MX record validation is available for emails.

Message Broker

Celery is listed as a non-functional requirement for asynchronous processing. Celery supports multiple message brokers, but the best supported, widely used and recommended by Celery are:

Redis

Redis is a high-performance message broker and supports most of the features. However as per its official persistence documentation http://redis.io/topics/persistence it favors low latency over keeping the latest written data in case of service failure and the recommended mode to improve that is AOF with fsync every second. They also list potential issues with the append-only mode. Redis will be used as result backend when results are needed, because of its good performance and for reducing CloudAMQP message consumption.

RabbitMQ

RabbitMQ is the recommended solution, it is widely deployed, tested and supports all features, including delivery acknowledgments. Focuses on reliability, however allows multiple configurations to improve performance. It also includes management and monitoring solutions. For the prototype purpose gossiping and mingle are disabled to fit the app in the CloudAMQP quotas.

Provider Priority

A modern mail delivery provider has relatively low server failure rates, so it is assumed that the service owner has a preferred mail provider with high message sending quota and keeps the additional services with lower quotas for failover. The provider priority algorithm tries to send the message with the top preferred provider that is assumed available. In case that fails, the provider is retried after a fixed time, preference considered. In case all providers are unavailable, waiting time until the next retry is applied. This simple approach allows a good balance between time to return to the primary service and requests count in case of service failure.

Deployment, etc.

Completing this step quickly being a priority, Heroku was chosen as a Cloud platform and Travis CI as a continuous integration tool. They have free versions and integrate pretty seamlessly with Github and with each other, while keeping the boilerplate code to a minimum. Heroku supports all of the products required for the development natively

Future releases

Front end and content support

Support for complex messages, rich text editor, file upload etc.

Security

Add HTTPS for the application, login for senders in the front end, api keys for the REST API

Reliability

  • Implement the status checks and retry sending a message if the provider did not deliver it
  • Storing sent messages in MongoDB for archiving purposes. Whether this is relevant depends on whether the service performs only forwarding or has a more extended functionality. (Mongo would be a good solution for this, since the archive may become pretty large and only simple querying would be needed)
  • Load balancing between different instances of the same provider

    • Testing

    Implement load tests, use a noreply email as a recipient

    .

    Clone this wiki locally