arch

Architecture: A High-level Overview

Tree-chain Architecture

Overlord organizes nodes, known as chains, into a tree-chain architecture. To communicate with a node that is part of the network, it's necessary to pass through each of the nodes that are located before the destination node, forming a chain. The network as a whole can be viewed as a tree.

This architecture was chosen over others because, first, it allows us to federate our nodes, so that a node can be maintained by another sysadmin, and second, it is very easy to connect one node to another, sharing resources between them.

Except for the operating system, i.e., FreeBSD, there is no requirement for our network to be homogeneous. In fact, one node can provide features that another does not have; for example, one node can deploy virtual machines, another provides better performance, and others are located in a data center with a fast connection to the outside world. Each node is labeled, which is easier than remembering IP addresses, and better yet, the label can give a clue as to what the node does and can even be used to label a group of nodes, not just a single node.

In this way, we can identify a faulty chain with a reactive and proactive approach and suspend its use, so that the user does not even notice (except in cases where the service is linked to the chain, of course).

Node Architecture

We have a basic idea of how Overlord works as a whole with a set of fictional chains described above, but what is inside the chain?

Except for metadata and other things, Overlord does not store state in the strict sense. It collects information about projects and jails in the system, so we can say that those are the state. However, that state is slow to retrieve synchronously, which is why memcached exists, to store things in memory and asynchronously through polling.

Each poll-* process retrieves a different set of information from the disk either directly or indirectly (via AppJail). Much of the disk activity is due to direct disk reads, as this is faster than using AppJail; however, some fields are complex or dynamic, so AppJail is used for those fields.

Below is a brief description of each poll-* process:

• poll-jails: Get the list of jails in your system.
• poll-jail-info: Get information about each jail using the list of jails retrieved by poll-jails.
• poll-jail-extras: Get more than just basic information about each jail using the list of jails retrieved by poll-jails.
• poll-jail-stats: Get the resource consumption provided by rctl(8) for each jail using the list of jails retrieved by poll-jails.
• poll-projects: Get the list of Director projects in your system.
• poll-project-info: Get information about each project using the list of projects retrieved by poll-projects.

There are a few other poll- processes that don't retrieve information per se, but perform other useful tasks for the network:

• poll-autoscale: Overlord is not only designed to limit the deployment of virtual machines and jails, but also to ensure that, based on certain user-defined rules, the project (and its jails) are deployed. This way, we can get much more out of our nodes, as we are not limited to just one to deploy our service, which could go down at any time.
• poll-heartbeat: Apart from circuit breakers (also known in Overlord as Smart Timeouts), this optional process can proactively disallow a chain by polling to check if it can be reached, so that the user doesn't have to wait for timeouts to expire, which can slow down deployment.

Now that we have an idea of how information is retrieved, stored, and shared, we come to the final question: how are projects deployed?

When the user submits a project through the server process, the project is queued as a job in beanstalkd, to be finally retrieved by watch-projects or watch-vm, neither of which performs any tasks, but rather create another job and send it through beanstalkd, and this job is retrieved by watch-commands which is the process that actually creates the project. In early versions of Overlord, we did not find watch-commands because the responsibility for this process was part of watch-vm and watch-projects. The reason for splitting them is that watch-commands executes an external script that does the actual work, and that additional separation allows us to update Overlord even if there is a job running. Without this separation (as in earlier versions), when we restart the processes, it is necessary to stop the job, which makes it difficult to update Overlord and, worse still, forces us to perform updates infrequently.

Below is a brief description of each watch-* process:

• watch-projects: Wait for jobs to create, destroy or cancel a project.
• watch-vm: Wait for jobs to create or destroy a VM.
• watch-commands: Runs the internal task that performs the actual work of creating virtual machines or jails.

Adaptive Polling

To avoid overloading the system with CPU and disk I/O operations when retrieving information about jails, projects, and virtual machines, Overlord implements a smarter, more dynamic, and adaptive polling system.

Overlord stores an internal timestamp for each poll-* process (and for poll-jail-extras, there is a timestamp for each element such as cpuset, devfs, expose, etc.). This timestamp is updated after the user requests information via the API to determine whether Overlord should perform polling or simply enter standby mode.

In Overlord's configuration file, there are parameters that affect the polling frequency (in addition to the interval for each poll-* process), such as the Polling Window, Idle Penalty, Maximum Idle Time, and Maximum Idle Penalty. The Polling Window is used to perform polling during the specified time only after the timestamp has been updated. The Idle Penalty increases the Maximum Idle Time. The Maximum Idle time indicates how many seconds the system is considered inactive. If Overlord detects this, it increases this time, but forces a poll. Once the Maximum Idle Penalty is reached, Overlord will force a timestamp update, performing a new poll and repeating the steps described.

In this way, Overlord will be more efficient, reflecting reality when necessary without losing its asynchronous advantages.

There is a special case: watch-projects and watch-vm will update all timestamps, as there is a possibility that a new jail, virtual machine, or project will be created or destroyed.