Twelve-Factor App

I am currently studying DevOps from the beginning again. I know how to use the tools and have experience with them. But I think I need to keep a record of my DevOps studies.

For the first step, I'm going to write about the Twelve-Factor App.

Most services have no time to take down the application for any reason, such as user experience, patching servers, additional resources, or scaling. For this to happen, our application needs to break free from the underlying infrastructure. We can host our application on-premises, on GCP, AWS, or Azure.

This makes it possible to run the same app in different environments without any settings like changing the source code or adjusting the environment settings.

To summarize, modern applications need to be portable, with minimal divergence when deployed between development, testing, and production environments to enable continuous deployment. They must be scalable by spinning up new instances.

Our application needs to be developed with certain principles in mind, and it is called the "Twelve-Factor App".

Codebase

Let's assume I'm creating a blog similar to the one you're currently looking at.

Now my code is only on my laptop. My blog is growing, and visitors are requesting more features.

So I brought my friends along as additional developers, and now everyone is working in their own environments, but on the same codebase. They are all copying their code to a central hub like GitHub whenever they're ready. We are working on the same codebase, which will be used to deploy to development, staging, and production environments.

Dependencies

Let's look at the second rule in the 'Twelve-Factor App.' This is to explicitly declare and isolate dependencies.

In succession, we are developing this blog in Python with the Flask framework. Therefore, it is essential to install the Flask Python framework before beginning development. Now we have one dependency for our blog.

The Twelve-Factor App recommends explicitly declaring and isolating dependencies. These dependencies are typically listed in a file named requirements.txt.

In the end, the pip install command will install all the dependencies listed in the requirements.txt file during the build process.

Now I'm going to tell you about isolating the dependencies.

Let's assume we are developing two separate services or applications on our laptop. What if one app requires one version of Flask and another app requires a different version of Flask?

For example, Python has a concept of a virtual environment.

We can ensure isolation of dependencies using the virtual environment.

However, what about tools that rely on dependencies outside of Python's ecosystem?

How do we control configurations that are outside of Python's capabilities within our dependencies?

The answer is a self-contained environment like Docker.

Docker allows us to run applications in a self-contained environment that is isolated from the host system. It is a more efficient and reliable way to manage dependencies.

Concurrency

Ok, we have containerized our application and executed it as a Docker container, and this instance of the application is able to serve several users. But what happens when we have more visitors to our site? I think we could scale up the resources vertically by increasing the resources on the server. However, we would have to take down the server to do that.

But we are able to provision more servers and spin up more instances of the application today with great ease, with new servers accessible within a matter of minutes.

We can have a load balancer in between that can balance the load across the different instances of the applications.

App processes are first-class citizens; applications should scale out horizontally and not vertically by running multiple instances of the application concurrently. The application itself should be built with that in mind.

Process

We have decided to add a new feature to our app to show the visitor count on our website every time a new visitor visits our page.

It works well when we have one process running because the visit count is stored in the memory of that process. But when we run multiple processes, we will encounter a problem: multiple processes have their own versions of the variable stored in them. So, they display different numbers for different users depending on which process served those users.

The same problem occurs with other details as well, such as user information. When a user logs in to the website, we store certain session information. This session information is needed on the server to keep that user logged in.

But if this is stored in the process memory or locally in the file system of that process, then when a future request from that user is directed to another process, the user may be considered logged out, as the session information isn't available there.

Now, there are load balancers that can redirect users to the same process each time. This is called a sticky session. However, it still has a problem: if a process crashes by some reason, locally stored data will be lost.

Which is why we never store anything in these processes. Instead, it should all be stored in an external backing service that can be easily accessed by all processes. An external service could be a database or a caching service like Redis.

Backing Services

Previously, we integrated Redis as a caching service for our app to store the visitor count.

There are many similar services available, such as S3, SMTP service, and others.

Ok, let's continue talking about Redis. Redis is an attached resource for our app, irrespective of where it is hosted—whether locally, in a cloud environment, or on a managed Redis service. It should work without having to change our application.

We should be able to point our app to another instance, and it must work.

Build, Release and Run

One of the recent changes pushed caused a very small issue, but this needs to be fixed as soon as possible. How do we roll back a recent change without pushing another commit? Sometimes we may not have enough time to push such changes and wait for them to get rebuilt, tested, and deployed.

We can do this if we have a clear separation of build and run phases. We write code on our laptop and convert text format files into binary or executable formats. This process is called building. The result becomes the release object along with the environment. So, a combination of the executable along with the configuration becomes the release object.

Every release should have a unique release ID.

The release ID consists of the release version or a timestamp, making it easy to recognize when it was created.

Now, we shouldn't create a new release for a minor error. By clearly separating our build and run phases, we can easily roll back to previous releases or redeploy a specific release as needed. This improves our overall ability to manage and maintain our software.

Port Binding

We can access the website using a URL and port number. We can also access other programs running on the same server using specific port numbers.

Disposability

App processes are disposable, meaning they can be started or stopped at a moment's notice.

We shouldn't rely on complex startup scripts to provision the app because we need to scale up and provision additional instances when requirements increase at a moment's notice, often in a matter of seconds.

Our app processes should shut down gracefully when they receive a SIGTERM signal from the process manager.

Dev / prod parity

The Twelve-Factor app is designed for continuous deployment by keeping the gap between development and production small, and the developer resists the urge to use different backing services between different environments. With continuous integration, continuous delivery, and deployment tools available today, we are able to reduce the time.

We must aim to keep the same tools as much as possible, with many tools being lightweight and containerization tools like Docker making it easy to set up development environments

Logs

So traditionally applications followed different approaches to storing logs one was to write the logs to a local file named log file or something like that.

The problem with this approach is that since we are living in the world of containers the container may be killed anytime and the logs are lost moreover.

So the 11th principle in the 12th of Factor app is about logs management at 12 Factor app never concerns itself with routing or storage of its output stream store logs in a centralized location in a structured format.

So the 12 Factor apps must not try to write to a specific file or be tied to a specific logging Solution 12 Factor apps must write all logs to its standard out or to a local file in a structured Json format that can then be used by an agent to transfer to a centralized location for consolidation and Analysis purposes.

Admin Processes

The admin processes principle of the 12 Factor app methodology recommends that administrative tasks should be kept separate from the application process and that they should be run in an identical setup and be automated scalable and reproducible.