This post was written by Adam Bergstein with input and feedback from Aimee Degnan and Ryan Bateman.
Hook 42 recently had a project that required the use of Kubernetes. Kubernetes is an orchestration tool that helps solve many enterprise problems when using multiple containers. People often want redundancy and scale across a series of containers or the same set of containers across multiple machines. Kubernetes helps solve this. It also can help perform orchestration tasks during failures or to distribute load between containers. Managing containers at scale can be a challenge and the goal of Kubernetes is to help.
We have long been tracking the efforts of Lagoon, a promising open source continuous integration and continuous delivery (CI/CD) hosting platform that is developed with Kubernetes. CI/CD is built around the concept of rapid deployments of ongoing, frequent changes. This lowers the risk presented by larger, fixed deployments by promoting smaller, more routine deployments. Lagoon not only offers hosting-related tools, but the platform is able to run locally as well. The CI/CD capabilities helped create testable environments as we pushed changes through our development workflow. We want to share our experience.
Understanding Considerations
There are some key concepts to understand how Lagoon works before diving in.
Persistence
Drupal applications require a “source of truth” for persistent content, which includes a Drupal application’s managed files and database. Production environments (or pre-production before a system is launched) conventionally serve as the source of truth environment within a development workflow. Content is subsequently pulled from the source of truth and it is only managed by changes made directly on the production system. Where code can be pushed through environments, content should always get pulled from production.
Repositories
Code repositories are critical for managing code and deployments. Each change is one or more commits that can be staged within specific branches. Changes can be merged into other branches, rolled back if there is an issue, and audited as a series of changes.
Hosting providers offer varying conventions and integrations tied to code repositories. As a simple example: both Pantheon and Acquia offer their own Git repositories commonly synchronized with a Github or Gitlab repository. The Github/Gitlab repositories offer development tools like pull requests or integrations to help support team-based workflows.
Hooks/Events
Both repositories and hosting platforms expose relevant hooks that are useful for performing actions during DevOps events. This is how automation can be built into specific changes. Automation is critical for any CI/CD infrastructure, as it’s not manageable or practical to manually rebuild environments as each frequent change occurs.
Creating and maintaining branches, tags, and pull requests tied to repositories create opportunities for automation that are commonly leveraged repository events in our DevOps infrastructure. Even synchronizing between two repositories can be a useful DevOps trigger, as this signifies code is ready for some degree of deployment or testing.
Any sort of deployment found in CI/CD workflows often require rebuilding containers. This is common for all environments. Persistent aspects may be left untouched, while the containers for a given environment are rebuilt as new changes are deployed.
On-demand Environments
Cloud infrastructures changed the traditional way of understanding environments. On-demand environments are a result of rapid change and are transient. This is in contrast to an environment traditionally configured on a bare metal server. On-demand environments are commonly provisioned with new branches, tags, or pull requests and destroyed when the changes are deployed. They are not intended to persist.
In a CI/CD workflow, incremental changes are verified before the production deployment. Development-specific branches and pull requests can build new environments known as on-demand environments. Persistent contents from the “source of truth” are copied with changes pushed to a branch on a new environment. This helps to verify and mimic production behavior with the new change. And, the environment subsequently goes away when the change is deployed.
Fixed Environments
Hosting providers still commonly offer “fixed” environments as people are not often comfortable merging an on-demand environment right into production. But, fixed environments exist for an intended purpose. As an example, a production environment is intended to be what end-users access. Other environments are commonly used to vet changes before a production launch. Vetting may include any of the following purposes: proper deployment, stakeholder approval, automated tests, and verified with the most recent code (changes pushed while the code was developed and/or staged). The same fixed environments can be uniquely configured and used in those capacities in a more permanent capacity.
Exploring Lagoon
Every hosting platform has a set of best practices tied to their platform that drives intended use. While any platform, like Lagoon, can seem opinionated, our focus is to connect the aforementioned concepts to their specific Lagoon equivalent.
Repositories, Hooks, and Events
Lagoon, at this point, attaches to a remote repository, like Github or Gitlab. Lagoon integrates through the repository platform, via a webhook or continuous integration system. The hook is intended to be invoked during specific events like branch creation and/or pull request to help create the CI/CD on-demand environments. This approach shaped our development and release workflow to the Lagoon platform, which we elaborate more on below.
Fixed Environments
The first thing our team did was set up fixed environments tied to specific branches. This met the need of having changes go through a conventional development, staging, and production release cycle. Within Github, there are subsequent branches for each fixed environment. As a best practice, each branch was locked within Github. This is to ensure a branch was not accidentally removed, which may remove a fixed environment entirely.
Github repositories often leverage “master” as the default branch. We’ve selected that as our branch for our development server. This is useful for pull requests, as “master” is selected by default and our team didn’t need to worry about selecting the wrong branch. Doing so may trigger a code deployment to another, unintended environment.
On-demand Environments
Lagoon maintains two events for provisioning on-demand environments. The first is pull requests. Pull requests seem to be useful if you are operating in a pure CI/CD environment where changes can be tested as an environment tied to the change. But, pull requests often are not created until a potential change is ready to be reviewed and possibly merged. This would be ideal for an environment to do smoke testing. But, we desired to have environments for work-in-progress as well, where anyone could push a commit to a branch, demonstrate some work, or get help on something. We opted not to use pull requests for that reason.
The second provisioning event is through a branch pattern. This leverages a naming convention to create on-demand environments. Lagoon monitors the creation of any branch that matches the pattern and creates an environment. This is helpful for the initial testing of changes.
Release Workflow
Our release workflow is based on staggered branches. Changes are pushed to the on-demand branches and prepared for initial review. A pull request is made for the code review and smoke testing occurs on the on-demand environment. Once the pull request is merged into master, our development server is rebuilt (any commits pushed to master trigger a rebuild of the development server). We close all on-demand branches at this point, which subsequently removes the environment on Lagoon.
Once the merge is complete, additional quality assurance, client review, and automated testing occurs through development and staging environments. This happens by making a pull request from the master to staging branches, creating a release candidate.
With all of the verification passing, we are able to initiate the release. This is done via a pull request from the staging to production branches. Once the pull request is accepted, the production release occurs. All of this is automated thanks to the hooks and events tied to the Lagoon platform.
Persistence
Lagoon rebuilds environments on every push to a branch. In our workflow, accepted pull requests push vetted code to branches. Persistence becomes a major factor when environments can be rebuilt in this fluid manner.
Lagoon also provides the ability to configure what environment is deemed the production environment. This is subsequently protected within Lagoon and persistent. The “production” branch, and its subsequent environment, represent the source of truth for database and files. Rebuilding the database and files on a production release is risky, so this mechanism needs to exist to differentiate from the other more fluid environments that get rebuilt more routinely. These protection mechanisms helped avoid production data being overwritten through the API or any unintended impact by a production deployment and subsequent rebuild. Not only that, but this helps identify what data needs to be routinely backed-up and maintain high fidelity.
DevOps
All non-production environments should automatically load a copy of the production database and files when new changes are pushed to the subsequent branch. Verifying changes before being released to production was a critical DevOps automation for us. We leverage hooks in Lagoon (defined in the .lagoon.yml file) with Drush commands and aliases to identify the environments that synchronize the database and files from production every time an environment is rebuilt.
Code artifacts were also a vital part of our DevOps automation. We leveraged Composer to build the Drupal codebase (using the great Composer template for Drupal projects - drupal-composer/drupal-project) and Gulp to build the theme. Custom code was committed to the repository. This allowed us to easily and routinely evaluate changes to core and contrib.
Once our code was built, we executed Drush commands to import the configuration, run database updates, and clear caches to ensure changes were properly deployed. While this does not catch every possible nuance in deploying code (e.g. rebuilding entities), we automated a significant portion of this that should minimize the need for running manual commands.
Observations
Lagoon is doing some innovative work, especially for Drupal teams looking for a CI/CD platform adopting rapid releases. Their “infrastructure as code” implementation, through their Docker images and .lagoon.yml configuration, enable rapid, effective change at the heart of DevOps that can help continuous learning and the subsequent predictability of automation.
As expected, coming from the perspective of using other hosting platforms like Acquia and Pantheon, there was some learning to adapt to the fluid CI/CD nature of the platform. Lagoon’s local implementation replaced our standard MAMP, DDEV, and Lando setups. Being able to add in restrictions, like branch locking, was beneficial to our teams transition when configuring the infrastructure. Also, there were some different conventions, like leveraging drush aliases through Lagoon’s CLI container and not through a local Drush, that were unique. But, many of our existing knowledge and concepts were the same or could easily be mapped to their Lagoon equivalent.
After some trial and error, we were able to share some feedback with the Amazee team on improvements to their documentation and relevant code samples we felt could help others. Hopefully shaping the path forward for others being onboarded can make it easier to digest for those new to the platform.
Overall, Lagoon shows a lot of promise for modern workflows. While different than other hosting platforms, Lagoon enabled our team to work effectively and efficiently from start to finish. We’re excited to see how the platform evolves and continues to provide solutions capable of rapidly changing to our customer’s needs.