The Challenge: QA wish list
Phillip Cave, VP of Product Development, needed a way to test new Springboard releases. Springboard has grown into a complex software package that uses no fewer than 4 different Github repos, 27 community-contributed Drupal modules, and several external libraries. Standing up a new site for testing a new feature request would take manual work, and was error prone. Phillip needed something simpler and more reliable, so he asked "Can you build me something like simplytest.me?"
David and I had previously evaluated Simplytest.me for our team's testing needs, but because it is restricted to prepackaged Drupal distributions and contrib modules, we found it was too limited for our needs. However, Simplytest.me served as an inspiration for a simple user experience. With simplicity in mind, we distilled our minimum requirements into an interface that includes only 4 fields, one for each of our primary Github repos.
Why not Aegir?
We first attempted to solve this problem with Aegir, and used it successfully for about a year. We realized eventually that Aegir wasn't really optimized for building out a QA environment. There's no easy way to update existing sites without performing a codebase migration, and while Aegir can build from makefiles, it doesn't monitor repos for changes. The output from the makefiles also wasn't in a portable format.
While we eventually stumbled upon DevShop, a fork of Aegir that attempts to solve some of the limitations we encountered, we also wanted to have a shared configuration with Springboard as a service, which was using Chef to manage the stack and production Springboard deployments. By using the same Chef cookbooks that our production environments use, we eliminate unwanted environment or deployment-specific bugs.
Jackson River has four public GitHub repos that make up Springboard, in addition to Drupal core and contrib modules:
- We have a repo for our Drush makefiles. These contain definitions of how to build Springboard and locks in versions of contrib modules and libraries. We also use these makefiles to patch and extend contrib modules when needed.
- We have a separate repo for our Springboard install profile. Using this profile on new installs enables required modules and sets up some initial permissions and other settings.
- Our Springboard modules repo contains the custom modules that make up most of Springboard.
- We have built Springboard themes for out-of-the-box Springboard look and feel.
The repos are built via Drush make, and then the codebase is stored in a tarball and uploaded to our Springboard package repository.
Storing them as fully assembled artifacts ensures we don't repeat ourselves, an essential principle of software development. We only need to run Drush make one time to have a codebase that can be deployed to testing sites, used for local development work, and if it's a canonical Springboard release, shared with our clients and their development teams for production deployments.
Deploying Springboard with Chef
Our utilization of Chef to manage our Springboard as a service infrastructure is worthy of an entirely separate blog post, but it's important to touch on a couple key points to why we're using Chef here, since it's not commonly used as a deployment system.
Springboard is more than just Drupal modules
If we step back from the code level, it's easy to see how Springboard is more than just a collection of Drupal modules. Springboard is also a collection of PHP extension requirements, infrastructure recommendations (e.g. Memcache, APC), and directory structure. We use Chef on our QA server to ensure that each piece of this stack matches our expectations and requirements. Chef makes it so that we test apples to apples with our Springboard as a service production environment.
We also use Chef on our QA server to install a couple extra pieces specific for testing, Xhprof and Xdebug, and to ensure that our Jenkins backend is up and configured correctly.
Chef installs Springboard
Once you have the code and the LAMP stack ready, the final step is to install Springboard. We use Chef here because it already has all the necessary information, or can render that information rather easily.
Since Chef created the database, it also has the database credentials for each QA site. The $databases array, private and public files directory paths, and config variable overrides are added to settings.php automatically by Chef, and required files like the encrypt key and sustainer key are created.
Finally, Chef runs drush site-install sbsetup, which uses the Springboard install profile to provision a brand new Springboard site on the newly created database.
Our Jenkins jobs tie everything together
By this point, we have a way to assemble a Springboard package, which contains the required Drupal and Springboard code, and we have a way to provision a new site on the QA server. The final piece of the puzzle is to connect all of this into a single deployment process. To accomplish this, we use Jenkins, a continuous integration server.
Monitor and package our Springboard repos
We have created packaging jobs and monitoring jobs. Our monitoring jobs can be configured to monitor any branch on any of our four Github repos for changes, and when new code is pushed, they can trigger packaging jobs. The packaging jobs use a predefined makefile, pulled from our build repo, and use drush make to assemble the codebase.
QA site C(R)UD: Create, update and delete
We created a full CRUD workflow for our QA sites using Jenkins:
Creating a new QA site is simple. When starting a new build, you are asked to input the Springboard version. This can represent a specific released version, like 7.x-4.4.1, or the job can accept branch names, commit hashes, and tags from each of the four repos if you need to build non-standard packages. You then provide a name for the new QA site, and provide an initial password for uid 1.
There are two options for updating QA sites. If you have new code you want to deploy to an existing QA site, but it's using the same package name (e.g. 7.x-4.4.1-beta1), you can use the Update QA site job. This only asks for the name of the QA site you wish to update. It then repackages the version of Springboard on the QA site and updates the codebase.
On the other hand, if you want to migrate an existing QA site to an entirely different codebase, you can use the Migrate QA site job. In this case, the job asks for the name of the QA site and the name of the new packaged version.
If you wish to delete a QA site, you can use the Delete QA site job. This removes the data bag item for the site. Chef automatically removes sites that aren't represented in the data bags.
Since most of the Jenkins work is restricted to modifying the data bag items used by Chef to provision the QA sites, the final step in the Create, Migrate and Delete jobs is to run chef-client.
Plans for the future
As with any process, there's always room for improvement. In the months since deploying this setup for internal use, we've identified a couple pain points.
It needs a better UI, with less exposure to Jenkins
Unless you're familiar with Jenkins, it can be an intimidating interface. We'd like to see the QA site building process distilled into a simpler interface that combines data pulled from Chef (e.g. the existing QA sites could be displayed in a list or table) and the available branches and tags pulled from Github, with better feedback on the QA site provisioning process. Better error reporting and feedback would be nice to have here as well.
More test coverage and more automation
We are working on introducing more test coverage into Springboard, which we hope will lessen the time spent on manual QA. The new testing automation will utilize parts of the process we've created here, including the Chef cookbooks and packaging jobs, to build sites and execute tests automatically.