This is the text of a talk that I gave at WordCamp Raleigh 2015, or a rough approximation of it anyway. It’s about the infrastructure I built for the SAS WordPress intranet.
If you’d rather, you can now watch the video of the talk: WordPress Continuous Deployment.
- Continuous Integration / Continuous Deployment / Agile & Scrum
- Source Control and Dependency Management
Hi, I’m Lisa Linn Allen and I’m an applications developer for the Intranet at SAS. We’re a software company located in Cary. We have 13,000 employees.
Several thousand of our employees are software developers and testers. We also employ SAS consultants, SAS instructors, a small army of tech support analysts, applications developers and system programmers, in other words a lot of people who feel comfortable writing code. Our Intranet is nearly 20 years old and has never been governed. That means that anyone who had the inclination and the ability could put pretty much anything up on the Intranet that they wanted to. The result is a big mess. Our Intranet search index has swelled into the millions of documents. SAS IT has never provided a web CMS for use on the Intranet outside of SharePoint. No governance over the Intranet content, aside from the top page, has been provided by any central authority.
In the spring of 2014, I was in my yearly review and my manager and I were discussing some long term ideas for the Intranet. We had been hearing second- and third-hand that our executives were asking why the Intranet wasn’t planning to adopt the big, commercial, Java-based web CMS that had been recently adopted for the sas.com web site with great success.
As a small team of PHP developers – there are three of us – the last thing we wanted was to have a commercial Java-based CMS forced on us. We had always used open source projects, including PHP, with great success and didn’t want to be shut out of any big effort to improve the Intranet.
So given the state of the Intranet and the rumblings about a commercial CMS for the Intranet, it seemed like a good idea to offer an open source CMS of our own choosing sooner rather than later, and without any plan in mind, completely off the cuff, I suggested to my manager that we just set up WordPress as a CMS and get something out there.
At the time my co-worker Charles was already managing a fairly large WordPress multisite install with about 500 internal blogs. A couple of years prior I had hacked the 2012 theme for a one-off special project. That seemed to constitute “WordPress experience” and be reason enough to go with WordPress over, say, Drupal.
We pitched it to the head of Internal Communications because she would be the logical sponsor for any large-scale Intranet projects. Our pitch was a little more successful than we planned. After my slide deck traveled to various executive offices (usually without me), word came back that we were to put the entire Intranet into WordPress and we were to make it look just like the external sas.com marketing site.
My response to this was
Immediately followed by fear, because it’s a big project that will go on for years.
I knew one of the first things I had to do was set up an infrastructure with dev, test and prod areas, because that system had been working well for us on other projects. I’d already been hearing about continuous integration and continuous deployment with a server called Bamboo from other developers in our department.
I went looking, but could not find a recipe for doing this with WordPress and Bamboo, so here is what I worked out.
I have to start by talking about Agile & Scrum, Continuous Integration and Continuous Deployment, because otherwise our way of doing things may not make much sense. The three work together. Although they are all big topics, just a small amount of information about each one is all you need to follow the rest of what I am going to be talking about.
Agile & Scrum
Agile is a software development methodology. It’s a set of principles that help developer teams solve a few common problems.
These are the guiding values of Agile:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
In a practical sense, this is what Agile does for us:
- ensures that the developers get constant feedback from clients so a project doesn’t go off the rails
- breaks development work down into small tasks called “stories” so everyone gets a sense of accomplishment on a frequent basis, and forward progress is very visible and very fast.
- uses brief iterations called “sprints” – a few stories are assigned to each sprint, and the expectation is that each story in a sprint can be completed by the end of the sprint. in our case, our sprints are three weeks long.
Scrum is a framework for implementing Agile. In Scrum, the team — which includes both developers and clients — has brief, frequent meetings during the course of the sprint, and everyone is kept accountable by stating what they have done and what they will do at each meeting.
In our case we meet three times a week, but many teams will meet daily. A client joins the meetings to give feedback and answer questions. This gives us very fast course corrections throughout the sprint.
Continuous integration refers to the frequent merging of new code with an existing code base. When multiple developers are changing the code base at the same time, there can be merge conflicts if each developer’s code branch becomes too far out of sync with the main code base. Continuous integration is meant to prevent or minimize those merge conflicts.
It basically means that each developer’s working version of the code is synced with the main code base frequently. This dovetails nicely with Agile “stories” as each story represents a small amount of code to be integrated. This means that there’s less likely to be a stressful train wreck of an integration when a new feature is complete.
On our team the main code base only changes a few times a week, usually, so it’s not necessary for us to sync with the main trunk several times a day as is the case on larger teams. We will merge the main trunk into our development branch before we are finished with a story if it seems wise to do so, to be sure we’re not in conflict with any new code that has been added.
Even if you’re the only developer working on a code base, this can help you. If you are working on a big new feature in a development branch, then have to stop work to do a hotfix in the main code base, CI just means you should merge your code base with the hotfix into your dev branch after the hotfix has been deployed. That way if you accumulate a bunch of hotfixes during the course of writing that big new feature, you won’t have to merge them all at once at the end.
There is a story about a pottery class that went around software development circles a while back. Agile folks love this story. At the beginning of the class, the class was divided into two groups. Half the class was told they would be judged by quality – they only had to make one pot, but it had to be perfect to get a passing grade. Half the class was told they would be judged by quantity alone, their pots would simply be weighed at the end of the semester. Much to everyone’s surprise, by the end of class the best quality work came from the quantity half of the class – not from the folks who were supposed to be making that one perfect pot. Why? because to get something right, you have to practice.
Continuous deployment amounts to about the same thing, you may not produce the exact features the clients need on your first attempt but when you get something in front of them quickly, and they use it, and give feedback then your next attempt will be closer to the mark. Keep iterating like this and you can break through the problems caused when your client has trouble articulating or deciding what they really need.
In our case. continuous deployment means that individual stories are deployed rather than large releases of bunches of stories bundled together. Deploys can usually be done with almost no downtime for the site. There is no deployment schedule. When a story is through QA, it can deploy. In a given sprint we might deploy ten times or one time – or not at all, if it’s a bad sprint.
This increases the continuous feedback loop between developers and clients. When clients get their hands on a feature and use it, we find issues and bad assumptions built into the feature that we can’t find during QA. We have found it helps break through logjams that occur when clients can’t articulate what they need due to their inexperience or other factors. If we can get something in front of them to use, even if it’s not what they really need, it will help them decide what they do really need.
For example, when we did the first story that implemented in-page tabs, we assumed that we’d never need more tabs than would fit across the page in one row. You can probably see where this is going! I believe my co-worker Sarah just did a fix to tabs so that when they wrap they at least line up neatly. We may have to go back to the designers on that one before we truly get it right, but obviously we didn’t think that one through very thoroughly until someone actually used it.
All three of these concepts revolve around short, iterative processes and they make it much easier to kick start a project from nothing – as we have done – and to keep the momentum going. They also make it easy to course correct. We often remind ourselves that we can try something new without a lot of risk – if it’s not working, in three weeks, when the next sprint starts, we’ll have a chance to change it.
To get into the details, I am going to start with code and dependency management. How the developers actually work with the system, what Bamboo does, and what technologies we use to make it all work.
The life cycle of a story
For developers, it all starts with a story. Each story has a life cycle. The story begins with the developer creating a branch from the main source code trunk, and that’s where they’ll write the code for the story. As they get each piece in place, they commit in their development branch. Once the story is complete, they merge their branch into master. Master is then pushed to the central repository and the continuous integration server picks up the new commits and automatically starts what we call a “build”. After we get a good build, we deploy.
Where does the story live its life?
We have eight different installs of WordPress in our infrastructure. Three are developer playpens. One install for automated testing at build time. Two are for testing by groups outside of our developer group, such as clients and the accessibility and usability groups at SAS. One is for content staging and one is the live site.
A story starts its life in the developer playpen, then is part of a build at autotest. Finally, it is deployed to the last four.
One detail that is relevant here — we do not run our developer playpens locally on our machines. They use the same file system and web servers as all the other installs. Although this is unusual, there are some advantages to this
- the playpens are always available, it doesn’t matter if the developer has shut down their machine
- they can be viewed by anyone on our network who knows the URL
- they use the same authentication system as all other installs
The last reason is fairly important already and will be very important when we start adding features to personalize and localize content.
What is a build?
I’ve mentioned the concept of a build a few times. The term build is borrowed from compiled languages. Since we aren’t working in a language that’s compiled, for us it means that all of our changes – to code, dependencies, and configuration – are made in one place at one time and checked to be sure there are no problems as a result of those changes. It’s more about integration than compilation.
We use a continuous integration / continuous deployment server called Bamboo for both the build and deploy processes.
This is an overview of how stories flow through the system. Once a story is merged with the main code trunk and pushed to our central source control repository, Bamboo starts a build. This build happens on our autotest install.
Once the build runs with no failures, Bamboo says that the build has passed and it creates a set of files to be copied out to our other installs – that’s called the Deployment Artifact. That’s also terminology from the world of compiled languages, but Bamboo uses it, so we do too.
Deploy is a separate task. If we are satisfied with the way autotest looks after a build, we can deploy the most recent Deployment Artifact to alpha, beta, stage and live.
This is what a build looks like when it’s running in Bamboo:
For us a build involves:
- pulling new code from source control – the stuff we write
- downloading dependencies – the stuff we get from other developers
- updating WordPress configuration – enabling plugins, setting settings and similar housekeeping tasks
Git, Composer and Bamboo are the technologies that our infrastructure is built around. We use Git for source control, and we use Composer for dependency management. Bamboo manages the build and deploy pipeline.
Source control allows us to have multiple developers working in the same code base without clobbering each other. It also allows us to keep records of everything that has happened to the code base, and – if we do a good job of writing our commit messages – we keep track of why things have happened. If we do a really good job and include the tracking ID of a story in a commit message, our systems will create links between Bamboo and our story tracking application, JIRA.
WordPress is not as friendly to source control as it could be, so we use something called WordPress-Skeleton to help with that. WordPress-Skeleton is a framework for running WordPress with WordPress core code sequestered into its own directory which makes it much, much easier to separate your code from WordPress core, and therefore makes it much, much easier to develop and deploy your code. Your theme and plugin code isn’t co-mingled with core.
We’ve probably all seen the file structure on the left before. Directory
wp-content/ houses all plugins and themes, so if you are developing theme or plugin code, your code is co-mingled with the rest of WordPress core and it’s difficult to keep it under source control without getting everything else in source control, too. If you wish to treat WordPress as a dependency and download it with a dependency manager, this structure is basically impossible to work with. Core needs to be in its own directory.
WordPress-Skeleton does just that, directory
wp/ is where core lives and your dependency manager can download to that location without clobbering your code, because your code is separated out into content/.
This brings up one of the concepts that helped me a lot while designing our infrastructure, that of “separation of concerns”. I had read about half of “The Pragmatic Programmer” when I started work on this project, enough to have been introduced to the idea of separation of concerns. While the authors were writing about program design, this principle also works in other kinds of design as well. Simply put it means grouping like things together and separating unlike things.
In this case, I separated code we write from all other code in the project. Code we write – our theme and a few plugins – is maintained in source control because of all the reasons that developers need source control. Other code is maintained by Composer, because all other code is a dependency, and Composer is a dependency manager.
One of the issues with WordPress-Skeleton is that it doesn’t use Composer, probably because Composer wasn’t really around (or used with WordPress) when WordPress-Skeleton was created. It uses Git submodules to link in all dependencies, which is a truly terrible way to manage dependencies and creates all kinds of angst on the part of the person who has to keep up with those submodules.
On my first attempt to set up this infrastructure I actually used Git subtrees because my IDE didn’t support submodules. It was a complete nightmare. Git submodules and subtrees are extremely difficult and painful to work with and I despaired of ever having anyone else on my team interested in helping to maintain our dependencies.
Then I discovered Composer and along with it a much easier way with much better separation of concerns – our code from dependencies.
So instead of submodules we have code that is brought in via Composer and for the most part that works extremely well. If you are not already familiar with using Composer with WordPress, there is a site called wpackagist.org where you can get Composer-friendly versions of all WordPress plugins that have been published on WordPress.org. You can also get WordPress core from the more general Composer package repository at packagist.org.
To find out more about usage of Composer with WordPress, take a look at composer.rarst.net. It was an instrumental resource for me as I set all of this up.
Even with WordPress-Skeleton there is the problem of having downloaded code in the plugins and themes directories along with the code that we have written. These things need to be kept separated. Our code needs to be in source control, but downloaded dependencies do not.
To keep Git and Composer out of each others way in the plugins and themes directories, our .gitignore file is set up to ignore everything that is not prefixed with our team’s acronym:
#ignore non-sww plugins and themes /content/plugins/* /content/mu-plugins/* /content/themes/* #do not ignore sww plugins and themes !sww-*
So now we have our code neatly separated out from the dependencies that we are downloading.
Back to how a build works. When a build runs in Bamboo, Bamboo steps through several tasks:
First it gets files from source control — the code we wrote.
This is what it looks like when the composer update part of the build is running. I really enjoy watching this part because boy was it hard to get it working!
Composer depends (ha, ha) on a manifest listing all of your dependencies and their version numbers. This manifest is called composer.json. We keep our composer.json in git, so Bamboo always has the latest dependency information after it downloads from source control. This means that if we need to update or add a dependency, we alter the composer.json file, commit the change in git and push to our central source control repository.
After composer update runs, all of the updated files are copied out to autotest, and then our configuration scripts run.
I mentioned that part of our build process is to make configuration changes – enable plugins, set settings, and general housekeeping. This is also part of the deploy process. These are all tasks that we have automated so Bamboo can do them for us.
Often when people talk about deployment schemes for WordPress there is some hand waving around configuration deployment and they will say to copy the database. While we can wholesale copy the database, there are reasons we don’t want to be tied to doing that.
For us, content development and code development are concurrent processes and completely separate from each other. The content on stage is always changing. Developers always have test content in their playpens. Even if we grabbed a fresh copy of everything from stage before beginning a new story, by the time we were done, we’d be out of sync.
But if a developer has configured a new plugin in their playpen as part of a story they are working on, we want to be able to use that playpen as the template for those plugin settings without having to manually reproduce them. So for config deploy it’s better to be more granular and deploy individual config settings tied to a specific story.
The key is WP-CLI which is a tool that allows you to control WordPress from the command line. Here’s what WP-CLI does for us:
- allows us to automate configuration changes
- keeps a record of all changes made
- enforces consistency – both describes and enforces a ‘policy’ across all installs
We set up scripts that run our WP-CLI commands, and Bamboo can run those scripts for us. We mainly use UNIX shell scripts, but there are many scripting languages that can run WP-CLI commands – anything that can run commands on the command line can be used.
How does Bamboo run WP-CLI tasks for us?
This shows the deploy tasks for the alpha install.
We have one script that always runs with every build and every deploy – it’s called current-deploy.sh and you can see it being run as the last deployment task in this screenshot.
We pass an argument to that script with the name of the install – in this case alpha. That allows us to target alpha with any configuration updates we have in our script. We have separate deploy jobs for each install.
current-deploy.sh in turn runs other scripts. It always runs an enable-plugins.sh script that ensures the correct plugins are enabled everywhere. In addition, it may call other scripts that run WP-CLI commands specific to the story being deployed.
#!/bin/bash # ================================================== # # current deploy tasks # runs for every instance, every deploy # ================================================== # # always run enable-plugins everywhere ./deploy/enable-plugins.sh "$1" # 9/24/15 - delete unwanted template SWWHOME-390 # ./deploy/SWWHOME-390-delete-sidebar-template/del-no-sidebar.sh "$1" # 8/21/15 grab megamenu settings from lisa playpen and distribute to the other installs # ./deploy/isi-logo-header/megamenu-settings.sh "$1" # 8/12/15 - regenerate thumbnails for SWWHOME-327 talkingheads # ./deploy/SWWHOME-327-talkingheads/regenerate-thumbnails.sh "$1" # 8/10/15 - regenerate thumbnails to create new sizes needed for new templates # ./deploy/SWWHOME-339-level1-cards/level1-cards.sh "$1" # 7/16/15 - remove some deleted plugins from leftnav development # ./deploy/leftnav/remove-section-links.sh "$1"
For example, if my co-worker Sarah had configured a new mega-menu plugin in her playpen, she would also write a script that gets the plugins settings from her playpen and updates those settings on the current deployment target. She would add a line to enable-plugins.sh to ensure the plugin is enabled, and she’d add a line to current-deploy.sh to run her settings script. She would commit all of this in git.
After her story to add a new mega-menu plugin to our site is completely deployed, she would go back to current-deploy.sh and comment out the line that runs her settings script – but not delete it, so we’d retain a record of when it was run and what it changes.
These scripts are kept in git, which keeps them in sync with the other code for our stories.
We treat content development and deployment as a completely separate process from code and configuration development and deployment. One of our goals in moving the Intranet to a CMS is to give content owners a way to maintain and control their own content, regardless of their skill set. We want the non-technical content owners to be able to directly edit their content using intuitive tools, we don’t want content management to be the responsibility of a single knowledgeable individual in a department because that model is partly responsible for where we are now, with a lot of out of date, misleading content on the Intranet.
However, we also have to protect our content owners from making mistakes publicly, and we need to give them a place to stage big content revisions safely and quietly. We have to give them a chance to review and approve content when necessary. That’s why we have a WordPress install that is specifically for content staging.
All content editing occurs on stage. To enforce this, we have disabled access to the admin screens of the live site for everyone who is not a super-admin, so it’s not possible for anyone but us developers to make changes directly to live.
To deploy content between stage and live we use a plugin by Crowd Favorite called RAMP. We haven’t used it heavily yet because the business users on the project aren’t ready to work with the live server.
RAMP shows all differences between your content staging site and your live site. It allows you to create ‘batches’ of changes and save them to push in the future. RAMP gives you a nice audit record of all the changes that have been moved over, and allows you to roll back the last change.
Once we have many content owners using the system, they will need a coordinator – a sort of traffic cop – to keep the world of RAMP running smoothly. I definitely do not think that every content maintainer should be able to use RAMP, and that may cause some consternation when folks who thought that WordPress was going to make web site updates completely self-service still have to go through the “traffic cop” to get their changes pushed to live.
I also think the RAMP UI is going to be difficult when there is a high volume of changes.
Maybe ‘think’ is not the right word.
There’s no way to filter pages by category or other criteria in the UI, there’s just a list of changed pages, posts, categories, etc. When it’s a lot of changes you have to pick through it to find what you need to batch up and push.
In SAS IT on any given project we talk about risks and mitigations for those risks. Now none of the things that I worry about put this project at risk to such a degree that I am going to raise them up to management and say that we need to tell the clients that we have a problem. But the risk / mitigation language is useful for expressing these concerns.
I’ve already mentioned some of the concerns I have regarding RAMP – here are some other areas that concern me.
We have a QA process but right now it is far too minimal for my comfort level. One of the big things that you’re supposed to have for continuous deployment is a bunch of automated testing. We are only just starting to create automated tests and I can see how big of a job it’s going to be to get tests in place that really give us a comfort level as the system becomes progressively more complex.
Scaling of the admin interface
We’re talking about having thousands of pages in a single WordPress site. Even with just over 100 pages I find the admin UI cumbersome and searching for pages is exasperating and slow. My co-worker Sarah has done some customization that makes it easier to filter pages in a way that makes sense, but I am concerned that we’ll need to make more changes. Fortunately, WordPress is very flexible and it will mainly be a matter of us, as developers, understanding what we can do and how to implement it.
At the moment we have no record of what has happened when changes are made in certain parts of the content staging install, particularly menus and page deletions, and no central report of all activity on the system. We have seen one plugin that we think will give us a central audit log (with rollback) but haven’t done anything about it yet.
In every case, there’s a way to solve these problems and, as we always remind ourselves, the beautiful thing about Agile is that every few weeks you get a clean slate and a chance to push an important concern to the top of the list of things to work on. That increases my confidence that we will get to these things – if we need them – and if we don’t, they can fall by the wayside without ever having wasted the team’s time.
“Art and Fear”, David Bayles
“The Pragmatic Programmer”, Andrew Hunt and David Thomas