the promise and the pain of automated testing

I am probably too tired from trying all day to get something useful out of Behat (an automated test technology) to write this effectively, but I still really want to complain about this right the heck now.

Automated testing is supposed to be a necessity with continuous integration / continuous deployment. So far we haven’t succeeded in getting any automated testing set up, but today I’d reached the point where the thought of manually testing yet another version of WordPress core was more than I could take. I’d gotten some kind of Behat test working a while back, albeit not one that did anything useful. I decided to take another crack at it.

Oh Behat, you are so deceptively simple…

Feature: prisoner

  @javascript @wip
  Scenario: As a user with the editor role I can create a new Prisoner template page
    Given I am logged in as "numbersix"
    And I go to ""
    And I fill in "post_title" with "Create Prisoner template page"
    And I fill in "content" with "I am in The Village."
    And I press "publish"
    And I select "page-templates/page-prisoner.php" from "page_template"
    And I press "publish"
    And I fill in "_prisoner_template_0_prisoner_feature_title" with "I am not a number, I am a free man"

It looks so promising, doesn’t it? So easy. If only it actually did something useful.

Some of it works. It gets to about the first I press “publish” and then there’s a problem. It took me all day to get this this point, after…

  • adding dependencies and getting versions straightened out in Composer
  • re-installing and configuring selenium server
  • updating php
  • updating the JDK
  • finding a useful example of using Behat with WordPress
    • going down a side alley wherein I found three separate git repos from three different people all claiming to have written the same Behat test code that actually does nothing, as far as I can tell
  • probably some other stuff that has since fallen out of my head

It’s not difficult to find articles, tutorials, and WordCamp talks about setting up automated testing. There’s always a technology like PHPUnit, or Codeception, or whatever. Sometimes with something like Behat it looks appealingly simple but like an Angular.js app, the simplicity sits on top of a giant stack of other technologies that a developer has to manage, somehow.

Meanwhile, code is continuously pushed, WordPress versions stack up. Launch deadlines loom.

In the past year and a half, I have spent a week getting PHPUnit running and talking to Selenium. I wrote a test that checked for Hello Dolly, got it running on Bamboo, it worked for a while and then mysteriously stopped working. Then our testing adviser pointed us toward Behat as a better way to handle the kind of testing we need to do. So I spent a day or so getting Behat going. Then I ran out of time and never wrote any tests. Then we have today, I got Behat working with Selenium and Mink and got half of one useful test written and working on my local machine. The thought of getting it to run on Bamboo is the stuff nightmares are made of, so I am not even considering that in the short term.

That’s all the time I’ve had to spend on automated testing in 18 months, and I’m the only person in our group who has the patience to deal with these huge, crazy stacks that sit under testing technologies.

Maybe tomorrow I’ll figure out how to force focus in Behat / Mink / Selenium and I can get somewhere. Or not. Or I’ll just tear my hair out and send out a torrent of sarcastic tweets and we’ll be no better off.