CFML/TestBox: running on actual proper tests

G'day:
Various people seem to be trying out TestBox on their old MXUnit unit tests today, and reporting back on Twitter. I'm working through our ones, and have a few findings. I have more that 140 chars of information, so I'll quickly blog instead.

I can't go into too many details regarding our tests, but we have a few thousand of them spread over a number of applications.

Running through our main tranche of... ooh... 2793 (!!) tests, I had the following issues:

assertIsQuery() is missing

This is a standard MXUnit assertion (assertIsQuery()), missing from TestBox. Lack of this probably breaks about 200 of our unit tests

assertIsEmpty() is missing

So is this (assertIsEmpty()). We use this less often, but it probably breaks 50-odd tests.

mxunit:expectedException missing

It was pointed out to me the other day ("I should pay attention to what my readers tell me! expectException()") that MXUnit has a better approach to dealing with this - and one that seems is indeed implemented in TestBox - expectException(). However we have a lot of tests, and a bunch of them still use the above annotation. Perhaps 50 tests "fail" due to this not being implemented.

makePublic() doesn't seem to work

I didn't investigate this thoroughly, but it seems TestBox has implemented makePublic(), but it doesn't actually work. I say this because I get no error on code calling makePublic(), but none of the methods made public actually are public, as a result. A few hundred failures.

mock() not implemented

MXUnit has built-in mocking. So does TestBox. However each are different, and it doesn't look like MXUnit's version has been implemented. We only use this once in our tests (we generally use MockBox).

If beforeTests() errors, testing halts

This in itself is completely the same as MXUnit, but it basically discounts any chance of us using TestBox as it stands, because given addAssertDecorator() is also not implemented, this prevents me from actually doing a test run. Because the whole thing falls over as soon as we try to bring in our custom assertions. That aside, lack of being able to call in custom assertions also kills about another 500 of our tests. Fixing beforeTests() would be really nice, but if addAssertDecorator() was implemented, that'd get us moving forward.

One really annoying side effect of this is that TestBox doesn't preserve the context of where the error happened, it just reports it as happening in the bowels of TestBox itself, which is not correct. It should bubble back the original exception. It was basically impossible for me to bandaid some of the errors I was seeing because TestBox wasn't telling me where they were happening!

The method init was not found in component TestCase.cfc

This is a very edge-case one. We have a helper CFC which is not run as a test case, but we want to run some of TestCase's methods. So we instantiate an instance of TestCase of our own. In MXUnit, TestCase.cfc has an init() method. In TestBox, it does not, so it errors. This is one of those crazy edge-cases one only ever discovers testing on real world code.

Others

I was getting a few other test failures too, in tests I know work fine. I wasn't too sure if those were a byproduct of all the stuff above, so I did not investigate. Once I have a stable system, I will look at those ones more closely.

Conclusion

So those results weren't great. About a quarter of our tests would not run without being monkeyed with, and we cannot do a test run at all. Still: this is because of a few minor glitches we use in key areas of our tests.

I'm outa lunchtime, so I will need to run the other tests later, and see if I come up with anything else. But until we get the issues above sorted out, TestBox is not a possibility for us. I'll continue to use it at home, though.

--
Adam

So what did I do today? (Nothing interesting. Here are the details)

G'day:
I faffed around on the computer - with ColdFusion and Railo and TestBox - all day today, and have concluded I have nothing interesting to show for it. Still... this is a log of what I've been up to with CFML, so you're gonna hear all about it anyhow. And what's "better"... I'm gonna spin it out to be two articles. I am the Peter Jackson of CFML blogs!

OK, so I got wind of something to investigate the other day, and to do the metrics, I needed to time some stuff. Normally I'd just use a getTickCount() before and after the code, but I thought I might need something more comprehensive this time, so I figured a wee stopwatch UDF was in order. I've also just installed TestBox and am in the process of seeing if I can migrate away from MXUnit, so decided to do another TDD exercise with it. Note: there is no great exposition in this article, or really much new about TDD etc. It's simply what I did today.

Note: I abandoned TesxtBox's MXUnit compatibility mode today, because of a show-stopper bug: "ADDASSERTDECORATOR not implemented". This is the function that allows the importing of custom assertions, which I am using here. On the basis of that, I decided to just go with TestBox syntax instead.

Let's have a quick look at the function, and the tests.

//makeStopwatch.cfm
struct function makeStopwatch(){
    var timeline        = [];

    var lap = function(string message=""){
        var ticksNow    = getTickCount();
        var lapCount    = arrayLen(timeline);
        var lap            = {
            currentClock    = ticksNow,
            lapDuration        = lapCount > 0 ? ticksNow - timeLine[lapCount].currentClock : 0,
            totalDuration    = lapCount > 0 ? ticksNow - timeLine[1].currentClock : 0,
            message            = message
        };
        arrayAppend(timeline, lap);
        return lap;
    };

    return {
        start        = function(string message="start"){
            return lap(message);
        },
        lap            = function(string message="lap"){
            return lap(message);
        },
        stop        = function(string message="stop"){
            return lap(message);
        },
        getTimeline    = function(){
            return timeLine;
        }
    };
};

Not much interesting here. I enjoyed finding another reason / excuse to use function expressions and a wee bit of closure around the timeline which "sticks" with the start() / lap() / stop() / getTimeline() functions.

What this function does is to log a struct at start, each lap, and another on stop. Here's it in action:

// useStopwatch.cfm
include "makeStopwatch.cfm";
stopwatch = makeStopwatch();

stopwatch.start("Begin timing");
sleep(500);
stopwatch.lap();
sleep(1500);
secondLap = stopwatch.lap("after another 1500ms");
writeDump(var=secondLap, label="secondLap");
sleep(2000);
stopwatch.start("Stop timing");
writeDump(var=stopWatch.getTimeline());

Output:

secondLap - struct
CURRENTCLOCK	1387658080067
LAPDURATION	1500
MESSAGE	after another 1500ms
TOTALDURATION	2000

array
1	struct CURRENTCLOCK 1387658078067 LAPDURATION 0 MESSAGE Begin timing TOTALDURATION 0
2	struct CURRENTCLOCK 1387658078567 LAPDURATION 500 MESSAGE lap TOTALDURATION 500
3	struct CURRENTCLOCK 1387658080067 LAPDURATION 1500 MESSAGE after another 1500ms TOTALDURATION 2000
4	struct CURRENTCLOCK 1387658082117 LAPDURATION 2050 MESSAGE Stop timing TOTALDURATION 4050

This is the bumpf I usually need when I'm timing stuff. Well: some or all of it. It'll be quite handy, I reckon.

One interesting facet here is that initially I thought I'd need three separate functions for the start() / lap() / stop() functions, I was doing TDD (seriously, I knew you'd wonder, so everything I write on this blog uses full TDD now), and having knocked out the first few tests to verify the method signatures for the returned functions, it occurred to me that stop() was actually redundant. It doesn't do anything that lap() wouldn't already do. I mean all this "stopwatch" does is take time marks... there's no clock starting or stopping really (getTickCount() keeps on ticking, we simply start paying attention and then stop paying attention at some point).

So, anyway, I decided before I starting messing around with a redesign, I'd get the test coverage done, get it working, and then refactor. This is something one of my colleagues (Brian, I mean you) has been drumming into me recently: don't get halfway through something, decide to do it differently, start again, refactor, waste time, and not have anything to show for it if I get interrupted (or, in a work situation, we get to the end of the sprint and need to release). So I banged out the rest of the tests, got everything working, and looked at my code some more.

Here are the tests:

Unit Testing / TDD - switching off MXUnit, switching on TestBox

G'day:
This article is more an infrastructure discussion, rather than examining more actual testing stuff. The ever-growing *Box empire has recently borged into yet another part of the CFML community: testing. They're released another box... TestBox. TestBox is interesting to me as it has a different approach to testing than MXUnit has... rather than xUnit style assertion-based testing, instead favouring a BDD approach. I've not done a lick of BDD, but people keep banging on about it, so I shall be looking at it soon. -ish. First I need to switch to TestBox.

One appealing thing I had heard about TestBox is that it's backwards compatible with MXUnit, so this should mean that I can just do the switch and continue with my current approach to testing, and ease my way into BDD as I learn more about it. So the first thing I decided to examine is how well this stands up, and how many changes I need to make to my existing tests to get them to run. Realistically, nothing is every completely backwards compatible... not even say between different versions of the same software (ColdFusion 9 to ColdFusion 10), let along a second system emulating another system (eg: Railo and ColdFusion). This is fine. I don't expect this migration to be seamless.

Here's what I worked through this morning to get up and running (spoilers: kinda running) on TestBox.

I preface this with the fact that I have always found Ortus's documentation to be a bit impenetrable (there's too much of it, it waffles too much), so I was hesitant about how long this would all take.

Locating, downloading and installing

Finding it

I googled "testbox", and the first link was the ColdBox Platform Wiki - TestBox. This is promising. Within a paragraph (and a to-the-point paragraph which just intros the product, so maybe the docs have got some improved focus: cool) there were download links. TestBox requires ColdFusion 10 / Railo 4.1, btw. I presume it uses closure or something? I'm not sure. But that's cool, I use CF10 and Railo [latest] for my work for this blog. It does preclude me from really giving it a test our with our 3000 unit tests at work though (which is a shame), because we're still on CF9 and will be for the foreseeable future.

Installing it

The installation instructions threw me a bit. The default suggestion is to put the testbox dir into the web root, but that's poor advice: only files specifically intended to be web browseable should ever be in your web root. Fortunately the also mention one can stick 'em anywhere, and map them in with a /testbox mapping. I wish this was their default suggestion. In fact I wish it was their only suggestion. They should not encourage poor practice.

There's a caveat with this though (and this is where I had problems), is that TestBox does have some web assets which need to be web browseable, so it does actually need a web mapping, not just a CF mapping. They do caveat this further down the page.

The first pitfall I had was which directory they're actually talking about. The zipfile has this baseline structure:

/testbox_1.0.0/
    testbox-1.0.0.00062-201312171237
    apidocs/
    browser/
    runner/
    runner-template/
    samples/
    testbox/
    license.txt
    mockbox.txt
    testbox.txt

So I homed this lot in my CF root (not web root, CF root) as /frameworks/testbox/1.0.0/, and added a /testbox CF mapping to that location.

WARNING (if you're following along and doing this at the same time): this is not the correct thing to do. Keep reading...

I then had a look around for which directory I needed to add a web server virtual directory for, and found web-servable assets in the following locations:

/apidocs/
/browser/
/runner/
/samples/
/testbox/system/testing/reports/assets/

(I searched for images, JS, CSS, HTML and index.cfm files; not perfect, but will give me an idea).

OK, so I figured he apidocs and samples are separate from the TestBox app, but that still leaves three disconnected (and laterally displaced) directories which need to be web browseable. This ain't great. So basically it looks like I need to make the entire /testbox dir web browseable. That's a bit shit, and a bit how we might have set up our CFML-driven websites... ten years ago. Oh well.

Configuring Tomcat

Here's a challenge (cue: Sean to get grumpy). I have no idea how to set up a virtual directory on Tomcat's built-in web server. Fortunately that's what Google is for, so I googled "tomcat web server virtual directories", and the very first link is a ColdFusion-10-specific document: "Getting Started with Tomcat in ColdFusion 10". I shuddered slightly that this is just in the ColdFusion Blog, rather than in the CF docs where it belongs, but it'll do. Fortunately the info in there is accurate, which is good.

Basically there's a file server.xml located at <ColdFusion_Home>/runtime/conf/server.xml, where <ColdFusion_Home> is the cfusion dir in your ColdFusion install directory. For me the conf dir is at: C:\apps\adobe\ColdFusion\10\cfusion\runtime\conf.

In there there's an XML note like this:

<Context
    path    = "/"
    docBase    = "<cf_home>\wwwroot"
    WorkDir    = "<cf_home>\runtime\conf\Catalina\localhost\tmp"
>
</Context>

It's commented out by default. All the instructions one needs are in the file itself, but basically it's uncomment it, put actual paths in, and add an aliases attribute:

<Context
    path    = "/"
    docBase    = "C:\apps\adobe\ColdFusion\10\cfusion\wwwroot"
    WorkDir    = "C:\apps\adobe\ColdFusion\10\cfusion\runtime\conf\Catalina\localhost\tmp"
    aliases    = "/testbox=C:\webroots\frameworks\testbox\1.0.0"
>
</Context>

I restarted CF and browsed to http://localhost:8500/testbox, and I got the files in my C:\webroots\frameworks\testbox\1.0.0 directory listing, so that worked. Good to know. I'll now forget about server.xml and aliases and stuff as I won't need to do it again for another six months. Shrug.

ColdFusion config

I put a mapping to the same place in my test app's Application.cfc:

// Application.cfc
component {

    this.mappings            = {
        "/cflib"    = getDirectoryFromPath(getCurrentTemplatePath()),
        "/testbox"    = expandPath("/testbox") // CF will use the virtual directory to resolve that. This is slightly cheating, but hey
    };

}