Showing posts with label TestBox. Show all posts
Showing posts with label TestBox. Show all posts

Saturday, 21 December 2013

So what did I do today? (Nothing interesting. Here are the details)

I faffed around on the computer - with ColdFusion and Railo and TestBox - all day today, and have concluded I have nothing interesting to show for it. Still... this is a log of what I've been up to with CFML, so you're gonna hear all about it anyhow. And what's "better"... I'm gonna spin it out to be two articles. I am the Peter Jackson of CFML blogs!

OK, so I got wind of something to investigate the other day, and to do the metrics, I needed to time some stuff. Normally I'd just use a getTickCount() before and after the code, but I thought I might need something more comprehensive this time, so I figured a wee stopwatch UDF was in order. I've also just installed TestBox and am in the process of seeing if I can migrate away from MXUnit, so decided to do another TDD exercise with it. Note: there is no great exposition in this article, or really much new about TDD etc. It's simply what I did today.

Note: I abandoned TesxtBox's MXUnit compatibility mode today, because of a show-stopper bug: "ADDASSERTDECORATOR not implemented". This is the function that allows the importing of custom assertions, which I am using here. On the basis of that, I decided to just go with TestBox syntax instead.

Let's have a quick look at the function, and the tests.

struct function makeStopwatch(){
    var timeline        = [];

    var lap = function(string message=""){
        var ticksNow    = getTickCount();
        var lapCount    = arrayLen(timeline);
        var lap            = {
            currentClock    = ticksNow,
            lapDuration        = lapCount > 0 ? ticksNow - timeLine[lapCount].currentClock : 0,
            totalDuration    = lapCount > 0 ? ticksNow - timeLine[1].currentClock : 0,
            message            = message
        arrayAppend(timeline, lap);
        return lap;

    return {
        start        = function(string message="start"){
            return lap(message);
        lap            = function(string message="lap"){
            return lap(message);
        stop        = function(string message="stop"){
            return lap(message);
        getTimeline    = function(){
            return timeLine;

Not much interesting here. I enjoyed finding another reason / excuse to use function expressions and a wee bit of closure around the timeline which "sticks" with the start() / lap() / stop() / getTimeline() functions.

What this function does is to log a struct at start, each lap, and another on stop. Here's it in action:

// useStopwatch.cfm
include "makeStopwatch.cfm";
stopwatch = makeStopwatch();

stopwatch.start("Begin timing");
secondLap = stopwatch.lap("after another 1500ms");
writeDump(var=secondLap, label="secondLap");
stopwatch.start("Stop timing");


secondLap - struct
MESSAGEafter another 1500ms
MESSAGEBegin timing
MESSAGEafter another 1500ms
MESSAGEStop timing

This is the bumpf I usually need when I'm timing stuff. Well: some or all of it. It'll be quite handy, I reckon.

One interesting facet here is that initially I thought I'd need three separate functions for the start() / lap() / stop() functions, I was doing TDD (seriously, I knew you'd wonder, so everything I write on this blog uses full TDD now), and having knocked out the first few tests to verify the method signatures for the returned functions, it occurred to me that stop() was actually redundant. It doesn't do anything that lap() wouldn't already do. I mean all this "stopwatch" does is take time marks... there's no clock starting or stopping really (getTickCount() keeps on ticking, we simply start paying attention and then stop paying attention at some point).

So, anyway, I decided before I starting messing around with a redesign, I'd get the test coverage done, get it working, and then refactor. This is something one of my colleagues (Brian, I mean you) has been drumming into me recently: don't get halfway through something, decide to do it differently, start again, refactor, waste time, and not have anything to show for it if I get interrupted (or, in a work situation, we get to the end of the sprint and need to release). So I banged out the rest of the tests, got everything working, and looked at my code some more.

Here are the tests:

CFML: Follow-up to last night's article about a TestBox "bug". PEBCAK strikes again!

Time for me to fall on my sword.

Yesterday I started looking at TestBox ("Unit Testing / TDD - switching off MXUnit, switching on TestBox"), and whilst messing around, I came across a coupla bugs (some legit, I hasten to add), one of which Luis asked me to clarify on the ColdBox mailing list. Which I did: "[Testbox v1.0.0] "Variable ISEXPECTEDEXCEPTION is undefined" on erroring test".

The gist of this is as follows:

// Tests.cfc
component {

    public void function beforeTests(){
        variables.beforeTestsRun = true;

    public void function setup(){
        variables.setupRun = true;

    public void function testInitFunctions(){
        assert(structKeyExists(variables, "beforeTestsRun"));
        assert(structKeyExists(variables, "setupRun"));

    public void function testThatErrors(){
        throw(type="TestException", message="This is a test exception", detail="Note that it is NOT being caught / tested for. This is by design");

    public void function testThatFails(){
        fail("This test should fail");


This is a distillation of my situation, not actual code. I am testing TestBox's MXUnit compatibility "mode", so here's a test that checks that beforeTests() and setup() run, plus how it deals with an erroring test, and a failing test.

When I was running this on ColdFusion, I got this output:

TestBox v1.0.0.00062

Global Stats (86 ms)

[ Bundles/Suites/Specs: 1/1/3 ] [ Pass: 1 ] [ Failures: 0 ] [ Errors: 2 ] [ Skipped: 0 ] [ Reset ] 

shared.git.blogExamples.unittests.testbox.Tests (18 ms)

[ Suites/Specs: 1/3 ] [ Pass: 1 ] [ Failures: 0 ] [ Errors: 2 ] [ Skipped: 0 ] [ Reset ]
Note how the erroring and failing tests were reporting as erroring, but with an error internal to TestBox, not with the expected output MXUnit would give. So basically when a test failed or errored, TestBox itself was failing.

At this point... can you see what I've done wrong here?

Here's the thing. For TestBox test CFCs, there is no need to extend anything. I guess TestBox just uses mix-ins to insert all its bits and pieces. So my CFC doesn't extend anything.

However with MXUnit, one does need to extend mxunit.framework.TestCase.

Obviously on all my "normal" MXUnit test CFCs I do have the correct inheritance in place. But this particular test CFC started out with me testing TestBox tests, not MXUnit tests, so I neglected to put the inheritance in when I switched over to running it with MXUnit compatibility "mode". My bad.

Once I sorted that out, my tests run fine:

TestBox v1.0.0.00062

Global Stats (82 ms)

[ Bundles/Suites/Specs: 1/1/3 ] [ Pass: 1 ] [ Failures: 1 ] [ Errors: 1 ] [ Skipped: 0 ] [ Reset ] 

shared.git.blogExamples.unittests.testbox.Tests (15 ms)

[ Suites/Specs: 1/3 ] [ Pass: 1 ] [ Failures: 1 ] [ Errors: 1 ] [ Skipped: 0 ] [ Reset ]

I fed this back to Luis, so that's case closed on that one. I also editorialised a bit, saying this:

It's a pity your test case CFCs don't need to be of a certain type, as that would have clarified that error from the outset. It would also help mitigate a failing MXUnit always had in my opinion: that it determines whether a CFC is a test CFC by its NAME rather than its TYPE (eg: it's a subclass of TestCase). This name-based approach isn't great OO, especially if the capability to do it "properly" is right there.

This has always annoyed me about MXUnit. And it'll annoy me about TestBox. It doesn't annoy me much, but that MXUnit always relied on a file-naming scheme to identify test CFCs always seemed a bit amateurish to me. [gallic shrug]... this is a very minor gripe, and I do not forget that MXUnit has still been one of the most useful CFML tools out there, and TestBox is almost certainly gonna be an appropriate successor.

And Luis... very awesome you followed up my mailing list question so promptly. Much appreciated. I need to venture out into an uncharacteristically windy London for coffee and food, but I shall be writing more tests today. Let's see how it goes...


Friday, 20 December 2013

Unit Testing / TDD - switching off MXUnit, switching on TestBox

This article is more an infrastructure discussion, rather than examining more actual testing stuff. The ever-growing *Box empire has recently borged into yet another part of the CFML community: testing. They're released another box... TestBox. TestBox is interesting to me as it has a different approach to testing than MXUnit has... rather than xUnit style assertion-based testing, instead favouring a BDD approach. I've not done a lick of BDD, but people keep banging on about it, so I shall be looking at it soon. -ish. First I need to switch to TestBox.

One appealing thing I had heard about TestBox is that it's backwards compatible with MXUnit, so this should mean that I can just do the switch and continue with my current approach to testing, and ease my way into BDD as I learn more about it. So the first thing I decided to examine is how well this stands up, and how many changes I need to make to my existing tests to get them to run. Realistically, nothing is every completely backwards compatible... not even say between different versions of the same software (ColdFusion 9 to ColdFusion 10), let along a second system emulating another system (eg: Railo and ColdFusion). This is fine. I don't expect this migration to be seamless.

Here's what I worked through this morning to get up and running (spoilers: kinda running) on TestBox.

I preface this with the fact that I have always found Ortus's documentation to be a bit impenetrable (there's too much of it, it waffles too much), so I was hesitant about how long this would all take.

Locating, downloading and installing

Finding it

I googled "testbox", and the first link was the ColdBox Platform Wiki - TestBox. This is promising. Within a paragraph (and a to-the-point paragraph which just intros the product, so maybe the docs have got some improved focus: cool) there were download links. TestBox requires ColdFusion 10 / Railo 4.1, btw. I presume it uses closure or something? I'm not sure. But that's cool, I use CF10 and Railo [latest] for my work for this blog. It does preclude me from really giving it a test our with our 3000 unit tests at work though (which is a shame), because we're still on CF9 and will be for the foreseeable future.

Installing it

The installation instructions threw me a bit. The default suggestion is to put the testbox dir into the web root, but that's poor advice: only files specifically intended to be web browseable should ever be in your web root. Fortunately the also mention one can stick 'em anywhere, and map them in with a /testbox mapping. I wish this was their default suggestion. In fact I wish it was their only suggestion. They should not encourage poor practice.

There's a caveat with this though (and this is where I had problems), is that TestBox does have some web assets which need to be web browseable, so it does actually need a web mapping, not just a CF mapping. They do caveat this further down the page.

The first pitfall I had was which directory they're actually talking about. The zipfile has this baseline structure:


So I homed this lot in my CF root (not web root, CF root) as /frameworks/testbox/1.0.0/, and added a /testbox CF mapping to that location.

WARNING (if you're following along and doing this at the same time): this is not the correct thing to do. Keep reading...

I then had a look around for which directory I needed to add a web server virtual directory for, and found web-servable assets in the following locations:


(I searched for images, JS, CSS, HTML and index.cfm files; not perfect, but will give me an idea).

OK, so I figured he apidocs and samples are separate from the TestBox app, but that still leaves three disconnected (and laterally displaced) directories which need to be web browseable. This ain't great. So basically it looks like I need to make the entire /testbox dir web browseable. That's a bit shit, and a bit how we might have set up our CFML-driven websites... ten years ago. Oh well.

Configuring Tomcat

Here's a challenge (cue: Sean to get grumpy). I have no idea how to set up a virtual directory on Tomcat's built-in web server. Fortunately that's what Google is for, so I googled "tomcat web server virtual directories", and the very first link is a ColdFusion-10-specific document: "Getting Started with Tomcat in ColdFusion 10". I shuddered slightly that this is just in the ColdFusion Blog, rather than in the CF docs where it belongs, but it'll do. Fortunately the info in there is accurate, which is good.

Basically there's a file server.xml located at <ColdFusion_Home>/runtime/conf/server.xml, where <ColdFusion_Home> is the cfusion dir in your ColdFusion install directory. For me the conf dir is at: C:\apps\adobe\ColdFusion\10\cfusion\runtime\conf.

In there there's an XML note like this:

    path    = "/"
    docBase    = "<cf_home>\wwwroot"
    WorkDir    = "<cf_home>\runtime\conf\Catalina\localhost\tmp"

It's commented out by default. All the instructions one needs are in the file itself, but basically it's uncomment it, put actual paths in, and add an aliases attribute:

    path    = "/"
    docBase    = "C:\apps\adobe\ColdFusion\10\cfusion\wwwroot"
    WorkDir    = "C:\apps\adobe\ColdFusion\10\cfusion\runtime\conf\Catalina\localhost\tmp"
    aliases    = "/testbox=C:\webroots\frameworks\testbox\1.0.0"

I restarted CF and browsed to http://localhost:8500/testbox, and I got the files in my C:\webroots\frameworks\testbox\1.0.0 directory listing, so that worked. Good to know. I'll now forget about server.xml and aliases and stuff as I won't need to do it again for another six months. Shrug.

ColdFusion config

I put a mapping to the same place in my test app's Application.cfc:

// Application.cfc
component {

    this.mappings            = {
        "/cflib"    = getDirectoryFromPath(getCurrentTemplatePath()),
        "/testbox"    = expandPath("/testbox") // CF will use the virtual directory to resolve that. This is slightly cheating, but hey