Unit testing: the importance of testing over assumption

G'day:
Today some code! That's better than rhetoric innit? I'm sure it won't be as popular though.

OK, so I have been too busy recently with ColdFusion 11 and various other odds 'n' sods to either continue writing about unit testing, and also I have been neglecting the CFLib.org submission queue. So today I am remedying both of those in one fell swoop.

This article is not really "TDD" or "BDD" as the code I am testing is a fait accompli, so the tests are not driving the development. However this is perhaps a good lesson in how to retrofit tests into functionality, and what sort of things to test for. There's a bit of TDD I suppose, because I end up completely refactoring the function, but first create a test bed so that that is a safe thing to do without regressions.

On CFLib there is already a function firstDayOfWeek(), but someone has just submitted a tweak to it so that it defaults to using today's date if no date is passed into it. Fair enough. Here's the updated code:

function firstDayOfWeek() {
    var dow = "";
    var dowMod = "";
    var dowMult = "";
    var firstDayOfWeek = "";
    if(arrayLen(arguments) is 0) arguments.date = now();
    date = trim(arguments.date);
    dow = dayOfWeek(date);
    dowMod = decrementValue(dow);
    dowMult = dowMod * -1;
    firstDayOfWeek = dateAdd("d", dowMult, date);

    return firstDayOfWeek;
}

Nothing earthshattering there, and I basically scanned down it going, "yep... uh-huh... oh, OK, yeah... cool... yeah that seems OK". And I was about to just approve it. Then I felt guilty and thought "ah... I really should test this. And I'll use TestBox and its BDD-style syntax to do so, and I can get a blog article out of this too.

It's just as well I did test it, because it has a bug. Can you see it?

I sat down and looked at the code, and went "right... what do I need to test?". I came up with this lot:

• if I don't pass in a date, it returns the first day of the current week;
• if I do pass in a date, it returns the first day of that week;
• if I pass in the date that is the first day of the week already, it passes back that same date;
• if I pass it "not a date", it errors predictably.

I think that about covers the testing need here? OK, so I started writing the tests:

// TestFirstDayOfWeek.cfc
component extends="testbox.system.testing.BaseSpec" {

    function beforeAll(){
        include "udfs/firstDayOfWeek.cfm";

        variables.testDate = now();
        variables.testFirstDayOfWeek = dateAdd("d", -(dayOfWeek(variables.testDate)-1), variables.testDate);
    }

    function run(){
        describe("Tests for firstDayOfWeek()", function(){

            it("returns most recent sunday for default date", function(){
                expect(
                    dateCompare(variables.testFirstDayOfWeek, firstDayOfWeek(), "d")
                ).toBe(0);
            });

            it("returns most recent sunday for current date", function(){
                expect(
                    dateCompare(variables.testFirstDayOfWeek, firstDayOfWeek(variables.testDate), "d")
                ).toBe(0);
            });
        });
    }

}

Notes:

• I've got the UDF saved in a separate file, so just include it at the beginning of the tests;
• I set some helper variables for using in the tests;
• I'll get to this.

I was running the tests as I went, and here's what I got on the test round with these first two tests:

TestBox, BDD-style tests, Railo member functions and bugs therein

G'day:
One of the promised features of ColdFusion 11 is to bring "member functions" to ColdFusion's inbuilt data types. Railo's already had a good go at doing this, and has reasonably good coverage. See "Member Functions" for details.

One concern I have is whether ColdFusion 11 will implement these the same way as Railo already has. I mean... they should do, there's not much wriggle room, but who knows. If there's a way to do it wrong, I'm sure Adobe can find it. With that in mind, I want to be able to run through some regression/transgression tests on both systems once ColdFusion 11 goes beta, so in prep for that, today I knocked out some unit tests for all the struct member functions.

What I did is to go into the ColdFusion 10 docs, go to the "Structure functions" section (to remind me what all the functions were), and write unit tests for the whole lot. I used the ColdFusion docs rather than the Railo ones because the CF10 docs are more comprehensive than Railo's (this is an indictment of Railo's docs, not a recommendation for the ColdFusion ones, btw!), plus I wanted to make sure I covered any functions Railo might have overlooked.

To make this more interesting (because, let's face it, it's not an interesting task; either to undertake, write up, or for you to read about!), I decided to have a look at TestBox's BDD-inspired testing syntax. The short version of this side of things is that I rather like this new syntax.  I don't think it's got anything to do with BDD (which is an approach to documentation and test design, not a syntax model), but it's interesting anyhow.

Anyway, stand-by for a raft of code (there's a Gist of this too: Struct.cfc):

Unit Testing / TDD: not testing stuff

G'day:
It's about bloody time I got back to this series on TDD and unit testing. I've already got the next few articles semi-planned in my head: topic, if not content. I have to say I am not in the most inspired writing mood today, and the words aren't exactly flowing from fingers through keyboard to screen - this is the third attempt at this paragraph - but we'll see how I go.

To find inspiration and to free up the fingers a bit, I'm at my local pub in Merlin Park in Galway, which - today - has shown its true colours as a football pub (to you Americans, that's "soccer". To me, it's "shite"). I've tried to like football, but it all just seems too effeminate to me. The place is chock-full of colours-wearing lads yelling at the screen. Either for or against one of Chelsea or Liverpool (the former lead 2-1 at half time). It's not conducive to writing, but the Guinness next to me will be.

I'll not list the previous articles in the series as it will take up too much room. They're all tagged with "TDD" and "unit testing" though.

On with the show...

CFML/TestBox: running on actual proper tests

G'day:
Various people seem to be trying out TestBox on their old MXUnit unit tests today, and reporting back on Twitter. I'm working through our ones, and have a few findings. I have more that 140 chars of information, so I'll quickly blog instead.

I can't go into too many details regarding our tests, but we have a few thousand of them spread over a number of applications.

Running through our main tranche of... ooh... 2793 (!!) tests, I had the following issues:

assertIsQuery() is missing

This is a standard MXUnit assertion (assertIsQuery()), missing from TestBox. Lack of this probably breaks about 200 of our unit tests

assertIsEmpty() is missing

So is this (assertIsEmpty()). We use this less often, but it probably breaks 50-odd tests.

mxunit:expectedException missing

It was pointed out to me the other day ("I should pay attention to what my readers tell me! expectException()") that MXUnit has a better approach to dealing with this - and one that seems is indeed implemented in TestBox - expectException(). However we have a lot of tests, and a bunch of them still use the above annotation. Perhaps 50 tests "fail" due to this not being implemented.

makePublic() doesn't seem to work

I didn't investigate this thoroughly, but it seems TestBox has implemented makePublic(), but it doesn't actually work. I say this because I get no error on code calling makePublic(), but none of the methods made public actually are public, as a result. A few hundred failures.

mock() not implemented

MXUnit has built-in mocking. So does TestBox. However each are different, and it doesn't look like MXUnit's version has been implemented. We only use this once in our tests (we generally use MockBox).

If beforeTests() errors, testing halts

This in itself is completely the same as MXUnit, but it basically discounts any chance of us using TestBox as it stands, because given addAssertDecorator() is also not implemented, this prevents me from actually doing a test run. Because the whole thing falls over as soon as we try to bring in our custom assertions. That aside, lack of being able to call in custom assertions also kills about another 500 of our tests. Fixing beforeTests() would be really nice, but if addAssertDecorator() was implemented, that'd get us moving forward.

One really annoying side effect of this is that TestBox doesn't preserve the context of where the error happened, it just reports it as happening in the bowels of TestBox itself, which is not correct. It should bubble back the original exception. It was basically impossible for me to bandaid some of the errors I was seeing because TestBox wasn't telling me where they were happening!

The method init was not found in component TestCase.cfc

This is a very edge-case one. We have a helper CFC which is not run as a test case, but we want to run some of TestCase's methods. So we instantiate an instance of TestCase of our own. In MXUnit, TestCase.cfc has an init() method. In TestBox, it does not, so it errors. This is one of those crazy edge-cases one only ever discovers testing on real world code.

Others

I was getting a few other test failures too, in tests I know work fine. I wasn't too sure if those were a byproduct of all the stuff above, so I did not investigate. Once I have a stable system, I will look at those ones more closely.

Conclusion

So those results weren't great. About a quarter of our tests would not run without being monkeyed with, and we cannot do a test run at all. Still: this is because of a few minor glitches we use in key areas of our tests.

I'm outa lunchtime, so I will need to run the other tests later, and see if I come up with anything else. But until we get the issues above sorted out, TestBox is not a possibility for us. I'll continue to use it at home, though.

--
Adam

So what did I do today? (Nothing interesting. Here are the details)

G'day:
I faffed around on the computer - with ColdFusion and Railo and TestBox - all day today, and have concluded I have nothing interesting to show for it. Still... this is a log of what I've been up to with CFML, so you're gonna hear all about it anyhow. And what's "better"... I'm gonna spin it out to be two articles. I am the Peter Jackson of CFML blogs!

OK, so I got wind of something to investigate the other day, and to do the metrics, I needed to time some stuff. Normally I'd just use a getTickCount() before and after the code, but I thought I might need something more comprehensive this time, so I figured a wee stopwatch UDF was in order. I've also just installed TestBox and am in the process of seeing if I can migrate away from MXUnit, so decided to do another TDD exercise with it. Note: there is no great exposition in this article, or really much new about TDD etc. It's simply what I did today.

Note: I abandoned TesxtBox's MXUnit compatibility mode today, because of a show-stopper bug: "ADDASSERTDECORATOR not implemented". This is the function that allows the importing of custom assertions, which I am using here. On the basis of that, I decided to just go with TestBox syntax instead.

Let's have a quick look at the function, and the tests.

//makeStopwatch.cfm
struct function makeStopwatch(){
    var timeline        = [];

    var lap = function(string message=""){
        var ticksNow    = getTickCount();
        var lapCount    = arrayLen(timeline);
        var lap            = {
            currentClock    = ticksNow,
            lapDuration        = lapCount > 0 ? ticksNow - timeLine[lapCount].currentClock : 0,
            totalDuration    = lapCount > 0 ? ticksNow - timeLine[1].currentClock : 0,
            message            = message
        };
        arrayAppend(timeline, lap);
        return lap;
    };

    return {
        start        = function(string message="start"){
            return lap(message);
        },
        lap            = function(string message="lap"){
            return lap(message);
        },
        stop        = function(string message="stop"){
            return lap(message);
        },
        getTimeline    = function(){
            return timeLine;
        }
    };
};

Not much interesting here. I enjoyed finding another reason / excuse to use function expressions and a wee bit of closure around the timeline which "sticks" with the start() / lap() / stop() / getTimeline() functions.

What this function does is to log a struct at start, each lap, and another on stop. Here's it in action:

// useStopwatch.cfm
include "makeStopwatch.cfm";
stopwatch = makeStopwatch();

stopwatch.start("Begin timing");
sleep(500);
stopwatch.lap();
sleep(1500);
secondLap = stopwatch.lap("after another 1500ms");
writeDump(var=secondLap, label="secondLap");
sleep(2000);
stopwatch.start("Stop timing");
writeDump(var=stopWatch.getTimeline());

Output:

secondLap - struct
CURRENTCLOCK	1387658080067
LAPDURATION	1500
MESSAGE	after another 1500ms
TOTALDURATION	2000

array
1	struct CURRENTCLOCK 1387658078067 LAPDURATION 0 MESSAGE Begin timing TOTALDURATION 0
2	struct CURRENTCLOCK 1387658078567 LAPDURATION 500 MESSAGE lap TOTALDURATION 500
3	struct CURRENTCLOCK 1387658080067 LAPDURATION 1500 MESSAGE after another 1500ms TOTALDURATION 2000
4	struct CURRENTCLOCK 1387658082117 LAPDURATION 2050 MESSAGE Stop timing TOTALDURATION 4050

This is the bumpf I usually need when I'm timing stuff. Well: some or all of it. It'll be quite handy, I reckon.

One interesting facet here is that initially I thought I'd need three separate functions for the start() / lap() / stop() functions, I was doing TDD (seriously, I knew you'd wonder, so everything I write on this blog uses full TDD now), and having knocked out the first few tests to verify the method signatures for the returned functions, it occurred to me that stop() was actually redundant. It doesn't do anything that lap() wouldn't already do. I mean all this "stopwatch" does is take time marks... there's no clock starting or stopping really (getTickCount() keeps on ticking, we simply start paying attention and then stop paying attention at some point).

So, anyway, I decided before I starting messing around with a redesign, I'd get the test coverage done, get it working, and then refactor. This is something one of my colleagues (Brian, I mean you) has been drumming into me recently: don't get halfway through something, decide to do it differently, start again, refactor, waste time, and not have anything to show for it if I get interrupted (or, in a work situation, we get to the end of the sprint and need to release). So I banged out the rest of the tests, got everything working, and looked at my code some more.

Here are the tests:

CFML: Follow-up to last night's article about a TestBox "bug". PEBCAK strikes again!

G'day:
Time for me to fall on my sword.

Yesterday I started looking at TestBox ("Unit Testing / TDD - switching off MXUnit, switching on TestBox"), and whilst messing around, I came across a coupla bugs (some legit, I hasten to add), one of which Luis asked me to clarify on the ColdBox mailing list. Which I did: "[Testbox v1.0.0] "Variable ISEXPECTEDEXCEPTION is undefined" on erroring test".

The gist of this is as follows:

// Tests.cfc
component {

    public void function beforeTests(){
        variables.beforeTestsRun = true;
    }

    public void function setup(){
        variables.setupRun = true;
    }

    public void function testInitFunctions(){
        assert(structKeyExists(variables, "beforeTestsRun"));
        assert(structKeyExists(variables, "setupRun"));
    }

    public void function testThatErrors(){
        throw(type="TestException", message="This is a test exception", detail="Note that it is NOT being caught / tested for. This is by design");
    }

    public void function testThatFails(){
        fail("This test should fail");
    }

}

This is a distillation of my situation, not actual code. I am testing TestBox's MXUnit compatibility "mode", so here's a test that checks that beforeTests() and setup() run, plus how it deals with an erroring test, and a failing test.

When I was running this on ColdFusion, I got this output:

TestBox v1.0.0.00062

 

Global Stats (86 ms)

[ Bundles/Suites/Specs: 1/1/3 ] [ Pass: 1 ] [ Failures: 0 ] [ Errors: 2 ] [ Skipped: 0 ] [ Reset ] 

shared.git.blogExamples.unittests.testbox.Tests (18 ms)

[ Suites/Specs: 1/3 ] [ Pass: 1 ] [ Failures: 0 ] [ Errors: 2 ] [ Skipped: 0 ] [ Reset ]

• +shared.git.blogExamples.unittests.testbox.Tests (18 ms)


testInitFunctions (2 ms)


testThatErrors (10 ms) - Variable ISEXPECTEDEXCEPTION is undefined. 


testThatFails (4 ms) - Variable ISEXPECTEDEXCEPTION is undefined. 

Note how the erroring and failing tests were reporting as erroring, but with an error internal to TestBox, not with the expected output MXUnit would give. So basically when a test failed or errored, TestBox itself was failing.

At this point... can you see what I've done wrong here?

Here's the thing. For TestBox test CFCs, there is no need to extend anything. I guess TestBox just uses mix-ins to insert all its bits and pieces. So my CFC doesn't extend anything.

However with MXUnit, one does need to extend mxunit.framework.TestCase.

Obviously on all my "normal" MXUnit test CFCs I do have the correct inheritance in place. But this particular test CFC started out with me testing TestBox tests, not MXUnit tests, so I neglected to put the inheritance in when I switched over to running it with MXUnit compatibility "mode". My bad.

Once I sorted that out, my tests run fine:

TestBox v1.0.0.00062

 

Global Stats (82 ms)

[ Bundles/Suites/Specs: 1/1/3 ] [ Pass: 1 ] [ Failures: 1 ] [ Errors: 1 ] [ Skipped: 0 ] [ Reset ] 

shared.git.blogExamples.unittests.testbox.Tests (15 ms)

[ Suites/Specs: 1/3 ] [ Pass: 1 ] [ Failures: 1 ] [ Errors: 1 ] [ Skipped: 0 ] [ Reset ]

• +shared.git.blogExamples.unittests.testbox.Tests (15 ms)


testThatErrors (8 ms) - This is a test exception 


testThatFails (4 ms) - This test should fail 


testInitFunctions (1 ms)

I fed this back to Luis, so that's case closed on that one. I also editorialised a bit, saying this:

It's a pity your test case CFCs don't need to be of a certain type, as that would have clarified that error from the outset. It would also help mitigate a failing MXUnit always had in my opinion: that it determines whether a CFC is a test CFC by its NAME rather than its TYPE (eg: it's a subclass of TestCase). This name-based approach isn't great OO, especially if the capability to do it "properly" is right there.

This has always annoyed me about MXUnit. And it'll annoy me about TestBox. It doesn't annoy me much, but that MXUnit always relied on a file-naming scheme to identify test CFCs always seemed a bit amateurish to me. [gallic shrug]... this is a very minor gripe, and I do not forget that MXUnit has still been one of the most useful CFML tools out there, and TestBox is almost certainly gonna be an appropriate successor.

And Luis... very awesome you followed up my mailing list question so promptly. Much appreciated. I need to venture out into an uncharacteristically windy London for coffee and food, but I shall be writing more tests today. Let's see how it goes...

--
Adam

Unit Testing / TDD - switching off MXUnit, switching on TestBox

G'day:
This article is more an infrastructure discussion, rather than examining more actual testing stuff. The ever-growing *Box empire has recently borged into yet another part of the CFML community: testing. They're released another box... TestBox. TestBox is interesting to me as it has a different approach to testing than MXUnit has... rather than xUnit style assertion-based testing, instead favouring a BDD approach. I've not done a lick of BDD, but people keep banging on about it, so I shall be looking at it soon. -ish. First I need to switch to TestBox.

One appealing thing I had heard about TestBox is that it's backwards compatible with MXUnit, so this should mean that I can just do the switch and continue with my current approach to testing, and ease my way into BDD as I learn more about it. So the first thing I decided to examine is how well this stands up, and how many changes I need to make to my existing tests to get them to run. Realistically, nothing is every completely backwards compatible... not even say between different versions of the same software (ColdFusion 9 to ColdFusion 10), let along a second system emulating another system (eg: Railo and ColdFusion). This is fine. I don't expect this migration to be seamless.

Here's what I worked through this morning to get up and running (spoilers: kinda running) on TestBox.

I preface this with the fact that I have always found Ortus's documentation to be a bit impenetrable (there's too much of it, it waffles too much), so I was hesitant about how long this would all take.

Locating, downloading and installing

Finding it

I googled "testbox", and the first link was the ColdBox Platform Wiki - TestBox. This is promising. Within a paragraph (and a to-the-point paragraph which just intros the product, so maybe the docs have got some improved focus: cool) there were download links. TestBox requires ColdFusion 10 / Railo 4.1, btw. I presume it uses closure or something? I'm not sure. But that's cool, I use CF10 and Railo [latest] for my work for this blog. It does preclude me from really giving it a test our with our 3000 unit tests at work though (which is a shame), because we're still on CF9 and will be for the foreseeable future.

Installing it

The installation instructions threw me a bit. The default suggestion is to put the testbox dir into the web root, but that's poor advice: only files specifically intended to be web browseable should ever be in your web root. Fortunately the also mention one can stick 'em anywhere, and map them in with a /testbox mapping. I wish this was their default suggestion. In fact I wish it was their only suggestion. They should not encourage poor practice.

There's a caveat with this though (and this is where I had problems), is that TestBox does have some web assets which need to be web browseable, so it does actually need a web mapping, not just a CF mapping. They do caveat this further down the page.

The first pitfall I had was which directory they're actually talking about. The zipfile has this baseline structure:

/testbox_1.0.0/
    testbox-1.0.0.00062-201312171237
    apidocs/
    browser/
    runner/
    runner-template/
    samples/
    testbox/
    license.txt
    mockbox.txt
    testbox.txt

So I homed this lot in my CF root (not web root, CF root) as /frameworks/testbox/1.0.0/, and added a /testbox CF mapping to that location.

WARNING (if you're following along and doing this at the same time): this is not the correct thing to do. Keep reading...

I then had a look around for which directory I needed to add a web server virtual directory for, and found web-servable assets in the following locations:

/apidocs/
/browser/
/runner/
/samples/
/testbox/system/testing/reports/assets/

(I searched for images, JS, CSS, HTML and index.cfm files; not perfect, but will give me an idea).

OK, so I figured he apidocs and samples are separate from the TestBox app, but that still leaves three disconnected (and laterally displaced) directories which need to be web browseable. This ain't great. So basically it looks like I need to make the entire /testbox dir web browseable. That's a bit shit, and a bit how we might have set up our CFML-driven websites... ten years ago. Oh well.

Configuring Tomcat

Here's a challenge (cue: Sean to get grumpy). I have no idea how to set up a virtual directory on Tomcat's built-in web server. Fortunately that's what Google is for, so I googled "tomcat web server virtual directories", and the very first link is a ColdFusion-10-specific document: "Getting Started with Tomcat in ColdFusion 10". I shuddered slightly that this is just in the ColdFusion Blog, rather than in the CF docs where it belongs, but it'll do. Fortunately the info in there is accurate, which is good.

Basically there's a file server.xml located at <ColdFusion_Home>/runtime/conf/server.xml, where <ColdFusion_Home> is the cfusion dir in your ColdFusion install directory. For me the conf dir is at: C:\apps\adobe\ColdFusion\10\cfusion\runtime\conf.

In there there's an XML note like this:

<Context
    path    = "/"
    docBase    = "<cf_home>\wwwroot"
    WorkDir    = "<cf_home>\runtime\conf\Catalina\localhost\tmp"
>
</Context>

It's commented out by default. All the instructions one needs are in the file itself, but basically it's uncomment it, put actual paths in, and add an aliases attribute:

<Context
    path    = "/"
    docBase    = "C:\apps\adobe\ColdFusion\10\cfusion\wwwroot"
    WorkDir    = "C:\apps\adobe\ColdFusion\10\cfusion\runtime\conf\Catalina\localhost\tmp"
    aliases    = "/testbox=C:\webroots\frameworks\testbox\1.0.0"
>
</Context>

I restarted CF and browsed to http://localhost:8500/testbox, and I got the files in my C:\webroots\frameworks\testbox\1.0.0 directory listing, so that worked. Good to know. I'll now forget about server.xml and aliases and stuff as I won't need to do it again for another six months. Shrug.

ColdFusion config

I put a mapping to the same place in my test app's Application.cfc:

// Application.cfc
component {

    this.mappings            = {
        "/cflib"    = getDirectoryFromPath(getCurrentTemplatePath()),
        "/testbox"    = expandPath("/testbox") // CF will use the virtual directory to resolve that. This is slightly cheating, but hey
    };

}

Two things: I'm thick; and one cannot use a Mockbox mocked method as a callback

G'day:
And to think someone said on Twitter the other day that I'm one of those people who are never wrong (the subtext being "actually you are, but just will never admit it"). Well here's an example of me being wrong, happily admitting it, and admitting to being a bit stupid to boot.

Weird issue with MockBox and interfaces: issue identified

G'day:
A coupla days back I wrote a vague article "Weird issue with Mockbox and interfaces", which detailed some weirdness I was seeing with using Mockbox to mock methods in CFCs which implement an interface.  The short version is that on the first use of a method mocked this way, I was getting this error:

coldfusion.runtime.InterfaceRuntimeExceptions$ArgumentsMistmatchException: Function argument mismatch.[etc]

However if one just re-ran the test, the problem went away. In fact it was only the first usage of the mocked method after ColdFusion was started that caused the problem. Subsequent runs: all good. Restart CF: problem. This is on ColdFusion 9 & 10, but not Railo.

Weird issue with Mockbox and interfaces

G'day:
This is not gonna be a very well-realised article, as it's posing a question that I've not really been able to flesh out yet.

Repro case for "contains" pseudo-reserved-word interfering with Mockbox

G'day:
This is mostly for Brad Wood, but it might be of passable interest to others, so I'll plonk it here.

Yesterday's article discussed how contains is kind of a reserved word, but kind of not in ColdFusion (it's just not in Railo). I observed to Brad that this actually bites us on the bum with Mockbox, and he asked for more info, so here it is.

Basically we use a caching system which has a method "contains" which checks to see if there's an item with a given key already in the cache, before trying to fetch it. We've actually since revised this approach, but we have some legacy code still using methods called "contains". So we need to unit test them, and indeed their responses play a part in other methods we test. When testing these other methods which use the caching system, we mock-out the cache, and the methods within it, and we use Mockbox to do this. Mockbox is cool, btw. You should be using it if yer not already.

We're still using Mockbox 1.3 (more about why further down), and it is impossible to directly mock a method called contains using Mockbox. We've worked around this, but it took a bloody long time to work out what the hell was going on, and that there was working-around to do.