This article is more an infrastructure discussion, rather than examining more actual testing stuff. The ever-growing *Box empire has recently borged into yet another part of the CFML community: testing. They're released another box... TestBox. TestBox is interesting to me as it has a different approach to testing than MXUnit has... rather than xUnit style assertion-based testing, instead favouring a BDD approach. I've not done a lick of BDD, but people keep banging on about it, so I shall be looking at it soon. -ish. First I need to switch to TestBox.
One appealing thing I had heard about TestBox is that it's backwards compatible with MXUnit, so this should mean that I can just do the switch and continue with my current approach to testing, and ease my way into BDD as I learn more about it. So the first thing I decided to examine is how well this stands up, and how many changes I need to make to my existing tests to get them to run. Realistically, nothing is every completely backwards compatible... not even say between different versions of the same software (ColdFusion 9 to ColdFusion 10), let along a second system emulating another system (eg: Railo and ColdFusion). This is fine. I don't expect this migration to be seamless.
Here's what I worked through this morning to get up and running (spoilers: kinda running) on TestBox.
I preface this with the fact that I have always found Ortus's documentation to be a bit impenetrable (there's too much of it, it waffles too much), so I was hesitant about how long this would all take.
Locating, downloading and installing
Finding it
I googled "testbox", and the first link was the ColdBox Platform Wiki - TestBox. This is promising. Within a paragraph (and a to-the-point paragraph which just intros the product, so maybe the docs have got some improved focus: cool) there were download links. TestBox requires ColdFusion 10 / Railo 4.1, btw. I presume it uses closure or something? I'm not sure. But that's cool, I use CF10 and Railo [latest] for my work for this blog. It does preclude me from really giving it a test our with our 3000 unit tests at work though (which is a shame), because we're still on CF9 and will be for the foreseeable future.Installing it
The installation instructions threw me a bit. The default suggestion is to put the testbox dir into the web root, but that's poor advice: only files specifically intended to be web browseable should ever be in your web root. Fortunately the also mention one can stick 'em anywhere, and map them in with a /testbox mapping. I wish this was their default suggestion. In fact I wish it was their only suggestion. They should not encourage poor practice.There's a caveat with this though (and this is where I had problems), is that TestBox does have some web assets which need to be web browseable, so it does actually need a web mapping, not just a CF mapping. They do caveat this further down the page.
The first pitfall I had was which directory they're actually talking about. The zipfile has this baseline structure:
/testbox_1.0.0/
testbox-1.0.0.00062-201312171237
apidocs/
browser/
runner/
runner-template/
samples/
testbox/
license.txt
mockbox.txt
testbox.txt
So I homed this lot in my CF root (not web root, CF root) as /frameworks/testbox/1.0.0/, and added a /testbox CF mapping to that location.
WARNING (if you're following along and doing this at the same time): this is not the correct thing to do. Keep reading...
I then had a look around for which directory I needed to add a web server virtual directory for, and found web-servable assets in the following locations:
/apidocs/
/browser/
/runner/
/samples/
/testbox/system/testing/reports/assets/
(I searched for images, JS, CSS, HTML and index.cfm files; not perfect, but will give me an idea).
OK, so I figured he apidocs and samples are separate from the TestBox app, but that still leaves three disconnected (and laterally displaced) directories which need to be web browseable. This ain't great. So basically it looks like I need to make the entire /testbox dir web browseable. That's a bit shit, and a bit how we might have set up our CFML-driven websites... ten years ago. Oh well.
Configuring Tomcat
Here's a challenge (cue: Sean to get grumpy). I have no idea how to set up a virtual directory on Tomcat's built-in web server. Fortunately that's what Google is for, so I googled "tomcat web server virtual directories", and the very first link is a ColdFusion-10-specific document: "Getting Started with Tomcat in ColdFusion 10". I shuddered slightly that this is just in the ColdFusion Blog, rather than in the CF docs where it belongs, but it'll do. Fortunately the info in there is accurate, which is good.Basically there's a file server.xml located at <ColdFusion_Home>/runtime/conf/server.xml, where <ColdFusion_Home> is the cfusion dir in your ColdFusion install directory. For me the conf dir is at: C:\apps\adobe\ColdFusion\10\cfusion\runtime\conf.
In there there's an XML note like this:
<Context
path = "/"
docBase = "<cf_home>\wwwroot"
WorkDir = "<cf_home>\runtime\conf\Catalina\localhost\tmp"
>
</Context>
It's commented out by default. All the instructions one needs are in the file itself, but basically it's uncomment it, put actual paths in, and add an aliases attribute:
<Context
path = "/"
docBase = "C:\apps\adobe\ColdFusion\10\cfusion\wwwroot"
WorkDir = "C:\apps\adobe\ColdFusion\10\cfusion\runtime\conf\Catalina\localhost\tmp"
aliases = "/testbox=C:\webroots\frameworks\testbox\1.0.0"
>
</Context>
I restarted CF and browsed to http://localhost:8500/testbox, and I got the files in my C:\webroots\frameworks\testbox\1.0.0 directory listing, so that worked. Good to know. I'll now forget about server.xml and aliases and stuff as I won't need to do it again for another six months. Shrug.
ColdFusion config
I put a mapping to the same place in my test app's Application.cfc:// Application.cfc
component {
this.mappings = {
"/cflib" = getDirectoryFromPath(getCurrentTemplatePath()),
"/testbox" = expandPath("/testbox") // CF will use the virtual directory to resolve that. This is slightly cheating, but hey
};
}
Running an xUnit test
Trying... and failing...
The next thing I decided to do was to ignore BDD stuff, and just check how well my xUnit tests would run under TestBox. I skimmed down the TestBox wiki page, and there was a link "TestBox xUnit Primer" which looked the business, so I followed that.(I did not, somehow, spot the heading saying "MXUnit Compatibility" immediately below that. Had I seen that I could have been going more quickly, but so be it!).
Ortus have (this is an actual quote from the docs, not my wording) "prepared an awesome PDF ref card for working with TestBox xUnit-style". Jesus. Self-congratulation isn't a great look, chaps: don't tell people you're awesome. Just strive to be awesome and let other people decide how awesome you actually are for themselves. But their docs tend to be a bit like that, so I'll ignore their narcissism. The PDF is quite good though. It certainly got me heading in the right direction enough to keep moving forward (I didn't have such a good experience with the WireBox one, but that's going to be the subject of a different article).
Basically, I came up with this test CFC:
// TestMakeStopwatch.cfc
component {
public void function beforeTests(){
variables.beforeTestsRan = true;
}
public void function setup(){
variables.setupRan = true;
}
public void function testInitialisation(){
assert(structKeyExists(variables, "beforeTestsRan"), "beforeTestsRan not set");
assert(structKeyExists(variables, "setupRan"), "setupRan not set");
}
}
The "stopwatch" reference is something I'm doing for another blog article. I have a few balls in the air at the moment.
This is all very familiar. I've got beforeTests(), setup() and assert()... all exactly how my previous test CFCs have been written. This code runs fine on MXUnit.
The ref card didn't explain how to run these tests in a browser... I was looking for the code equivalent to how I'd run them in MXUnit, eg:
<!--- runTests.cfm --->
<cfset thisDirectory = getDirectoryFromPath(getCurrentTemplatePath())>
<cfoutput>
#new mxunit.runner.DirectoryTestSuite().run(
directory = thisDirectory,
componentPath = listLast(thisDirectory, "/\")
).getResultsOutput("html")#
</cfoutput>
Which is what I'd do in MXUnit.
I flicked back to the web-based docs, and found this code example:
<!--- runTestsViaTestBox.cfm --->
<cfoutput>
#new testbox.system.testing.TestBox(
bundles="TestMakeStopwatch"
).run(
reporter="simple"
)#
</cfoutput>
And running that I got...
... an error:
| The following information is meant for the website developer for debugging purposes. | |
| Error Occurred While Processing Request | |
|
And fair enough... looking at the code, it's trying find a CFC testbox.system.testing, and there's no such file in the directory I've mapped to /testbox (ColdFusion mapping, remember; not virtual directory. Despite the fact they're pointing to the same place). Hmmm. I was one directory out: I'm mapping /testbox to C:\webroots\frameworks\testbox\1.0.0 (which is what the instructions seem to say), but I should have mapped it to C:\webroots\frameworks\testbox\1.0.0\testbox. But this gives me another potential problem: the CF mapping /testbox needs to point to one place, but the virtual directory needs to stay pointing where it currently is: a different place. I tried to get this working, but CF (and I!) got very confused, so I decided life was too short and I don't care about the pretty-pretty CSS stuff and images and shit, so I'd go without the virtual directory initially, and see if I can get my tests running, even if not looking great.
And actually getting it to work
So I reverted my change to server.xml, and re-did my /testbox mapping in Application.cfc to point at the right place. And restarted CF. This time when I ran the test file, I got better results:
TestBox v1.0.0.00062
Global Stats (7676 ms)
[ Bundles/Suites/Specs: 1/1/1 ] [ Pass: 1 ] [ Failures: 0 ] [ Errors: 0 ] [ Skipped: 0 ] [ Reset ]cflib.makeStopwatch.TestMakeStopwatch (8 ms)
[ Suites/Specs: 1/1 ] [ Pass: 1 ] [ Failures: 0 ] [ Errors: 0 ] [ Skipped: 0 ] [ Reset ]Woohoo!
So that's cool. It looks like the MXUnit syntax just works. And then I noticed the "MXUnit Compatiblity" instructions that I should have followed in the first place. I hasten to add that not spotting these was completely my own fault. I've actually found the TestBox docs to be pretty helpful so far.
MXUnit Compatibility
Had I RTFMed properly, I would have noticed this:TestBox is fully complaint with MXUnit xUnit test cases (if not, let us know). In order to leverage it you will need to create or override the /mxunit mapping and make it point to the /testbox/system/testing/compat folder. That's it, everything should continue to work as expected.Sigh.
So to test this I got rid of my /testbox mapping, and just did what I was supposed to do: revise my /mxunit mapping. This wasn't quite right (and the docs are open to interpretation here), because it's not solely that I need to change the /mxunit mapping, I did actually still need the /testbox one too. This is no surprise, but I think the docs are - as I say - not quite clear there.
So that I am being 100% clear, my Application.cfc now says this:
// Application.cfc
component {
this.mappings = {
"/cflib" = getDirectoryFromPath(getCurrentTemplatePath()),
"/testbox" = expandPath("/frameworks/testbox/1.0.0/testbox"),
"/mxunit" = expandPath("/frameworks/testbox/1.0.0/testbox/system/testing/compat")
};
}
And my normal runTests.cfm file for MXUnit looks like this:
<!--- runTests.cfm --->
<cfset thisDirectory = getDirectoryFromPath(getCurrentTemplatePath())>
<cfoutput>
#new mxunit.runner.DirectoryTestSuite().run(
directory = thisDirectory,
componentPath = "cflib." & listLast(thisDirectory, "/\")
).getResultsOutput("html")#
</cfoutput>
When I run this via TestBox I get...
... an error:
| The following information is meant for the website developer for debugging purposes. | |
| Error Occurred While Processing Request | |
|
I deduced this was from the argument value passed to getResultOutput(), so changed that to the one the TestBox version of the test file used: "simple". Now I rerun the code and get better results:
TestBox v1.0.0.00062
Global Stats (21 ms)
[ Bundles/Suites/Specs: 1/1/1 ] [ Pass: 1 ] [ Failures: 0 ] [ Errors: 0 ] [ Skipped: 0 ] [ Reset ]cflib.makeStopwatch.TestMakeStopwatch (3 ms)
[ Suites/Specs: 1/1 ] [ Pass: 1 ] [ Failures: 0 ] [ Errors: 0 ] [ Skipped: 0 ] [ Reset ]Cool! I'm in!
However this was a contrived test set up specifically for this blog article. What about some of my older tests? I adjusted the code for my createLocalisedDayOfWeekAsInteger() (which can be found on GitHub here: https://github.com/daccfml/scratch/tree/master/cflib/createLocalisedDayOfWeekAsInteger) so that it used "simple" rather than "html" (and otherwise didnt change anything else) and ran the tests.
This demonstrates a bit of an glitch in TestBox's MXUnit compat:
TestBox v1.0.0.00062
Global Stats (36 ms)
[ Bundles/Suites/Specs: 2/2/0 ] [ Pass: 0 ] [ Failures: 0 ] [ Errors: 0 ] [ Skipped: 0 ] [ Reset ]createLocalisedDayOfWeekAsInteger.TestCreateLocalisedDayOfWeekAsInteger (2 ms)
[ Suites/Specs: 1/0 ] [ Pass: 0 ] [ Failures: 0 ] [ Errors: 0 ] [ Skipped: 0 ] [ Reset ]Note how it's looked for tests in two CFCs - Application.cfc and the actual test one, but not actually found any tests. MXUnit has two rules for locating tests which TestBox is getting reversed here:
- Test CFCs need to have the word "test" in their file name. Hence "TestCreateLocalisedDayOfWeekAsInteger.cfc". TestBox is looking for tests in just any old CFC.
- Test functions can be called anything. They just need to be public. However TestBox only runs functions which have the word test in them. My tests don't. Because in a CFC specifically intended for testing, it'd be tautological to also include the word "test" in the function name.
TestBox v1.0.0.00062
Global Stats (103 ms)
[ Bundles/Suites/Specs: 2/2/11 ] [ Pass: 11 ] [ Failures: 0 ] [ Errors: 0 ] [ Skipped: 0 ] [ Reset ]createLocalisedDayOfWeekAsInteger.TestCreateLocalisedDayOfWeekAsInteger (64 ms)
[ Suites/Specs: 1/11 ] [ Pass: 11 ] [ Failures: 0 ] [ Errors: 0 ] [ Skipped: 0 ] [ Reset ]That's better! I'm not exactly pushing the boat out with these tests, but it's an initial proof of concept anyhow.
Conclusion
I'm gonna rename my tests, and switch to using TestBox for my testing now. There's a coupla incompats with MXUnit, but nothing I'm that bothered about. I'm more bothered by the continued notion that it's an OK approach to structuring CFML apps so randomly that they need their entire codebase web-browseable, but it's not like Ortus are the only people who do this. And this is just a dev box, so my annoyance is purely one of principle here... it's not gonna stop me using the software.I know Brad and possibly Luis will be reading this, so I'll await their feedback on the coupla hiccups I had along the way.
All in all: nice work, lads. I am looking forward to moving onto the BDD part of this series, but I have a coupla more general TDD things to look at first.
But first... lunch. And a completely different blog article next...
--
Adam
