Various people seem to be trying out TestBox on their old MXUnit unit tests today, and reporting back on Twitter. I'm working through our ones, and have a few findings. I have more that 140 chars of information, so I'll quickly blog instead.
I can't go into too many details regarding our tests, but we have a few thousand of them spread over a number of applications.
Running through our main tranche of... ooh... 2793 (!!) tests, I had the following issues:
assertIsQuery()), missing from TestBox. Lack of this probably breaks about 200 of our unit tests
assertIsEmpty()). We use this less often, but it probably breaks 50-odd tests.
I should pay attention to what my readers tell me! expectException()") that MXUnit has a better approach to dealing with this - and one that seems is indeed implemented in TestBox - expectException(). However we have a lot of tests, and a bunch of them still use the above annotation. Perhaps 50 tests "fail" due to this not being implemented.
makePublic(), but it doesn't actually work. I say this because I get no error on code calling makePublic(), but none of the methods made public actually are public, as a result. A few hundred failures.
mocking. So does TestBox. However each are different, and it doesn't look like MXUnit's version has been implemented. We only use this once in our tests (we generally use MockBox).
addAssertDecorator() is also not implemented, this prevents me from actually doing a test run. Because the whole thing falls over as soon as we try to bring in our custom assertions. That aside, lack of being able to call in custom assertions also kills about another 500 of our tests. Fixing beforeTests() would be really nice, but if addAssertDecorator() was implemented, that'd get us moving forward.
One really annoying side effect of this is that TestBox doesn't preserve the context of where the error happened, it just reports it as happening in the bowels of TestBox itself, which is not correct. It should bubble back the original exception. It was basically impossible for me to bandaid some of the errors I was seeing because TestBox wasn't telling me where they were happening!
init() method. In TestBox, it does not, so it errors. This is one of those crazy edge-cases one only ever discovers testing on real world code.
OthersI was getting a few other test failures too, in tests I know work fine. I wasn't too sure if those were a byproduct of all the stuff above, so I did not investigate. Once I have a stable system, I will look at those ones more closely.
ConclusionSo those results weren't great. About a quarter of our tests would not run without being monkeyed with, and we cannot do a test run at all. Still: this is because of a few minor glitches we use in key areas of our tests.
I'm outa lunchtime, so I will need to run the other tests later, and see if I come up with anything else. But until we get the issues above sorted out, TestBox is not a possibility for us. I'll continue to use it at home, though.