G'day:
In the rest of my Kotlin articles thusfar I've been using Kotest for my testing. I went that direction instead of JUnit as the mindset behind xUnit style tests always seems to steer one towards writing tests of the code, rather than writing test cases for features. I think the RSpec-style better helps one focus on features when describing what the test case implementation will be when one starts with a descriptive statement rather than a method name. Plus the test output is a lot more user-friendly seeing the test case descriptions rather than test method names. Another area that RSpec-style beats xUnit hands-down when it comes to code organisation, given one can categorise one's tests within a test file via nested describe blocks. Lastly I think the "expectation" style of testing values - eg: expect(actual).toBe(expected) or actual shouldBe expected - is easier to read and more logical than assertEquals(expected, actual). That assertEquals approach is how Yoda would write an expectation, not a human.
However JUnit is the baseline testing framework for Java (and by extension Kotlin) so I'm gonna have a look at it to see how it shapes up. I've got some confirmation bias going on with my opinions here, I freely admit this. But I should sideline those when making the call as to which testing framework we should use for my day job.
First test / config
I'm following along "Test code using JUnit in JVM – tutorial", although my target here is to re-implement the Kotest tests I have so far in JUnit. But I figure there's be some config stuff I'll need to do upfront, and the tutorial will guide me through that.
There's a dependency and a task to add in my build.gradle.kts file, but I already have those in place via my Kotest installation:
// ...
dependencies {
testImplementation(kotlin("test"))
testImplementation("io.kotest:kotest-runner-junit5:5.4.2")
testImplementation("io.kotest:kotest-assertions-core:5.4.2")
testImplementation("com.github.stefanbirkner:system-lambda:1.2.1")
testImplementation("org.apache.commons:commons-lang3:3.12.0")
}
tasks.withType<Test>().configureEach {
useJUnitPlatform()
}
(I've just noticed that Kotest uses the JUnit5 test runner already).
There's some JUnit support baked-in to IntelliJ (of course there is), so to create a test, apparently all I need to do is to select a method › right-click › Generate › Test… and IntelliJ will sort it out for me:
Apparently I have to install that JUnit5 library… I wonder how that differs from the dependency I have in the build file already? Let's press it and find out…
… it's also added these two dependencies now:
dependencies {
testImplementation(kotlin("test"))
testImplementation("io.kotest:kotest-runner-junit5:5.4.2")
testImplementation("io.kotest:kotest-assertions-core:5.4.2")
testImplementation("com.github.stefanbirkner:system-lambda:1.2.1")
testImplementation("org.apache.commons:commons-lang3:3.12.0")
testImplementation("org.junit.jupiter:junit-jupiter:5.8.1")
testImplementation("org.junit.jupiter:junit-jupiter:5.8.1")
}
Looks like that's just JUnit5 internal housekeeping. Okey doke.
OK, I've renamed the suggested test class from MainKtTest to MainTest, and told it to go in the junit package in my app (so in src/test/kotlin/junit/MainTest.kt). Letting this run lands me with this skeleton:
package junit
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.Assertions.*
internal class MainTest {
@Test
fun main() {
}
}
It got the name of the test method wrong, but that's an easy fix (ie: it should be testMain) I sling a failing test into that to make sure the runner does it's thing properly:
@Test
fun testMain() {
assertTrue(false)
}
org.opentest4j.AssertionFailedError: expected: <true> but was: <false> at app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55) at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:40) at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:35) at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:179) at app//junit.MainTest.testMain(MainTest.kt:11)
All good. I wonder if I can right-click the top level of the test dir and run all tests: Kotest and JUnit ones:
⨯ Test Results ✓ PropertiesTest ✓ kotest.MainTest ✓ Tests of Main class ✓ outputs G'day World & its args ✓ works OK with no args ✓ kotest.language.ScopeFunctionsTest ✓ kotest.language.classes.AbstractTest ✓ kotest.language.classes.ClassesTest ✓ kotest.language.classes.InheritanceTest ✓ kotest.language.collections.CollectionTest ✓ kotest.language.functions.FunctionSyntaxTest ✓ kotest.language.properties.BackingPropertiesTest ✓ kotest.language.properties.LateInitPropertiesTest ✓ kotest.language.types.FunctionsTest ✓ kotest.language.types.NumberTest ✓ kotest.language.types.strings.StringTest ✓ kotest.language.variables.VariablesTest ✓ kotest.system.SystemTest ⊘ kotest.system.kotest.ConfigTest ✓ kotest.system.kotest.KotestTest ✓ kotest.system.kotest.MatcherTest ⨯ MainTest ⨯ testMain()
Cool! Yes it will.
OK, now to re-implement those two tests I need on main:
package junit
import com.github.stefanbirkner.systemlambda.SystemLambda
import main
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.Assertions.*
import org.junit.jupiter.api.DisplayName
@DisplayName("Tests of Main class")
internal class MainTest {
val EOL = System.lineSeparator()
@Test
@DisplayName("it outputs G'day World & its args")
fun testMainOutputsCorrectly() {
val testArgs = arrayOf("some arg", "some other arg")
val output = SystemLambda.tapSystemOut {
main(testArgs)
}
assertEquals("G'day World!${EOL}Program arguments: some arg, some other arg", output.trim())
}
@Test
fun `works OK with no args`() {
val testArgs = arrayOf<String>()
val output = SystemLambda.tapSystemOut {
main(testArgs)
}
assertEquals("G'day World!${EOL}Program arguments:", output.trim())
}
}
I'm pleased with this, and a couple of my confirmation-bias-based concerns have been mitigated already:
- That @DisplayName annotation allows the descriptive label on the test class and test function.
- And - bravo Kotlin - one can even use a string literal for a function name, using back ticks as delimiters.
When I run these tests, the output is pretty much on-point for what I want to see:
✓ Test Results ✓ Tests of Main class ✓ it outputs G'day World & its args ✓ it works OK with no args()
Second challenge solved: hierarchical tests
In my intro, one of the challenges I predicted xUnit tests would have compared to RSpec style ones is the lack of hierarchical organisation in xUnit tests. I have become used to this sort of thing (implementation code elided for brevity):
class SystemTest : DescribeSpec({
describe("Tests of kotest installation") {
it("should return the size of a string") {
// …
}
it("should test for the prefix of a string") {
// …
}
}
describe("tests of system-lambda") {
it("checks system-lambda is working OK") {
// …
}
}
})
Here I have two distinct sets of tests in the one test class. This aids code organisation, as well as making the output easier to deal with:
✓ Test Results ✓ kotest.system.SystemTest ✓ Tests of kotest installation ✓ should return the size of a string ✓ should test for the prefix of a string ✓ tests of system-lambda ✓ checks system-lambda is working OK
Last time I checked xUnit, this wasn't a thing. But I'm pleased to discover it is a thing with JUnit5:
class SystemTest {
@Nested
@DisplayName("Tests of JUnit installation")
inner class JUnitInstallationTest {
@Test
fun `the length of the string should be 5`() {
assertEquals(5, "hello".length)
}
@Test
fun `the string should start with "wor"`() {
assertTrue("world".startsWith("wor"))
}
}
@Nested
@DisplayName("Tests of system-lambda")
inner class SystemLambdaTest {
@Test
fun `system-lambda should be able to capture stdout`() {
val testString = "string to capture"
val output = SystemLambda.tapSystemOut {
print(testString)
}
assertEquals(testString, output)
}
}
}
✓ Test Results ✓ SystemTest ✓ Tests of JUnit installation ✓ the length of the string should be 5() ✓ the string should start with "wor"() ✓ Tests of system-lambda ✓ system-lambda should be able to capture stdout()
This is all down to that @nested annotation, and nesting classes.
There's actually a win specifically for JUnit here too. Using the it syntax with RSpec-style tests, the pedant in me always tries to make a grammatically-correct sentence out of the function call, eg:
it("is a grammatically correct sentence that describes the test case", () => {}))
This often works well. But sometimes it's just daft to wangle a correct-sounding test description starting with "it". As the JUnit descriptions or back-ticked method names have no constraint that the case description "must" start with it, I can actually use better test case labels here. Cool. This is minor, and down to me being a pedant I know. But still: I like it.
One thing to note about that though: I get a warning on the `the string should start with "wor"` test:
So I better keep an eye on that. The test did run though (on both Windows and Linux).
Third challenge: conditional tests
Kotest can do this sort of thing:
it ("only runs on linux").config(enabled = SystemUtils.IS_OS_LINUX) {
SystemUtils.IS_OS_LINUX shouldBe true
}
That test only runs when the OS is Linux. I needed to do this as I had a test that needed to deal with OS-specific line-endings (Linux: LF; Windows: CRLF). How to do this in JUnit? Excuse me whilst I google etc.…
[five minutes pass]
…OK, it's pretty easy, just use @EnabledOnOs:
@Test
@DisplayName("The test is only executed on Linux")
@EnabledOnOs(OS.LINUX)
fun testOnLinux() {
assertTrue(SystemUtils.IS_OS_LINUX)
}
On Linux:
On Windows:
Random things
There's a few wee things that don't warrant much discussion, but they're stuff I tested in Kotest so wanna make sure I can do the same thing with JUnit.
fail and assertThrows<AssertionError>
@Test
fun `It has a fail method and its exception can be caught and asserted against`() {
val exception = assertThrows<AssertionError> {
fail("This is a failure")
}
assertEquals("This is a failure", exception.message)
}
Messages on assertions
These are done with clues in Kotest:
withClue("this is the clue") {
fail("this is a failure message")
}
And with the message argument on the assertion with JUnit:
fun `A message can be supplied with the assertion to make failures clearer`() {
val exception = assertThrows<AssertionError> {
assertEquals(1, 2, "the two numbers should be equal")
}
assertEquals("the two numbers should be equal ==> expected: <1> but was: <2>", exception.message)
}
@TestFactory
I had a brain fart with one of my Kotest tests and was gonna test one thing, but then tested something different. So let's not worry about the Kotest version of this. But what I was wanting to do is to pass a list of inputs to a test and have the test run for each of them. Easy with JUnit:
@TestFactory
fun testPrimeFactorsOf210() = listOf(2, 3, 5, 7).map {
DynamicTest.dynamicTest("Dividing by $it") {
assertEquals(0, 210 % it)
}
}
✓ testPrimeFactorsOf210() ✓ Dividing by 2 ✓ Dividing by 3 ✓ Dividing by 5 ✓ Dividing by 7
Grouping multiple assertions with assertAll
This is the equivalent of Kotest's Soft assertions:
describe("Soft assertions") {
it("can use soft assertions to let multiple assertions fail and report back on all of them") {
val actual :Int = 15
try {
assertSoftly {
actual.shouldBeTypeOf<Number>()
withClue("should be 15.0") {
actual.equals(15.0).shouldBeTrue()
}
actual shouldBe 15
actual shouldBe 16
}
} catch (e :MultiAssertionError) {
assertSoftly(e.message) {
shouldContain("1) 15 should be of type kotlin.Number")
shouldContain("2) should be 15.0")
shouldContain("expected:<true> but was:<false>")
shouldContain("3) expected:<16> but was:<15>")
}
}
}
}
})
JUnit version:
fun `It can use soft assertions to let multiple assertions fail and report back on all of them`() {
val actual: Int = 15
val exception = assertThrows<AssertionError> {
assertAll(
{ assertEquals(Number::class, actual::class) },
{ assertEquals(15.0, actual) },
{ assertEquals(15, actual) }, // this one is OK
{ assertEquals(16, actual) }
)
}
assertTrue(exception.message!!.contains("expected: <class kotlin.Number> but was: <class kotlin.Int>"))
assertTrue(exception.message!!.contains("expected: <15.0> but was: <15>"))
assertTrue(exception.message!!.contains("expected: <16> but was: <15>"))
}
I'm gonna pause things here. I've got other stuff to do this afternoon, plus I'm happy that JUnit5 meets the organisation requirements I expect from a testing framework at this point. I'm not so happy with its native assertions library as they seem pretty limited, and I still think the assertEquals(expected, actual) approach to things is clunky. The next thing I need to look at is options for extending the assertion suite, and I have already seen there's a coupla options. I reckon that will be an article to itself, though.
The code for this lot is up on GitHub @ /src/test/kotlin/junit.
Righto.
--
Adam
