Friday, 14 October 2022

Kotlin: looking at delegated properties

G'day:

In my "Kotlin/Gradle: abstracting versions into a config file, and wondering what delegated properties are" article from a few days ago, I landed upon some syntax in my build.gradle.kts file that I didn't quite get:

val kotestVersion: String by project
val systemLambdaVersion: String by project
val commonsLang3Version: String by project
// …

I wasn't so sure what that by project jazz was all about. project is just an object as it turns out - the properties of which are defined in gradle.properties - and this by thing is something to do with "Delegated properties. Let's have a look at those.

The intro in the docs there say this:

With some common kinds of properties, even though you can implement them manually every time you need them, it is more helpful to implement them once, add them to a library, and reuse them later.

Erm: fine. But I am none the wiser. I'll try some code. Well: I'll get Copilot to try some code, as it turns out.


Property delegates

Here's a test

fun `It can delegate the Maori translation of a word`() {
    val one = Number(1)
    one.mi shouldBe "tahi"
    one.en shouldBe "one"
}

Different from my usual test Number classes, this one only sets the digit. The rest… gets done for me. How?

class Number(val id: Int) {
    val mi: String by MaoriNumber()
    val en: String by EnglishNumber()
}
class MaoriNumber {
    operator fun getValue(thisRef: Number, property: kotlin.reflect.KProperty<*>): String {
        return when (thisRef.id) {
            1 -> "tahi"
            2 -> "rua"
            3 -> "toru"
            4 -> "wha"
            5 -> "rima"
            6 -> "ono"
            7 -> "whitu"
            8 -> "waru"
            9 -> "iwa"
            10 -> "tekau"
            else -> "unknown"
        }
    }
}

(EnglishNumber is the same, just in English).

I've specified the getValue function as an operator because the docs told me to, btw. Well in fact Copilot wrote 95% of that for me, and then I checked back with the docs to make sure it nailed it. Copilot wanted MaoriNumber to extend Number, for some reason, which is unnecessary so I removed that. It did everything else. It makes sense: it receives the object it's a delegate for, and the property of the object it's a delegate for, looks up the id value in its internal map and returns the value.

Next I want to throw an exception for unsupported values, instead of returning "unknown". I have NFI how to do this, so I am gonna encourage Copilot to help out.

fun `It throws an exception when the Maori translation is not found`() {
    val eleven = Number(11)
    shouldThrow<IllegalArgumentException> {
        eleven.mi
    }.message shouldBe "No Maori translation for 11"
}

I had to type as far as "It throws and exception", then Copilot did the rest. Except it did it for "2" not "11", which was a slight disappointment. The code supports "2".

The key thing to notice here is that the exception doesn't get thrown until the relevant property is accessed; not when the object is first created. mi is not initialised until it's needed.

The update for the delegate is reasonably obvious:

7 -> "whitu"
8 -> "waru"
9 -> "iwa"
10 -> "tekau"
else -> throw IllegalArgumentException("No Maori translation for ${thisRef.id}")

Copilot handled it after "throw", but did not work out to correlate the message back to the test. This is fine. Let's see if it nails the English version now that I've explained stuff better:

7 -> "seven"
8 -> "eight"
9 -> "nine"
10 -> "ten"
else -> throw IllegalArgumentException("No English translation for ${thisRef.id}")

Nailed it. It could not work out that I wanted to include an expectation for this in the test, so I hand-cranked that.

That's quite cool.

Seems there are some more specialised delegate types as well…


Lazy properties

A property that delegates to lazy will use lazy's lambda to initialise the property first time round,

@Test
fun `It is only initialise when accessed`() {
    class TestClass {
        val lazyProperty: String by lazy {
            telemetry.add("lazyProperty initialised")
            "lazyProperty"
        }
        val telemetry: MutableList<String> = mutableListOf("initialising telemetry")
    }

    val testClass = TestClass()
    testClass.telemetry.first() shouldBe "initialising telemetry"
    testClass.lazyProperty shouldBe "lazyProperty"
    testClass.telemetry.last() shouldBe "lazyProperty initialised"
    testClass.telemetry.size shouldBe 2
    testClass.lazyProperty shouldBe "lazyProperty"
    testClass.telemetry.size shouldBe 2
}

Note for this:

  • Lazy properties cannot be mutable, so have to be vals, and obviously can't have a setter. If I try I get this error in IntelliJ:

    Type 'Lazy<TypeVariable(T)>' has no method 'setValue(DelegatedPropertiesTest.LazyProperties.TestClass, KProperty<*>, String)' and thus it cannot serve as a delegate
  • The telemetry tells the story here. Only the first access to lazyProperty's getter actually executes the code in the lazy lambda: there is no second telemetry entry for the second access to it.

I've augmented this test in the version of this code in source control, but whilst it demonstrates some stuff, this first iteration is clearer, hence using this one.

This next test shows that if the lazy initialisation code fails, it will re-try subsequent times:

@Test
fun `It can throw an exception on initialisation but will continue to attempt to initialise the property on subsequent calls`() {
    class TestClass {
        private var initAttempts = 0

        val lazyProperty: String by lazy {
            initAttempts++
            telemetry.add("lazyProperty initialisation called ($initAttempts)")
            if (initAttempts <= 2) {
                throw IllegalStateException("lazyProperty initialisation failed ($initAttempts)")
            }
            telemetry.add("lazyProperty initialisation succeeded ($initAttempts)")
            "lazyProperty"
        }
        val telemetry: MutableList<String> = mutableListOf("initialising telemetry")
    }

    val testClass = TestClass()
    testClass.telemetry[0] shouldBe "initialising telemetry"

    shouldThrow<RuntimeException> {
        testClass.lazyProperty
    }.message shouldBe "lazyProperty initialisation failed (1)"

    testClass.telemetry.size shouldBe 2
    testClass.telemetry[1] shouldBe "lazyProperty initialisation called (1)"

    shouldThrow<RuntimeException> {
        testClass.lazyProperty
    }.message shouldBe "lazyProperty initialisation failed (2)"
    testClass.telemetry.size shouldBe 3
    testClass.telemetry[2] shouldBe "lazyProperty initialisation called (2)"

    shouldNotThrow<RuntimeException> {
        testClass.lazyProperty
    }
    testClass.telemetry.size shouldBe 5
    testClass.telemetry[3] shouldBe "lazyProperty initialisation called (3)"
    testClass.telemetry[4] shouldBe "lazyProperty initialisation succeeded (3)"

    shouldNotThrow<RuntimeException> {
        testClass.lazyProperty
    }
    testClass.telemetry.size shouldNotBeGreaterThan 5
    testClass.telemetry.last() shouldBe "lazyProperty initialisation succeeded (3)"
}

This looks like a lot of code, but there's not much to it.

  • The lazy initialisation lambda throws an exception the first two times it's called.
  • After that, it initialises the value successfully.
  • The test calls the getter 1, 2, 3, 4 times, and checks what telemetry has been logged each time.
  • After the third call (which was successful), the lambda is not called again, ie: there is no further telemetry since the first successful initialisation.

Observable properties

This sounded interesting to me when I saw the heading ("Ooh! Baked-in Observer Pattern!"), but I was underwhelmed by the implementation.

inner class TestClass {
    var observableProperty: String by Delegates.observable("initial value") {
        property, oldValue, newValue ->
        telemetry.add("${property.name} changed from $oldValue to $newValue")
    }
    val telemetry: MutableList<String> = mutableListOf("initialising telemetry")
}

Test:

@Test
fun `It shows that observable properties can be used to track changes to a property`() {
    val testClass = TestClass()

    testClass.telemetry.first() shouldBe "initialising telemetry"
    testClass.observableProperty shouldBe "initial value"
    testClass.telemetry.size shouldBe 1

    testClass.observableProperty = "new value"
    testClass.telemetry.size shouldBe 2
    testClass.telemetry[1] shouldBe "observableProperty changed from initial value to new value"
    testClass.observableProperty shouldBe "new value"

    testClass.observableProperty = "newer value"
    testClass.telemetry.size shouldBe 3
    testClass.telemetry[2] shouldBe "observableProperty changed from new value to newer value"
    testClass.observableProperty shouldBe "newer value"

    testClass.observableProperty = "newer value"
    testClass.telemetry.size shouldBe 4
    testClass.telemetry[3] shouldBe "observableProperty changed from newer value to newer value"
    testClass.observableProperty shouldBe "newer value"
}

So… you can have a setter method. I… already knew that. How does that observableProperty property differ from this:

var standardProperty : String = "initial value"
    set(value) {
        telemetry.add("standardProperty changed from $field to $value")
        field = value
    }

I got Copilot to write the test for me, and it's the same as the observableProperty one :-(

To be clear, that seems all there is to it. One can only "attach" that single listener to the "observable" property. There's no mechanism to go observableProperty.attach({property, oldValue, newValue -> /* stuff here */}) to attach more handlers later, which would be handy.

Hopefully I'm missing something, but googling about, I get the same vibe from other people too.


Vetoable properties

These were only mentioned in passing on that previous docs page I had been looking at. I only found out about them because I asked Copilot "anything else about delegated properties I should know about?", and it autofilled something about vetoable properties, so did a google.

class TestClass {
    var vetoableProperty: String by Delegates.vetoable("initial value") {
        property, oldValue, newValue ->
        telemetry.add("${property.name} changed from $oldValue to $newValue")
        newValue != "veto value"
    }
    val telemetry: MutableList<String> = mutableListOf("initialising telemetry")
}

Here if the lambda returns false, the property doesn't actually get updated:

val testClass = TestClass()
testClass.telemetry.first() shouldBe "initialising telemetry"
testClass.vetoableProperty shouldBe "initial value"
testClass.telemetry.size shouldBe 1

testClass.vetoableProperty = "new value"
testClass.telemetry.size shouldBe 2
testClass.telemetry[1] shouldBe "vetoableProperty changed from initial value to new value"
testClass.vetoableProperty shouldBe "new value"

testClass.vetoableProperty = "veto value"
testClass.telemetry.size shouldBe 3
testClass.telemetry[2] shouldBe "vetoableProperty changed from new value to veto value"
testClass.vetoableProperty shouldBe "new value"

testClass.vetoableProperty = "newer value"
testClass.telemetry.size shouldBe 4
testClass.telemetry[3] shouldBe "vetoableProperty changed from new value to newer value"
testClass.vetoableProperty shouldBe "newer value"

See how after trying to set it to "veto value", the value was not updated from the previous one.


Setting properties from a map

Probably just the tests speak for themselves here, as I think it's pretty obvious what's going on, and why/how:

fun `It shows that map properties can be used to access a map by key`() {
    class Number(val map: Map<String, Any?>) {
        val id: Int by map
        val en: String by map
        val mi: String by map
    }

    val number = Number(
        mapOf(
            "id" to 1,
            "en" to "one",
            "mi" to "tahi"
        )
    )
    number.id shouldBe 1
    number.en shouldBe "one"
    number.mi shouldBe "tahi"
}
@Test
fun `It shows that map properties can be used to access a map by key with a default value from a function`() {
    class Number(val map: Map<String, Any?>) {
        val id: Int by map
        val en: String by map
        val mi: String by map
        val ga: String by map.withDefault { key -> "unknown $key" }
    }

    val number = Number(
        mapOf(
            "id" to 1,
            "en" to "one",
            "mi" to "tahi"
        )
    )
    number.id shouldBe 1
    number.en shouldBe "one"
    number.mi shouldBe "tahi"
    number.ga shouldBe "unknown ga"
}
@Test
fun `It shows that map properties can be used to access a map by key with a default value from a function that throws an exception`() {
    class Number(val map: Map<String, Any?>) {
        val id: Int by map
        val en: String by map
        val mi: String by map
        val ga: String by map.withDefault { key -> throw IllegalArgumentException("unknown key $key") }
    }

    val number = Number(
        mapOf(
            "id" to 1,
            "en" to "one",
            "mi" to "tahi"
        )
    )
    number.id shouldBe 1
    number.en shouldBe "one"
    number.mi shouldBe "tahi"
    shouldThrow<IllegalArgumentException> {
        number.ga
    }.message shouldBe "unknown key ga"
}

Copilot wrote all that code for me btw. I changed the third language from the French it suggested to Irish.


OK, so that's that. The concept of property delegation is interesting / useful, But I didn't quite "get" why they have bothered with the pre-packaged ones (lazy, observation, vetoable),if I'm honest. I suspect I am missing something still…

The code is on GitHub, here: DelegatedPropertiesTest.kt.

Righto.

--
Adam

Kotlin / JUnit5: making a test conditional based on a custom condition

G'day:

I sat down to continue my "next" blog article, which will be about delegated properties in Kotlin (will cross link here once it's done), and as a first step I ran my tests to remind myself where I got to.

To my dismay I saw a bunch of failing tests. This in itself is OK as I am half-way through doing something, but the tsts that were failing were the ones for a different article: "Kotlin / TDD: writing the tests for a small web service". I slapped my forehead on this because obviously they would fail because I don't have that web service running ATM. And that's legit, so I needed to be able to conditionalise (is that a word? Is now I guess) those tests based on whether the web service is up.

I have previously looked at conditionalising tests in both Kotest ("Kotlin / Gradle / Kotest random exploration"), and then JUnit ("Kotlin: looking at JUnit instead of Kotest"), but in the case of the JUnit ones, they were all "canned" conditions, eg: @EnabledOnOs(OS.LINUX). In this case, I need to base the conditionality on my own logic.

Some quick googling landed me on the ever-helpful Baeldung site, with JUnit 5 Conditional Test Execution with Annotations › 6.2. Conditions. That said, I didn't find that article very helpful, and their examples were writing JS (yes: JS in a Java test; using Nashorn) in a string which seemed like a shit way of doing things to me, and I "knew" there must be a better way. I looked directly at the JUnit5 docs and found what I wanted: JUnit 5 User Guide › 2.8.6. Custom Conditions, their example was this:

@Test
@EnabledIf("customCondition")
void enabled() {
    // ...
}

@Test
@DisabledIf("customCondition")
void disabled() {
    // ...
}

boolean customCondition() {
    return true;
}

I can work with that. I set about writing an isUp function to call on my tests. I did TDD the behaviour of Ktor's HttpClient when getting no response from the server, along these lines:

@Test
fun `it throws an exception if the web service can't be reached`() {
    shouldThrow<ConnectException> {
        runBlocking {
            HttpClient().use { client ->
                client.get(webServiceBaseUrl)
            }
        }
    }.message shouldStartWith "Connection refused"
}

But this did not make the cut for my final code as it was a bit tricky to test the function that I use to conditionally enable the tests based on its own response (if you see what I mean: it's a bit self-referential). So only partial TDD on this I guess. Oops.

The final version of the isUp function was this:

fun isUp(): Boolean {
    return try {
        runBlocking {
            HttpClient().use { client ->
                client.head(webServiceBaseUrl)
            }
        }
        true
    } catch (e: ConnectException) {
        false
    }
}

I still love how a try/catch is an expression in Kotlin, so I can return it.

I had to horse around a bit when it came to actually using it in the annotation though. Unlike the examples in the docs, I did not want to make just a test method conditional, I wanted the whole class to be conditional, eg:

@DisplayName("Tests of WebService class")
@EnabledIf("isUp")
internal class WebServiceTest {

That's fine. But I had the function inside WebServiceTest, and the annotation could not find it:

Failed to evaluate condition [org.junit.jupiter.api.condition.EnabledIfCondition]: Cannot invoke non-static method [public final boolean junit.practical.WebServiceTest.isUp()] on a null target.

I don't really know what "on a null target" means here, but clearly I could not use a object method here. I thought it might be OK if I hauled the function out of the class and had it at the top level, but then it lost access to webServiceBaseUrl and I didn't want to repeat the URL in two places, so… noticed the error said non-static method, and so decided to sling the thing into a companion object (see "Kotlin: there's no such thing as static, apparently") of WebServiceTest:

@EnabledIf("isUp")
internal class WebServiceTest {

    private val webserviceUrl = webServiceBaseUrl + "numbers/"

    @Serializable
    data class SerializableNumber(val id: Int, val en: String, val mi: String)

    companion object {
        const val webServiceBaseUrl = "http://localhost:8080/"
        @JvmStatic
        fun isUp(): Boolean {
            return try {
                runBlocking {
                    HttpClient().use { client ->
                        client.head(webServiceBaseUrl)
                    }
                }
                true
            } catch (e: ConnectException) {
                false
            }
        }
    }
	// …
}    

And that worked fine. Note the @JvmStatic there. Just having the companion object works fine for Kotlin code, but I guess something to do with how the JUnit code is run requires that annotation to make the method seem static from Java's perspective too.

When I have the web service running:

And when it's off:

Compared to before:

Excellent.

Time for a break, and then I'll get back to that delegated properties article. The code for this is on GitHub @ WebServiceTest.kt.

Righto.

--
Adam

Sunday, 9 October 2022

Kotlin / TDD: using JetBrains/Exposed to read from the DB

G'day:

Earlier today I wrote "Kotlin / TDD: writing the tests for a small web service". I focused on learning how to make HTTP requests and test the results thereof. The last bit I needed to do for the exercise I had at hand was to write a test to verify my POST request was actually writing to the DB. To do this I'd need to work out how to actually make a DB call in Kotlin, and I've not got that far yet. I decided to leave that to the next article, and come back to it later. Then I got to thinking: everything else has proven to be really easy so far: I bet this will be too. And it started to bug me so I decided "to hell with it: let's find out". So here I am again. Spoilers: it was easy, and this will be a short article.


It saves the new object to the database

Full disclosure: no it doesn't. I have not modified the test web service to do anything of the sort: it does not touch the DB, it just responds saying "yeah, done", and I already have the test data in there. So the data is "mocked" I guess. All I'm doing here is testing that I can fetch something from the DB.

This test is more complicated. Firstly I have to define the entity that represents the data:

class TranslatedNumber(id: EntityID<Int>) : IntEntity(id) {
    object TranslatedNumbers : IntIdTable("numbers") {
        val en: Column<String> = varchar("en", 50)
        val mi: Column<String> = varchar("mi", 50)
    }
    companion object : IntEntityClass<TranslatedNumber>(TranslatedNumbers)

    var en by TranslatedNumbers.en
    var mi by TranslatedNumbers.mi
}

I pretty much got all that from the docs: JetBrains/Exposed › Getting Started › Your first Exposed DAO, although I refactored things a bit.

There's two parts to this:

It's all pretty clear, I think?

Now the test:

@Test
@EnabledIfEnvironmentVariable(
    named = "MARIADB_PASSWORD",
    matches = ".*",
    disabledReason = "This test requires a MariaDB database, so it needs the password"
)
fun `It saves the new object to the database`() {
    val six = SerializableNumber(6, "six", "ono")
    runBlocking {
        HttpClient() {
            install(ContentNegotiation) {
                json()
            }
        }.use { client ->
            val response = client.post(webserviceUrl) {
                contentType(ContentType.Application.Json)
                setBody(six)
            }

            response.status shouldBe HttpStatusCode.Created

        }
        Database.connect(
            "jdbc:mysql://localhost:3308/db1",
            driver = "com.mysql.cj.jdbc.Driver",
            user = "user1",
            password = System.getenv("MARIADB_PASSWORD")
        )
        transaction {
            addLogger(StdOutSqlLogger)

            TranslatedNumber.findById(six.id).asClue {
                it shouldNotBe null
                it!!.en shouldBe six.en
                it.mi shouldBe six.mi
            }
        }
    }
}

I've greyed-out the stuff that's mostly the same as the previous tests from the earlier article. The interesting bits are:

  • I don't want to hard-code my DB password here (especially as it's going into source control), so I use an environment variable. And if the env variable ain't set: skip the test. I was quite let down that the reason message doesn't display in the test output.
  • If I add that logger there, all the SQL statements echo to std out. This makes debugging issues easier.

    SQL: SELECT numbers.id, numbers.en, numbers.mi FROM numbers WHERE numbers.id = 6
  • Apparently all Exposed DB calls need to occur within one of these transaction blocks. That's not me deciding to run a single select query in a transaction, it's a hard requirement of Exposed.
  • The !! operator throws an exception at runtime if the expression is null. I guess we only need the one of them as it's implied that it's an all/nothing thing if the object has been initialised. Copilot wrote that code [cough]. However if I take it out, I get an error:

    Only safe (?.) or non-null asserted (!!.) calls are allowed on a nullable receiver of type TranslatedNumber?

    If I use ?. instead, then I need it on both en and mi.

One thing I am left wondering about here: there doesn't seem to be any direct coupling between the DB connection and the transaction block. What if I had multiple DB connections? I see there is a discussion on GitHub: Multiple databases #93, and I share the opinion that I am not in love with how they have resolved this: Transactions › Working with a multiple databases. Basically it's this:

val conn = Database.connect(
    // etc
)
transaction(conn) {
   // etc
}

I was expecting this sort of thing:

Database.connect(
    // etc
).transaction {
   // etc
}

// or

val conn = Database.connect(
    // etc
)

// ...

conn.transaction {
   // etc
}

That looks more Kotlin-idiomatic to me (he says, having been using Kotlin for like a coupla weeks… ;-).


OK that was easy. One last observation: "Exposed" is a shit name. It reads like a red-banner-tabloid headline, and… it's a verb (this might have been lost on the JetBrains crew, as I don't think they're native English speakers). It's also not hugely google-able without prefixing it with "kotlin". Ah well.

And now I am giving up for the day. That's enough Kotlin. Code is here: WebServiceTest.kt and TranslatedNumber.kt. This latest tag also includes some housekeeping changes, but those are the important bits.

Righto.

--
Adam

Kotlin / TDD: writing the tests for a small web service

G'day:

One of my tasks @ work is to check out how to test a web service. I started with this yesterday's article: Kotlin: getting the khttp library installed and running… then… getting rid of it and using something else, but that was justa "proof of concept" of making an HTTP call, and examining its results. Today I'm gonna write the actual tests we need for the interface for a web service. Note: I still don't know enough about Ktor to create a web service with it, so I'm gonna fall back to using CFML for that end of things. I'm not gonna bother with the code for that here; the tests will demonstrate whether or not the web service is fulfilling its contract. But I will be TDDing this. Currently I have zero tests and zero code to test. Let's get on with it.


It should return a 200-OK on the root URI on a valid request

fun `It should return a 200-OK on the root URI on a valid request`() {
    runBlocking {
        HttpClient().use { client ->
            val response = client.get(webserviceUrl)
            response.status shouldBe HttpStatusCode.OK
        }
    }
}

NB: I will only comment if there's something note-worthy or not obvious. All tests will initially fail until I implement the relevant bit of the web service, eg:

expected:<200 OK> but was:<404 Not Found>
Expected :200 OK
Actual   :404 Not Found

It should return a 406-NOT-ACCEPTABLE and suggest the expected type if the Accepts header is not application/json

@Test
fun `It should return a 406-NOT-ACCEPTABLE and suggest the expected type if the Accepts header is not application-json`() {
    runBlocking {
        HttpClient().use { client ->
            val response = client.get(webserviceUrl) {
                header("Accept", "text/plain")
            }

            response.status shouldBe HttpStatusCode.NotAcceptable
            response.body() as String shouldBe """["application/json"]"""
        }
    }
}

Note: It's a Kotlin/JVM limitation that I have to use application-json rather than application/json in the method name there. See Why Kotlin does not allow slash in identifiers, which in turn points the reader to Java Virtual Machine Specification › Chapter 4. The class File Format:

4.2.2. Unqualified Names

Names of methods, fields, local variables, and formal parameters are stored as unqualified names. An unqualified name must contain at least one Unicode code point and must not contain any of the ASCII characters . ; [ / (that is, period or semicolon or left square bracket or forward slash).

When I initially just referenced response.body() shouldBe """["application/json"]""" Kotlin was saying "Not enough information to infer type variable", and after some googling I landed on Type checks and casts › "Unsafe" cast operator, which explains that as String thing I have there.

Also note I needed to update the 200-OK test to pass the correct Accept header.


It returns an array of Numbers as a JSON array

This one was a bit trickier, but the docs were reasonably helpful, and I'm pleased with the outcome.

@Test
fun `It returns an array of Numbers as a JSON array`() {
    @Serializable
    data class Number(val id: Int, val en: String, val mi: String)

    runBlocking {
        HttpClient() {
            install(ContentNegotiation) {
                json()
            }
        }.use { client ->
            val response = client.get(webserviceUrl) {
                header("Accept", "application/json")
            }
            response.status shouldBe HttpStatusCode.OK
            response.body() as List<Number> shouldBe listOf(
                Number(1, "one", "tahi"),
                Number(2, "two", "rua"),
                Number(3, "three", "toru"),
                Number(4, "four", "wha")
            )
        }
    }
}

I had guessed that there'd be a way to deliver objects straight to the app from an HTTP request, and a quick google set me on the right path here, landing me on these docs: Content negotiation and serialization. What's super helpful in these docs as they link through to examples of everything (including the build.gradle.kt file, as I needed to add some dependencies - see further down), eg: ktor-documentation/codeSnippets/snippets/client-json-kotlinx/src/main/kotlin/com/example/Application.kt and ktor-documentation/codeSnippets/snippets/client-json-kotlinx/build.gradle.kts. I just followed those and changed the bits I needed to change.

Steps:

  • tell the client that I'm gonna expect it to work out what the content represents (boilerplate).
  • Create a data class that the client will use to deserialize the data as. NB: it needs to be tagged as being serializable, because well: that's what we're doing here.
  • Specify that type - or in this case a list of that type - as the body value.

Done.

I will admit than initially I thought I had messed-up because instead of getting a "nup, it's not a list of Numbers" in my failing test, I got this monstrousity:

io.ktor.client.call.NoTransformationFoundException: No transformation found: class io.ktor.utils.io.ByteBufferChannel -> class kotlin.collections.List

But it turns out that just means "um… yer request ain't returning JSON". Which it indeed was not. Once I got it to return something (anything) and the correct content-type, I got a more on-point error:

Expected start of the array '[', but had 'EOF' instead at path: $
JSON input: ""  

And from there I tweaked the web service to return a coupla wrong things to see how Ktor reported on deserialization failures, and it was all helpful.

All the examples I saw for this was only deserializing one object, so I was slightly cautious as to how to deal with a JSON array, but I took a punt on just specifying response.body() as List<Number>, thinking Ktor's probably clever enough to expect this sort of thing, and - lo - it did. Nice one.

As I mentioned above, I also had to tweak the dependencies and plugins a bit (build.gradle.kts):

plugins {
    kotlin("jvm") version "1.7.10"
    kotlin("plugin.serialization").version("1.7.10")
    application
}

// …

dependencies {
    // …
    implementation("io.ktor:ktor-client-core:$ktorVersion")
    implementation("io.ktor:ktor-client-cio:$ktorVersion")
    implementation("io.ktor:ktor-client-content-negotiation:$ktorVersion")
    implementation("io.ktor:ktor-serialization-kotlinx-json:$ktorVersion")
    // …
}

It will accept a POST request of an object as JSON and return the same object as confirmation, and its URL

This was really straight-forward, having done all the hard-bit in the previous one:

@Test
fun `It will accept a POST request of an object as JSON and return the same object as confirmation, and its URL`() {
    val five = Number(5, "five", "rima")
    runBlocking {
        HttpClient() {
            install(ContentNegotiation) {
                json()
            }
        }.use { client ->
            val response = client.post(webserviceUrl) {
                contentType(ContentType.Application.Json)
                setBody(five)
            }
            response.status shouldBe HttpStatusCode.Created
            response.body() as Number shouldBe five
            response.headers["Location"] shouldBe "${webserviceUrl}5"
        }
    }
}

By now there's absolutely nothing "unexpected" in this, I think.


OK that's enough for a Sunday afternoon. This weekend I've managed to work out how to make HTTP requests in my tests, how to set/check headers, response codes and the body of the responses. I've posted an object and received objects back again, letting Ktor handle the (de)serialization.

These tests are only testing the interface of the web webservice, which is fine and an essential part of building a web service. However the next thing on the list is to do end to end tests: check the underlying data store that new objects are being created (cos they absolutely are not at the moment ;-)), and the correct data is being returned, etc. I have no idea how to make a DB call in Kotlin yet. Something for the coming week, I guess.

The code is on GitHub @ /src/test/kotlin/junit/practical/WebServiceTest.kt, and the web stub service I was creating to make the tests pass is this lot: adamcameron/Numbers.cfc.

Righto.

--
Adam

Saturday, 8 October 2022

TDD: are tests that just "measure twice, cut once" legit?

G'day:

There's a bit of archaeology going on here: I started writing this in Jan 2022, but never got past the first para. I have a few articles like this, and I've decided to either complete them if they have merit, or get rid if they don't.

Part of my problem previously on this blog is that if I didn't think I could spin the article out to be say ~1500 words, I didn't think it was worth writing about. I think that was wrong-headed, and there's nothing wrong with taking a quick look at something I've been thinking about. Let's see if this works.

The situation I describe herein actually happened after I had wondered about the subject matter. It bore out/validated my hypothesis quite nicely.

Today I had an interesting TDD case. We have a features that rely on finding the "next working day" in various countries. The code behind these features ultimately calls a function isBankHoliday, and the logic in that function maintains a list of bank holidays within its implementation. This is perhaps not the ideal approach, but it's the approach we have.

I was loading in the holidays for 2023, and me being me, the first thing I did was to look for the unit tests for the function. We didn't have one. That was odd because we had a job a few months prior to add the 2022 holidays, and I can recall discussing the need for tests with the dev concerned. Apparently this never happened, and I can't follow-up because said dev is no longer with us. Harrumph.

I can't reproduce any code relating to this here because it's my employer's code, but I can describe what I did. I opened the official govt sites that list public holidays (eg in the UK it is https://www.gov.uk/bank-holidays), and wrote a test that iterated through the list of 2023(*) holidays there and called isBankHoliday on each of them, expecting it to be true. I then picked some edge-case days around bank holidays and tested those expected the result to be false. This gives both a control group (the tests returning false), and the TDD tests for the work I was about to do. I got the expected pass/failures: all good.

(*) why did I not do all of them, and only tested the 2022 ones I was adding? We have a policy of not backfilling missing unit tests, because we'd be there all day/month/year if we did. We only have maybe 25% code coverage of our codebase for… "historical reasons".

OK, so the tests are in place. I then closed the test file, went back to https://www.gov.uk/bank-holidays and re-copied all the holiday dates again. I did not pull the prior work out of the test and refactor to suit the method implementation. I even made sure to specifically use a different data structure in the tests than I knew we had in the method implementation: the tests used an array because I'm not a muppet; the method implementation used a comma-delimited string (ugh).

Why did I do this? Well: measure twice, cut once. If I wrote the tests, then lifted the test data out of the tests and put them in the implementation, all I'd be doing is testing my ability to copy and paste. I would not be testing that I had keyed the source data I was testing against correctly. A typo in the test data would translate to a typo in the implementation, but the test would still pass.

That code went into production and has not caused us any problems.

Back in May this year we had a system glitch where a bunch of processing didn't run. It was 2022-05-03. We looked into it and found that the dev who had loaded in the 2022 holidays had miskeyed, and had entered the Queen's Platinum Jubilee holiday as 2022-05-03 not 2022-06-03. And with no tests: we did not catch it. It went through code review, that said, and we didn't actually catch it there either.

This situation caused a bunch of work for us because not only did we have to remediate the unprocessed work from 2022-05-03, but the system had also started to queue-up work for 2022-06-03 when it shouldn't have, and we had to unpick that lot too.

All because we didn't measure twice, and cut once.

Righto.

--
Adam

Kotlin/Gradle: abstracting versions into a config file, and wondering what delegated properties are

G'day:

Currently I have just been following the instructions fairly slavishly when it comes to my dependency management in my application. If some readme file says "put testImplementation("io.kotest:kotest-runner-junit5:5.5.0") in yer Gradle build file to include this dependency", then that's exactly what I do. Sometimes I monkey with the version if IntelliJ says there's a newer version. So I've ended up with this lot:

dependencies {
    testImplementation(kotlin("test"))
    testImplementation("io.kotest:kotest-runner-junit5:5.5.0")
    testImplementation("io.kotest:kotest-assertions-core:5.5.0")
    testImplementation("com.github.stefanbirkner:system-lambda:1.2.1")
    testImplementation("org.apache.commons:commons-lang3:3.12.0")
    testImplementation("org.junit.jupiter:junit-jupiter:5.9.0")
    testImplementation("org.junit.jupiter:junit-jupiter:5.9.0")
    testImplementation("io.kotest:kotest-framework-datatest:5.5.0")
    implementation("io.ktor:ktor-client-core:2.1.2")
    implementation("io.ktor:ktor-client-cio:2.1.2")
}

That's fine. However then I looked at the sample project that the Ktor Project Generator generates, and I see this in my its build file:

val ktor_version: String by project
val kotlin_version: String by project
val logback_version: String by project

// ...

dependencies {
    implementation("io.ktor:ktor-server-core-jvm:$ktor_version")
    implementation("io.ktor:ktor-server-netty-jvm:$ktor_version")
    implementation("ch.qos.logback:logback-classic:$logback_version")
    testImplementation("io.ktor:ktor-server-tests-jvm:$ktor_version")
    testImplementation("org.jetbrains.kotlin:kotlin-test-junit:$kotlin_version")
}

What's all this by project carry-on? I looked for a string match for ktor_version, and found this lot in gradle.properties:

ktor_version=2.1.2
kotlin_version=1.7.20
logback_version=1.2.11
kotlin.code.style=official

(Mine currently only has that last line)

OK so that seems like a good idea, I'm gonna run with this approach for the dependencies in my scratch project too:

kotlin.code.style=official
kotest_version=5.5.0
kotlin_version=1.7.10
system_lambda_version=1.2.1
commons_lang_3_version=3.12.0
junit_jupiter_version=5.9.0
ktor_client_core_version=2.1.2
dependencies {
    testImplementation(kotlin("test"))
    testImplementation("io.kotest:kotest-runner-junit5:$kotest_version")
    testImplementation("io.kotest:kotest-assertions-core:$kotest_version")
    testImplementation("io.kotest:kotest-framework-datatest:$kotest_version")
    testImplementation("com.github.stefanbirkner:system-lambda:$system_lambda_version")
    testImplementation("org.apache.commons:commons-lang3:$commons_lang_3_version")
    testImplementation("org.junit.jupiter:junit-jupiter:$junit_jupiter_version")
    testImplementation("org.junit.jupiter:junit-jupiter:$junit_jupiter_version")
    implementation("io.ktor:ktor-client-core:$ktor_client_core_version")
    implementation("io.ktor:ktor-client-cio:$ktor_client_core_version")
}

When I rebuilt: it all went fine, and all my tests still pass.

I noted in the Ktor build file the Kotlin plug-in version was still a static inline string:

plugins {
    application
    kotlin("jvm") version "1.7.20"
    // …
}

I tried to make that dynamic as well:

plugins {
    kotlin("jvm") version kotlin_version
    application
}

But INtelliJ barfed at this saying:

'val kotlin_version: String' can't be called in this context by implicit receiver. Use the explicit one if necessary

I googled that, and landed on Why can’t I use val inside Plugins {}? which pointed me to this: Using Gradle Plugins › Limitations of the plugins DSL, and the important bit there is:

This requires that plugins be specified in a way that Gradle can easily and quickly extract, before executing the rest of the build script. It also requires that the definition of plugins to use be somewhat static.

(My emphasis).

Fair cop. I'll not try to do that then.

All this is lovely, but how does this work. What magic is "by project" doing? I guessed that by is some inline function, and project is its argument, similar to in a test where I have x shouldBe y. I can drill down on by, and it takes me to org.gradle.kotlin.dsl.ProjectExtensions:

operator fun Project.provideDelegate(any: Any?, property: KProperty<*>): PropertyDelegate =
    propertyDelegateFor(this, property)

I'm none-the-wiser looking at that, but it gave me something to google (googling on phrases where "by" is the important word was not a particularly edifying or fruitful endeavour). I found Delegated properties, the summary of which is:

With some common kinds of properties, even though you can implement them manually every time you need them, it is more helpful to implement them once, add them to a library, and reuse them later.

…

Kotlin supports delegated properties:

class Example {
    var p: String by Delegate()
}

OK, that answers that. There is also a bunch more interesting stuff in that article which I will need to look at later. Oh and what's the "project" part of "by project"? It's just an object that wraps up the values I set in gradle.properties. Slick.

I'm really glad to understand what's going on there, and also looking forward to looking at "delegated properties" some more. Later.


That was painless and short. I quite like that I have de-duped my versions there, and separated them from their usage. The changes made for this exercise are in GitHub @ kotlin_scratch, tag 1.12. Not that it's very intertesting code today, I s'pose ;-)

Righto.

--
Adam

Friday, 7 October 2022

Kotlin: getting the khttp library installed and running... then... getting rid of it and using something else

G'day:

So 50% of this was a largely fruitless exercise in the end. Other than the fact that I learned some stuff that I think is worth knowing for future reference.

The next thing on my Kotlin/testing list is a more practical exercise: I need to be able to write tests for web service endpoints. To do that I need to be able to make HTTP requests. I second-guessed there'd be really awful boilerplate-laiden Java ways of doing things, but I wanted a Kotlin-idiomatic way of doing it.


khttp

I googled "kotlin http requests", and one of the first links was to an article on the Baeldung website: "HTTP Requests with Kotlin and khttp". Their articles ae usually pretty solid, so I didn't look any further. It was especially appealing because they made a point of saying:

On the JVM we have several available options, from lower-level to very high-level libraries[…]. However, most of them are targeted primarily at Java programs.

In this article, we’re going to look at khttp, an idiomatic Kotlin library for consuming HTTP-based resources and APIs.

Perfect.

First I had to install this khttp thing. The instructions were for a hand-cranked Maven install, but I can't be arsed with that at the moment, and I wanna use Gradle via my build.gradle.kts file. Other things I've looked at have supplied the installation instructions for that, which is really handy for a n00b like me as I don't really know what I'm doing and what the various values mean in the Maven XML. And TBH: I don't want to know. But I figured I could work out from the XML:

<dependency>
    <groupId>khttp</groupId>
    <artifactId>khttp</artifactId>
    <version>0.1.0</version>
</dependency>

<repository>
    <id>central</id>
    <url>http://jcenter.bintray.com</url>
</repository>

I need the repository cos it's not on Maven Central yet apparently. Cool: I know that one can specify repositories and dependencies in the Gradle file, so how hard could it be.

repositories {
    mavenCentral()
    maven("https://jcenter.bintray.com")
}

dependencies {
    // …
    testImplementation("khttp:khttp:0.1.0")
}

All I needed to do is to change the repo's scheme from http to https, because with http I was getting an error:

Using insecure protocols with repositories, without explicit opt-in, is unsupported.

And that's fair enough. It installed OK over https.

I created my test:

@DisplayName("Tests of WebService class")
internal class WebServiceTest {

    val serviceUrl = "https://example.com/"
    @Test
    fun `it returns a 200`() {
        val response = khttp.get(serviceUrl)

        response.statusCode shouldBe 200
    }
}

And ran it:

Unable to make field private java.lang.String java.net.URL.host accessible: module java.base does not "opens java.net" to unnamed module @645aa696 java.lang.reflect.InaccessibleObjectException: Unable to make field private java.lang.String java.net.URL.host accessible: module java.base does not "opens java.net" to unnamed module @645aa696 at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)

Not what I wanted to see. Not least of all cos I have NFI what that means. I poss should have noticed the mention of reflection in which case I might have recalled that there's been warnings emiting from Java about this when I start-up Lucee:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by [x] to method [y]
WARNING: Please consider reporting this to the maintainers of [x]
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

Had I thought about it I would have landed on the fix sooner. It took me about an hour to find a comment on the khttp projects GitHub issues: ascclemens/khttp issue #88. I am running on a Java 16 JDK, and that reflective thing got blocked from that version on. The best answer on the issue was:

For enabling khttp 1.0 with JDK 17.0 I use these VM arguments:

--add-opens java.base/java.net=ALL.UNNAMED --add-opens java.base/sun.net.www.protocol.https=ALL-UNNAMED

Cool. I had a hunt around in the settings of IntelliJ a bit to find out where I could sling those. I found a few places that took VM arguments, which looked promising, but none worked when I rebuild the project. I also found a way of putting them in my build file:

application {
    mainClass.set("MainKt")
    listOf(
        "--add-opens java.base/java.net=ALL.UNNAMED",
        "--add-opens java.base/sun.net.www.protocol.https=ALL-UNNAMED"
    ).also { applicationDefaultJvmArgs = it }
}

Which is good to know, but also didn't work.

In the end I downloaded Java 15 JDK and used that instead. For future me, the setting that matter is the Gradle JVM:

It is not just a matter of changing the Java or Kotlin byte-code versions:

I could switch those to 15, and it didn't make a difference. But once I changed Gradles JVM to 15, it worked:

But do you know what? I wasn't so happy with reverting my JVM version to accommodate a library I was using for testing, so I decided to see if there was another one. And this is when I wished I had looked below the fold on the first page of Google results…


Ktor

Ktor is the framework we have decided to use for our project @ work, so it kinda makes sense to use its own tooling. I'm gonna have a look at their offering.

Installation was simple:

implementation("io.ktor:ktor-client-core:2.1.2")
implementation("io.ktor:ktor-client-cio:2.1.2")

The first one is self-explanatory; the second is the underlying engine the client uses it seems. There's some docs: Ktor Client / Developing applications / Setting up a client / Engines.

From there the test was a bit of horsing around:

fun `it returns a 200`() {
    runBlocking {
        HttpClient().use { client ->
            val response = client.get("https://jsonplaceholder.typicode.com/todos/1")
            response.status shouldBe HttpStatusCode.OK
        }
    }
}

The CIO engine is asynchronous, so all this is using coroutines which I don't have headspace for ATM, but it seems the "async await" approach to making async code run in non-async code is to sling a runBlocking block around it. Fine.

One cool Kotlin thing here is that I did have this code written like this:

fun `it returns a 200`() {
    runBlocking {
        val client = HttpClient()
        try {
            val response = client.get("https://jsonplaceholder.typicode.com/todos/1")
            response.status shouldBe HttpStatusCode.OK
        } finally {
            client.close()
        }
    }
}

IntelliJ had a squiggly line under the try, and the hint said "try-finally can be replaced with 'use()'". Um: sure. Can it? OK. Let's do that then. And it changed it for me to the code I initially showed.

I was bemused that the client.close() call had gone completely. I RTFMed a bit, and use is quite cool:

Executes the given block function on this resource and then closes it down correctly whether an exception is thrown or not.

It can be called on any object that implements the Closeable interface (although use is a Kotlin extension to this). If one looks at the implementation of Closeable.use we see what's going on (this is heavily elided):

fun use() {
    try {
        return block(this)
    } finally {
        try {
            close()
        } catch (closeException: Throwable) {
        }
    }
}

This use approach just makes sense.


Thoughts

In contrasting the very simple khttp approach to the Ktor approach, I prefer the khttp one. This reminds me of GuzzleHttp for PHP: it started off being a simple and useful and just… did HTTP requests. No async, no promises, it just got on with it. Then it decided async was all the rage so it took a very simple interface and made it overly complex for the task at hand (not to mention not backwards compatible). Ktor's approach seems similar. When I reflect upon how often I really need my HTTP requests to be done asynchronously to my mainline code, the answer is fucking never. Or so infrequently then I'm happy to write an async wrapper for the call if I need it.

But: we're already using Ktor, and it's more mature, and I don't have to retrograde to an old version of Java to use it. So Ktor and its unecessary complexity it is.

I'll try to do something more useful with this stuff tomorrow, but getting everything working (twice) took more time than I wanted to spend on this, so I'm stopping here for now.

The code for this is on GitHub @ /src/test/kotlin/junit/practical/WebServiceTest.kt.

Righto.

--
Adam