Friday, 23 September 2022

Kotlin: abstract classes, backing fields/properties, lateinit, and frustration

G'day:

I'm just continuing on in the Kotlin "Classes" docs, carrying on from my earlier article "Kotlin: investigating classes".


Abstract classes

I have pretty much run with the example in the docs just to see it in action. I don't wanna copy and paste stuff cos I don't learn so well just by looking at the code. Making myself remember what I need to type - even if it's just class syntax with a method or two - helps me learn it.

The thing the example had was to have an abstract class that has a concrete implementation of a method, extended by another abstract class that actually makes that method abstract again, followed by a concrete implementation class. To get my brain around that, I wrote it in Java first (all this experimentation is making my Java better too btw: I'm writing more Java-comparison code for these articles than I'm necessarily showing in them). My Java is limited to one-off class files with a main method though. I can't be arsed setting up a Java project in IntelliJ and working out how to get JUnit working, etc. Sorry. (Dammit I feel guilty now). Anyway, Java version:

public class TestAbstract {
    public static void main(String[] args){
        MyWildShape mws = new MyWildShape();
        mws.draw();
    }
}

abstract class Shape {
    public void draw() {
        System.out.println("Shape.draw"); // MyWildShape.draw
    }
}


abstract class WildShape extends Shape {
    abstract public void draw();
}  


class MyWildShape extends WildShape {
    @Override
    public void draw() {
        System.out.println("MyWildShape.draw");
    }
}

And the Kotlin version:

abstract class Shape {
    open fun draw() = print("Shape.draw")
}

abstract class WildShape : Shape() {
    abstract override fun draw()
}

class MyWildShape : WildShape() {
     override fun draw() = print("MyWildShape.draw")
}

class AbstractTest : DescribeSpec({
    describe("abstract test") {
        it("permits concrete subclasses to override abstract methods that are not overridden in the base class") {
            val mws = MyWildShape()
            val output = SystemLambda.tapSystemOut {
                mws.draw()
            }
            output shouldBe "MyWildShape.draw"
        }
    }
})

I wondered what the need for open was in that. Kotlin's been pretty good an letting boilerplate be inferrable, and this seemed like boilerplate. I found the answer in the docs:

By default, Kotlin classes are final – they can't be inherited. To make a class inheritable, mark it with the open keyword

Kotlin / Concepts / Classes and objects / Inheritance

As to why that's the case, I had to dig slightly deeper, but I found a discussion on it here: Classes final by default, and at the top of that discussion someone links to the book "Effective Java" wherein I guess someone recommends making stuff final by default. I'm not entirely convinced with this as it kinda presupposes all code is written in a specific way: implemented to an interface, and nothing every expects a concrete implementation, basically. I'm OK writing my code like that, however there's no guarantee the author of some lib I need to use also has. And in my experience (granted: in PHP) it's not the case. But so be it. I guess it does encourage one to stop and think. And design to interfaces. Which is a good thing.


Companion objects

I messed with these in a coupla other articles:

I've nothing else I feel I need to check-out on these right now.


Back to backing fields (and backing properties)

Fields

In an earlier article ("Kotlin: there's no such thing as static, apparently"), I learned that a property has a backing field: the underlying variable that stores the value. One just refers to it as field in accessor implementations:

class Person {
    var firstName = ""
        get() = field
        set(value) {
            field = value
        }
    var lastName = ""
        get() = field
        set(value) {
            field = value
        }
}

Under the hood Kotlin is creating appropriately-named discrete private variables for each property's field.

One cool thing IntelliJ can do is to show the bytecode that the Kotlin code would generate (Tools › Kotlin › Show Kotlin Byte Code), and from there one can decompile that back to Java. The result in this case being (I've trimmed some irrelevant stuff):

final class Person {
   private String firstName = "";
   private String lastName = "";

   public final String getFirstName() {
      return this.firstName;
   }

   public final void setFirstName(String value) {
      this.firstName = value;
   }

   public final String getLastName() {
      return this.lastName;
   }

   public final void setLastName(String value) {
      this.lastName = value;
   }
}

One thing I wondered about before when I found myself getting a stack overflow when doing this:

var firstName = ""
    get() = field
    set(value) {
    	// was field = value
        firstName = value
    }

Whether this was because my reference to firstName there was actually calling "itself". It turns out it is, and IntelliJ was even telling me:


Backing properties

This was the next thing that threw me. The docs also talk about backing properties, and really didn't make it at all clear why these existed.

If you want to do something that does not fit into this implicit backing field scheme, you can always fall back to having a backing property

They give an example, and I could understand the code, but it didn't really sink in as to what specifically showing me. I did a google and was pleased to find out I was not the only person a bit flumoxed by all this, and found a good answer here: I have a hard time understanding the purpose of a "backing property". And I was able to contrive a test case to demonstrate it (largely to myself) in action:

class MyList {
    private var _myList = mutableListOf<String>()
    val immutable: List<String>
        get() = _myList
    var mutable: MutableList<String> = mutableListOf()
        set(value) {
            field = value
            _myList = value
        }
}

There's really no trick to it. It's just when an accessor method needs to store its value in some other variable, not its backing field. In the example above I have a class that has a property called mutable that has a setter which will take a MutableList of strings, and another property called immutable which actually accesses the same underlying variable, but returns it as an immutable List. This is a daft example, but it demonstrates the point. I've two properties, and they both access the same underlying data, so it has to be in a specific private property rather than using field which intrinsically is a different variable for each property. Also note I am setting mutable's backing field too. Not for any reason, just to show I can.

Of course I wrote a test for this:

class BackingPropertiesTest : DescribeSpec({
    it("should take a MutableList and return an ImmutableList") {
        val mutableList = mutableListOf("tahi", "rua", "toru")
        val myList = MyList()
        myList.mutable = mutableList

        myList.immutable.shouldBeInstanceOf<List<String>>()
        myList.immutable shouldBe listOf("tahi", "rua", "toru")

        mutableList.add("wha")
        myList.immutable.shouldBeInstanceOf<List<String>>()
        myList.immutable shouldBe listOf("tahi", "rua", "toru", "wha")
    }
})

Straight forward stuff: just demonstrating what the code is supposed to be doing. One thing I thought was neat was how I could append that new element to the mutable list, and when I accessed it again, I got a new immutable list out, also with my new value. It almost seems like I am changing the immutable list, but of course it's a new list each time.


lateinit

This is the solution to a problem that I had to work around the other day. In the end I was "just doing it wrong" for the given situation (I forget what it was now), but I had thought "but I don't want to initalise that with a non-null value now just to satisfy the compiler. I'll deal with it later. Trust me". And of course the "trust me" part is implemented via the lateinit modifier.

class Colour(val en: String) {
    lateinit var mi: String

    fun setMaori(value: String) {
      mi = value
    }
}

Here I have then English version of the colour as a normal property handled by the primary constructor, and another property mi representing the Maori word for the colour, and that isn't handled by any constructor, and is also not initialised. It's happy to stay undefined until "later". If I didn't have the lateinit there, the code would not even compile:

e: [...]\LateInitPropertiesTest.kt: (52, 5): Property must be initialized or be abstract

IntelliJ makes it clear too:

I have a coupla tests demonstrating this in action:

it("allows me to create a colour object without an mi property") {
    val red  = Colour("red")
    red.en shouldBe "red"
}

it("lets me set the mi property later") {
    val orange  = Colour("orange")
    orange.setMaori("karaka")

    orange.mi shouldBe "karaka"
}

What if I try to read that property before it's been initialised?

it("throws UninitializedPropertyAccessException if access mi before it's been set") {
    val yellow  = Colour("yellow")

    val exception = shouldThrow<UninitializedPropertyAccessException> {
        yellow.mi shouldNotBe "kōwhai"
    }
    exception.message shouldBe "lateinit property mi has not been initialized"
}

It throws an exception at runtime. Cool. And it's a specific exception, with a clear message.

For good measure I also checked with a method call with the reference to mi within the class. I did not expect any different behaviour, but I like to check these things.

class Colour(val en: String) {
    lateinit var mi: String

    fun setMaori(value: String) {
      mi = value
    }

    fun getMaori() = mi
}
it("throws UninitializedPropertyAccessException if use mi before it's been set") {
    val green  = Colour("green")

    val exception = shouldThrow<UninitializedPropertyAccessException> {
        green.getMaori() shouldNotBe "kakariki"
    }
    exception.message shouldBe "lateinit property mi has not been initialized"
}

One can also check to see if the property is initialised before using it:

class Colour(val en: String) {
    lateinit var mi: String

    fun setMaori(value: String) {
      mi = value
    }

    fun getMaori() = mi

    fun isMiInitialized() = ::mi.isInitialized
}
it("can have initialisation status checked") {
    val green  = Colour("green")

    green.isMiInitialized() shouldBe false
    green.setMaori("kakariki")
    green.isMiInitialized() shouldBe true
}

I thought I might be able to go green::mi.isInitialized, but I get told off:

Looking in the docs, this is because:

This check is only available for properties that are lexically accessible when declared in the same type, in one of the outer types, or at top level in the same file.

Ooookay. I cannot actually work out how to write code where I can call isInitialized where the property is "declared… in one of the outer types, or at top level in the same file". I always get "Backing field of 'var someValue: String' is not accessible at this point". And that is a coupla hours or so of my life I am never going to get back.


And as that was a bit frustrating at the end there, I'm gonna go and do something else now. You need to picture me with my bottom lip jutted-out in a put-upon way when I say that. I tell you: if I had a ball, I would take it and go home ;-)

Righto.

--
Adam

Tuesday, 20 September 2022

CFWheels: a recommendation for their dev team

 G'day

I'm going to pass this on to the CFWheels Dev Team, but it's applicable in anyone's code, so posting it here first.

A lot of the code in the CFWheels codebase hasn't really been "designed". It's been written as if the developer concerned just went "OK, I have this file open… I'll start typing code in this file…", rather than giving thought to the notions of design patterns or the SOLID principles and stuff like that that exist to keep codebases… usable.

It's legacy code, and we have all written code like this in the past, so let's not dwell too much on how the codebase got to where it is now. Let's just accept that it could be better than it is.

But It's 2022 now, and there's no real excuse for perpetuating coding practices like this.

An example in front of me ATM is the onApplicationStart method. It is 1000 lines long. That is ludicrous.

The design problem in this particular method (and this is the same for a lot of code in the CFWheels codebase) is that it confuses "the things I need to get done", with "how to do those things". It's onApplicationStart's job to… start the application. There are numerous steps to this, but instead of it just calling all the steps it needs to call to define "starting the application", it also has all the implementation of the steps inline in it as well. This is an anti-pattern.

One can even see what each of the steps are. There are comments identifying them. In fact the first two are done "right":

// Abort if called from incorrect file.
$abortInvalidRequest();
// Setup the CFWheels storage struct for the current request.
$initializeRequestScope();

(Except the comments here are pointless as they simply repeat what the method names already say clearly).

But then the wheels (sic) fall off:

if (StructKeyExists(application, "wheels")) {
    // Set or reset all settings but make sure to pass along the reload password between forced reloads with "reload=x".
    if (StructKeyExists(application.wheels, "reloadPassword")) {
        local.oldReloadPassword = application.wheels.reloadPassword;
    }
    // Check old environment for environment switch
    if (StructKeyExists(application.wheels, "allowEnvironmentSwitchViaUrl")) {
        local.allowEnvironmentSwitchViaUrl = application.wheels.allowEnvironmentSwitchViaUrl;
        local.oldEnvironment = application.wheels.environment;
    }
}
application.$wheels = {};
if (StructKeyExists(local, "oldReloadPassword")) {
    application.$wheels.reloadPassword = local.oldReloadPassword;
}

That could should not be inline. There should be a method call inline, called something like configureReloadOptions (or whatever this is doing… a case in point is that I can't really tell).

The thing is onApplicationStart should not be defining the implementation of this, something else should be doing that, and onApplicationStart should just know how to call it.

Why is this an issue?

Because for us, one of the sections of processing in onApplicationStart is wrong, and sufficiently wrong it's been flagged up as a pen vector.

If each part of the application-start-up sequence was in its own functions, then we could override the function with our own implementation to address the issue. And the change would be local to that one small function, so the testing surface area would be small. But instead I now have to hack a third-party codebase to solve this. It's 100s of lines into this method, and accordingly its test surface area is huge (and largely unreachable, I think).

This is basic Open/Closed Principle stuff. The O in SOLID.

My recommendation / rule of thumb is that whenever one is tempted to put a comment in that explains a block of code (or delimit it from other blocks), then that block of code should be in its own function. If one follows this rule of thumb, one seldom gets into the situation that one has Open/Closed Principle issues. It's not foolproof, but it's a good start.

Also having a code design rule that functions should start to raise a flag if they are over 20 lines long. They are probably doing too much, or aren't factored well. If they are more than 50 lines? Stop. Recode them.

If a function has unrelated flow-control blocks in them? Also a red flag. Each block is likely more appropriate to be in their own functions.

A function like onApplicationStart that needs to carry-out a _lot_ of steps to consider the app to be "started" should look something like this:

function onApplicationStart() {
    doTheThing()
    doAnotherThing()
    doSomethingElse()
    doThisThingRelatingToStuff()
    doOtherStuff()
    doThis()
    doThat()
   // ...
}

It really shouldn't be implementing any stuff or things or this or that itself.

Lastly, someone drew my attention to this statement on the CFWheels home page:

Good Organization

Stop thinking about how to organize your code and deal with your business specific problems instead.

This possibly explains a lot of the way the CFWheels codebase got to be the way it is. It is really bad advice. One should always be thinking about how one's application ought to be designed. It's important.

NB: this is not just related to onApplicationStart. That's just what currently has me sunk. It relates to really a lot of the code in the codebase.

This is also something that can be tackled in a piecemeal fashion. If one has to maintain some code in a really long method: extract the block of code in question into its own function, and maintain (and test that).

Righto.

--
Adam

Monday, 19 September 2022

Kotlin: investigating classes

G'day:

In my other random explorations of the Kotlin language, I've already used simple classes to facilitate testing other language features, but my next koans exercise (man I am progressing slowly with those!) links to the docs page for classes, so I'm gonna have a breeze through that, so I can extend my understanding of how they operate beyond the superficial level I currently have. Anything that intrigues me or is non-obvious, I'll write a test and sling it in here.


Primary and secondary constructors

I touched on this in "Kotlin: another Friday afternoon, another round of random investigation › Primary and secondary constructors", but only in as much as I messed up my syntax for my constructor and IntelliJ fixed it for me.

Primary constructor

This is baked-into the class declaration:

class Suffragist(firstName:String, lastName:String) {
    val fullName = "$firstName $lastName"
}

class ClassesTest : DescribeSpec({
    describe("Constructor tests") {
        it("takes values that can be used in initialisation code") {
            val suffragist = Suffragist("Kate", "Sheppard")

            suffragist.fullName shouldBe "Kate Sheppard"
        }
    }
})

It's important to understand that the primary constructor here is just the bit in parentheses. The braces are not the constructor implementation, that is the class implementation, for example in Java that constructor (in context) would be:

class Suffragist {

    String firstName;
    String lastName;

    public Suffragist(String firstName, String lastName) {
        this.firstName = firstName;
        this.lastName = lastName;
    }
}

So the primary constructor identifies the properties needed for initialisation of the object.

In my class above, firstName and lastName are not properties of the Suffragist object. In my test I could not do this:

suffragist.firstName shouldBe "Kate"
suffragist.lastName shouldBe "Sheppard"

This gives a compile error:

Unresolved reference: firstName

To make them properties, I need to put a val or var qualifier in the primary constructor:

class Scientist(val firstName:String, val lastName:String)

class ClassesTest : DescribeSpec({
    describe("Constructor tests") {
        // ...
        it("needs to qualify params to make them properties") {
            val scientist = Scientist("Siouxsie", "Wiles")

            scientist.firstName shouldBe "Siouxsie"
            scientist.lastName shouldBe "Wiles"
        }
    }
})

Back to the first example:

class Suffragist(firstName:String, lastName:String) {
    val fullName = "$firstName $lastName"
}

That line is not part of the constructor, it's initialising the fullName property. It just happens to have access to the firstName and lastName values from the constructor when it's executed.


Initializer blocks

As the primary constructor can't have any code in it, any code that needs to run when the object is created needs to be in initializer blocks in the class body:

class Politician(firstName:String, lastName:String) {
    init {
        println("First name: $firstName")
    }
    init {
        println("Last name: $lastName")
    }
}

class ClassesTest : DescribeSpec({
    describe("Constructor tests") {
        
        // ...
        
        it("can have code in initializer blocks") {
            val EOL = System.lineSeparator()
            val output = SystemLambda.tapSystemOut {
                Politician("Jacinda", "Ardern")
            }

            output shouldBe "First name: Jacinda${EOL}Last name: Ardern${EOL}"
        }
    }
})

Unsurprisingly they are executed in the order they are in the file.


Secondary constructors

Kotlin can have secondary constructors if one needs to do more than take some arguments and pass them to initializer blocks. Secondary constructors use the constructor keyword to define their behaviour.

Here I have a secondary constructor that takes a map of firstName / lastName pairs, and this extracts the values from the map for each of the firstName / lastName parameters of the primary constructor.

class FilmMaker(firstName:String, lastName:String) {
    val fullName = "$firstName $lastName"
    constructor(names: Map<String, String>) : this(names["firstName"]!!, names["lastName"]!!)
}

class ClassesTest : DescribeSpec({
    describe("Constructor tests") {

        // ...
        
        it("can have a secondary constructor") {
            val names = mapOf(Pair("firstName", "Jane"), "lastName" to "Campion")
            val filmMaker = FilmMaker(names)

            filmMaker.fullName shouldBe "Jane Campion"
        }
    }
})

It's important to note that because I have a primary constructor defined, then ultimately one of the secondary constructors must call it, even if there's a sequence of secondary constructors calling one another in a chain, ultimately the primary constructor must be called.

I did wonder why it's not possible to do this sort of thing:

constructor(names: Map<String, String>) {
    val firstName = names["firstName"]
    val lastName = names["lastName"]

    this(firstName!!, lastName!!)
}

IE: to have the secondary constructor do some work and then call the primary constructor. I could not find an answer for this.

BTW, that !! operator is a not-null assertion operator. In this usage it tells the compiler to error-out if the map doesn't have the key/value pair I'm using there. Otherwise I'd need to make the following code handle nulls for those values, and that is incorrect.


Here's an example where there's no primary constructor, which enables a secondary constructor to do more than simply invoke the primary one, inline:

class ChiefJustice {
    var firstName = ""
    var lastName = ""
    var fullName = ""

    constructor(firstName:String, lastName:String) {
        this.firstName = firstName
        this.lastName = lastName

        fullName = "$firstName $lastName"
    }

    constructor(names: List<String>) : this(names.first(), names.last())
}

class ClassesTest : DescribeSpec({
    describe("Constructor tests") {

        // ...

        it("can call another secondary constructor") {
            val names = listOf("Helen", "Winkelmann")
            val chiefJustice = ChiefJustice(names)

            chiefJustice.fullName shouldBe "Helen Winkelmann"
        }
    }
})

Here we have two secondary constructors. One takes separate String values for the properties, and the second takes a list of names, and extracts first and second to pass to the previous secondary constructor to actually then do the initialisation work.


Companion objects and factory methods

I touched on companion objects before in my "Kotlin: there's no such thing as static, apparently" article. One thing I read a few times when reading about secondary constructors, is that they're a bit frowned-up for situations where there's constructors taking different repesentations of the property data values (like all the examples I offer here!). Instead of having a constructor that takes a map of name/value pairs, one would instead have a factory method, using a companion object. So instead of this example:

class FilmMaker(val firstName:String, val lastName:String) {
    val fullName = "$firstName $lastName"
    constructor(names: Map<String, String>) : this(names["firstName"]!!, names["lastName"]!!)
}

One might better have:

class FilmMaker(firstName:String, lastName:String) {
    val fullName = "$firstName $lastName"

    companion object Factory {
        fun createFromMap(names: Map<String, String>) = FilmMaker(names["firstName"]!!, names["lastName"]!!)
    }
}

class ClassesTest : DescribeSpec({
    describe("Constructor tests") {
        it("can have a secondary constructor") {
            val names = mapOf(Pair("firstName", "Jane"), "lastName" to "Campion")
            val filmMaker = FilmMaker.createFromMap(names)

            filmMaker.fullName shouldBe "Jane Campion"
        }
    }
})

Now I have a factory method createFromMap which is a bit more clear in its intent / purpose than having multiple vague constructors.


I'm gonna leave it there today. There's some more sections on that docs page regarding inheritance and abstract classes and stuff, but I'm not in the mood for that right now. Am struggling a bit with focus just now: not quite sure why.

The code for today is on GitHub as ClassesTest.kt (1.5 tag) except the last example which was a refactor of the earlier code, and is tagged as 1.5.1: ClassesTest.kt.

Righto.

--
Adam

Thursday, 15 September 2022

Kotlin: scope functions (mostly)

G'day:

More Kotlin learning/investigation, following on from these other Kotlin articles. This time I didn't even get to start the next koans exercise before being sidetracked.


Scope functions

There's a bunch of functions - let, with, run, apply, and also - that one calls on an object; they each take a lambda, and within the lambda expressions act on the object either via it (a parameter of the lambda), or this (a reference to the object), or simply implicitly (no reference to the object at all!). They all seem to do similar stuff - the docs even say so - they're all just variants that are possibly better-suited to one use case or another. I have written a test for each of them, keeping the "feature" being tested similar across all of them to show the differences.

let

it("provides an object to run the statements in the lambda with") {
    val result = listOf(
        Colour("whero"),
        Colour("karaka"),
        Colour("kōwhai")
    ).first()
    .let {
        it.english = "red"
        "${it.maori}: ${it.english}"
    }
    result shouldBe "whero: red"
}

The intent of let is to be able to chuck it in a chain of other method calls. In this contrived example I have a list of colours, I get the first one, and I call let on that. The let lambda just fills in the English value, and returns a string of "[Maori]: [English]" for that colour.


with

it("calls the statements in the lambda with the given object") {
    val green = Colour()

    with(green) {
        maori = "kākāriki"
        english = "green"
    }
    "${green.maori}: ${green.english}" shouldBe "kākāriki: green"
}

with is for when one wants to go "with this object, do some stuff to it". Here the statements are applied to this (the argument of with), implicitly. I could explicitly say this.maori = "kākāriki". I also found the term for those functions that are just functions (doSomething(someObject)), rather than being used like how I'd expect a method to be used: someObject.doSomething(). The term is "non-extension function".

It can also return a value (as per usual with lambdas in Kotlin: the last expression is returned):

it("will return something") {
    val purple = with(Colour()) {
        maori = "papura"
        english = "purple"

        this
    }
    "${purple.maori}: ${purple.english}" shouldBe "papura: purple"
}

For the return, I need to be explicit about this otherwise Kotlin won't know I mean to return it.

run

This is like a combo of with and let: it's an extension function, so can be called in a chain, but it receives the value of the chain as its (implicit) this:

it("applies the statements in the lambda to the object") {
    val blue = listOf(
        Colour("whero"),
        Colour("kākāriki"),
        Colour("kikorangi")
    ).last()
    .run {
        english = "blue"

        this
    }
    "${blue.maori}: ${blue.english}" shouldBe "kikorangi: blue"
}

This is similar to the let example except no need to use "it" when referencing the object.


apply

describe("Tests of apply function") {
    it("applies the assignments in the lambda to the object") {
        val yellow = Colour().apply {
            maori = "kōwhai"
            english = "yellow"
        }
        "${yellow.maori}: ${yellow.english}" shouldBe "kōwhai: yellow"
    }
}

apply seems like an extension function (look at me with my new jargon) version of with. It implicitly returns the original object though. So - and I'm just paraphrasing the docs here - it's idiomatic usage seems to be for configuring an object. One is applying the assignemnts in the lambda to the object.


also

it("also runs the statements in the block with the object") {
    val indigo = Colour().also {
        it.maori = "tūāuri"
        it.english = "indigo"
    }
    "${indigo.maori}: ${indigo.english}" shouldBe "tūāuri: indigo"
}

I'm just gonna quote the docs for this one, as it describes it kinda like how it occurred to me to do so:

When you see also in the code, you can read it as "and also do the following with the object."

Cos it's an extension function, it too could be slung in the middle of some other chain of calls. NB: it implicitly returns the original object.


Callable references

Something cool that the docs used which I will repeat here is passing the object to a callable reference to a function:

it("can use a callable reference to use a function as the callback") {
    val orange = Colour("karaka", "orange")

    val output = SystemLambda.tapSystemOut {
        "${orange.maori}: ${orange.english}".also(::print)
    }

    output shouldBe "karaka: orange"
}

See how I'm passing my string to also, and the lambda is passing it to a reference to print. One would not do this in real life, but it demonstrates the point. If one prefixes a function with the :: operator, one gets a callable reference to that function.


takeIf and takeUnless

The "Scope functions" page of the docs has these two tacked onto the end of it, so I'll do the same.

takeIf

it("returns the object if the predicate is true, otherwise null") {
    var number = 210

    var checked = number.takeIf { it % 2 == 0 }
    checked shouldBe number

    checked = checked?.takeIf { it % 3 == 0 }
    checked shouldBe number

    checked = checked?.takeIf { it % 5 == 0 }
    checked shouldBe number

    checked = checked?.takeIf { it % 7 == 0 }
    checked shouldBe number

    checked = checked?.takeIf { it % 11 == 0 }
    checked.shouldBeNull()
}

The test case kinda explains what this one does. And I admit the actual test is a bit daft.

Note how I need to use the null-safe operator after the first takeIf expression, as the object could be null after that. The code won't compile unless I do. Indeed IntelliJ tells me when I've got it wrong:

(I am really loving the static type-checking in Kotlin, btw).


takeUnless

takeUnless works the other way around: it returns the object if the predicate is false (otherwise null). Why have both? Sometimes the code just reads better with different wording.

it("returns the object if the predicate is false, otherwise null") {
    val number = 210

    number.asClue {
        var checked = it.takeUnless { it % 2 != 0 }
        withClue("should be a multiple of 2") {
            checked shouldBe it
        }

        checked = checked?.takeUnless { it % 3 != 0 }
        withClue("should be a multiple of 3") {
            checked shouldBe it
        }

        checked = checked?.takeUnless { it % 5 != 0 }
        withClue("should be a multiple of 5") {
            checked shouldBe it
        }

        checked = checked?.takeUnless { it % 7 != 0 }
        withClue("should be a multiple of 7") {
            checked shouldBe it
        }

        checked = checked?.takeUnless { it % 11 != 0 }
        withClue("should NOT be a multiple of 11") {
            checked.shouldBeNull()
        }
    }
}

For a change here I'm using those asClue and withClue constructs that Kotest has, as mentioned in the previous article. To recap: this just adds context and hinting to test failures. Let me break a condition there to show you. I've slung this in as the last condition in that previous test:

checked = checked?.takeUnless { it % 13 != 0 }
withClue("should be a multiple of 13") {
    checked shouldBe it
}

And this yields:

210
java.lang.AssertionError: 210
should be a multiple of 13
Expected 210 but actual was null

And that was the bottom of that page of docs, and this is long enough, so I'll leave it here. I suspect I will be back tomorrow with more…

All the code shown here is on Github: /src/test/kotlin/language/ScopeFunctionsTest.kt

Righto.

--
Adam

Something for my Namesake

https://tinyurl.com/45karnk7

Wednesday, 14 September 2022

CFML: working MySQL datasource in Application.cfc (this is just a note-to-self/Google)

G'day:

No content in this one, I just want something I can find when I search for how to config a MySQL datasource in Application.cfc in ColdFusion. I always forget/lose the code, and I can never find anything on Google. It'd be marvellous of Adobe would document this stuff, but it's a bit much to expect of them, I guess.

component {

    setsettings()
    loadDatasources()

    private void function setSettings() {
        this.name = "app1"
    }

    private void function loadDataSources() {
        this.datasources["dsn1"] = {
            driver = "mysql",
            class = "com.mysql.jdbc.Driver",
            url = "jdbc:mysql://database.backend:3306/"
                & "#server.system.environment.MARIADB_DATABASE#"
                & "?useUnicode=true&characterEncoding=UTF-8",
            username = server.system.environment.MARIADB_USER,
            password = server.system.environment.MARIADB_PASSWORD
        }
        this.datasource = "dsn1"
    }
}

The bit I always forget is the class bit, so I end up with an error along these lines:

No driver or URL found.

java.sql.SQLException: No driver or URL found.
 at coldfusion.server.j2ee.sql.pool.JDBCPool.createPhysicalConnection(JDBCPool.java:586)
 at coldfusion.server.j2ee.sql.pool.ConnectionRunner$RunnableConnection.run(ConnectionRunner.java:67)
 at java.base/java.lang.Thread.run(Thread.java:834)

This is also on GitHub @ Application.cfc.

Righto.

--
Adam

Saturday, 10 September 2022

Kotlin: looking at Numbers, more Kotest stuff, and something about "function literals with receivers"?

G'day:

Here's some new unguided Kotlin experimentation, following on from other similar articles.

Firstly I inadvertantly finished yesterday's article when I was sitting on the last exercise in the first set of Kotlin Koans exercises, and finishing that off was about a dozen obvious keystrokes (it % 2 == 0) so I probably should have finished that article triumphantly with that.

Instead I look at it this morning and went "oh that was super easy, barely an inconvenience", and finished it off. So much for story pacing.


Numbers in Kotlin

The code for these examples is on GitHub: NumberTest.kt.

I did need to do a bit of research to nail those 11 keystrokes for the last exercise. It was to finish a lambda that was the predicate for an any call to determine there there were any even numbers in a set. I initially wondered - given Kotlin seems to have a method for everything else - whether it might have an isEven method, so looked into that. No, it doesn't. Just use modulo division. But anyway, I landed on the docs page for Numbers, and had a look at that.

As always, my investigation was implemented via tests:

it("demonstrates ints and longs of same 'human value' are not equal") {
    val i :Int = 24
    val l :Long = 24L

    shouldThrow<AssertionError> {
        i.equals(l).shouldBeTrue()
    }
}

Note I don't need the explicit types in the variable declaration there, I felt it made it more clear what I was doing. Note the L suffix on the Long literal.


it("demonstrates Kotest will handle the casting between ints and longs internally") {
    val i :Int = 17
    val l :Long = 17L

    i shouldBe l
}

it("demonstrates Kotest will handle the casting between ints and floats internally") {
    val i :Int = 17
    val f :Float = 17.0F

    i shouldBe f
}

it("demonstrates Kotest will handle the casting between ints and doubles internally") {
    val i :Int = 17
    val d :Double = 17.0

    i shouldBe d
}

Am quite pleased Kotest takes care of the type check so disparate types can be checked in a human sense. Note the difference in literals for floats (F suffix) and Doubles (just a decimal number).


it("demonstrates one can convert an int to a double") {
    val i :Int = 17
    val d :Double = 17.0

    i.toDouble().equals(d).shouldBeTrue()
}

There are conversion methods to/from all numeric types.


it("demonstrates literals can have _ as a separator") {
    val i :Int = 123_456_789

    i shouldBe 123456789
}

it("demonstrates _ is not just a 1000s separator") {
    val d :Double = 3_6_5.2_4_2_5

    d shouldBe 365.2425
}

One thoughtful thing is that for clarity one can put an underscore in any numeric literal to make groupings clear. Not that my latter example is more clear than just 365.2425, but it's demonstrating the point.


it("has binary and hex literals") {
    val b = 0b1111_1111_1111_1111
    var h = 0xff_ff

    b shouldBe h
}

it("demonstrates binary and hex numbers of same 'human value' ARE equal") {
    val b = 0b1111_1111_1111_1111
    val h = 0xff_ff

    b.equals(h).shouldBeTrue()
    b.shouldBeTypeOf<Int>()
    h.shouldBeTypeOf<Int>()
}

Kotlin also has binary and hex literals (but not octal ones. No great loss in my books). Those literals are just ways of defining an Int, they are not their own numeric sub-type.

There's undoubtedly more to look at with numbers, but my attention got shifted when working out how that shouldBeTypeOf<Int> matcher works.


Type-checking in Kotest matchers

The noteworthy ones are:

  • shouldBeTypeOf<type> matches the exact type (subclasses will not match).
  • shouldBeInstanceOf<type> matches within the type-hierarchy (subclasses will match).

Examples:

describe("Tests of type-checking") {
    it("checks exact type with shouldBeTypeOf") {
        val i = 24

        i.shouldBeTypeOf<Int>()
    }

    it("checks subtype with shouldBeInstanceOf") {
        val i = 17

        shouldThrow<AssertionError> {
            i.shouldBeTypeOf<Number>()
        }
        i.shouldBeInstanceOf<Number>()
    }
}

The code for this example is on GitHub: MatcherTest.kt.


Kotest clues

One neat trick one can do in Kotest is to provide a clue for an expectation:

it("can fail with a clue") {
    try {
        withClue("this is the clue") {
            fail("this is a failure message")
        }
    } catch (e :AssertionError) {
        e.message shouldBe """
            this is the clue
            this is a failure message
        """.trimIndent()
    }
}

This is useful both in the code to make it more clear why we're bothering to assert what we are, and it also comes out in the test failure feedback too. This is what I get if I take out the try/catch from that test above, and run it:

C:\Users\camer\.jdks\openjdk-17.0.1\bin\java.exe
Testing started at 16:28 ...

Misc Kotest tests -- emits a failure message is excluded by test filter(s): Excluded by test path filter: 'Misc Kotest tests -- can fail with a clue'

this is the clue
java.lang.AssertionError: this is the clue
this is a failure message
	at system.kotest.KotestTest$1$1$2.invokeSuspend(KotestTest.kt:25)

I realise Jasmine-esque test framework support this, eg:

expect(actual).toBe(expected, "careful now")

So does xUnit-ish frameworks:

asertEquals(expected, actual, "down with this sort of thing")

But I think Kotest's approach is clearer / tidier / less-of-an-afterthought.


One can also use the asClue method to pass an object into a block of expectations, and that object is then reflected as the clue for the failure:

it("can attach asClue to an object to use it for multiple expectations") {
    val colours = listOf("whero","karaka","kōwhai","kākāriki","kikorangi","poropango","papura")

    try {
        colours.asClue {
            withClue("should not contain green") {
                it.none{it == "kākāriki"}.shouldBeTrue()
            }
        }
    } catch (e :AssertionError) {
        e.message shouldBe """
            [whero, karaka, kōwhai, kākāriki, kikorangi, poropango, papura]
            should not contain green
            expected:<true> but was:<false>
        """.trimIndent()
    }
}

Here I have a clue on the object, and also on the specific expecation.

Note the outer it is a reference to the object being used as the clue. The inner it is a reference to current element of iterating over… the outer it… for the none (none iterates over a collection and returns false as soon as the predicate returns true of any element). The it usage is slightly confusing there, but that's down to me being lazy. And so I could write this paragraph. I could have written that block with some clearer parameter names:

colours.asClue {
    colours ->
    withClue("should not contain green") {
        colours.none{colour -> colour == "kākāriki"}.shouldBeTrue()
    }
}

TBH, I don't know that that is any better.

The code for these examples is on GitHub: KotestTest.kt.


Kotest soft assertions

This solves a problem that has caused me grief a lot of times with other testing frameworks. A test demonstrates how this works:

it("can use soft assertions to let multiple assertions fail and report back on all of them") {
    val actual :Int = 15

    try {
        assertSoftly {
            actual.shouldBeTypeOf<Number>()
            withClue("should be 15.0") {
                actual.equals(15.0).shouldBeTrue()
            }
            actual shouldBe 15
            actual shouldBe 16
        }
    } catch (e :MultiAssertionError) {
        assertSoftly(e.message) {
            shouldContain("1) 15 should be of type kotlin.Number")
            shouldContain("2) should be 15.0")
            shouldContain("expected:<true> but was:<false>")
            shouldContain("3) expected:<16> but was:<15>")
        }
    }
}

Here all of those expectations fail. On other testing frameworks I have used they always bail-out as soon as the first expectation fails, and the rest aren't tested. Often this is fine and exactly correct because subsequent expectations might rely on earlier ones passing. But this is not always they case. Obvs one could create one's own custom expectation to check a whole wodge of stuff at once, but it's nicer to have a simple block construct that JFDI.

Another thing to note is the two forms of assertSoftly. The first rendition just takes a lambda with no argument, and the body of the lambda is implemented the same as one usually would in a test. The second variant takes and argument, and that argument is used as the input for all the expectations. I actually dunno how this syntax works, and I have some reading to do. For now: handy to know, and a good reduction of code repetition. And is also encourages the expectations to stay focused, given they must all be applied to the same object.

Be mindful that my test is contived there: it's testing the behaviour of the failure, rather than being a failing test. Normally the test would be more like this:

it("can use soft assertions to let multiple assertions fail and report back on all of them") {
    val actual :Int = 15

    assertSoftly {
        actual.shouldBeTypeOf<Number>()
        withClue("should be 15.0") {
            actual.equals(15.0).shouldBeTrue()
        }
        actual shouldBe 15
        actual shouldBe 16
    }
}

And given that's a failing test now, we get that in the test-run feedback:

io.kotest.assertions.MultiAssertionError: 
The following 3 assertions failed:
1) 15 should be of type kotlin.Number
	at system.kotest.KotestTest$1$2$1.invokeSuspend(KotestTest.kt:78)
2) should be 15.0
expected:<true> but was:<false>
	at system.kotest.KotestTest$1$2$1.invokeSuspend(KotestTest.kt:63)
3) expected:<16> but was:<15>
	at system.kotest.KotestTest$1$2$1.invokeSuspend(KotestTest.kt:66)

Good stuff.

The code for these examples is on GitHub: KotestTest.kt.


Kotest shouldThrowMessage

This is just a quick one. I didn't think it fitted in any of the other sections. I'm catching a lot of exceptions to test how things "fail" in tests, but still have a passing test. Generally I've been expectimng a specofoc exception type to be thrown, but one can also just look for a specific message:

it("should throw an exception with a specific message") {
    class InquisitionException(message: String?) : Exception(message)

    shouldThrowMessage("No-one expects the... InquisitionException") {
        throw InquisitionException("No-one expects the... InquisitionException")
    }
}

Handy.

The code for this example is on GitHub: MatcherTest.kt.


Function literals with receivers

Further up I was bemused by how this code works:

assertSoftly(e.message) {
    shouldContain("1) 15 should be of type kotlin.Number")
    shouldContain("2) should be 15.0")
    shouldContain("expected:<true> but was:<false>")
    shouldContain("3) expected:<16> but was:<15>")
}

Somehow shouldContain is implicitly being called on the e.message argument I am passing to assertSoftly. IE: to my ignorant eyes I am expecting to be seeing something like this:

assertSoftly(e.message) {
    someThingHere shouldContain("1) 15 should be of type kotlin.Number")
    someThingHere shouldContain("2) should be 15.0")
    someThingHere shouldContain("expected:<true> but was:<false>")
    someThingHere shouldContain("3) expected:<16> but was:<15>")
}

Turns out it's not just magic, it's a feature of function literals in Kotlin. Now I only barely grasp what these docs are saying, but I've made enough sense of them to write some code to test it:

Function types with receiver, such as A.(B) -> C, can be instantiated with a special form of function literals – function literals with receiver.

As mentioned above, Kotlin provides the ability to call an instance of a function type with receiver while providing the receiver object.

Inside the body of the function literal, the receiver object passed to a call becomes an implicit this, so that you can access the members of that receiver object without any additional qualifiers, or access the receiver object using a this expression.

Just to start with, here's a no-surprises function literal:

it("is a baseline test using a simple function expression") {
    val add = {op1:Int, op2:Int -> op1 + op2}

    add(17, 24) shouldBe 41
}

It takes two operands, and adds them.

Stealing the example from the docs, we have this alternative:

it("uses function literals with receivers") {
    val add: Int.(Int) -> Int = { other -> this.plus(other) }

    add(17, 24) shouldBe 41
    17.add(24) shouldBe 41
}

This is nice syntax from Kotlin. Not only can one call this function the "normal" way, one can also call that function with the first argument as a receiver (I guess that's what they mean as a receiver?), and it looks like a method on Int class (the literal 17 is still an Int object).

The implementation above doesn't use idiomatic-Kotlin though, in how I have explicitly referenced this (a reference to the first argument). Here's what they mean about "so that you can access the members of that receiver object without […] using a this expression":

it("uses function literals with receivers and implicit this") {
    val add: Int.(Int) -> Int = { other -> plus(other) } // see there's no this here any more

    add(17, 24) shouldBe 41
    17.add(24) shouldBe 41
}

That's all that assertSoftly function expression is doing.

Just so I was clear on this, I created another test:

it("checks the values of the operands in the function implementation") {
    var op1 = 0
    var op2 = 0
    val add: Int.(Int) -> Int = {
        other ->
        op1 = this
        op2 = other
        this.plus(other)
    }

    17.add(24) shouldBe 41

    op1 shouldBe 17
    op2 shouldBe 24
}

This passes, so my hypothesis is right.

My grasp of how the docs explain this is still tenuous. I didn't really see how it worked until I tried it, so I need to think some more about this stuff. Or perhaps better understanding will come from just doing. I have to say that this synxtax doesn't mean much to me: Int.(Int) -> Int = {}, and I can't correlate it back to add(17, 24) and 17.add(24).

I guess looking at the assigment from the test:

val add: Int.(Int) -> Int = { other -> plus(other) }

That's just this:

val variableName: type = value

Where:

  • add is the variableName
  • Int.(Int) -> Int is the type
  • { other -> plus(other) } is the value

The only confusing bit is the type, but I guess if I use different types it might make more sense:

it("clarifies the syntax a bit by using different types") {
    val repeat: String.(Int) -> String = { other -> repeat(other) }

    "Z".repeat(11) shouldBe "ZZZZZZZZZZZ"
}

This is a bit clearer. The type String.(Int) -> String is, I guess, a function that takes a String and an Int, and returns a String. And the String.(Int) syntax means that it can be called using the arg1.function(arg2) syntax.

I think.

The code for these examples is on GitHub: FunctionSyntaxTest.kt.


Anyway, I've been at this a while now: the amount of writing I do (blog and code) only represents about 1/3 of the time I spend reading stuff. So I've been at this most of the day now. Time for a beer. And maybe dinner.

Righto.

--
Adam


PS: I made zero new progress on the koans today, but I learned a lot I reckon.

Friday, 9 September 2022

Kotlin: another Friday afternoon, another round of random investigation

G'day:

Because no-one has specifically screams "FFS stop it, Cameron", I'm gonna continue with another random Kotlin noobie investigation / brain dump. Previous similar articles here.


The Nothing and Unit types

Nothing

This came up in the next koans exercise. I read the docs, and looked at an answer on Stack Overflow ("Kotlin - Void vs. Unit vs. Nothing). The key bit seems to be the word cannot in this sentence: "If a function has return type Nothing, then it cannot return normally. It either has to throw an exception, or enter an infinite loop". And the next sentence is an interesting one: "Code that follows a call to a function with return type Nothing will be marked as unreachable by the Kotlin compiler". I wrote a test (obviously ;-)):

class NothingTest : DescribeSpec( {
    describe("Tests of the Nothing type") {
        val EOL = System.lineSeparator()

        val output = SystemLambda.tapSystemOut {
            try {
                outputUntilDie(3)
            } catch (e: Exception) {
                println(e.message)
            }
        }
        output.trim() shouldBe "Iteration 1${EOL}Iteration 2${EOL}Iteration 3${EOL}It died"
    }
})

fun outputUntilDie(max :Int) :Nothing {
    var current = 0
    while (true) {
        current++
        println("Iteration $current")
        if (current >= max) {
            throw Exception("It died")
        }
    }
}

The code below is on Github @ NothingTest.kt

Note in outputUntilDie there is never a normal return condition. The loop either goes forever (assuming I could somehow pass it a max of infinity), or the code bombs out. So whatever comes after the call to outputUntilDie can never be run. If I try to add some, IntelliJ lets me know:

And the compiler gives a warning:

> Task :compileTestKotlin
w: C:\src\kotlin\scratch\src\test\kotlin\language\types\NothingTest.kt: (15, 17): Unreachable code

If I was to change the return type to be Unit (returns no value, as opposed to returns nothing), then I don't get the visual cue or warning.

So I guess it's handy in that regard.

OK, the koans exercise demonstrates a more realistic situation where the code being considered unreachable is important.

import java.lang.IllegalArgumentException

fun failWithWrongAge(age: Int?) {
    throw IllegalArgumentException("Wrong age: $age")
}

fun checkAge(age: Int?) {
    if (age == null || age !in 0..150) failWithWrongAge(age)
    println("Congrats! Next year you'll be ${age + 1}.")
}

fun main() {
    checkAge(10)
}

Here the code won't even compile, because it's possible to get to ${age + 1} with age being null:

C:\src\kotlin\Kotlin Koans\Introduction\Nothing type\src\Task.kt:9:50
Kotlin: Operator call corresponds to a dot-qualified call 'age.plus(1)' which is not allowed on a nullable receiver 'age'.

Telling it that failWithWrongAge has a return type of Nothing mitigates this as processing could have halted already by then.

Unit

And the Unit type is a type that is equivalent to void, it's just a more OO way of doing things, cos it is actually a type, and there is a Unit object. This means everything returns an object, and there's no "special" handling like in Java with void. It's also the default return type in Kotlin according to the docs, so one doesn't need to specify it.


Lambdas

Here I've just messed around with some common higher-order functions available on collections. The code below is on Github @ FunctionsTest.kt

it("is a basic PoC test") {
    val strings = listOf("whero", "kākāriki", "kikorangi")
    val lengths = strings.map {it.length}

    lengths.joinToString(",") shouldBe "5,8,9"
}

map just takes a single param that is the mapping function, so no need for parentheses. And because the callback only receives the one argument, one can just use the default it name. So {it.length} is a lambda that returns the length property value of its it argument (ie: String.length).


class Number(var value: Int, var en: String, var mi: String) {}

it("chains some together") {
    val jumbledNumbers = listOf(
        Number(2, "two", "rua"),
        Number(4, "four", "wha"),
        Number(3, "three", "toru"),
        Number(1, "one", "tahi")
    )
    val maoriNumbers = jumbledNumbers.sortedBy{it.value}.map{it.mi}

    maoriNumbers.joinToString(",") shouldBe "tahi,rua,toru,wha"
}

More of the same except I'm chaining some methods together here.


it("shows using _ to ignore a param in a lambda") {
    val numbers = mapOf(
        Pair("one", "tahi"),
        Pair("two", "rua"),
        Pair("three", "toru"),
        Pair("four", "wha")
    )
    val output = SystemLambda.tapSystemOut {
        numbers.forEach{(_, maori) -> print("$maori")}
    }

    output.trim() shouldBe "tahiruatoruwha"
}

The docs made a point of mentioning this, so I might as well. It's right in the Kotlin idiom to use _ as a parameter name when your lambda receives some arguments it doesn't need. Here the key is the first parameter (so the English version of the number), and I don't need that so this is made explicit by just using _. It's only the second argument I want: the Māori version


class Number(var value: Int, var en: String, var mi: String) {}

it("uses an anonymous function") {
    val jumbledNumbers = listOf(
        Number(2, "two", "rua"),
        Number(4, "four", "wha"),
        Number(3, "three", "toru"),
        Number(1, "one", "tahi")
    )

    val maoriNumbers = jumbledNumbers.sortedWith(fun(e1, e2) = e1.value - e2.value).map{it.mi}

    maoriNumbers.joinToString(",") shouldBe "tahi,rua,toru,wha"
}

For the hell of it here I use the anonymous function syntax (so with the fun keyword) for sortedWith comparator lambda. As the comparator method is just one expression, I don't need the parentheses or the return statement.


it("is an example of a fold") {
    val rgb = listOf("whero", "kākāriki", "kikorangi")

    val colours = rgb.fold("") {
        colours, colour -> if (colours.isEmpty()) colour else "$colours,$colour"
    }

    colours shouldBe "whero,kākāriki,kikorangi"
}

And with the block syntax I had no idea how to define my own params (instead of just the situation where there's only one, and the default it would do), but it's pretty straight forward as it turns out. The bit I didn't guess right was the parameter definitions are inside the block, not before it like I kinda wanted to do.

I also found out that there is no ternary operator (? :) in Kotlin. if is an expression, so it does the same thing. Slightly more verbose, but it's OK. I don't like using else in my code though (but I don't might the "else" operand when using ? :. Hrm…?


it("is an example of a reduce") {
    val redWhiteAndBlue = listOf("whero", "mā", "kikorangi")

    val colours = redWhiteAndBlue.reduce {colours, colour -> "$colours,$colour"}

    colours shouldBe "whero,mā,kikorangi"
}

This shows ths difference between fold and reduce:

  • fold takes a specific initial value as a first argument.
  • reduce uses the first element of the collection as the initial value.

groupBy higher-order function

Ooh I just spotted this in the docs. Kotlin has a gropupBy higher-order function. It does what it says on the tin. here's a contrived example:

it("groups the words by their first character, capitalising each word in the result") {
    val words = listOf(
        "tahi","rua","toru","wha","rima","ono","whitu","waru","iwa","tekau", // 1-10
        "whero","karaka","kōwhai","kākāriki","kikorangi","poropango","papura", // ROYGBIV
        "rāhina","rātū","rāapa","rāpare","rāmere","rāhoroi","rātapu" // Mon-Sun
    ).sortedWith(Collator.getInstance(Locale("mi_NZ")))

    val groupedByLetter = words.groupBy(
        { word -> word.first()},
        {word ->  word.replaceFirstChar(Char::titlecase)}
    ).toSortedMap()

    groupedByLetter shouldBe mapOf(
        Pair('i', listOf("Iwa")),
        Pair('k', listOf("Kākāriki", "Karaka", "Kikorangi", "Kōwhai")),
        Pair('o', listOf("Ono")),
        Pair('p', listOf("Papura", "Poropango")),
        Pair('r', listOf("Rāapa", "Rāhina", "Rāhoroi", "Rāmere", "Rāpare", "Rātapu", "Rātū", "Rima", "Rua")),
        Pair('t', listOf("Tahi", "Tekau", "Toru")),
        Pair('w', listOf("Waru", "Wha", "Whero", "Whitu"))
    )
}

I like that.


Primary and secondary constructors

You saw I had that wee Number class in my tests:

 class Number(var value: Int, var en: String, var mi: String) {}

Initially I had written that long-form like this:

class Number {
    var value = 0
    var en = ""
    var mi = ""

    constructor(value: Int, en: String, mi: String) {
        this.value = value
        this.en = en
        this.mi = mi
    }
}

But IntelliJ admonished me with this:

And I went "um… primary and secondary constructors are a thing, are they? OK. Go for it pal". And it squizhed everything up for me.

I RTFMed and right there on the class page it says this:

A class in Kotlin can have a primary constructor and one or more secondary constructors. The primary constructor is a part of the class header, and it goes after the class name and optional type parameters.

[…]

The primary constructor cannot contain any code. …

Ah OK this makes sense if one just wants simple class with some property values (like I do here), there's no need for so much boiler plate to achieve this. Nice. I'm gonna need to read the rest of that section about initializer blocks and the like. But: tomorrow. I've messed around enough for one afternoon.


That was quite short, but it reflects the forward progress I made in a few hours exploration. The good thing is: the more I find out about Kotlin, the more I want to find out about it.

Righto.

--
Adam

Thursday, 8 September 2022

CFML: speaking of application scope proxies

G'day:

OK so you probably weren't talking about application scope proxies, but I was in my previous article: CFML: looking at how CFWheels messes up a loop. In that article I look at some very uncontrolled (and buggy: hence the article) application-scope access. And I made the observation that one should never access the application scope in one's application code, other than via a proxy.

This got me thinking. How hard is it actually to write one of these proxies that handles the locking for you, keeps yer code testable, and minimises the chance of having shared scope errors? All whilst minimising locking boilerplace and stuff.

Turns out it's pretty simple to get a proof of concept together (code on Github @ ApplicationScopeProxy.cfc / ApplicationScopeProxyTest.cfc):

component {

    public function init(struct applicationScope) {
        variables.applicationScope = arguments.applicationScope
    }

    public function set(required string key, required any value) {
        lock scope="application" type="exclusive" timeout=5 throwontimeout=true {
            "variables.applicationScope.#key#" = value
        }
    }

    public any function get(required string key) {
        lock scope="application" type="readonly" timeout=5 throwontimeout=true {
            if (isDefined("variables.applicationScope.#key#")) {
                return structGet("variables.applicationScope.#key#")
            }
            throw(type="ApplicationScopeProxy.KeyNotFoundException", message="Key [#key#] not found in application scope");
        }
    }

    public void function withReadOnlyLock(required function task) {
        lock scope="application" type="readonly" timeout=5 throwontimeout=true {
            task()
        }
    }

    public void function withExclusiveLock(required function task) {
        lock scope="application" type="exclusive" timeout=5 throwontimeout=true {
            task()
        }
    }
}

And it's got superficial first-pass red/green tests:

import testbox.system.BaseSpec
import cfml.cfmlLanguage.scopes.ApplicationScopeProxy

component extends=BaseSpec {

    function run() {
        describe("Tests of ApplicationScopeProxy", () => {
            describe("Tests for the set method", () => {
                it("sets a variable", () => {
                    testApp = {}
                    proxy = new ApplicationScopeProxy(testApp)
                    proxy.set("testKey", "TEST_VALUE")

                    expect(testApp).toHaveKey("testKey")
                    expect(testApp.testKey).toBe("TEST_VALUE")
                })

                it("sets a variable with a deep path", () => {
                    testApp = {}
                    proxy = new ApplicationScopeProxy(testApp)
                    proxy.set("one.two.three", "TEST_VALUE")

                    expect(testApp).toHaveKey("one")
                    expect(testApp.one.two.three).toBe("TEST_VALUE")
                })
            })

            describe("Tests for the get method", () => {
                it("gets a variable", () => {
                    testApp = {testKey = "TEST_VALUE"}
                    proxy = new ApplicationScopeProxy(testApp)
                    result = proxy.get("testKey")

                    expect(result).toBe("TEST_VALUE")
                })

                it("gets a variable with a deep path", () => {
                    testApp = {one={two={three = "TEST_VALUE"}}}
                    proxy = new ApplicationScopeProxy(testApp)
                    result = proxy.get("one.two.three", "TEST_VALUE")

                    expect(result).toBe("TEST_VALUE")
                })

                it("throws an exeption if the key is not found", () => {
                    testApp = {}
                    proxy = new ApplicationScopeProxy(testApp)

                    expect(() => {
                        proxy.get("one.two.three", "TEST_VALUE")
                    }).toThrow(
                        type = "ApplicationScopeProxy.KeyNotFoundException",
                        regex = "Key \[one\.two\.three\] not found in application scope"
                    )
                })
            })

            describe("Tests for the withReadOnlyLock method", () => {
                it("waits until it can get a lock before continuing", () => {
                    proxy = new ApplicationScopeProxy({})

                    telemetry = ["test start"]
                    t1 = runAsync(() => {
                        telemetry.append("t1 started")
                        lock scope="application" type="exclusive" timeout=5 throwontimeout=true {
                            sleep(1000)
                        }
                        telemetry.append("t1 ended")
                    })
                    t2 = runAsync(() => {
                        telemetry.append("t2 started")

                        proxy.withReadOnlyLock(() => {
                            telemetry.append("withReadOnlyLock ran")
                        })

                        telemetry.append("t2 ended")
                    })
                    t1.get()
                    t2.get()

                    lockTelemetry = telemetry.find("withReadOnlyLock ran")
                    t1EndTelemetry = telemetry.find("t1 ended")
                    expect(lockTelemetry).toBeGt(t1EndTelemetry, serializeJson(telemetry))
                })
            })

            describe("Tests for the withExclusiveLock method", () => {
                it("waits until it can get a lock before continuing", () => {
                    proxy = new ApplicationScopeProxy({})

                    telemetry = ["test start"]
                    t1 = runAsync(() => {
                        telemetry.append("t1 started")
                        lock scope="application" type="readlonly" timeout=5 throwontimeout=true {
                            sleep(1000)
                        }
                        telemetry.append("t1 ended")
                    })
                    t2 = runAsync(() => {
                        telemetry.append("t2 started")

                        proxy.withExclusiveLock(() => {
                            telemetry.append("withExclusiveLock ran")
                        })

                        telemetry.append("t2 ended")
                    })
                    t1.get()
                    t2.get()

                    lockTelemetry = telemetry.find("withExclusiveLock ran")
                    t1EndTelemetry = telemetry.find("t1 ended")
                    expect(lockTelemetry).toBeGt(t1EndTelemetry, serializeJson(telemetry))
                })
            })
        })
    }
}

The tests kinda summarise its usage, but there's four methods (five including constructor):

  • Constructor: pass in the scope from the application-boot code (from the DI config or just in onApplicationStart etc). This way even the proxy is uncoupled from the scope. It also aids testing, which you'll've noticed in the tests.
  • set - sets a value in the scope at the location of the specified key. The key can be a dotted-path, and that will be unpacked into substructs. Is run in an exclusive lock.
  • get - as above, but gets it back out again. Is run in a read-only lock.
  • withReadOnlyLock - to avoid race conditions like in the CFWheels code, this takes a block of code and runs it all in a read-only lock. Good for multiple accesses to the application scope that rely on the state not being changed between each access.
  • withExclusiveLock - same as above, except with an exclusive lock. This would handly situations where code is reading and writing to the scope, and needs to be run as an atom before other accesses try to act on the data themselves.

This is just a proof of concept I knocked together in about 1hr without giving too much thought to it (the best way to write code to present to the public as if I know what I'm doing ;-)). There is some room for improvement, obvs:

  • It's not for the methods themselves to set the timeout / throwontimeout values on the locks. I just set those for the sake of expediency. A real-world implementation would likely pass those with each call, and probably optionally fall back to values set by init, or just the default for lock (if there is one? I think maybe not, for timeout).
  • If the returned value is a complex object, then it'll still be prone to race conditions as it'll just be a reference to the underlying shared value. So one would either need to advise to use duplicate when fetching the value, or make it an option on the get, or have a getCopy method too.
  • Having methods like exists, delete, etc would be handy too.
  • Maybe some mechanism to pass in a callback that receives variables.applicationScope as an argument, to do more complex stuff, and the implementation of that just ensures the lock is made?

There'd be plenty of options. The good thing with TDD and incremental delivery, is that one can just deliver what's needed now, and worry about the rest later.

What CFWheels needs now is that withExclusiveLock method, I think…

Righto.

--
Adam

CFML: looking at how CFWheels messes up a loop

G'day:

This exercise came about from a bug in CFWheels we encountered today.

We got a notification that a user had received a 500-error, and I was looking into it. It was happening in /wheels/global/internal.cfm, in this bit of code:

for (local.key in application.wheels.cache[arguments.category]) {
    if (Now() > application.wheels.cache[arguments.category][local.key].expiresAt) {

On the second line we are getting an error: key [81AB90EB428410D6222B18AAE12EC014] doesn't exist.

It should be obvious to even a journeyman dev what the problem is here.

When the first line of code runs, there is a key 81AB90EB428410D6222B18AAE12EC014 in application.wheels.cache. By the time the second line of code runs, some other request has come along and removed 81AB90EB428410D6222B18AAE12EC014 from application.wheels.cache, and hence we get the error. This is why everyone (?) knows that one must synchronise access to shared scopes like the application scope, so that if a process is writing to it, then other processes can't read from it. Or competing writes can't be happening simultaneously. IE: Chuck a lock block around this sort of thing. This is CFML 101 sort of stuff, CFWheels Team.

On the face of it, this was just super bad timing. What are the odds that one request will run the first line, and then before that request gets to the second line, another request has blitzed what the first line "knew" was there to use.

But it's not really super bad timing. It's just bad coding. This code is a loop. When does a for (key in struct) loop decide which keys exist in the struct, and accordingly which keys to loop over? At the first time the loop is entered. And only then. So if there are 10000 items in that struct when our request first ran, any other process has from the first iteration until the 9999th iteration to delete the 10000th key, and we'll get this error. That's a lot of opportunity for failure. Also, due to the very nature of this code, there's a good chance that a bunch of requests are gonna be running it simultaneously: the key deletions come after a determination that a shared cache is full and needs culling (let's not get into matter of why a function called addToCache is deleting stuff anyhow. This is not the worst thing about this cluster🦆, I assure you), and at the moment in time the cache seems to be full, then every call to addToCache (sic) will try to delete stuff. Ugh.

To demonstrate what's going on here, I've knocked a quick example together (code on github):

// Application.cfc
component {
    this.name = "testLoopingOverAppScopedStructs"
}
// firstRequest.cfm
cfflush(interval=16)
application.numbers = [
    "one" = "tahi",
    "two" = "rua",
    "three" = "toru",
    "four" = "wha"
]

for (en in application.numbers) {
    writeOutput("keys @ top of loop: #application.numbers.keyList()#<br>")
    writeOutput("current key: #en#<br>")
    writeOutput("current value: #application.numbers[en]#<br>")
    sleep(2000)
    writeOutput("keys after sleep: #application.numbers.keyList()#<br>")
}
// secondRequest.cfm
application.numbers["five"]  = "rima"
application.numbers.delete("four")
writeDump(application.numbers)

All very straight forward. In secondRequest.cfm I only add a key so there's a visual cue in the output of firstRequest.cfm when the second request ran.

The test is to run firstRequest.cfm, then whist that's running, run secondRequest.cfm. The output from a typical run of firstRequest.cfm would be like:

keys @ top of loop: one,two,three,four
current key: one
current value: tahi
keys after sleep: one,two,three,four
keys @ top of loop: one,two,three,four
current key: two
current value: rua
keys after sleep: one,two,three,five <--- here is where I ran secondRequest.cfm
keys @ top of loop: one,two,three,five
current key: three
current value: toru
keys after sleep: one,two,three,five
keys @ top of loop: one,two,three,five
current key: four

[...]
Error Occurred While Processing Request
Element FOUR is undefined in a CFML structure referenced as part of an expression.

You can see that the keys the loop iterates over is set at the beginning, given it's unaffected by changes to the struct's keys during the loop.


What should the CFWheels codebase do to mitigate this. Well: here's a list of things that are… erm… "less good than they could be "… in that code.

  • Framework functionality should be in classes, not in include files. In this case cache-handing code should be in a CachingService or somesuch.
  • internal.cfm is an awful name for a library. A filename should describe what the code does (like, say "CachingService"), not how it's used ("internally", apparently. Sounds uncomfortable, not to mention a bit unhygienic).
  • Related to this, that file is a hotchpotch mess of all sorts of shite that should be homed somewhere else. This describes most files in the CFWheels codebase, unfortunately.
  • The only code that touches the application scope should be in an ApplicationScopeProxy object. BTW this makes controlling the scope-locking way easier to manage. See also: SessionScopeProxy / ServerScopeProxy / etc
  • Within that proxy, access to any shared scope should be appropriately synchronised (with a lock). This should also be extended to mitigating race conditions like we have here.
  • If your function is called addToCache then stick to doing that. Code that removes stuff from the cache does not belong there.
  • All the code in that method is conditional. All of it. There is no "mainline" or happy path. Simplify the conditions and bail out if they're not met. Don't put the code you actually expect to run inside a conditional block.
  • Connected to the two preceding points: most of the code in that method is not actually anything to do with adding something to a cache.
  • For the love of god stop prefixing functions with $ to indicate they're private. Just make them private.

I don't think anything there is particularly controversial or complicated. And whilst it's beyond "CFML 101", I think it's within the low bar of "Writing Frameworks 101".

Anyway. There you got. A coupla hours in the life of someone who has to use CFWheels.

Righto.

--
Adam