Showing posts with label ColdFusion. Show all posts
Showing posts with label ColdFusion. Show all posts

Friday 17 March 2023

FAO ColdFusion users: you need to address a critical vuln in your system

G'day:

Just in case you happen to read this blog, but not Charlie Arehart's one, pls go over to his blog and read his article "ColdFusion March 2023 emergency update, and what to do about it", and follow-up.

This is serious, don't ignore it. I don't have time or inclination to look into it myself - I don't use ColdFusion any more - but figured I should do my bit to get the info out there.

It does not impact Lucee.

Righto.

--
Adam

Friday 10 March 2023

CFML: ColdFusion 2023 is in public beta now

G'day:

As posted in the CFML Slack channel by Adobe ColdFusion star, Mark Takata, CF2023 is in public beta now. One can sign up for it here: https://www.adobeprerelease.com/beta/48C2D737-9CB4-445E-E39C-90CB6381919A.

I've not done much on it yet: I've tested some bugfixes I was interested in, and that's about it.

The first thing I did was to set up a Docker container running ColdFusion 2023 with a directory mapped into it from my host PC so I can run scratch code to test stuff. I prefer to do this than install stuff onto the host machine itself.

The project is on Github @ https://github.com/adamcameron/cf2023/tree/1.0. The readme.md for it is:

CF2023 container

This is the config for a minimal(-ish) Docker container running CF2023.

I've installed vim in it as well as I never don't need it, so why dick around not having it?

I add bash aliases for ll and cls as they're handy (sorry the latter is cos I'm a perennial Windows user. Shrug).

I've exposed the container's port 8500 as 8523 on the host.

I map the src directory of this project to the /app/src directory in the container. The webroot is /app. So you ought to be able to browse to http://localhost:8523/src/dumpServer.cfm to verify the install. And then, like, chuck other code in there.

I expose the log dir /opt/coldfusion/cfusion/logs as var/log/coldfusion/: I always find it helpful to have the logs right there to look at when I'm testing new stuff.

I copy the MySQL community driver into the container so one can make MySQL DSNs without fussing.

I've installed the CF debugger module as otherwise one can't switch on Robust Exception Handling, which is a hard requirement for dev, I think.

There's a shell script docker/rebuildContainer.sh. Use this to (re)build the container:

~/cf2023/docker$ ./rebuildContainer.sh 123

All it does is remove any existing container, rebuild it, and bring it up.

That's it. If this is useful to anyone getting up and running and being able to pitch-in testing ColdFusion 2023, then: cool. If not: shrug. Good on you for reading down to here though.

Righto.

--
Adam

Thursday 16 February 2023

ColdFusion: bug in runAsync

G'day:

In my article yesterday ("CFML: runAsync doesn't seem very asynchronous on Lucee") I mentioned in passing that I'd found a bug in ColdFusion's runAsync implementation. I wasn't focusing on ColdFusion at the time, so I didn't try to repro it; however I have now.

Don't worry about the wall of code, it's the same thing three times to increase the odds of the bug situation to surface:

// /vendor/ColdFusion/2021/runAsyncBug/repro.cfm
telemetry = []
recordTelemetry = (label) => {
    telemetry.append(label)
}

recordTelemetry("Begin")

recordTelemetry("f1 before define")
f1 = runAsync(() => {
    recordTelemetry("f1 main")
}).then(() => {
    recordTelemetry("f1 then1")
}).then(() => {
    recordTelemetry("f1 then2")
})
recordTelemetry("f1 after define")

recordTelemetry("f2 before define")
f2 = runAsync(() => {
    recordTelemetry("f2 main")
}).then(() => {
    recordTelemetry("f2 then1")
}).then(() => {
    recordTelemetry("f2 then2")
})
recordTelemetry("f2 after define")

recordTelemetry("f3 before define")
f3 = runAsync(() => {
    recordTelemetry("f3 main")
}).then(() => {
    recordTelemetry("f3 then1")
}).then(() => {
    recordTelemetry("f3 then2")
})
recordTelemetry("f3 after define")

recordTelemetry("f1 before get")
f1.get()
recordTelemetry("f1 after get")

recordTelemetry("f2 before get")
f2.get()
recordTelemetry("f2 after get")

recordTelemetry("f3 before get")
f3.get()
recordTelemetry("f3 after get")

recordTelemetry("End")

writeDump(telemetry)

tallies = [
    "Begin" = 0,
    "f1 before define" = 0,
    "f1 main" = 0,
    "f1 then1" = 0,
    "f1 then2" = 0,
    "f1 after define" = 0,
    "f2 before define" = 0,
    "f2 main" = 0,
    "f2 then1" = 0,
    "f2 then2" = 0,
    "f2 after define" = 0,
    "f3 before define" = 0,
    "f3 main" = 0,
    "f3 then1" = 0,
    "f3 then2" = 0,
    "f3 after define" = 0,
    "f1 before get" = 0,
    "f1 after get" = 0,
    "f2 before get" = 0,
    "f2 after get" = 0,
    "f3 before get" = 0,
    "f3 after get" = 0,
    "End" = 0
]
errors = tallies
    .map((k) => telemetry.findAll(k).len())
    .filter((_, v) => v != 1)

writeDump(errors)

This code is very similar to the cod ein the earlier article I linked to. I create three futures: f1, f2, f3; and each of those has a main task and two then handlers. All the task and then handlers do is append to a telemetry array. This is pared down from some code that actually did more stuff that I wanted to be async.

All going well (and 75% of the time, this is the case), we should end up with an array of 23 telemetry entries, one each for the entries in the code. We can't guarantee the order cos most of it is running asynchronously.

What I was seeing though, is sometimes entries were missing, but some others were doubled up. Here's an example of that:

See how the telemetry array has no entries for "f2 then1" and "f3 before get"; but instead has two each of "f3 then1" and "f2 before get". Odd.

Also about 5% of the time I get this:

Variable LABEL is undefined.

 
The error occurred in /app/cfml/vendor/ColdFusion/2021/runAsyncBug/repro.cfm: line 4
2 : telemetry = []
3 : recordTelemetry = (label) => {
4 :     telemetry.append(label)
5 : }
6 : 


There are always 23 entries; just some are wrong. There seems to be any number of wrong ones, but usually only a couple. The example above is a fairly typical run.

I tried a variant of the test with a lock around the write to the telemetry array:

recordTelemetry = (label) => {
    lock type="exclusive" name="telemetry" timeout=1 throwontimeout=true {
        telemetry.append(label)
    }
}

But this did not make a difference.

Also the fact I am using a function expression using closure over the telemetry array is significant. If I just have a function statement:

function recordTelemetry(label) {
    telemetry.append(label)
}

There is no problem.

I dunno what to make of this, but it'll give the Adobe ColdFusion Team something to work with anyhow.

Oh: this is not an issue with Lucee.

Righto.

--
Adam

Wednesday 15 February 2023

CFML: runAsync doesn't seem very asynchronous on Lucee (5.3.10.120)

G'day:

I was wanting to compare how other languages dealt with the promise-style async processing, as I was not completely convinced by what I was seeing when investigating spatie/async in the previous article: "PHP: looking at spatie/async". It seems solid, just doesn't behave how I'd expect in some situations. More on that in another article. I checked what JS does, and it was fine, and then I recalled in the back of my mind that CFML added something new to do with async processing a while ago. CF2018 as it turns out. runAsync. And this was also added to Lucee at some point too (their docs don't say when something was added).

I knocked together a quick repro of what I was looking at in spatie/async and JS as a baseline; and then wtote an equivalent in CFML. But when I ran it on Lucee it didn't seem to be behaving how I'd expect, but in a different way from what I was trying to compare to in the other languages. When I ran the same code on ColdFusion though, it seemed to behave the way I expected. It seemed I have found a Lucee rabbit hole to go down.

I ended up with this repro that demonstrates what I'm seeing. There's a chunk of code, but it's a bunch of largely the same operations repeated a few times so as to provide usable telemetry. I'll go through it piecemeal though.

This is all being run on Lucee 5.3.10.120.

start = getTickCount()
writeLog(file="runAsync", text="Begin @ #getTickCount() - start#")

I'll time things as I go.

writeLog(file="runAsync", text="f1 before define @ #getTickCount() - start#")
f1 = runAsync(() => {
    writeLog(file="runAsync", text="f1 main before query @ #getTickCount() - start#")
    queryExecute("CALL sleep_and_return(1)", [], {datasource="test"});
    writeLog(file="runAsync", text="f1 main after query @ #getTickCount() - start#")
}).then(() => {
    writeLog(file="runAsync", text="f1 then1 before query @ #getTickCount() - start#")
    queryExecute("CALL sleep_and_return(1)", [], {datasource="test"});
    writeLog(file="runAsync", text="f1 then1 after query @ #getTickCount() - start#")
}).then(() => {
    writeLog(file="runAsync", text="f1 then2 before query @ #getTickCount() - start#")
    queryExecute("CALL sleep_and_return(1)", [], {datasource="test"});
    writeLog(file="runAsync", text="f1 then2 after query @ #getTickCount() - start#")
})
writeLog(file="runAsync", text="f1 after define @ #getTickCount() - start#")
  • I create a future
  • And it has a main task, and two then handlers.
  • Each of these do exactly the same thing:
    • Call a proc in the DB. See below for the implementation code, but you can probably tell it's a proc specifically designed to take time to execute. It's basically a DB-borne sleep call.
    • And log the current run time before and after the proc call.
  • Oh and I log the timing either side of the whole oepration too.

Here's the proc:

DROP PROCEDURE IF EXISTS sleep_and_return;
DELIMITER //
CREATE PROCEDURE sleep_and_return(IN seconds INT)
BEGIN
    DO SLEEP(seconds);
    SELECT seconds;
END //
DELIMITER ;

It just sleeps for the specified time and then returns the value it was passed.

Then I do exactly the same thing I did with f1 for f2 and f3 as well. Three identical sets of runAsync / then / then with the logging and the DB call.

After that, I sit back and get each of them, timing around each get:

writeLog(file="runAsync", text="f1 before get @ #getTickCount() - start#")
f1.get()
writeLog(file="runAsync", text="f1 after get @ #getTickCount() - start#")

writeLog(file="runAsync", text="f2 before get @ #getTickCount() - start#")
f2.get()
writeLog(file="runAsync", text="f2 after get @ #getTickCount() - start#")

writeLog(file="runAsync", text="f3 before get @ #getTickCount() - start#")
f3.get()
writeLog(file="runAsync", text="f3 after get @ #getTickCount() - start#")

And one final entry in the log at the end:

writeLog(file="runAsync", text="End @ #getTickCount() - start#")

All the code together looks like this:

start = getTickCount()
writeLog(file="runAsync", text="Begin @ #getTickCount() - start#")


writeLog(file="runAsync", text="f1 before define @ #getTickCount() - start#")
f1 = runAsync(() => {
    writeLog(file="runAsync", text="f1 main before query @ #getTickCount() - start#")
    queryExecute("CALL sleep_and_return(1)", [], {datasource="test"});
    writeLog(file="runAsync", text="f1 main after query @ #getTickCount() - start#")
}).then(() => {
    writeLog(file="runAsync", text="f1 then1 before query @ #getTickCount() - start#")
    queryExecute("CALL sleep_and_return(1)", [], {datasource="test"});
    writeLog(file="runAsync", text="f1 then1 after query @ #getTickCount() - start#")
}).then(() => {
    writeLog(file="runAsync", text="f1 then2 before query @ #getTickCount() - start#")
    queryExecute("CALL sleep_and_return(1)", [], {datasource="test"});
    writeLog(file="runAsync", text="f1 then2 after query @ #getTickCount() - start#")
})
writeLog(file="runAsync", text="f1 after define @ #getTickCount() - start#")


writeLog(file="runAsync", text="f2 before define @ #getTickCount() - start#")
f2 = runAsync(() => {
    writeLog(file="runAsync", text="f2 main before query @ #getTickCount() - start#")
    queryExecute("CALL sleep_and_return(1)", [], {datasource="test"});
    writeLog(file="runAsync", text="f2 main after query @ #getTickCount() - start#")
}).then(() => {
    writeLog(file="runAsync", text="f2 then1 before query @ #getTickCount() - start#")
    queryExecute("CALL sleep_and_return(1)", [], {datasource="test"});
    writeLog(file="runAsync", text="f2 then1 after query @ #getTickCount() - start#")
}).then(() => {
    writeLog(file="runAsync", text="f2 then2 before query @ #getTickCount() - start#")
    queryExecute("CALL sleep_and_return(1)", [], {datasource="test"});
    writeLog(file="runAsync", text="f2 then2 after query @ #getTickCount() - start#")
})
writeLog(file="runAsync", text="f2 after define @ #getTickCount() - start#")


writeLog(file="runAsync", text="f3 before define @ #getTickCount() - start#")
f3 = runAsync(() => {
    writeLog(file="runAsync", text="f3 main before query @ #getTickCount() - start#")
    queryExecute("CALL sleep_and_return(1)", [], {datasource="test"});
    writeLog(file="runAsync", text="f3 main after query @ #getTickCount() - start#")
}).then(() => {
    writeLog(file="runAsync", text="f3 then1 before query @ #getTickCount() - start#")
    queryExecute("CALL sleep_and_return(1)", [], {datasource="test"});
    writeLog(file="runAsync", text="f3 then1 after query @ #getTickCount() - start#")
}).then(() => {
    writeLog(file="runAsync", text="f3 then2 before query @ #getTickCount() - start#")
    queryExecute("CALL sleep_and_return(1)", [], {datasource="test"});
    writeLog(file="runAsync", text="f3 then2 after query @ #getTickCount() - start#")
})
writeLog(file="runAsync", text="f3 after define @ #getTickCount() - start#")


writeLog(file="runAsync", text="f1 before get @ #getTickCount() - start#")
f1.get()
writeLog(file="runAsync", text="f1 after get @ #getTickCount() - start#")

writeLog(file="runAsync", text="f2 before get @ #getTickCount() - start#")
f2.get()
writeLog(file="runAsync", text="f2 after get @ #getTickCount() - start#")

writeLog(file="runAsync", text="f3 before get @ #getTickCount() - start#")
f3.get()
writeLog(file="runAsync", text="f3 after get @ #getTickCount() - start#")


writeLog(file="runAsync", text="End @ #getTickCount() - start#")

When I run this on CF2021, I get exactly what I'd expect:

Begin @ 0
f1 before define @ 1
f1 main before query @ 2
f1 after define @ 3
f2 before define @ 4
f2 main before query @ 6
f2 after define @ 8
f3 before define @ 10
f3 after define @ 12
f3 main before query @ 12
f1 before get @ 13
f1 main after query @ 1006
f1 then1 before query @ 1008
f2 main after query @ 1011
f2 then1 before query @ 1012
f3 main after query @ 1015
f3 then1 before query @ 1016
f1 then1 after query @ 2010
f1 then2 before query @ 2012
f2 then1 after query @ 2014
f2 then2 before query @ 2016
f3 then1 after query @ 2023
f3 then2 before query @ 2024
f1 then2 after query @ 3014
f1 after get @ 3016
f2 before get @ 3017
f2 then2 after query @ 3018
f2 after get @ 3020
f3 before get @ 3021
f3 then2 after query @ 3026
f3 after get @ 3028
End @ 3029

Each of the f1, f3, f3 operations are being executed in a "first come, first served", and independently of each other (ie: asynchronously). And it's all over and done with at about the time it takes the slowest one to finish (allowing for intrinsic overhead).

It's a different story when I run this on Lucee:

Begin @ 1
f1 before define @ 19
f1 main before query @ 24
f1 main after query @ 1244
f1 then1 before query @ 1247
f1 then1 after query @ 2252
f1 after define @ 2255
f1 then2 before query @ 2255
f2 before define @ 2255
f2 main before query @ 2258
f1 then2 after query @ 3258
f2 main after query @ 3263
f2 then1 before query @ 3266
f2 then1 after query @ 4270
f2 after define @ 4272
f3 before define @ 4273
f2 then2 before query @ 4274
f3 main before query @ 4276
f2 then2 after query @ 5277
f3 main after query @ 5278
f3 then1 before query @ 5281
f3 then1 after query @ 6284
f3 after define @ 6286
f3 then2 before query @ 6286
f1 before get @ 6286
f1 after get @ 6287
f2 before get @ 6288
f2 after get @ 6289
f3 before get @ 6290
f3 then2 after query @ 7288
f3 after get @ 7289
End @ 7290

Here f1 gets underway, but it really seems to mostly block until [f1 then2 before query] until f2 even gets a look-in. This is two seconds of synchronous execution before Lucee even finds out there's something else asynchronous to be cracking on with. Then f2 blocks for second before f3 gets a chance to be looked at. Then there's a mess of stopping and starting, sucking up even more time, for an end result of ~7sec for an "asynchronous" operation that would take ~9sec if it was synchronous. Not great.

I also hasten to add that this was not a case of bad luck how the "async-ness" happened to run this time. I've run this code dozens of times, and it's always around the same. I also have run it with just CFML's own sleep instead of the DB call, and it's still the same. It's not the work the tasks are doing, it's how Lucee is implementing the asynchronous thread management, I think.

Also if ColdFusion's handling of the same code was similar to Lucee's, I'd just chalk it up to "sometimes asynchronous code doesn't even up seeming that asynchronous". But given ColdFusion nails it, I really think there's something up with Lucee. Pity: I had a good use case for this stuff in our app. But now I'm thinking I need to go back to our simple <cfthread> implementations and check exactly how asynchronous that code is…

Full disclosure: I also found what seems to be a bug with how ColdFusion handles enclosed variable references asynchronously, but I can't work out what's going on with that yet, and I was half way through this bit of code, so I wanted to finish this first. I might come back to the ColdFusion thing "tomorrow".

The code here is reasonably portable (one just needs to create that proc, and a DSN), so I'd be dead keen to hear from other people if they can run it in their own Lucee environments, if you can be arsed, that is.

Or am I missing something? Entirely possible. Let me know.

Righto.

--
Adam

Wednesday 14 September 2022

CFML: working MySQL datasource in Application.cfc (this is just a note-to-self/Google)

G'day:

No content in this one, I just want something I can find when I search for how to config a MySQL datasource in Application.cfc in ColdFusion. I always forget/lose the code, and I can never find anything on Google. It'd be marvellous of Adobe would document this stuff, but it's a bit much to expect of them, I guess.

component {

    setsettings()
    loadDatasources()

    private void function setSettings() {
        this.name = "app1"
    }

    private void function loadDataSources() {
        this.datasources["dsn1"] = {
            driver = "mysql",
            class = "com.mysql.jdbc.Driver",
            url = "jdbc:mysql://database.backend:3306/"
                & "#server.system.environment.MARIADB_DATABASE#"
                & "?useUnicode=true&characterEncoding=UTF-8",
            username = server.system.environment.MARIADB_USER,
            password = server.system.environment.MARIADB_PASSWORD
        }
        this.datasource = "dsn1"
    }
}

The bit I always forget is the class bit, so I end up with an error along these lines:

No driver or URL found.

java.sql.SQLException: No driver or URL found.
 at coldfusion.server.j2ee.sql.pool.JDBCPool.createPhysicalConnection(JDBCPool.java:586)
 at coldfusion.server.j2ee.sql.pool.ConnectionRunner$RunnableConnection.run(ConnectionRunner.java:67)
 at java.base/java.lang.Thread.run(Thread.java:834)

This is also on GitHub @ Application.cfc.

Righto.

--
Adam

Saturday 7 May 2022

ColdFusion: probable bug with the implementation of the rest operator

G'day:

ColdFusion 2021 added the spread and rest operators. These are implemented as two different usages of .... In this article I am going to be making an observation about how the implementation of the rest operator is incomplete and faulty. I first raised this in the CFML Slack channel, but I've now wittered on enough about it to copy it to here.

What does the rest operator do? The docs say:

[The] Rest Operator operator is similar to [the] Spread Operator but behaves in [the] opposite way, while Spread syntax expands the iterables into individual element[s], the Rest syntax collects and condenses them into a single element.

ibid.

That's not so useful out of the context of a discussion on the spread operator, that said. So a test should clarify:

function testRest(first, ...rest) {
    return arguments
}

function run() {
    describe("Testing CF's rest operator, eg: function testRest(first, ...rest)", () => {
        it("combines all latter argument values into one parameter value", () => {
            actual = testRest("first", "second", "third", "fourth")
            expect(actual).toBe([
                first = "first",
                rest = ["second", "third", "fourth"]
            ])
            
             writeDump(var=actual, label="actual/expected")
        })
    })
}

tinyTest.runTests()
  • The rest operator is ... as a prefix to the last parameter in a function signature.
  • Any arguments passed from that position on are combined into that one parameter's value.

So far so good. So what's the problem? I had a use case where I needed this to work for named arguments, not positional ones:

it("combines all latter argument values into one parameter value when using named arguments", () => {
    actual = testRest(first="first", two="second", three="third", four="fourth")
    expect(actual.first).toBe("first")
    expect(actual).toHaveKey("rest")
    expect(actual.rest).toBe({two="second", three="third", four="fourth"})
})

Same as before, just the arguments have names now. This test fails. Why? Because CF completely ignores the rest operator when one uses named arguments. This "passing" test shows the actual (wrong) behaviour:

it("doesn't work at all with named arguments", () => {
    actual = testRest(first="first", two="second", three="third", four="fourth")
    expect(actual.first).toBe("first")
    expect(actual?.rest).toBeNull()
    expect(actual.keyList().listSort("TEXTNOCASE")).toBe("FIRST,four,REST,three,two")
    
     writeDump(var=actual, label="actual")
     writeDump(var={first="first", rest={two="second", three="third", four="fourth"}}, label="expected")
})

As I said, I raised this on the CFML Slack channel. I got one useful response:

I think it's unusual to use rest with named parameters, but, CF supports named parameters as well as positional so I would expect it to work. I'd settle for it being fully documented though as only working with positional.

John Whish

Fair point, and as I said in reply:

My reasoning was remarkably similar. I started thinking "it's a bit unorthodox to use named arguments here", but then I stopped to think… why? And my conclusion was "because in other languages I use this operation there's no such thing as named arguments, so I've not been used to thinking about it", and that was the only reason I could come up with for me to think that. So I binned that thought (other than deciding to ask about it here).

The thing is there's no good reason I can think of that named arguments should not work. One cannot mix named and positional arguments which would be one wrinkle, so it's 100% reliable to take a set of named arguments, and match the argument names to the parameter names in the method signature. There is no ambiguity: any args that have the same name as a param are assigned as that parameter value. All the rest - instead of being passed in as ad-hoc arguments - are handled by the ... operation.

I cannot see a way that there's any ambiguity either. It's 100% "match the named ones, pass all others in the rest param".

What happens if the method call actually specifies a named argument that matched the name of the "rest" param? Same as if one specifies a positional argument in the position of the "rest" param: it doesn't matter. all arguments that don't match other named params are passed in the rest argument value.

I also think that if for some reason named arguments are not supported for use on function using the rest operator, then an exception should be thrown; not simply the code being ignored.

And whatever the behaviour is needs to be documented.

However one spins it, there are at least two bugs here:

  • Either it should work as I'd expect (or some variation thereof, if I have not thought of something), or it should throw an exception.
  • The behaviour should be documented.

I have not raised tickets for these as I'm not really a user of CF any more, so I don't care so much. Enough to raise it with them; not enough to care what they do about it. But CFers probably should care.

NB: I did not realise Lucee did not support the spread and rest operators at all, so I had to take a different approach to my requirement anyhow. I've not decided on the best way as yet.

There is a ticket for them to be implemented in Lucee: LDEV-2201.

The tests I wrote for this can be run on trycf.com.

Righto.

--
Adam

Sunday 20 June 2021

Making the code from the previous article work on both ColdFusion and Lucee

G'day:

A quick one today. I showed just a ColdFusion-only solution to some code in my previous article the other day: "CFML: emulating query-of-query group-by with higher-order functions". I figured when coming up with the completed code in a runnable example, I should make it work on both platforms. Here it is.

<cfscript>
function getUngroupedRecords(required numeric rows) {
    createRows = (number) => repeatString(",", rows).listToArray(",", true)
    date = () => now().add("d", randRange(-365, 365))
    amount = () => randRange(1,10000) / 100

    fakedDbData = queryNew(
        "settlementDate,LongTermGainLoss,ShortTermGainLoss",
        "Date,Double,Double",
        createRows(rows).map((_) => [date(), amount(), amount()])
    )
    return fakedDbData
}
ungroupedRecords = getUngroupedRecords(20)
writeDump(ungroupedRecords)

groupedRecords = ungroupedRecords.reduce((grouped={}, row) => {
    y = row.settlementDate.year()
    m = row.settlementDate.month()
    key = "#y#-#m#"
    grouped[key] = grouped[key] ?: {stgl = 0, ltgl = 0}
    grouped[key].stgl = grouped[key].stgl + row.ShortTermGainLoss
    grouped[key].ltgl = grouped[key].ltgl + row.LongTermGainLoss

    return grouped
}).reduce(
    (records, key, values) => {
        records.addRow({
            month = key.listLast("-"),
            year = key.listFirst("-"),
            ltgl = values.ltgl,
            stgl = values.stgl
        })
        return records
    },
    queryNew("month,year,ltgl,stgl", "Integer,Integer,Double,Double")
).sort((r1, r2) => {
    yearDiff = r1.year - r2.year
    if (yearDiff != 0) {
        return yearDiff
    }
    return r1.month - r2.month
})

writeDump(groupedRecords)
</cfscript>

Some notes on that:

  • I was using ColdFusion-only syntax for arrow functions with only one parameter: CF - correctly - does not require parentheses for the function signature here. Lucee however does, so I added them.
  • As previously-mentioned, Lucee didn't like arrayNew(1).resize(number) here, so I've changed it to being a bit shit, but at least it works on Lucee.
  • Lucee still doesn't return the updated query from Query.addRow, it returns the number of rows added (as per queryAddRow). ColdFusion changed this back in CF2018, so Lucee has some catch-up to do here. Anyway I needed to split this into two statements to make it work on Lucee.
  • Originally I had the sort callback as one expression to demonstrate a "trick" with variable assignment expressions, but whilst this worked in Lucee, ColdFusion choked on it. The callback was: (r1, r2) => (yearDiff = r1.year - r2.year) ? yearDiff : r1.month - r2.month. This pushed well past the bounds of what is clearly understandable, and I think the long-hand version I used is better code. But it was a bug in ColdFusion that the short version didn't work.

Anyway… this version of the code works on both ColdFusion and Lucee.

Righto.

--
Adam

Wednesday 9 June 2021

Repro for code that breaks in ColdFusion but works in Lucee

G'day:

In the last article (CFML: messing around with mixins (part 2)) I used some code that worked in Lucee, but didn't work in ColdFusion. It was not my code's fault, it's ColdFusion's fault for not being able to parse it. I didn't think I could be arsed looking at it this evening, but someone's asked me about it already so I had a look. I can reproduce the issue, and work around it.

Here's the code wot works on Lucee but breaks on ColdFusion (on trycf.com):

obj = {
    myMethod = function () {return arguments}
}    
methodName = "myMethod"
    
useProxy = true

callMe = useProxy
    ? () => obj[methodName](argumentCollection=arguments)
    : obj[methodName]

result = callMe("someArg")
writeDump(result)

This errors with: Invalid CFML construct found on line 10 at column 8. on line 10

It's pointing to the arrow function shorthand, but if I change it back to a normal function expression using the function keyword, it still errors.

It's also not the dynamic method call either. In the past that would have caused problems on ColdFusion, but they've fixed it at some point.

It's also not the lack of semi-colons ;-)

This adjustment works (on trycf.com):

obj = {
    myMethod = function () {return arguments}
}
methodName = "myMethod"
    
useProxy = true

proxy = () => obj[methodName](argumentCollection=arguments)
callMe = useProxy
    ? proxy
    : obj[methodName]

result = callMe("someArg")
writeDump(result)

It seems like ColdFusion really didn't like that arrow function / function expression in the ?: expression.

I'd say this is definitely a bug, and will point this repro case to Adobe for them to… do whatever they like with it.

Righto

--
Adam

Sunday 30 May 2021

CFML: weirdness with… <cfquery> and single-quote escaping

G'day:

I was looking around for a good (read: bad) example of one of those exercises in "building a dynamic SQL string inside a <cfquery>". Firstly, one should pretty much never do this. You are almost certainly tightly-coupling business logic with DB-tier logic, and that's an anti-pattern right there. And it also means you can't test it without hitting an actual database, which is rubbish. So just: don't do it. Build the SQL string first, testing all the logic as you go, and once the string is built, pass the completed thing to queryExecute (let's all just pretend <cfquery> doesn't exist: it's a horrific construct, and should have been left to die in the C20th where it belongs).

I found an excellent example of this in the CFWheels codebase.

<cfquery attributeCollection="#arguments.queryAttributes#">

    <cfset local.$wheels.pos = 0>

    <cfloop array="#arguments.sql#" index="local.$wheels.i">

        <cfset local.$wheels.pos = local.$wheels.pos + 1>

        <cfif IsStruct(local.$wheels.i)>
            <cfset local.$wheels.queryParamAttributes = $queryParams(local.$wheels.i)>

            <cfif
                NOT IsBinary(local.$wheels.i.value)
                AND local.$wheels.i.value IS "null"
                AND local.$wheels.pos GT 1
                AND (
                    Right(arguments.sql[local.$wheels.pos-1], 2) IS "IS"
                    OR
                    Right(arguments.sql[local.$wheels.pos-1], 6) IS "IS NOT"
                )
            >
                NULL
            <cfelseif StructKeyExists(local.$wheels.queryParamAttributes, "list")>
                <cfif arguments.parameterize>
                    (<cfqueryparam attributeCollection="#local.$wheels.queryParamAttributes#">)
                <cfelse>
                    (#PreserveSingleQuotes(local.$wheels.i.value)#)
                </cfif>
            <cfelse>
                <cfif arguments.parameterize>
                    <cfqueryparam attributeCollection="#local.$wheels.queryParamAttributes#">
                <cfelse>
                    #$quoteValue(str=local.$wheels.i.value, sqlType=local.$wheels.i.type)#
                </cfif> 
            </cfif>
        <cfelse>
            <cfset local.$wheels.i = Replace(PreserveSingleQuotes(local.$wheels.i), "[[comma]]", ",", "all")>
            #PreserveSingleQuotes(local.$wheels.i)#
        </cfif>
        #chr(13)##chr(10)#
    </cfloop>
    <cfif arguments.limit>
        LIMIT #arguments.limit#
        <cfif arguments.offset>
            #chr(13)##chr(10)#
            OFFSET #arguments.offset#
        </cfif>
    </cfif>
    <cfif Len(arguments.comment)>
        #arguments.comment#
    </cfif>
</cfquery>

(note that in the CFWheels codebase, that is all one single line of code, which I find to be… astonishing)

Also note that there are more "opportunities for improvement" there than just embedding very complicated string-building logic into a DB call. But it's a perfect example of what I'm talking about. And it's by no means the only example I've seen of this; it's all over the place in the CFML world. In its defence, at least the logic here is not really business logic; it's all SQL-string-building-logic. So it could be homed close to the DB call - maybe even a private method the same class - but it def doesn't belong inline like that.

Why? Because it's really complicated, ought to have been developed using TDD strategies, but to test this logic, one needs to actually hit a database, because all the logic is munged into the DB call! Ugh.

I thought to myself "but it's so easy to have built this in a purposeful string-builder method; I wonder why it wasn't done that way?". There's a coupla things that offer a challenge to this:

  • it's slinging the parameter values inline with <cfqueryparam> which is a bit "old skool", and also hasn't been necessary since CF11 (and its Railo / Lucee counterparts), provided one uses queryExecute which can correctly separate the SQL statement from the parameters for the statement.
  • It's got that preserveSingleQuotes in there.

preserveSingleQuotes is harder to deal with. Firstly a refresher. For some reason best known to someone at Allaire, all single quotes in any expressions within <cfquery> tags are automatically escaped. Whether appropriate to do so or not. To prevent this from happening, one needs to wrap the expression with a call to preserveSingleQuotes. But this only works if the call to preserveSingleQuotes is made within the <cfquery> tag. One cannot use preserveSingleQuotes outside that context (something not mentioned in any of the ColdFusion, Lucee or CFDocs, btw). This is such a daft thing for ColdFusion to have instigated. <cfquery> should just pass the string it's given to the DB. The end. If there's an expression within the SQL string that has single quotes that need to be escaped, then the code building the SQL string should do the escaping. Do it when it's needed. Don't do it unilaterally and then provide a mechanism to not do it. Blimey.

Anyway, that means that with all that string-building logic within that <cfquery>, it's not just a matter of lifting the string out into its own method, and building it in a (unit) testable fashion.

This got me looking at a way of stopping <cfquery> from escaping stuff, so that the preserveSingleQuotes call ain't needed.

Lucee provides a tag attribute on <cfquery> to disable it: psq, which if false, will not do the escaping. I do so wish they'd put more thought into the naming of that, but hey. Maybe escapeSingleQuotes or something. Not just a mostly meaningless TLA. Ah well: at least it's there and it works.

ColdFusion implements no such functionality, as far as I can tell.

During my googling as to how to address this, I came across one of my own old articles: "UnexpectedBuggy ColdFusion behaviour with CFML statements in <cfquery> tags". And within the grey box down at the bottom, there was mention of using UDFs within a <cfquery> tag varying the behaviour of quote-escaping. Then I thought… "hang on a sec…?" and tried an experiment.

Here's a baseline example showing the problem CFML causes for itself with quote escaping:

<cfscript>
numbers = queryNew("id,en,mi", "integer,varchar,varchar", [
    [1,"one","tahi"],
    [2,"two","rua"],
    [3,"three","toru"],
    [4,"four","wha"],
    [5,"five","rima"],
    [6,"six","ono"],
    [7,"seven","whitu"],
    [8,"eight","waru"],
    [9,"nine","iwa"],
    [10,"ten","tekau"]
])

sqlStatement = "
    SELECT *
    FROM numbers
    WHERE mi IN ('rua', 'toru', 'rima', 'whitu')
"
</cfscript>

<cftry>
    <cfquery name="primes" dbtype="query">#sqlStatement#</cfquery>
    <cfset dump =  primes>
    <cfcatch>
        <cfset dump = [cfcatch.message, cfcatch.detail, cfcatch.extendedinfo]>
    </cfcatch>
    <cffinally>
        <cfdump var="#dump#" label="no handling">
    </cffinally>
</cftry>

Here my SQL string has single quotes in it. It's supposed to have. They're legit. That's syntactically correct. But ColdFusion (and Lucee, but I'm showing this on ColdFusion) thinks it knows better, and messes it up:

It's "helpfully" escaped all the single quotes for me. Thus breaking the SQL statement. Cheers for that.

I can prevent this from happening by using preserveSingleQuotes:

<cfquery name="primes" dbtype="query">#preserveSingleQuotes(sqlStatement)#</cfquery>

Fine. And in reality, this could almost certainly be the way CFWheels handles the string building exercise. Build the string in a string builder method, then preserveSingleQuotes the whole lot.

But what's this thing about UDFs messing with stuff. This is the thing that kinda surprises me. Check this out:

<cfscript>
    function returnTheArgument(theArgument) {
        return theArgument
    }
</cfscript>

<!--- ... --->
<cfquery name="primes" dbtype="query">#returnTheArgument(sqlStatement)#</cfquery>
<!--- ... --->

This… works. My function doesn't do anything… but simply calling it on the SQL statement inline within the <cfquery> tags is enough to prevent the CFML engine from escaping any single quotes.

W … T.a. … F?

And this works on both CF and Lucee.


Anyway. That's "interesting". My conclusion as to how to solve the problem with the embedded preserveSingleQuotes thing is probably to just not embed it into the middle of things; just to wrap the whole statement with them. Which means all that nasty untestable logic can be lifted out of being inline there, and put in its own function that does the one thing. "Single Responsibility Principle" craziness, I know. But the road to this conclusion today was a winding one, that's for sure.

[mutters to self] I dunno what the CFML engines are thinking here. I really don't.

Righto.

--
Adam

Tuesday 18 May 2021

I play the game "how long will it take me to find a new bug in ColdFusion" again, after a few years

G'day:

Answer: about two minutes to guess what would likely be broken, and I think I had an initial repro created in another minute after that.

Last time I tried this it took 44min: "ColdFusion bug challenge: how quickly can one find a bug in ColdFusion 11?".

This time I figured "ah what's a language feature in CF2021? I reckon anything more than superficial use of one of those will break. Um… the rest operator. I bet it can't handle type-checking properly". Actually that didn't even take two minutes to type, but it was about how long I thought about it. So: given I was actually right about that guess, it took me - what - 20seconds? I'm not claiming to be smart or anything here. It's just so frickin easy to find things wrong with CFML that that is as long as it might take.

For context, here's the rest operator in action in a simple situation:

function untypedVariadicArguments(string s, ...x) {
    return arguments
}
result = untypedVariadicArguments("some string", 1, "two", ["three"])

writeDump(result)

The rest operator enables one to define variadic functions: ones that can take any number of arguments. Of course CFML already supported this: you can pass whatever you like to a UDF / method, and all the values will be there in the arguments scope, whether you specify them in the function signature or not. The difference is that in a variadic function, all the trailing arguments are put in the same one parameter value as an array. The output of above is:

 


 That is working properly.

The bug comes in when I try to type-check the that x parameter, eg: it can have any number of x values but they all need to be numbers. I would expect the syntax to be like this:

function typedVariadicArguments(string s, numeric ...x) {
    return arguments
}

In most curly-brace languages I can find that do type-checked variadic functions, the type declaration is done before the parameter declaration (irrespective of whether the parameter syntax is ...paramName or paramName..., as that can vary). See the examples on that page on Wikipedia I link to above on variadic functions. If I run that code in ColdFusion though, I get this:

Invalid CFML construct found on line 2 at column 52. on line 2

(Ha, by the by I also just found a wee bug in Lucee. Lucee does not support the rest operation, and when I accidentally tried to run this on Lucee cos I was not paying attention to what I was doing on trycf.com, it errored with "invalid argument definition on line 2": it's a parameter not an argument. The parameter is the definition. The argument is the value you pass)

So I went "OK well there's a good chance they just did it wrong with ColdFusion", so I tried it like this:

function typedVariadicArguments(string s, ...numeric x) {
    return arguments
}

Now that at least compiled, however it seems to be completely ignoring the rest operator. I'm testing it with the same test code as above:

result = typedVariadicArguments("some string", 1, "two", ["three"])

writeDump(result)

This is how one would expect it to work with just a signature like this:

function typedVariadicArguments(string s, numeric x) {
    return arguments
}

ColdFusion really ought not be going "buggered if I know what those dots are doing there: I'll just ignore them". Note that is understanding that as the rest operator. If you try two dots or four dots, you'll get a compile error.

I'm going with two bugs here, actually:

  • An implementation omission in that numeric ...x should work. That's the syntax they should have used, and it's not valid to not implement the type check there.
  • ColdFusion should not be ignoring th rest operator in a method signature if it doesn't know what to do with it. It should error.
  • OK a third bug I think. It'll allow the rest operator on the non-last parameter without erroring (untypedVariadicArguments(...s, x)): it doesn't work, but again it just ignores it.

I'm gonna stop looking now.

Am I missing something? There are all bugs right? I'm not getting something wrong?

Righto.

--
Adam

Sunday 9 May 2021

Setting up a MariaDB JDBC datasource in ColdFusion 2021 in Application.cfc

G'day:

This is how I needed to set my datasource in Application.cfc to get ColdFusion 2021 to connect to MariaDB. I'm writing this because I could not - for the life of me - find this information in any one place in the docs. Nor could I find it anywhere else via Google. I have pieced this together from various sources (including Lucee docs, which were more helpful than the Adobe ColdFusion documentation), and want to put it in one place for my own future reference, or should anyone else need to know how to set up a MariaDB datasouce in ColdFusion using their JDBC driver. The params should also work for any other JDBC datasource.

Apologies for that fairly SEO-heavy opening paragraph.

OK, so it's this:

component {

    this.datasources["test"] = {
        host = "localhost",
        port = 3306,
        driver = "mariadb",
        class = "org.mariadb.jdbc.Driver",
        url = "jdbc:mariadb://localhost:3306/database_name?user=user_name&password=user_password",
        database = "database_name",
        username = "user_name",
        password = "user_password", 
        custom = {
            useUnicode = true,
            characterEncoding = "UTF-8"
        }
    }
    this.datasource = "test"
}

After checking up on what Sean says in the comment below, I was able to pare this back to:

this.datasources["test"] = {
    class = "org.mariadb.jdbc.Driver",
    url = "jdbc:mariadb://localhost:3306/cfmlindocker?useUnicode=true&characterEncoding=UTF-8",
    username = "cfmlindocker",
    password = server.system.environment.MYSQL_PASSWORD
}

It failed if I did not have the username and password keys; but I did not need them in the URL. And I can pass the custom stuff in the URL, rather than specifying them separately.

Cheers for encouraging me to investigate further, Sean!

I got the MariaDB jar file from Download and Install MariaDB Connector/J. I also found the class name for the URL in there. That jar should be dropped into [coldfusion directory]/]WEB-INF/lib

The driver value seems to just be a label.

Note that on Lucee they seem to use connectionString rather than url. I dunno if url also works, and CBA checking right now.

ColdFusion might give an unhelpful error if it does't like the datasource settings. Instead of saying "I don't like those datasource settings", or "they don't work", it says "Datasource test could not be found". It was easy to find, pal: it's right there in the Application.cfc file. I just had a param wrong. I especially saw this when I had connectionString rather then url.

I would normally store the password in the environment, not hard-code into a source-controlled file, eg: server.system.environment.TEST_DSN_PASSWORD (both ColdFusion and Lucee put any environment variables in server.system.environment. Or you could use an environment-specific non-source controlled config file or something. Just don't put it directly in Application.cfc

Here's some more SEO for you: so that was the Application.cfc configuration settings for ColdFusion 2021 to connect to a MariaDB database using their JDBC connection.

Righto.

--
Adam

Thursday 6 May 2021

Lucee: what now for goodness sake

G'day:

I'm just writing this here because it's too long to put in a message in the CFML/Lucee Slack channel, and so I can get some eyes on it.

Consider this code (test.cfm):

<cfoutput>
<cfset relativeFilePath = "../getCanonicalPathIssue/targetfile.cfm">
Relative file path: [#relativeFilePath#]<br>
Does relative path exist: [#fileExists(relativeFilePath)#]<br>
<br>

<cfset expandedPath = expandPath(relativeFilePath)>
Expanded file path: [#expandedPath#]<br>
Does expanded path exist: [#fileExists(expandedPath)#]<br>
<br>

<cfset canonicalPathFromRelativeFilePath = getCanonicalPath(relativeFilePath)>
Canonical path from relative path: [#canonicalPathFromRelativeFilePath#]<br>
Does canonical pathh from relative path exist: [#fileExists(canonicalPathFromRelativeFilePath)#]<br>
<br>

<cfset canonicalPathFromExpandedPath = getCanonicalPath(expandedPath)>
Canonical path from expanded path: [#canonicalPathFromExpandedPath#]<br>
Does canonical path from expanded path exist: [#fileExists(canonicalPathFromExpandedPath)#]<br>
<br>

<cfset directory = getDirectoryFromPath(canonicalPathFromRelativeFilePath)>
Directory: #directory#<br>
Directory contents and do they exist:<br>
#directoryList(directory).reduce((buffer="", filePath) => buffer & "#filePath#: #fileExists(filePath)#<br>")#
</cfoutput>

And this is its output:

Relative file path: [../getCanonicalPathIssue/targetfile.cfm]
Does relative path exist: [false]

Expanded file path: [/var/www/public/nonWheelsTests/getCanonicalPathIssue/targetfile.cfm]
Does expanded path exist: [true]

Canonical path from relative path: [/var/www/public/nonWheelsTests/getCanonicalPathIssue/targetFile.cfm]
Does canonical path from relative path exist: [false]

Canonical path from expanded path: [/var/www/public/nonWheelsTests/getCanonicalPathIssue/targetfile.cfm]
Does canonical path from expanded path exist: [true]

Directory: /var/www/public/nonWheelsTests/getCanonicalPathIssue/
Directory contents and do they exist:
/var/www/public/nonWheelsTests/getCanonicalPathIssue/test.cfm: true
/var/www/public/nonWheelsTests/getCanonicalPathIssue/targetfile.cfm: true

Basically I've got a second file (targetfile.cfm) in the same directory as I'm running this code from (test.cfm), and I'm giving Lucee a (valid) relative path to it, and asking some questions. Everything goes OK until I get to the result of getCanonicalPath(relativeFilePath): it's upper-cased part of the file name!??! Also note that expandPath(relativeFilePath) gets it right. And that getCanonicalPath(expandedPath) also gets it right.

The last bit of the code just shows what's def in the directory concerned.

I actually know what's causing the problem. A coupla hours ago I renamed the file from targetFile.cfm to targetfile.cfm. note the change in capitalisation, and how the old version matches the bad value getCanonicalPath is coming up with. So Lucee is caching that for some reason. I also found where to uncache it, but am buggered if I know why this setting caches file paths:

That's supposed to be about how often the contents of the file are checked; nothing about file paths. But if I set it to "Always (Bad)", then the problem goes away. It's clearly this lesser used definition of "bad", that means "actually works".

So that's an hour or so of my life I'll never get back. Thanks.


For shits and giggles I decided to run this on ColdFusion:

I'd give ColdFusion a pass here for choking on a relative path if not for two things:

  • it's specifically documented as dealing with them: "Absolute or relative path of a directory or to a file."
  • The path is a valid relative path. There's no trickery with the casing going on here, or caching or anything like that. The relative path I'm providing is the path to that file, relative to where the path is being used.

Oh well. I suppose I'll check with ppl to see if there's a reason why this behaviour on each platform, isn't wrong in the way I claim it is… and then raise some bugs. I'll cross ref once I've done that, but it'll be tomorrow now.

Righto.

--
Adam

Friday 23 October 2020

ColdFusion: looking at an issue Mingo had with ehcache and cachePut and cacheGet

 G'day:

Bloody Coldfusion. OK so why am I writing about a ColdFusion issue? Well about 80% of it is "not having a great deal else to do today", about 10% of being interested in this issue Mingo found. And 10% it being an excuse to mess around with Docker a bit. I am currently - and slowly - teaching myself about Docker, so there's some practise for me getting a ColdFusion instance up and running on this PC (which I no-longer have any type of CFML engine installed on).

OK so what's the issue. Mingo posted this on Twitter:


Just in case you wanna run that code, here it is for copy and paste:
foo = { bar = 1 };
cachePut( 'foobar', foo );

foo.bar = 2;

writeDump( cacheGet( 'foobar' ) );


Obviously (?) what one would expect here is {bar:1}. What gets put into cache would be a copy of the struct right?

Well... um... here goes...

/opt/coldfusion/cfusion/bin $ ./cf.sh
ColdFusion started in interactive mode. Type 'q' to quit.
cf-cli>foo = { bar = 1 };
struct
BAR: 1

cf-cli>cachePut( 'foobar', foo );
cf-cli>foo.bar = 2;
2
cf-cli>writeDump( cacheGet( 'foobar' ) );
struct
BAR: 2

cf-cli>

... errr... what?

It looks like ColdFusion is like only putting a reference to the struct into cache. So any code changing the data in the struct is changing it in CFML as well as changing it in cache. This does not seem right.

I did a quick google and found a ticket in Adobe's system about this: CF-3989480 - cacheGet returns values by reference. The important bit to note is that it's closed with


Not a great explanation from Adobe there, fortunately Rob Bilson had commented further up with a link to a cached version of an old blog article of his, explaining what's going on: "Issue with Ehcache and ColdFusion Query Objects". It's a slightly different situation, but it's the same underlying "issue". Just to copy and paste the relevant bit from his article:

Update: It looks like this is actually expected behavior in Ehcache. Unfortunately, it's not documented in the ColdFusion documentation anywhere, but Ehcache actually has two configurable parameters (as of v. 2.10) called copyOnRead and copyOnWrite that determine whether values returned from the cache are by reference or copies of the original values. By default, items are returned by reference. Unfortunately we can't take advantage of these parameters right now as CF 9.0.1 implements Ehcache 2.0.


I decided to have a look what we could do about this on ColdFusion 2018, hoping that its embedded implementation of Ehcache has been updated since Rob wrote that in 2010.

Firstly I checked the Ehcache docs for these two settings: copyOnWrite and copyOnRead. This is straight forward (from "copyOnRead and copyOnWrite cache configuration"):


<cache name="copyCache"
    maxEntriesLocalHeap="10"
    eternal="false"
    timeToIdleSeconds="5"
    timeToLiveSeconds="10"
    copyOnRead="true"
    copyOnWrite="true">
  <persistence strategy="none"/>
  <copyStrategy class="com.company.ehcache.MyCopyStrategy"/>
</cache>


The docs also confirm these are off by default

Next where's the file?

/opt/coldfusion $ find . -name ehcache.xml
./cfusion/lib/ehcache.xml
/opt/coldfusion $

Cool. BTW I just guessed at that file name.

So in there we have this (way down at line 471):


<!--
Mandatory Default Cache configuration. These settings will be applied to caches
created programmtically using CacheManager.add(String cacheName).

The defaultCache has an implicit name "default" which is a reserved cache name.
-->
<defaultCache
    maxElementsInMemory="10000"
    eternal="false"
    timeToIdleSeconds="86400"
    timeToLiveSeconds="86400"
    overflowToDisk="false"
    diskSpoolBufferSizeMB="30"
    maxElementsOnDisk="10000000"
    diskPersistent="false"
    diskExpiryThreadIntervalSeconds="3600"
    memoryStoreEvictionPolicy="LRU"
    clearOnFlush="true"
    statistics="true"
/>


That looked promising, so I updated it to use copyOnWrite:



<defaultCache
    maxElementsInMemory="10000"
    eternal="false"
    timeToIdleSeconds="86400"
    timeToLiveSeconds="86400"
    overflowToDisk="false"
    diskSpoolBufferSizeMB="30"
    maxElementsOnDisk="10000000"
    diskPersistent="false"
    diskExpiryThreadIntervalSeconds="3600"
    memoryStoreEvictionPolicy="LRU"
    clearOnFlush="true"
    statistics="true"
    copyOnWrite="false"
/>


Whatever I put into cache, I want it to be decoupled from the code immediately, hence doing the copy-on-write.

I restarted CF and ran the code again:

/opt/coldfusion/cfusion/bin $ ./cf.sh
ColdFusion started in interactive mode. Type 'q' to quit.
cf-cli>foo = { bar = 1 };
struct

BAR: 1

cf-cli>cachePut( 'foobar', foo );
cf-cli>foo.bar = 2;
2

cf-cli>writeDump( cacheGet( 'foobar' ) );

struct
BAR: 1

cf-cli>


Yay! We are getting the "expected" result now: 1

Don't really have much else to say about this. I'm mulling over writing down what I did to get ColdFusion 2018 up and running via Docker instead of installing it. Let's see if I can be arsed...

Righto.

-- 

Adam

Saturday 29 April 2017

CFML: Lucee 13 v 414 v 6 ColdFusion. We have a "winner"

G'day:
Argh. CFML again. I'm doing this because I was a bit of a meanie to Brad on the CFML Slack Channel, and promised I'd try to play nice with one of our current exercises set out by Ryan:

Todays challenge: post something you can do in CFML that you don’t think the majority of people know. Maybe its a little-known function, or some java integration, or a technique from another language that isn’t common in CFML - or even just something that you’ve learned from this slack group - if you didn’t know it before someone else probably doesn’t know it now. Doesn’t even have to be a good idea or something you’ve ever actually used but bonus points for useful stuff. Can apply to ACF [ed: Adobe ColdFusion, or could just  be ColdFusion or CF. Dunno why ppl say "ACF"] or Lucee but bonus points for something universal.
This is a great idea. And whilst I am trying to not use CFML any more due to it not being a good use of my time, I still try to help other people with it on the Slack channel, and CFML does do some good stuff.

I couldn't think of anything interesting that no-one else would know about, but I figured I'd show people a different approach to something. Just as food for thought.

In CFML, when hitting the DB one receives the data back not as an array of objects or an array of structs, but as a single "Query Object", which internally contains the data, and exposes access to said data by various functions, methods, and statements. This is fine which restricted to CFML code, but increasingly code these days needs to interchange with other systems, and sometimes it's a bit of a pain converting from a Query object to something more interchangeable like... an array of objects (or structs in CFML). There's no native method to do this, but it's easy enough to do with a reduction:

// fake a DB call
numbers = queryNew("id,en,mi", "integer,varchar,varchar", [
    [1,"one","tahi"],
    [2,"two","rua"],
    [3,"three","toru"],
    [4,"four","wha"]
]);

numbersAsArray = numbers.reduce(function(elements, row){
    return elements.append(row);
}, []);

writeDump({
    numbers=numbers,
    numbersAsArray=numbersAsArray
});

(run this yerself on trycf.com)

Here I'm faking the DB hit, but that can be chained together too:

numbersAsArray = queryExecute("SELECT * FROM numbers")
    .reduce(function(rows, row){
        return rows.append(row);
    }, []);

All good, and nice and compact. Not very inspiring though.

So as a next step I decided to add some more method calls in there. Let's say I only wanted numbers greater than 5, I wanted the keys in the structs to be different from the ones in the DB, and I wanted it sorted in reverse order. Obviously this is all easily done in the DB:

numbers = queryExecute("
    SELECT id AS value, en AS english, mi AS maori
    FROM numbers
    WHERE id > 5
    ORDER BY id DESC
");

But sometimes we have to play the hand we've been dealt, and we cannot change the recordset we're getting back.

Of course we could also do this with a query-of-query too, but CFML's internal SQL implementation offers... "challenges" of its own, so let's forget about that.

Anyway, I ended up with this:

numbersAsArray = queryExecute("SELECT * FROM numbers")
    .filter(function(row){
        return row.id > 5;
    })
    .map(function(row){
        return {value=row.id, english=row.en, maori=row.mi};
    }, queryNew("value,english,maori"))
    .reduce(function(rows=[], row){
        return rows.append(row);
    })
    .sort(function(e1,e2){
        return e2.value - e1.value;
    })
;

That's all straight forward. We retain only the rows we want with filter, we change the column names with map, and again convert the result to be an array of structs with reduce, then finally we re-order them with sort.

That's cool. And the result is...

YES

Huh? I mean literally... the value of numbersAsArray was "YES". Groan. For the uninitiated, the string "YES" is a boolean truthy value in CFML (CFML also has true and false, but it favours "YES" and "NO" for some common-sense-defying reason). And indeed some old-school CFML functions which should have been void functions instead return "YES". But the method versions should not: methods - where sensible - should return a result so they can be chained to the next method call. Like I'm trying to do here: the end of the chain should be the answer.

I could see how this would continue, so I decided to start keeping score of the bugs I found whilst undertaking this exercise.

ColdFusion: 1.

I pared my code back to get rid of any irrelevant bits for the sake of a SSCCE, and ran it on Lucee for comparison. I just got an error, but not one I expected.

I have an Application.cfc set up her to define my datasource, and I had this:

component {
    this.name = getCurrentTemplatePath().hash();
    this.datasource = "scratch_mysql";
}

Now I didn't need the DSN for this SSCCE, but the Application.cfc was still running obviously. And it seems Lucee does not implement the hash method:

Lucee 5.1.3.18 Error (expression)
MessageNo matching Method/Function for String.hash() found

Lucee joins the scoring:

ColdFusion 1 - 1 Lucee

I use the hash function instead of the method to name my application, and at least now Lucee gets to the code I want to run.

numbers = queryNew("id,en,mi", "integer,varchar,varchar", [
    [1,"one","tahi"],
    [2,"two","rua"],
    [3,"three","toru"],
    [4,"four","wha"]
]);

reversed = numbers.sort(function(n1,n2){
    return n2.id - n1.id;
});

writeDump([reversed,numbers]);

And the result:

Array
1
Query
Execution Time: 0 ms
Record Count: 4
Cached: No
Lazy: No 
idenmi
14fourwha
23threetoru
32tworua
41onetahi
2
Query
Execution Time: 0 ms
Record Count: 4
Cached: No
Lazy: No 
idenmi
14fourwha
23threetoru
32tworua
41onetahi

Hang on. Lucee's changed the initial query as well. If it's returning the result, then it should not also be changing the initial value. But I'm gonna say this is due to a sort of sideways compatibility with ColdFusion:

array
1YES
2
query
enidmi
1four4wha
2three3toru
3two2rua
4one1tahi

As it doesn't return the value from the method, it makes sense to act on the initial value itself.

But if Lucee's gonna copy ColdFusion (which it should be) then it should be copying it properly.

ColdFusion 1 - 2 Lucee

To mitigate this, I decide to duplicate the initial query first:

reversed = numbers.duplicate();
reversed.sort(function(n1,n2){
    return n2.id - n1.id;
});

This works fine on ColdFusion:

array
1
query
ENIDMI
1four4wha
2three3toru
3two2rua
4one1tahi
2
query
enidmi
1one1tahi
2two2rua
3three3toru
4four4wha

But breaks on Lucee:

 Error:
No matching Method/Function for Query.duplicate() found on line 9

Hmmm. Well I s'pose the duplicate method doesn't seem to be documented, but it was added in CF2016. This is getting in my way, so I'm still chalking it up to an incompat in Lucee:

ColdFusion 1 - 3 Lucee

(I probably should add a documentation bug with ColdFusion too, but that's a separate matter).

Anyway, that's mostly an aside. In my example what I am sorting is an intermediary value anyhow, so it doesn't matter that it gets sorted as well as being returned. For my purposes I am not using ColdFusion any more, just Lucee, as I'm specifically showing the method chaining thing, and we already know ColdFusion messes this up with how sort works.

So here we go, all done:

numbers = queryExecute("SELECT * FROM numbers")
    .map(function(row){
        return {value=row.id, english=row.en, maori=row.mi};
    }, queryNew("value,english,maori"))
    .filter(function(row){
        return row.value > 5;
    })
    .sort(function(e1,e2){
        return e2.value - e1.value;
    })
    .reduce(function(rows, row){
        return rows.append(row);
    }, [])
;

writeDump(numbers);

And the output:

Array
1
Struct
en
Empty:null
english
stringten
id
Empty:null
maori
stringtekau
mi
Empty:null
value
number10
2
Struct
en
Empty:null
english
stringnine
id
Empty:null
maori
stringiwa
mi
Empty:null
value
number9
3
Struct
en
Empty:null
english
stringeight
id
Empty:null
maori
stringwaru
mi
Empty:null
value
number8
4
Struct
en
Empty:null
english
stringseven
id
Empty:null
maori
stringwhitu
mi
Empty:null
value
number7
5
Struct
en
Empty:null
english
stringsix
id
Empty:null
maori
stringono
mi
Empty:null
value
number6

OK, now WTF is going on? Lucee hasn't remapped the columns properly. it's added the new ones, but it's also included the old ones. It ain't supposed to do that. Contrast ColdFusion & Lucee with some more simple code:

numbers = queryNew("id,en,mi", "integer,varchar,varchar", [
    [1,"one","tahi"],
    [2,"two","rua"],
    [3,"three","toru"],
    [4,"four","wha"]
]);
remapTemplate = queryNew("value,english,maori"); 

reMapped = numbers.map(function(row){
    return {value=row.id, english=row.en, maori=row.mi};
}, remapTemplate);

writeDump(reMapped);

ColdFusion:

query
ENGLISHMAORIVALUE
1onetahi1
2tworua2
3threetoru3
4fourwha4

Lucee:

Query
Execution Time: 0 ms
Record Count: 4
Cached: No
Lazy: No 
valueenglishmaoriidenmi
11onetahi
Empty:null
Empty:null
Empty:null
22tworua
Empty:null
Empty:null
Empty:null
33threetoru
Empty:null
Empty:null
Empty:null
44fourwha
Empty:null
Empty:null
Empty:null

Sigh. What's supposed to be returned by a map operation on a query is a new query with only the columns from that remapTemplate query. That's what it's for.

ColdFusion 1 - 4 Lucee

On a whim I decided to check what Lucee did to the remapTemplate:

Query
Execution Time: 0 ms
Record Count: 4
Cached: No
Lazy: No 
valueenglishmaoriidenmi
11onetahi
Empty:null
Empty:null
Empty:null
22tworua
Empty:null
Empty:null
Empty:null
33threetoru
Empty:null
Empty:null
Empty:null
44fourwha
Empty:null
Empty:null
Empty:null

ColdFusion 1 - 5 Lucee

This situation is slightly contrived as I don't care about that query anyhow. But what if I was using an extant query which had important data in it?

remapTemplate = queryNew("value,english,maori", "integer,varchar,varchar", [
    [5, "five", "rima"]
]);

So here I have a query with some data in it, and for whatever reason I want to remap the numbers query to have the same columns as this one. But obviously I don't want it otherwise messed with. Lucee mungs it though:

Query
Execution Time: 0 ms
Record Count: 5
Cached: No
Lazy: No 
valueenglishmaoriidenmi
15fiverima
21onetahi
Empty:null
Empty:null
Empty:null
32tworua
Empty:null
Empty:null
Empty:null
43threetoru
Empty:null
Empty:null
Empty:null
54fourwha
Empty:null
Empty:null
Empty:null

Not cool.

But wait. We're not done yet. Let's go back to some of my original code I was only running on ColdFusion:

numbersAsArray = queryExecute("SELECT id,mi FROM numbers LIMIT 4")
    .reduce(function(rows=[], row){
        return rows.append(row);
    })
;
writeDump(numbersAsArray);

This was part of the first example, I've just ditched the filter, map and sort: focusing on the reduce. On ColdFusion I get what I'd expect:

array
1
struct
ID1
MItahi
2
struct
ID2
MIrua
3
struct
ID3
MItoru
4
struct
ID4
MIwha

On Lucee I get this:

Lucee 5.1.3.18 Error (expression)
Messagecan't call method [append] on object, object is null
StacktraceThe Error Occurred in
queryReduceSimple.cfm: line 4 
2: numbersAsArray = queryExecute("SELECT id,mi FROM numbers LIMIT 4")
3: .reduce(function(rows=[], row){
4: return rows.append(row);
5: })
6: ;

Hmmm. What's wrong now? Oh. It's this:

reduce(function(rows=[], row)

Notice how I am giving a default value to the first argument there. This doesn't work in Lucee.

ColdFusion 1 - 6 Lucee

This is easy to work around, because reduce functions take an optional last argument which is the initial value for that first argument to the callback, so I can just re-adjust the code like this:

.reduce(function(rows , row){
    return rows.append(row);
}, [])

OK, at this point I give up. Neither implementation of CFML here - either ColdFusion's or Lucee's - is good enough to do what I want to do. Oddly: Lucee is far worse on this occasion than ColdFusion is. That's disappointing.

So currently the score is 1-6 to Lucee. How did I get to 4-13?

I decided to write some test cases with TestBox to demonstrate what ought to be happening. And with the case of duplicate, I tested all native data-types I can think of:

  • struct
  • array
  • query
  • string
  • double (I guess "numeric" in CFML)
  • datetime
  • boolean
  • XML

Lucee failed the whole lot, and ColdFusion failed on numerics and booleans. As this is undocumented behaviour this might seem a bit harsh, but I'm not counting documentation errors against ColdFusion in this case. Also there's no way I'd actually expect numerics and booleans to have a duplicate method... except for the fact that strings do. Now this isn't a method bubbling through from java.lang.String, nor is it some Java method of ColdFusion's string implementation (they're just java.lang.Strings). This is an actively-created CFML member function. So it seems to me that - I guess for the sake of completeness - they implemented for "every" data type... I mean it doesn't make a great deal of sense on a datetime either, really, does it? So the omission of it from numerics and booleans is a bug to me.

This leaves the score:

ColdFusion 3 - 13 Lucee

The last ColdFusion point was cos despite the fact that with the sort operation it makes sense to alter the initial object if the method doesn't return the sorted one... it just doesn't make sense that the sort method has been implemented that way. It should leave the original object alone and return a new sorted object.

ColdFusion 4 - 13 Lucee

My test cases are too long to reproduce here, but you can see 'em on Github: Tests.cfc.

Right so...

Ah FFS.

... I was about to say "right, so that's that: not a great experience coming up with something cool to show to the CFMLers about CFML. Cos shit just didn't work. I found 17 bugs instead".

But I just had a thought about how sort methods work in ColdFusion, trying to find examples of where sort methods return the sorted object, rather than doing an inline support. And I I've found more bugs with both ColdFusion and Lucee.

Here are the cases:

component extends="testbox.system.BaseSpec" {
    function run() {
        describe("Other sort tests", function(){
            it("is a baseline showing using BIFs as a callback", function(){
                var testString = "AbCd";
                var applyTo = function(object, operation){
                    return operation(object);
                };

                var result = applyTo(testString, ucase);
                
                expect(result).toBeWithCase("ABCD");
            });
            describe("using arrays", function(){
                it("can use a function expression calling compareNoCase as a string comparator when sorting", function(){
                    var arrayToSort = ["d","C","b","A"];
                    
                    arrayToSort.sort(function(e1,e2){
                        return compareNoCase(e1, e2);
                    });
                    
                    expect(arrayToSort).toBe(["A","b","C","d"]);
                });
                it("can use the compareNoCase BIF as a string comparator when sorting", function(){
                    var arrayToSort = ["d","C","b","A"];
                    
                    arrayToSort.sort(compareNoCase);
                    
                    expect(arrayToSort).toBe(["A","b","C","d"]);
                });
            });
            describe("using lists", function(){
                it("can use a function expression calling compareNoCase as a string comparator when sorting", function(){
                    var listToSort = "d,C,b,A";
                    
                    var sortedList = listToSort.listSort(function(e1,e2){
                        return compareNoCase(e1, e2);
                    });
                    
                    expect(sortedList).toBe("A,b,C,d");
                    expect(listToSort).toBe("d,C,b,A");
                });
                it("can use the compareNoCase BIF as a string comparator when sorting", function(){
                    var listToSort = "d,C,b,A";
                    
                    var sortedList = listToSort.listSort(compareNoCase);
                    
                    expect(sortedList).toBe("A,b,C,d");
                    expect(listToSort).toBe("d,C,b,A");
                });
            });
        });
    }
}

What I'm doing here is using CFML built-in functions as the callbacks for a sort operation. This should work, because the sort operation needs a comparator function which works exactly like compare / compareNoCase: returns <0, 0, >0 depending on whether the first argument is "less than", "equal to" or "greater than" the second object according to the sort rules. As far as strings go, the built-in functions compare and compareNoCase do this. So they should be usable as callbacks. Since I think CF2016 built-in-functions have been first-class functions, so should be usable wherever something expects a function as an argument.

The first test demonstrates this in action. I have a very contrived situation where I have a function applyTo, which takes an object and a function to apply to it. In the test I pass-in the built-in function ucase as the operation. This test passes fine on ColdFusion; fails on Lucee.

ColdFusion 4 - 14 Lucee

So after I've demonstrated the technique should work, I try to use compareNoCase as the comparator for an array sort. it just doesn't work: it does nothing on ColdFusion, and on Lucee it still just errors (not gonna count that against Lucee, as it's the same bug as in the baseline test).

ColdFusion 5 - 14 Lucee

Next I try to use it on a listSort. This time ColdFusion errors as well. So this is a different bug than the doesn't-do-anything one for arrays.

ColdFusion 6 - 14 Lucee

Here are the results for just this latter tranche of test cases:

ColdFusion:



Lucee:



Fuck me, I've giving up.

This has been the most shit experience I've ever had trying to get CFML to do something. I don't think any of this code is edge-case stuff. Those higher-order functions are perhaps not as commonly used as they ought to be by the CFMLers out there, but I'm just... trying to use them.

So... sorry Brad & Ryan... I tried to come up with something worth showing to the mob that's useful in CFML, but I've failed. And my gut reaction to this exercise is that CFML can go fuck itself, basically.

Righto.

--
Adam