TIL: two things about reduce operations

G'day:

When working through yesterday's article ("TDD, code puzzle, recursing reductions"), I learned two new things about how reduce operations work.

The first thing was that I was having problems with defaulting the initial value of my callback. I had something like this:

array = ["tahi", "rua", "toru", "wha"]

list = array.reduce((result="", value) => {
    return result.listAppend(value.ucase())
})
writeDump(list)

When I was running this on Lucee I was getting an error: can't call method [listAppend] on object, object is null. The callback was not receiving its default value.

After a while I clocked that I had Lucee's "Complete Null Support" switched on, and this was ruining things for everyone. Why?

Because the method signature for Lucee's Array.reduce has two forms:

Array.reduce(callback)

Array.reduce(callback, initialValue)

I was using the first form, but Lucee still needs to cater for the second form. What's happening is that when Lucee encounters the first form, it's saying "OK so no initialValue. But… null support is switched on, so I'll just pass null for that". So when the callback is called, the initial value is null, so the default in the function signature for the callback is "not needed". Harrumph. I don't like this much, but I understand why it is the way it is I s'pose.

The easy fix is to just use the second form all the time.

---

Brad Wood pointed out another pitfall when using the callback's default value for that first argument. Consider this variant of the code from above:

array = []

list = array.reduce((result="", value) => {
    return result.listAppend(value.ucase())
})
writeDump(list)

There's no elements in the array, so the callback is never called, so the initial value for the result is never set. So on Lucee with null support set, the result is null. On ColdFusion, and on Lucee with null support switched off, it errors along the lines of variable [list] doesn't exist. It's worthwhile remembering that the default value on the callback parameter is only for the callback, and it simply "coincidentally" works as a default value for the entire operation if the callback is used. To give the operation a default starting value, you need to set it on the reduce call, not the callback signature.

Cheers for pointing this out to me, Brad. It seems obvious once you mentioned it, but it had never occurred to me.

That's it. Two things I learned. Cool.

Righto.

--
Adam

CFML: Lucee 13 v 414 v 6 ColdFusion. We have a "winner"

G'day:
Argh. CFML again. I'm doing this because I was a bit of a meanie to Brad on the CFML Slack Channel, and promised I'd try to play nice with one of our current exercises set out by Ryan:

Todays challenge: post something you can do in CFML that you don’t think the majority of people know. Maybe its a little-known function, or some java integration, or a technique from another language that isn’t common in CFML - or even just something that you’ve learned from this slack group - if you didn’t know it before someone else probably doesn’t know it now. Doesn’t even have to be a good idea or something you’ve ever actually used but bonus points for useful stuff. Can apply to ACF [ed: Adobe ColdFusion, or could just  be ColdFusion or CF. Dunno why ppl say "ACF"] or Lucee but bonus points for something universal.

This is a great idea. And whilst I am trying to not use CFML any more due to it not being a good use of my time, I still try to help other people with it on the Slack channel, and CFML does do some good stuff.

I couldn't think of anything interesting that no-one else would know about, but I figured I'd show people a different approach to something. Just as food for thought.

In CFML, when hitting the DB one receives the data back not as an array of objects or an array of structs, but as a single "Query Object", which internally contains the data, and exposes access to said data by various functions, methods, and statements. This is fine which restricted to CFML code, but increasingly code these days needs to interchange with other systems, and sometimes it's a bit of a pain converting from a Query object to something more interchangeable like... an array of objects (or structs in CFML). There's no native method to do this, but it's easy enough to do with a reduction:

// fake a DB call
numbers = queryNew("id,en,mi", "integer,varchar,varchar", [
    [1,"one","tahi"],
    [2,"two","rua"],
    [3,"three","toru"],
    [4,"four","wha"]
]);

numbersAsArray = numbers.reduce(function(elements, row){
    return elements.append(row);
}, []);

writeDump({
    numbers=numbers,
    numbersAsArray=numbersAsArray
});

(run this yerself on trycf.com)

Here I'm faking the DB hit, but that can be chained together too:

numbersAsArray = queryExecute("SELECT * FROM numbers")
    .reduce(function(rows, row){
        return rows.append(row);
    }, []);

All good, and nice and compact. Not very inspiring though.

So as a next step I decided to add some more method calls in there. Let's say I only wanted numbers greater than 5, I wanted the keys in the structs to be different from the ones in the DB, and I wanted it sorted in reverse order. Obviously this is all easily done in the DB:

numbers = queryExecute("
    SELECT id AS value, en AS english, mi AS maori
    FROM numbers
    WHERE id > 5
    ORDER BY id DESC
");

But sometimes we have to play the hand we've been dealt, and we cannot change the recordset we're getting back.

Of course we could also do this with a query-of-query too, but CFML's internal SQL implementation offers... "challenges" of its own, so let's forget about that.

Anyway, I ended up with this:

numbersAsArray = queryExecute("SELECT * FROM numbers")
    .filter(function(row){
        return row.id > 5;
    })
    .map(function(row){
        return {value=row.id, english=row.en, maori=row.mi};
    }, queryNew("value,english,maori"))
    .reduce(function(rows=[], row){
        return rows.append(row);
    })
    .sort(function(e1,e2){
        return e2.value - e1.value;
    })
;

That's all straight forward. We retain only the rows we want with filter, we change the column names with map, and again convert the result to be an array of structs with reduce, then finally we re-order them with sort.

That's cool. And the result is...

YES

Huh? I mean literally... the value of numbersAsArray was "YES". Groan. For the uninitiated, the string "YES" is a boolean truthy value in CFML (CFML also has true and false, but it favours "YES" and "NO" for some common-sense-defying reason). And indeed some old-school CFML functions which should have been void functions instead return "YES". But the method versions should not: methods - where sensible - should return a result so they can be chained to the next method call. Like I'm trying to do here: the end of the chain should be the answer.

I could see how this would continue, so I decided to start keeping score of the bugs I found whilst undertaking this exercise.

ColdFusion: 1.

I pared my code back to get rid of any irrelevant bits for the sake of a SSCCE, and ran it on Lucee for comparison. I just got an error, but not one I expected.

I have an Application.cfc set up her to define my datasource, and I had this:

component {
    this.name = getCurrentTemplatePath().hash();
    this.datasource = "scratch_mysql";
}

Now I didn't need the DSN for this SSCCE, but the Application.cfc was still running obviously. And it seems Lucee does not implement the hash method:

Lucee 5.1.3.18 Error (expression)
Message	No matching Method/Function for String.hash() found

Lucee joins the scoring:

ColdFusion 1 - 1 Lucee

I use the hash function instead of the method to name my application, and at least now Lucee gets to the code I want to run.

numbers = queryNew("id,en,mi", "integer,varchar,varchar", [
    [1,"one","tahi"],
    [2,"two","rua"],
    [3,"three","toru"],
    [4,"four","wha"]
]);

reversed = numbers.sort(function(n1,n2){
    return n2.id - n1.id;
});

writeDump([reversed,numbers]);

And the result:

Array
1	Query Execution Time: 0 ms Record Count: 4 Cached: No Lazy: No  id en mi 1 4 four wha 2 3 three toru 3 2 two rua 4 1 one tahi
2	Query Execution Time: 0 ms Record Count: 4 Cached: No Lazy: No  id en mi 1 4 four wha 2 3 three toru 3 2 two rua 4 1 one tahi

Hang on. Lucee's changed the initial query as well. If it's returning the result, then it should not also be changing the initial value. But I'm gonna say this is due to a sort of sideways compatibility with ColdFusion:

array
1	YES
2	query en id mi 1 four 4 wha 2 three 3 toru 3 two 2 rua 4 one 1 tahi

As it doesn't return the value from the method, it makes sense to act on the initial value itself.

But if Lucee's gonna copy ColdFusion (which it should be) then it should be copying it properly.

ColdFusion 1 - 2 Lucee

To mitigate this, I decide to duplicate the initial query first:

reversed = numbers.duplicate();
reversed.sort(function(n1,n2){
    return n2.id - n1.id;
});

This works fine on ColdFusion:

array
1	query EN ID MI 1 four 4 wha 2 three 3 toru 3 two 2 rua 4 one 1 tahi
2	query en id mi 1 one 1 tahi 2 two 2 rua 3 three 3 toru 4 four 4 wha

But breaks on Lucee:

 Error:
No matching Method/Function for Query.duplicate() found on line 9

Hmmm. Well I s'pose the duplicate method doesn't seem to be documented, but it was added in CF2016. This is getting in my way, so I'm still chalking it up to an incompat in Lucee:

ColdFusion 1 - 3 Lucee

(I probably should add a documentation bug with ColdFusion too, but that's a separate matter).

Anyway, that's mostly an aside. In my example what I am sorting is an intermediary value anyhow, so it doesn't matter that it gets sorted as well as being returned. For my purposes I am not using ColdFusion any more, just Lucee, as I'm specifically showing the method chaining thing, and we already know ColdFusion messes this up with how sort works.

So here we go, all done:

numbers = queryExecute("SELECT * FROM numbers")
    .map(function(row){
        return {value=row.id, english=row.en, maori=row.mi};
    }, queryNew("value,english,maori"))
    .filter(function(row){
        return row.value > 5;
    })
    .sort(function(e1,e2){
        return e2.value - e1.value;
    })
    .reduce(function(rows, row){
        return rows.append(row);
    }, [])
;

writeDump(numbers);

And the output:

Array
1	Struct en Empty:null english string ten id Empty:null maori string tekau mi Empty:null value number 10
2	Struct en Empty:null english string nine id Empty:null maori string iwa mi Empty:null value number 9
3	Struct en Empty:null english string eight id Empty:null maori string waru mi Empty:null value number 8
4	Struct en Empty:null english string seven id Empty:null maori string whitu mi Empty:null value number 7
5	Struct en Empty:null english string six id Empty:null maori string ono mi Empty:null value number 6

OK, now WTF is going on? Lucee hasn't remapped the columns properly. it's added the new ones, but it's also included the old ones. It ain't supposed to do that. Contrast ColdFusion & Lucee with some more simple code:

numbers = queryNew("id,en,mi", "integer,varchar,varchar", [
    [1,"one","tahi"],
    [2,"two","rua"],
    [3,"three","toru"],
    [4,"four","wha"]
]);
remapTemplate = queryNew("value,english,maori"); 

reMapped = numbers.map(function(row){
    return {value=row.id, english=row.en, maori=row.mi};
}, remapTemplate);

writeDump(reMapped);

ColdFusion:

query
	ENGLISH	MAORI	VALUE
1	one	tahi	1
2	two	rua	2
3	three	toru	3
4	four	wha	4

Lucee:

Query Execution Time: 0 ms Record Count: 4 Cached: No Lazy: No
	value	english	maori	id	en	mi
1	1	one	tahi	Empty:null	Empty:null	Empty:null
2	2	two	rua	Empty:null	Empty:null	Empty:null
3	3	three	toru	Empty:null	Empty:null	Empty:null
4	4	four	wha	Empty:null	Empty:null	Empty:null

Sigh. What's supposed to be returned by a map operation on a query is a new query with only the columns from that remapTemplate query. That's what it's for.

ColdFusion 1 - 4 Lucee

On a whim I decided to check what Lucee did to the remapTemplate:

Query Execution Time: 0 ms Record Count: 4 Cached: No Lazy: No
	value	english	maori	id	en	mi
1	1	one	tahi	Empty:null	Empty:null	Empty:null
2	2	two	rua	Empty:null	Empty:null	Empty:null
3	3	three	toru	Empty:null	Empty:null	Empty:null
4	4	four	wha	Empty:null	Empty:null	Empty:null

ColdFusion 1 - 5 Lucee

This situation is slightly contrived as I don't care about that query anyhow. But what if I was using an extant query which had important data in it?

remapTemplate = queryNew("value,english,maori", "integer,varchar,varchar", [
    [5, "five", "rima"]
]);

So here I have a query with some data in it, and for whatever reason I want to remap the numbers query to have the same columns as this one. But obviously I don't want it otherwise messed with. Lucee mungs it though:

Query Execution Time: 0 ms Record Count: 5 Cached: No Lazy: No
	value	english	maori	id	en	mi
1	5	five	rima
2	1	one	tahi	Empty:null	Empty:null	Empty:null
3	2	two	rua	Empty:null	Empty:null	Empty:null
4	3	three	toru	Empty:null	Empty:null	Empty:null
5	4	four	wha	Empty:null	Empty:null	Empty:null

Not cool.

But wait. We're not done yet. Let's go back to some of my original code I was only running on ColdFusion:

numbersAsArray = queryExecute("SELECT id,mi FROM numbers LIMIT 4")
    .reduce(function(rows=[], row){
        return rows.append(row);
    })
;
writeDump(numbersAsArray);

This was part of the first example, I've just ditched the filter, map and sort: focusing on the reduce. On ColdFusion I get what I'd expect:

array
1	struct ID 1 MI tahi
2	struct ID 2 MI rua
3	struct ID 3 MI toru
4	struct ID 4 MI wha

On Lucee I get this:

Lucee 5.1.3.18 Error (expression)
Message	can't call method [append] on object, object is null
Stacktrace	The Error Occurred in queryReduceSimple.cfm: line 4  2: numbersAsArray = queryExecute("SELECT id,mi FROM numbers LIMIT 4") 3: .reduce(function(rows=[], row){ 4: return rows.append(row); 5: }) 6: ;

Hmmm. What's wrong now? Oh. It's this:

reduce(function(rows=[], row)

Notice how I am giving a default value to the first argument there. This doesn't work in Lucee.

ColdFusion 1 - 6 Lucee

This is easy to work around, because reduce functions take an optional last argument which is the initial value for that first argument to the callback, so I can just re-adjust the code like this:

.reduce(function(rows , row){
    return rows.append(row);
}, [])

OK, at this point I give up. Neither implementation of CFML here - either ColdFusion's or Lucee's - is good enough to do what I want to do. Oddly: Lucee is far worse on this occasion than ColdFusion is. That's disappointing.

So currently the score is 1-6 to Lucee. How did I get to 4-13?

I decided to write some test cases with TestBox to demonstrate what ought to be happening. And with the case of duplicate, I tested all native data-types I can think of:

• struct
• array
• query
• string
• double (I guess "numeric" in CFML)
• datetime
• boolean
• XML

Lucee failed the whole lot, and ColdFusion failed on numerics and booleans. As this is undocumented behaviour this might seem a bit harsh, but I'm not counting documentation errors against ColdFusion in this case. Also there's no way I'd actually expect numerics and booleans to have a duplicate method... except for the fact that strings do. Now this isn't a method bubbling through from java.lang.String, nor is it some Java method of ColdFusion's string implementation (they're just java.lang.Strings). This is an actively-created CFML member function. So it seems to me that - I guess for the sake of completeness - they implemented for "every" data type... I mean it doesn't make a great deal of sense on a datetime either, really, does it? So the omission of it from numerics and booleans is a bug to me.

This leaves the score:

ColdFusion 3 - 13 Lucee

The last ColdFusion point was cos despite the fact that with the sort operation it makes sense to alter the initial object if the method doesn't return the sorted one... it just doesn't make sense that the sort method has been implemented that way. It should leave the original object alone and return a new sorted object.

ColdFusion 4 - 13 Lucee

My test cases are too long to reproduce here, but you can see 'em on Github: Tests.cfc.

Right so...

Ah FFS.

... I was about to say "right, so that's that: not a great experience coming up with something cool to show to the CFMLers about CFML. Cos shit just didn't work. I found 17 bugs instead".

But I just had a thought about how sort methods work in ColdFusion, trying to find examples of where sort methods return the sorted object, rather than doing an inline support. And I I've found more bugs with both ColdFusion and Lucee.

Here are the cases:

component extends="testbox.system.BaseSpec" {
    function run() {
        describe("Other sort tests", function(){
            it("is a baseline showing using BIFs as a callback", function(){
                var testString = "AbCd";
                var applyTo = function(object, operation){
                    return operation(object);
                };

                var result = applyTo(testString, ucase);
                
                expect(result).toBeWithCase("ABCD");
            });
            describe("using arrays", function(){
                it("can use a function expression calling compareNoCase as a string comparator when sorting", function(){
                    var arrayToSort = ["d","C","b","A"];
                    
                    arrayToSort.sort(function(e1,e2){
                        return compareNoCase(e1, e2);
                    });
                    
                    expect(arrayToSort).toBe(["A","b","C","d"]);
                });
                it("can use the compareNoCase BIF as a string comparator when sorting", function(){
                    var arrayToSort = ["d","C","b","A"];
                    
                    arrayToSort.sort(compareNoCase);
                    
                    expect(arrayToSort).toBe(["A","b","C","d"]);
                });
            });
            describe("using lists", function(){
                it("can use a function expression calling compareNoCase as a string comparator when sorting", function(){
                    var listToSort = "d,C,b,A";
                    
                    var sortedList = listToSort.listSort(function(e1,e2){
                        return compareNoCase(e1, e2);
                    });
                    
                    expect(sortedList).toBe("A,b,C,d");
                    expect(listToSort).toBe("d,C,b,A");
                });
                it("can use the compareNoCase BIF as a string comparator when sorting", function(){
                    var listToSort = "d,C,b,A";
                    
                    var sortedList = listToSort.listSort(compareNoCase);
                    
                    expect(sortedList).toBe("A,b,C,d");
                    expect(listToSort).toBe("d,C,b,A");
                });
            });
        });
    }
}

What I'm doing here is using CFML built-in functions as the callbacks for a sort operation. This should work, because the sort operation needs a comparator function which works exactly like compare / compareNoCase: returns <0, 0, >0 depending on whether the first argument is "less than", "equal to" or "greater than" the second object according to the sort rules. As far as strings go, the built-in functions compare and compareNoCase do this. So they should be usable as callbacks. Since I think CF2016 built-in-functions have been first-class functions, so should be usable wherever something expects a function as an argument.

The first test demonstrates this in action. I have a very contrived situation where I have a function applyTo, which takes an object and a function to apply to it. In the test I pass-in the built-in function ucase as the operation. This test passes fine on ColdFusion; fails on Lucee.

ColdFusion 4 - 14 Lucee

So after I've demonstrated the technique should work, I try to use compareNoCase as the comparator for an array sort. it just doesn't work: it does nothing on ColdFusion, and on Lucee it still just errors (not gonna count that against Lucee, as it's the same bug as in the baseline test).

ColdFusion 5 - 14 Lucee

Next I try to use it on a listSort. This time ColdFusion errors as well. So this is a different bug than the doesn't-do-anything one for arrays.

ColdFusion 6 - 14 Lucee

Here are the results for just this latter tranche of test cases:

ColdFusion:

Lucee:

Fuck me, I've giving up.

This has been the most shit experience I've ever had trying to get CFML to do something. I don't think any of this code is edge-case stuff. Those higher-order functions are perhaps not as commonly used as they ought to be by the CFMLers out there, but I'm just... trying to use them.

So... sorry Brad & Ryan... I tried to come up with something worth showing to the mob that's useful in CFML, but I've failed. And my gut reaction to this exercise is that CFML can go fuck itself, basically.

Righto.

--
Adam

---

Bug tickets

• ColdFusion: sort higher order functions should return the sorted object, not simply "YES"
• ColdFusion: sort member functions should accept compare and compareNoCase functions as comparator callbacks
• ColdFusion: duplicate member functions missing from boolean and numeric types
• Lucee: query.map mishandles the "template" query
• Lucee: query.sort() should return a new query, not adjust the original query
• Lucee: duplicate member functions missing from native data types
• Lucee: Query.reduce does not allow default value on the callbacks' first argument
• Lucee: String.hash() method missing
• Lucee: built-in functions should work as sort comparators