A few days ago I had a look at this - apparent - myth that evaluate() is terribly terribly slow and one should not use it on that basis alone (not forgetting there's no real justification to use it these days anyhow). I tested on ColdFusion 9 and 10 to gather my results (the final results posted were from CF10, but the two were much of a muchness). I speculated that given there was not really good-enough evidence to support these performance issue claims in modern versions of ColdFusion that perhaps the problem existed "back in the day" and it's since been remedied. So over the weekend I blew the dust off my CFMX7 install, and ran the tests again on a like-for-like config, alongside re-running the tests on CF10. And for the sake of completeness, here are the results (they should be read in conjunction with the original article, as I do not go into great detail as to what was being tested).
Dynamically-named Methods
These are the tests in which I call a dynamically-named method via each of the mechanisms listed. CFMX7 does not have an invoke() function, so that test was omitted. For all results tables, the main number is the CFMX7 result, the one in parentheses are the results for CFMX10. They are an aggregation of 75 separate measurements of 10000 iterations of tests.| Stat\Test | reference | invoke() | evaluate() | <cfinvoke> |
|---|---|---|---|---|
| Min | 448 (173) | - (91) | 359 (132) | 339 (109) |
| 25% | 789 (451) | - (145) | 752 (332) | 563 (239) |
| Median | 1095 (740) | - (334) | 1006 (701) | 887 (525) |
| 75% | 1452 (1252) | - (735) | 1200 (1146) | 1120 (994) |
| Max | 2894 (6355) | - (3315) | 2604 (3595) | 3761 (4590) |
| Mean | 1186.2 (993.9) | - (543.7) | 1053.1 (890.8) | 950.7 (752.9) |
Interesting here - and generally across the board - is how bloody slow CFMX7 is compared to CF10. That said, outlier values (the max) on CF10 tend to out-lie further than with CFMX7. I would not read too much into that, as that sort of thing could be down to the PC or JVM deciding to do something else at the same time as the test was running, so there was a resource trough at the time.
Also interesting is here again using evaluate() is faster than using function reference insertion. I still haven't investigated as to why.
valueList()
This one is comparing using evaluate(), or using associative array notation for dynamic query variable names, and arrayToList() in lieu of valueList() for dynamic column names.| Stat\Test | reference | evaluate() |
|---|---|---|
| Min | 84 (45) | 130 (66) |
| 25% | 162 (46) | 247 (72) |
| Median | 229 (132) | 314 (170) |
| 75% | 304 (589) | 460 (511) |
| Max | 845 (3213) | 1182 (2232) |
| Mean | 245.9 (394.6) | 382 (354.8) |
Again the outlier values (upper quartile and max values) are worse on CF10 than CFMX7. But in general: CF10 is very much faster than CFMX7. But there's no resounding performance hit incurred by using evaluate().
Dynamic scope referencing
These tests compare using scope-references compared to using evaluate().| Stat\Test | reference | evaluate() |
|---|---|---|
| Min | 1010 (111) | 1020 (132) |
| 25% | 2748 (316) | 2708 (322) |
| Median | 4022 (487) | 4270 (592) |
| 75% | 5366 (852) | 5792 (995) |
| Max | 12088 (2477) | 9739 (3406) |
| Mean | 4280.4 (637.7) | 4562.7 (816.5) |
Here the performance on CF10 is really a lot faster than CFMX7... odd because it's really just a method call (object creations are not measured). But there's still hardly any difference between using references or evaluate(). And there's very little performance hit when using evaluate() in the best-case scenarios.
Dynamic variables
Finally, simply comparing using associative array notation to reference a dynamic variable, or using evaluate().| Stat\Test | evaluate() | array |
|---|---|---|
| Min | 776 (35) | 642 (23) |
| 25% | 1339 (37) | 1340 (24) |
| Median | 1677 (39) | 1907 (25) |
| 75% | 2219 (414) | 2372 (145) |
| Max | 6040 (1940) | 5049 (2154) |
| Mean | 1886.4 (291.6) | 1992.6 (200.6) |
Most interesting here is the comparison between CFMX7 (slow) and CF10 (fast), more than anything to do with evaluate(), which has comparable performance to using array notation.
Conclusion
My over all conclusion is that whoever first decided evaluate() was slow probably based it on loop-de-looping over a whole swag of iterations of some test, saw a difference in the timings and started wailing and gnashing their teeth, which was then picked up by the community. However what no-one noticed is that for all intents and purposes in the real world: there simply was no issue.Interesting.
I will reiterate that those "loop over really a lot of iterations of some code to measure its performance" is an intrinsically flawed approach to doing performance analysis. If you need to loop many times over something to be able to amplify its performance as as to be able to measure it: there is no performance issue.
Oh, btw, the code for all this is in my github repository: https://github.com/adamcameroncoldfusion/scratch/tree/master/blogExamples/evaluate, should you wish to look at it.
--
Adam
