Showing posts with label Javascript. Show all posts
Showing posts with label Javascript. Show all posts

Sunday 14 August 2022

JS: Server-sent events

G'day:

Yes, it's very odd for me to have something to say about JS stuff. I don't imagine there's anything new here for people that actually do JS development, but I don't, so this feature is new to me. Maybe it'll be new to some of my readers too.

I was looking at a question on Stack Overflow tagged with "ColdFusion": Server Side Events and Polling Database for new records with Lucee/Coldfusion. There was no answer, and I didn't know what the questioner was on about, so I decided to have a look.

Firstly, I RTFMed the JS words I didn't recognise:

The from ferreting around in the docs further, I found a PHP example of the server-side part: MDN › References › Web APIs › Server-sent events › Using server-sent events

From that lot it was easy to knock-together a CFML example.

<!-- test.html -->

<div id="result"></div>

<script>
var source = new EventSource('testInLoop.cfm');

source.addEventListener('message', function(e){
    document.body.innerHTML += `${e.data}<br>`
});
</script>
// testInLoop.cfm

header name="Content-Type" value="text/event-stream";

requestStartedAt = now()
for (i=1; i <= 10; i++) {
    data = {
        "uuid" = createUuid(),
        "now" = now().timeFormat("HH:mm:ss"),
        "requestStartedAt" = requestStartedAt.timeFormat("HH:mm:ss")
    }
    writeOutput("event: message" & chr(10))
    writeOutput('data: #serializeJson(data)#' & chr(10) & chr(10))
    flush;
    sleep(1000)
}
writeOutput("event: message" & chr(10))
writeOutput('data: #serializeJson({"complete": true})#' & chr(10) & chr(10))
flush;

Miraculaously, this works:

{"uuid":"4E1A4D17-6E4A-410B-B9C26B420A0DD5B4","now":"11:15:40","requestStartedAt":"11:15:40"}
{"uuid":"65C3F605-8823-4BF5-B32550FEABD0CECC","now":"11:15:41","requestStartedAt":"11:15:40"}
{"uuid":"D22E86D0-872F-4E5F-A89B29F44314BE05","now":"11:15:42","requestStartedAt":"11:15:40"}
{"uuid":"C4B6C1A6-0EF6-4CFD-AC9D4791F14A74AA","now":"11:15:43","requestStartedAt":"11:15:40"}
{"uuid":"FE0DB9CA-3535-416C-B63066EC4913E87D","now":"11:15:44","requestStartedAt":"11:15:40"}
{"uuid":"3E4E2DED-5B57-40AB-B8F5886DC7983053","now":"11:15:45","requestStartedAt":"11:15:40"}
{"uuid":"19611965-34D0-4ED4-9B9260B7EE3D8941","now":"11:15:46","requestStartedAt":"11:15:40"}
{"uuid":"81C7066F-55AD-47F1-BBBEC6EDAAA7A7C4","now":"11:15:47","requestStartedAt":"11:15:40"}
{"uuid":"2477899D-50EB-4CB0-AAF8E681DF168087","now":"11:15:48","requestStartedAt":"11:15:40"}
{"uuid":"7ABF54BD-1618-458D-A6BBF5D07812564D","now":"11:15:49","requestStartedAt":"11:15:40"}
{"complete":true}
{"uuid":"44AA5416-8EA9-451B-95C60E2BC1F4CEF8","now":"11:15:53","requestStartedAt":"11:15:53"}
{"uuid":"8829A145-93D5-4175-9305414B32CBE83E","now":"11:15:54","requestStartedAt":"11:15:53"}
{"uuid":"15F96712-263C-4798-ABCDBDC0C0AA68F7","now":"11:15:55","requestStartedAt":"11:15:53"}
{"uuid":"AF3A1831-8817-400E-B1292B4FCBD8BDDC","now":"11:15:56","requestStartedAt":"11:15:53"}
{"uuid":"C3D9D100-C8A9-4A92-87F6E62138DB5C6F","now":"11:15:57","requestStartedAt":"11:15:53"}
{"uuid":"1E60C5D6-E16A-429C-804D32B278272872","now":"11:15:58","requestStartedAt":"11:15:53"}
{"uuid":"DDB2DA3C-D064-4F6E-80227E337B41D657","now":"11:15:59","requestStartedAt":"11:15:53"}
{"uuid":"E7034B05-7912-4971-9ECD76E07403AD49","now":"11:16:00","requestStartedAt":"11:15:53"}
{"uuid":"D0D17279-6531-42B9-AB21FFB76DA5D939","now":"11:16:01","requestStartedAt":"11:15:53"}
{"uuid":"D804394F-DCB7-4FE3-BB6AAD86A4865D7D","now":"11:16:02","requestStartedAt":"11:15:53"}
{"complete":true}
{"uuid":"C25CF442-18FB-4BDD-8B55811E6DDCCD13","now":"11:16:06","requestStartedAt":"11:16:06"}

One thing to note is that once the server-side request completes, the browser (Chrome for me) re-polls the server after seemingly 4sec. I could not find anything in the docs about this (or how to set the interval). I've just tested in Edge and Firefox too, and their behaviour is the same as Chrome, except Firefox's interval seemed to be 6sec not 4sec.

That's all I have to say on this. Pleased to know about it, and pleased to be able to answer that Stack Overflow question now.

Righto.

--
Adam

Thursday 21 January 2021

Listening to the console log of a page loaded with Puppeteer

G'day:

This is a follow-up from something I touched on yesterday ("Polishing my Vue / Puppeteer / Mocha / Chai testing some more"). In that exercise I was using Puppeteer to load a web page I was testing, and then pulling some DOM element values out and checking they matched expectations. The relevant bits of code are thus:

describe.only("Tests of githubProfiles page using github data", function () {
    let browser;
    let page;
    let expectedUserData;

    before("Load the page and test data", async function () {
        await loadTestPage();
        expectedUserData = await loadTestUserFromGithub();
    });

    let loadTestPage = async function () {
        browser = await puppeteer.launch( {args: ["--no-sandbox"]});
        page = await browser.newPage();

        await Promise.all([
            page.goto("http://webserver.backend/githubProfiles.html"),
            page.waitForNavigation()
        ]);
    }

    it("should have the expected person's name", async function () {
        let name = await page.$eval("#app>.card>.content>a.header", headerElement => headerElement.innerText);
        name.should.equal(expectedUserData.name);
    });

  • Load the page with Puppeteer
  • Example test checking the page's DOM

This code seemed to be running fine, and the tests were passing. As I was adding more code ot my Vue component on the client end, I suddenly found the tests started to fail. Sometimes. If I ran them ten times, they'd fail maybe three times. At the same time, if I was just hitting the page in the browser, it was working 100% of the time. Odd. I mean clearly I was doing something wrong, and I'm new to all this async code I'm using, so figured I was using values before they were available or something. But it seemed odd that this was only manifesting sometimes. The way the tests were failing was telling though:

1) Tests of githubProfiles page using github data
       should have the expected person's name:

      AssertionError: expected '' to equal 'Alex Kyriakidis'
      + expected - actual

      +Alex Kyriakidis

The values coming from the DOM were blank. And note that it's not a case of the DOM being wrong, because if that was the case, the tests would barf all the time, with something like this:

Error: Error: failed to find element matching selector "#app>.card>.content>a.header"

The relevant mark-up here is:

<a class="header" :href="pageUrl">{{name}}</a>

So {{name}} isn't getting its value sometimes.

I faffed around for a bit reading up on Vue components, and their lifecycle handlers in case created was not the right place to load the data or something like that, but the code seemed legit.

My JS debugging is not very sophisticated, and it's basically a matter of console.logging stuff and see what happens. I chucked a bunch of log calls in to see what happens:

created () {
    console.debug(`before get call [${this.username}]`);
    axios.get(
        `https://api.github.com/users/${this.username}`,
        {
            auth: {
                username: this.$route.query.GITHUB_PERSONAL_ACCESS_TOKEN
            }
        }
    )
    .then(response => {
        console.debug(`beginning of then [${response.data.name}]`);
        this.name = response.data.name;
		// [etc...]
        console.debug("end of then");
    });
    console.debug("after get call");
}

Along with some other ones around the place, these all did what I expected when I hit the page in the browser:

beginning of js
githubProfiles.js:46 before VueRouter
githubProfiles.js:52 before Vue
githubProfiles.js:23 before get call [hootlex]
githubProfiles.js:43 after get call
githubProfiles.js:63 end of js
githubProfiles.js:33 beginning of then [Alex Kyriakidis]
githubProfiles.js:41 end of then

I noted that the then call was being fulfilled after the mainline code had finished, but in my test I was waiting for the page to fully load, so I'd catered for this. Repeated from above:

await Promise.all([
    page.goto("http://webserver.backend/githubProfiles.html"),
    page.waitForNavigation()
]);

I ran my tests, and was not seeing anything in the console which momentarily bemused me. But then I was just "errr… duh, Cameron. That stuff is logging in the web page's console. Not Node's console from the test run". I'm really thick sometimes.

This flumoxed me for a bit as I wondered how the hell I was going to get telemetry out of the page that I was calling in the Puppeteer headless browser. Then it occurred to me that I would not be the first person to wonder this, so just RTFMed.

It's really easy! The Puppeteer Page object exposes event listeners one can hook into, and one of the events is console. Perfect. All I needed to do is put this into my test code:

page = await browser.newPage();

page.on("console", (log) => console.debug(`Log from client: [${log.text()}] `));

await Promise.all([
    page.goto("http://webserver.backend/githubProfiles.html"),
    page.waitForNavigation()
]);

Then when I ran my tests, I was console-logging the log entries made in the headless browser as they occurred. What I was seeing is:

  Tests of githubProfiles page using github data
Log from client: [beginning of js]
Log from client: [before VueRouter]
Log from client: [before Vue]
Log from client: [before get call [hootlex]]
Log from client: [after get call]
Log from client: [end of js]
    1) should have the expected person's name
    2) should have the expected person's github page URL
    3) should have the expected person's avatar
    4) should have the expected person's joining year
    5) should have the expected person's description
Log from client: [beginning of xxxxxx then [Alex Kyriakidis]]
Log from client: [end of then]
    6) should have the expected person's number of friends
     should have the expected person's friends URL


  1 passing (4s)
  6 failing

Note how the tests get underway before the then call takes place. And shortly after that, the tests start passing because by then the dynamic values have actually been loaded into the DOM. This is my problem! that page.waitForNavigation() is not waiting long enough! My first reaction was to blame Puppeteer, but I quickly realised that's daft and defensive of me, given this is the first time I've messed with this stuff, almost certainly I'm doing something wrong. Then it occurred to me that a page is navigable once the various asset files are loaded, but not necessarily when any code in them has run. Duh. I figured Puppeteer would have thought of this, so there'd be something else I could make it wait for. I googled around and found the docs for page.waitForNavigation, and indeed I needed to be doing this:

page.waitForNavigation({waitUntil: "networkidle0"})

After I did that, I found the tests still failing sometimes, but now due to a time out:

  Tests of githubProfiles page using github data
Log from client: [beginning of js]
Log from client: [before VueRouter]
Log from client: [before Vue]
Log from client: [before get call [hootlex]]
Log from client: [after get call]
Log from client: [end of js]
Log from client: [beginning of then [Alex Kyriakidis]]
Log from client: [end of then]
    1) "before all" hook: Load the page for "should have the expected person's name"


  0 passing (4s)
  1 failing

  1) Tests of githubProfiles page using github data
       "before all" hook: Load the page for "should have the expected person's name":
     Error: Timeout of 5000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/usr/share/fullstackExercise/tests/functional/public/GithubProfilesTest.js)

I had the time out set for five seconds, but now the tests are waiting for the client to finish its async call as well, I was just edging over that five second mark sometimes. So I just bumped it to 10 seconds, and thenceforth the tests all passed all the time. I've left the telemetry in for one last successful run here:

  Tests of githubProfiles page using github data
Log from client: [beginning of js]
Log from client: [before VueRouter]
Log from client: [before Vue]
Log from client: [before get call [hootlex]]
Log from client: [after get call]
Log from client: [end of js]
Log from client: [beginning of then [Alex Kyriakidis]]
Log from client: [end of then]
     should have the expected person's name
     should have the expected person's github page URL
     should have the expected person's avatar
     should have the expected person's joining year
     should have the expected person's description
     should have the expected person's number of friends
     should have the expected person's friends URL


  7 passing (5s)

OK so that was a bit of a newbie exercise, but I'm a noob so yer gonna get that. It was actually pretty fun working through it though. I'm really liking all this tooling I'm checking out ATM, so yer likely get a few more of these basic articles from me.

Righto.

--
Adam

Tuesday 28 November 2017

That array_map quandary implemented in other languages

G'day:
A coupla days ago I bleated about array_map [having] a dumb implementation. I had what I thought was an obvious application for array_map in PHP, but it couldn't really accommodate me due to array_map not exposing the array's keys to the callback, and then messing up the keys in the mapped array if one passes array_map more than one array to process.

I needed to remap this:

[
    "2008-11-08" => "Jacinda",
    "1990-10-27" => "Bill",
    "2014-09-20" => "James",
    "1979-05-24" => "Winston"
]

To this:

array(4) {
  '2008-11-08' =>
  class IndexedPerson#3 (2) {
    public $date =>
    string(10) "2008-11-08"
    public $name =>
    string(7) "Jacinda"
  }
  '1990-10-27' =>
  class IndexedPerson#4 (2) {
    public $date =>
    string(10) "1990-10-27"
    public $name =>
    string(4) "Bill"
  }
  '2014-09-20' =>
  class IndexedPerson#5 (2) {
    public $date =>
    string(10) "2014-09-20"
    public $name =>
    string(5) "James"
  }
  '1979-05-24' =>
  class IndexedPerson#6 (2) {
    public $date =>
    string(10) "1979-05-24"
    public $name =>
    string(7) "Winston"
  }
}

Note how the remapped object also contains the original key value. That was the sticking point. Go read the article for more detail and more whining.

OK so my expectations of PHP's array higher order functions are based  on  my experience with JS's and CFML's equivalents. Both of which receive the key as well as the value in all callbacks. I decided to see how other languages achieve the same end, and I'll pop the codee in here for shits 'n' giggles.


CFML

Given most of my history is as a CFML dev, that one was easy.

peopleData = ["2008-11-08" = "Jacinda", "1990-10-27" = "Bill", "2014-09-20" = "James", "1979-05-24" = "Winston"]

people = peopleData.map((date, name) => new IndexedPerson(date, name))

people.each((date, person) => echo("#date# => #person#<br>"))

Oh, this presupposes the IndexedPerson component. Due to a shortcoming of how CFML works, components must be declared in a file of their own:

component {

    function init(date, name) {
        this.date = date
        this.name = name
    }

    string function _toString() {
        return "{date:#this.date#; name: #this.name#}"
    }
}


But the key bit is the mapping operation:

people = peopleData.map((date, name) => new IndexedPerson(date, name))

Couldn't be simpler (NB: this is Lucee's CFML implementation, not ColdFusion's which does not yet support arrow functions).

The output is:


2008-11-08 => {date:2008-11-08; name: Jacinda}
1990-10-27 => {date:1990-10-27; name: Bill}
2014-09-20 => {date:2014-09-20; name: James}
1979-05-24 => {date:1979-05-24; name: Winston}

Also note that CFML doesn't have associative arrays, it has structs, so the keys are not ordered. This does not matter here. (Thanks to Zac for correcting me here: CFML does have ordered structs these days).


JS

The next language I turned to was JS as that's the I'm next most familiar with. One thing that hadn't occurred to me is that whilst JS's Array implementation has a map method, we need to use an object here as the keys are values not indexes. And whilst I knew Objects didn't have a map method, I didn't know what the equivalent might be.

Well it turns out that there's no real option to use a map here, so I needed to do a reduce on the object's entries, Still: it's pretty terse and obvious:

class IndexedPerson {
    constructor(date, name) {
        this.date = date
        this.name = name
    }
}

let peopleData = {"2008-11-08": "Jacinda", "1990-10-27": "Bill", "2014-09-20": "James", "1979-05-24": "Winston"}

let people = Object.entries(peopleData).reduce(function (people, personData) {
    people.set(personData[0], new IndexedPerson(personData[0], personData[1]))
    return people
}, new Map())

console.log(people)

This returns what we want:

Map {
  '2008-11-08' => IndexedPerson { date: '2008-11-08', name: 'Jacinda' },
  '1990-10-27' => IndexedPerson { date: '1990-10-27', name: 'Bill' },
  '2014-09-20' => IndexedPerson { date: '2014-09-20', name: 'James' },
  '1979-05-24' => IndexedPerson { date: '1979-05-24', name: 'Winston' } }

TBH I think this is a misuse of an object to contain basically an associative array / struct, but so be it. It's the closest analogy to the PHP requirement. I was able to at least return it as a Map, which I think is better. I tried to have the incoming personData as a map, but the Map prototype's equivalent of entries() used above is unhelpful in that it returns an Iterator, and the prototype for Iterator is a bit spartan.

I think it's slightly clumsy I need to access the entries value via array notation instead of some sort of name, but this is minor.

As with all my code, I welcome people showing me how I should actually be doing this. Post a comment. I'm looking at you Ryan Guill ;-)

Java

Next up was Java. Holy fuck what a morass of boilterplate nonsense I needed to perform this simple operation in Java. Deep breath...

import java.util.HashMap;
import java.util.Map;
import java.util.stream.Collectors;

class IndexedPerson {
    String date;
    String name;
    
    public IndexedPerson(String date, String name) {
        this.date = date;
        this.name = name;
    }
    
    public String toString(){
        return String.format("{date: %s, name: %s}", this.date, this.name);
    }
}

class Collect {

    public static void main(String[] args) {

        HashMap<String,String> peopleData = loadData();

        HashMap<String, IndexedPerson> people = mapToPeople(peopleData);
            
        dumpIdents(people);
    }
    
    private static HashMap<String,String> loadData(){
        HashMap<String,String> peopleData = new HashMap<String,String>();
        
        peopleData.put("2008-11-08", "Jacinda");
        peopleData.put("1990-10-27", "Bill");
        peopleData.put("2014-09-20", "James");
        peopleData.put("1979-05-24", "Winston");
        
        return peopleData;
    }
    
    private static HashMap<String,IndexedPerson> mapToPeople(HashMap<String,String> peopleData) {
        HashMap<String, IndexedPerson> people = (HashMap<String, IndexedPerson>) peopleData.entrySet().stream()
            .collect(Collectors.toMap(
                e -> e.getKey(),
                e -> new IndexedPerson(e.getKey(), e.getValue())
            ));
            
        return people;
    }
    
    private static void dumpIdents(HashMap<String,IndexedPerson> people) {
        for (Map.Entry<String, IndexedPerson> entry : people.entrySet()) {
            System.out.println(String.format("%s => %s", entry.getKey(), entry.getValue()));
        }
    }
    
}

Result:
1979-05-24 => {date: 1979-05-24, name: Winston}
2014-09-20 => {date: 2014-09-20, name: James}
1990-10-27 => {date: 1990-10-27, name: Bill}
2008-11-08 => {date: 2008-11-08, name: Jacinda}

Most of that lot seems to be just messing around telling Java what types everything are. Bleah.

The interesting bit - my grasp of which is tenuous - is the Collectors.toMap. I have to admit I derived that from reading various Stack Overflow articles. But I got it working, and I know the general approach now, so that's good.

Too much code for such a simple thing though, eh?


Groovy

Groovy is my antidote to Java. Groovy makes this shit easy:

class IndexedPerson {
    String date
    String name

    IndexedPerson(String date, String name) {
        this.date = date;
        this.name = name;
    }

    String toString(){
        String.format("date: %s, name: %s", this.date, this.name)
    }
}

peopleData = ["2008-11-08": "Jacinda", "1990-10-27": "Bill", "2014-09-20": "James", "1979-05-24": "Winston"]

people = peopleData.collectEntries {date, name -> [date, new IndexedPerson(date, name)]}

people.each {date, person -> println String.format("%s => {%s}", date, person)}

Bear in mind that most of that is getting the class defined, and the output. The bit that does the mapping is just the one line in the middle. That's more like it.

Again, I don't know much about Groovy… I had to RTFM to find out how to do the collectEntries bit, but it was easy to find and easy to understand.

I really wish I had a job doing Groovy.

Oh yeah, for the sake of completeness, the output was thus:

2008-11-08 => {date: 2008-11-08, name: Jacinda}
1990-10-27 => {date: 1990-10-27, name: Bill}
2014-09-20 => {date: 2014-09-20, name: James}
1979-05-24 => {date: 1979-05-24, name: Winston}


Ruby

Ruby's version was pretty simple too as it turns out. No surprise there as Ruby's all about higher order functions and applying blocks to collections and stuff like that.

class IndexedPerson

    def initialize(date, name)
        @date = date
        @name = name
    end

    def inspect
        "{date:#{@date}; name: #{@name}}\n"
    end
end

peopleData = {"2008-11-08" => "Jacinda", "1990-10-27" => "Bill", "2014-09-20" => "James", "1979-05-24" => "Winston"}

people = peopleData.merge(peopleData) do |date, name|
    IndexedPerson.new(date, name)
end

puts people

Predictable output:

{"2008-11-08"=>{date:2008-11-08; name: Jacinda}
, "1990-10-27"=>{date:1990-10-27; name: Bill}
, "2014-09-20"=>{date:2014-09-20; name: James}
, "1979-05-24"=>{date:1979-05-24; name: Winston}
}

I wasn't too sure about all that block nonsense when I first started looking at Ruby, but I quite like it now. It's easy to read.


Python

My Python skills don't extend much beyond printing G'day World on the screen, but it was surprisingly easy to google-up how to do this. And I finally got to see what Python folk are on about with this "comprehensions" stuff, which I think is quite cool.

class IndexedPerson:
    def __init__(self, date, name):
        self.date = date
        self.name = name

    def __repr__(self):
        return "{{date: {date}, name: {name}}}".format(date=self.date, name=self.name)

people_data = {"2008-11-08": "Jacinda", "1990-10-27": "Bill", "2014-09-20": "James", "1979-05-24": "Winston"}

people = {date: IndexedPerson(date, name) for (date, name) in people_data.items()}

print("\n".join(['%s => %s' % (date, person) for (date, person) in people.items()]))


And now that I am all about Clean Code, I kinda get the "whitespace as indentation" thing too. It's clear enough if yer code is clean in the first place.

The output of this is identical to the Groovy one.

Only one more then I'll stop.

Clojure

I can only barely do G'day World in Clojure, so this took me a while to work out. I also find the Clojure docs to be pretty impentrable. I'm sure they're great if one already knows what one is doing, but I found them pretty inaccessible from the perspective of a n00b. It's like if the PHP docs were solely the user-added stuff at the bottom of each docs page. Most blog articles I saw about Clojure were pretty much just direct regurgitation of the docs, without much value-add, if I'm to be honest.

(defrecord IndexedPerson [date name])

(def people-data (array-map "2008-11-08" "Jacinda" "1990-10-27" "Bill" "2014-09-20" "James" "1979-05-24" "Winston"))

(def people
  (reduce-kv
    (fn [people date name] (conj people (array-map date (IndexedPerson. date name))))
    (array-map)
    people-data))

(print people)

The other thing with Clojure for me is that the code is so alien-looking to me that I can't work out how to indent stuff to make the code clearer. All the examples I've seen don't seem very clear, and the indentation doesn't help either, I think. I guess with more practise it would come to me.

It seems pretty powerful though, cos there's mot much code there to achieve the desired end-goal.

Output for this one:

{2008-11-08 #user.IndexedPerson{:date 2008-11-08, :name Jacinda},
1990-10-27 #user.IndexedPerson{:date 1990-10-27, :name Bill},
2014-09-20 #user.IndexedPerson{:date 2014-09-20, :name James},
1979-05-24 #user.IndexedPerson{:date 1979-05-24, :name Winston}}


Summary

This was actually a very interesting exercise for me, and I learned stuff about all the languages concerned. Even PHP and CFML.

I twitterised a comment regarding how pleasing I found each solution:


This was before I did the Clojure one, and I'd slot that in afer CFML and before JS, making the list:
  1. Python
  2. Ruby
  3. Groovy
  4. CFML
  5. Clojure
  6. JS
  7. PHP
  8. Java

Python's code looks nice and it was easy to find out what to do. Same with Ruby, just not quite so much. And, really same with Groovy. I could order those three any way. I think Python tips the scales slightly with the comprehensions.

CFML came out suprisingly well in this, as it's a bloody easy exercise to achieve with it.

Clojure's fine, just a pain in the arse to understand what's going on, and the code looks a mess to me. But it does a lot in little space.

JS was disappointing because it wasn't nearly so easy as I expected it to be.

PHP is a mess.

And - fuck me - Java. Jesus.

My occasional reader Barry O'Sullivan volunteered some input the other day:


Hopefully he's still up for this, and I'll add it to the list so we can have a look at that code too.

Like I said before, if you know a better or more interesting way to do this in any of the languages above, or any other languages, make a comment and post a link to a Gist (just don't put the code inline in the comment please; it will not render at all well).

I might have another one of these exercises to do soon with another puzzle a friend of mine had to recently endure in a job-interview-related coding test. We'll see.

Righto.

--
Adam

Monday 19 September 2016

Another Friday puzzle; and the importance of tests that try to break code

G'day:
There was another Friday Puzzle on the CFML Slack channel this week. The question was:

The maximum sum subarray problem consists in finding the maximum sum of a contiguous subsequence in an array or list of integers:

maxSequence([-2, 1, -3, 4, -1, 2, 1, -5, 4])
// should be 6: [4, -1, 2, 1]

Apparently the word "contiguous" is something that will flummox some people (judging by the discussion on the Slack channel). In this sense a contiguous sub-sequence is one that is made from adjacent array elements. So you can see they highlighted sub-sequence is a subset of adjacent array elements.

Eyeballing the example series here reveals the tricky bit one needs to watch for... negative values. Take a sub-sequence of 3,4. That has a sum of 7. Now if the next number is negative: 3,4,-1 (sum 6), the sum of this longer sub-sequence is less than that of the preceding shorter one, so the shorter subsequence is still the "winner". However if the next number (or numbers) sum to greater than the negative number, eg; 3,4,-1,2 (sum 8) then we've got the highest sum again. And of course there could be more negative numbers followed by positive numbers with a higher sum repeatedly.

Update:

I did not thoroughly read the requirement here, so got some of this wrong. See "TIL: Cameron is bloody wrong again".

A couple of the bods on the Slack channel quickly came up with solutions. Here they are, respectively: Isaiah's and John's. I'll only repeat one of them here because both used pretty much the same algorithm, just one used procedural code, and the other a more functional approach (in that they used higher-order functions). But they've both got the same logic error in them, because the approach is flawed. Let's have a look at John's one:

numeric function maxSequence(required array arr){
    var currentSum = 0;
    return arr.reduce(function(maxSum, number){
        currentSum = max(currentSum+number, 0);
        return max(currentSum, maxSum);
    }, 0);
}

WriteDump(maxSequence([-2, 1, -3, 4, -1, 2, 1, -5, 4]));

WriteDump(maxSequence([-2, 1, -3, 4, -1, 2, 1, -5, 4, 3]));

WriteDump(maxSequence([-2, -3, -1, -5]));

WriteDump(maxSequence([1, 4, 2, 1, 4, 3]));

This is nice and simple, and it took me a while to get what he's doing here. But we're keeping a running sum of the sequence outside the callback, and the callback itself returns the greater of the running sum or the whatever has previously been the maximum sum. So taking the first series there, we get the following iterations:

maxSumnumbercurrentSumnewMax
0-200
0111
1-301
1444
4-134
4255
5166
6-516
6456

Superficially this seems OK, except for one thing: it considers the minimum possible sum to be 0. Which is not correct. In a sequence with only negative numbers... the maximum sum will be the highest negative number: in [-1,-2,-3] the highest sum there is -1, but sums less than zero are ignored by this algorithm, so it incorrectly returns 0. Equally for an empty sequence, the answer is null or undefined, so this algorithm will fail there too, as it returns 0 for this too (as it defaults to a maximum sum of zero, whereas we don't know there'll be a sum when we start the process. So I guess that's two slightly different logic errors there.

I'll come back to John's solution in a bit, but first here's my solution. It's not as elegant as John's, and I'm not entirely happy with it, but it does work.

Here's the code (I've opted to do mine in ES2015 rather than CFML):

let sumLongestContiguousSubsequence = function (array) {
    let subSequences = array.map((_,i,a)=>a.slice(i));

    return subSequences.reduce(function(max, subsequence){
        let runningSum = subsequence[0];
        let maximumSubSequenceSum = subsequence.reduce(function(max, element){
            return (runningSum += element) > max ? runningSum : max;
        });
        return Math.max(max||maximumSubSequenceSum, maximumSubSequenceSum);
    }, null);
};

My conceit is that one cannot make assumptions about any earlier maximums, so I don't default the answer to anything.

Also I don't default the running sum to anything, I just take its starting point as the first element of the sequence I'm checking.

The general approach here is slightly more ham-fisted than John's approach.
  • I figure I need to scan each separate sub-sequence of the main sequence, starting each sub-sequence from each subsequent value in the main sequence. EG: if I have a sequence of [1,2,3,4], then subSequences will be [[1,2,3,4],[2,3,4] [3,4],[4]].
  • For each of those I do much the same thing as John: just run a cumulative sum, and remember whatever the highest value for that was.
  • So that'll bring me back to an array of maximums (in my [1,2,3,4] example, this'd be [10,9,7,4].
  • And as I reduce that lot, I simply return whichever is the higher of the previous ones and the current one.

I also wrote actual tests here. And this is where my approach differs from John's (or Isaiah's for that matter). When I test things I don't try to demonstrate to myself it works, I try to break the thing.

Here are my tests:

"use strict";

let assert = require("chai").assert;

let sumLongestContiguousSubsequence = require("./sumLongestContiguousSubsequence.js");

describe("Test of puzzle requirement", function(){
    it("returns the highest contiguous subseries sum for baseline requirement", function(){
        let sequence = [-2, 1, -3, 4, -1, 2, 1, -5, 4];
        let expectation = 4 + -1 + 2 + 1; // 6
        let result = sumLongestContiguousSubsequence(sequence);
        assert.equal(expectation, result);
    });
});

describe("Tests of other puzzle submission sequences", function(){
    it("returns the highest contiguous subseries sum for variation 1", function(){
        let sequence = [-2, 1, -3, 4, -1, 2, 1, -5, 4, 3];
        let expectation = 4 + -1 + 2 + 1 + -5 + 4 + 3; // 8
        let result = sumLongestContiguousSubsequence(sequence);
        assert.equal(expectation, result);
    });
    it("returns the highest contiguous subseries sum for variation 2", function(){
        let sequence = [-2, -1, -3, -5];
        let expectation = -1;
        let result = sumLongestContiguousSubsequence(sequence);
        assert.equal(expectation, result);
    });
    it("returns the highest contiguous subseries sum for variation 3", function(){
        let sequence = [1, 4, 2, 1, 4, 3];
        let expectation = 1 + 4 + 2 + 1 + 4 + 3; // 15
        let result = sumLongestContiguousSubsequence(sequence);
        assert.equal(expectation, result);
    });
});

describe("Test edge cases", function(){
    it("returns the highest contiguous subseries sum with an empty array", function(){
        let sequence = [];
        let expectation = null;
        let result = sumLongestContiguousSubsequence(sequence);
        assert.equal(expectation, result);
    });
    it("returns the highest contiguous subseries sum with just zero", function(){
        let sequence = [0];
        let expectation = 0;
        let result = sumLongestContiguousSubsequence(sequence);
        assert.equal(expectation, result);
    });
    it("returns the highest contiguous subseries sum with just -1", function(){
        let sequence = [-1];
        let expectation = -1;
        let result = sumLongestContiguousSubsequence(sequence);
        assert.equal(expectation, result);
    });
    it("returns the highest contiguous subseries sum with just 1", function(){
        let sequence = [1];
        let expectation = 1;
        let result = sumLongestContiguousSubsequence(sequence);
        assert.equal(expectation, result);
    });
});

describe("Better described tests", function(){
    it("returns the highest contiguous subseries sum when the sequence has negative values", function(){
        let sequence = [1,2,3,-11];
        let expectation = 1 + 2 + 3; // 6
        let result = sumLongestContiguousSubsequence(sequence);
        assert.equal(expectation, result);
    });
    it("returns the highest contiguous subseries sum when the sequence has negative values followed by a greater positive value", function(){
        let sequence = [1,2,3,-4,5];
        let expectation = 1 + 2 + 3 + -4 + 5; // 7
        let result = sumLongestContiguousSubsequence(sequence);
        assert.equal(expectation, result);
    });
    it("returns the highest contiguous subseries sum when the sequence has negative values followed by a subseries positive values that are net greater than the negative one", function(){
        let sequence = [2,4,6,-8,3,7];
        let expectation = 2 + 4 + 6 + -8 + 3 + 7; // 14
        let result = sumLongestContiguousSubsequence(sequence);
        assert.equal(expectation, result);
    });
    it("returns the highest contiguous subseries sum when the sequence has repeated negative values followed by a subseries positive values that are net greater than the negative one", function(){
        let sequence = [12,14,16,-8,3,7,-12,5,9];
        let expectation = 12 + 14 + 16 + -8 + 3 +7 + -12 + 5 + 9; // 46
        let result = sumLongestContiguousSubsequence(sequence);
        assert.equal(expectation, result);
    });
});

These are fairly deliberate in what they test, but don't try to do anything clever (so there's a lot of repeated code). I make a point of testing edge cases like empty sequences, and sequences with only one element, and all negative values and all positive values etc. My philosophy with testing is that I'm not trying to demonstrate the code works, I'm trying to demonstrate it doesn't break. This amounts to the same thing, but it starts from a slightly different mindset. Testing is one situation where "glass half empty" rather than "glass half full" is the better way to look at things.

So this version of the solution works, but I'm not happy about it. For one thing, I don't think one can look at it and go "right, it's clear what's going on there". I looked for refactoring opportunities, but am not seeing any. Maybe subSequences could have a better name? I looked at extracting the callbacks into named function expressions, but that didn't look any clearer to me either.

let sumLongestContiguousSubsequence = function (array) {
    let runningSum;
    
    let getCurrentMaxSum = function(max, element){
        return (runningSum += element) > max ? runningSum : max;
    };
    
    let findSumOfLongestSub = function(max, subsequence){
        runningSum = subsequence[0];
        let maximumSubSequenceSum = subsequence.reduce(getCurrentMaxSum);
        return Math.max(max||maximumSubSequenceSum, maximumSubSequenceSum);
    };

    let subSequencesSlicedAtEachIndex = array.map((_,i,a)=>a.slice(i));

    return subSequencesSlicedAtEachIndex.reduce(findSumOfLongestSub, null);
};

What do you think: is that clearer? I guess it's a bit better, innit? It's straying away from "elegance in simplicity" though.

Ha, just for a laugh I took the code clarity in the opposite direction. How about this mess:

let f=a=>a.map((_,i,a)=>a.slice(i)).reduce((m1,s)=>{
    let t=s[0],m2=s.reduce((m,v)=>(t+=v)>m?t:m)
    return Math.max(m1||m2,m2)
},null)

It's the same logic as my original version.

The other issue that Sean and Ryan are likely to pull me up on is that one of my reductions relies on side-effects spilling out into the intermediary calling code:

let sumLongestContiguousSubsequence = function (array) {
    let subSequences = array.map((_,i,a)=>a.slice(i));

    return subSequences.reduce(function(max, subsequence){
        let runningSum = subsequence[0];
        let maximumSubSequenceSum = subsequence.reduce(function(max, element){
            return (runningSum += element) > max ? runningSum : max;
        });
        return Math.max(max||maximumSubSequenceSum, maximumSubSequenceSum);
    }, null);
};


This didn't actually bother me until I did the PHP version of this, where the side-effects are more glaring:

$sumLongestContiguousSubsequence = function ($array) {
    $subSequences = array_map(function($i) use ($array) {
        return array_slice($array, $i);
    }, array_keys($array));

    return array_reduce($subSequences, function($max, $subSequence){
        $runningSum = 0;
        $maximumSubSequenceSum = array_reduce($subSequence, function($max, $element) use (&$runningSum){
            return ($runningSum += $element) > $max ? $runningSum : $max;
        }, $subSequence[0]);
        return max($max?:$maximumSubSequenceSum, $maximumSubSequenceSum);
    }, null);
};

PHP sux a bit because it doesn't do closure implicitly, one needs to tell it what to enclose, which is clumsy IMO. But it's also a bit of an orange flag. This orange flag becomes a red flag when I need to actually use a reference here, cos I'm changing the original value, not simply using it.

That guilt-tripped me into revising my JS version so as to not need the side effects:

let sumLongestContiguousSubsequence = function (array) {
    let subSequences = array.map((_,i,a)=>a.slice(i));

    return subSequences.reduce(function(max, subsequence){
        let maximumSubSequence = subsequence.reduce(function(working, element){
            working.runningSum += element;
            working.max = working.max || working.runningSum;
            working.max = working.runningSum > working.max ? working.runningSum : working.max
            return {max:working.max, runningSum:working.runningSum};
        }, {runningSum:0});
        return Math.max(max||maximumSubSequence.max, maximumSubSequence.max);
    }, null);
};


Now I'm carrying both the runningSum and the max value through the first argument in the reduction, by using an object rather than just the simple value. It's a small change but gets rid of the smell. To be completely honest though... I prefer my original version. I realise it's frowned-upon to bleed side-effects (which I then leverage), but this refactoring seems like an exercise in pedantic/dogmatic busy-work, and makes the code slightly less clear in the process. I dunno.

Oh, for completeness I did a CFML conversion too:

sumLongestContiguousSubsequence = function (array) {
    var subSequences = array.map((_,i,a) => a.slice(i));
    return subSequences.reduce(function(maxSum, subSequence){
        var runningSum = 0;
        var maximumSubSequenceSum = subSequence.reduce(
            (maxSum, element) => (runningSum += element) > maxSum ? runningSum : maxSum,
            subSequence[1]
        );
        return max(maxSum ?: maximumSubSequenceSum, maximumSubSequenceSum);
    }, null);
};

Note that this solution only works on Lucee, not ColdFusion, for three reasons:
  • ColdFusion does not have arrow functions;
  • ColdFusion has a bug in its parser which breaks on this expression (see 4190163);
  • Lucee has a null keyword whereas ColdFusion does not.

The Lucee version is OK though. Same as the JS version for the most part: just 1-based arrays, not 0-based; and Lucee has the ?: operator whereas JS uses ||.

That's about all I can think to say about this. I might see if I can mess with John's approach to make it work for those two edge-cases it fails on. I much prefer his way compared to my way... but only provided it can be made to work! I'd also be keen to see treatments of this puzzle in other languages, or better/different solutions for JS, PHP or CFML.

Righto.

--
Adam

Saturday 3 September 2016

JS: next Friday puzzle: displaying bytes in "nearest unit"

G'day:
Here's my answer for the next Friday puzzle that was posted on the CFML Slack channel. I skipped last week's one as... well... I couldn't be arsed, but this was an easy one and I could sort it out whilst cursing my jet lag (I've just done London -> Auckland this week).

The details of the puzzle are in this gist, but basically it's to take a number of bytes and "round" it to the appropriate closest unit of bytes (eg: kB, MB, GB etc), up to PB. There's more detail than that, but that's the bit I'm paying attention do. Doing CFML is a waste of time, so this week I've decided to do it in JavaScript instead, and re-polish my piss-poor ES2015 / Node.js / Mocha skills. Although not too much.

First I tried a solution which treated the number as a string and just reduced it down to the most significant figures and slapped a unit onto the end of it. Whilst this did what I set out to do, it only dealt with decimal groupings, not 1024-based ones. Here's the code anyhow. It's not polished up cos I abandoned it half-way through:

f = function(x) {
    var units = ["B", "kB", "MB", "GB", "TB", "PB"];
    var ordersOfMagnitude = 3;

    var numberAsString = x.toString();
    var lengthOfNumber = numberAsString.length; 

    var numberOfDigits = lengthOfNumber % ordersOfMagnitude;
    var unitIndex = Math.floor(lengthOfNumber / ordersOfMagnitude);

    if (numberOfDigits == 0) {
        numberOfDigits = 3;
        unitIndex--;
    }
    if (unitIndex+1 > units.length){
        unitIndex = units.length - 1;
        numberOfDigits = lengthOfNumber - (unitIndex * ordersOfMagnitude);
    }
    
    var digits = numberAsString.substring(0, numberOfDigits);
    var unitToUse = units[unitIndex];

    var result = digits + unitToUse;
    
    return result;
}

This was more just a spike to get me thinking.

Then I tried another version where I did a reduction on the units array, but that was crap as it was quite side-effect-y, and I don't really think it was a good use of a reduce operation. I did not keep the code for that one, so I can't show you.

Next I decided to stop messing around and come up with an answer that actually worked and wasn't daft, so I knocked this one together:

"use strict";
 
let numberToMemoryUnits = function(bytes) {
    let units = ["kB", "MB", "GB", "TB", "PB"];
    let binaryDivisor = 1024;
    let numberOfBytesAsUnit = bytes;
    let unit = "B";
    while (numberOfBytesAsUnit >= binaryDivisor && units.length){
        numberOfBytesAsUnit /= binaryDivisor;
        unit = units.shift();
    }
    let roundedValue = Math.floor(numberOfBytesAsUnit);

    return `${roundedValue}${unit}`;
}

module.exports = numberToMemoryUnits;

That's OK-ish I guess. The only thing I don't like is how I have that first assignment of the units separate to the loop. It seems like dodgy code doubling-up to me, and I should get rid of that somehow, but can't be bothered thinking it through (I'm slightly hungover, which doesn't help).

Update:

I've tweaked this slightly since I first posted it:
  • Improved the argument name from a rather lazy-arsed x to the more descriptive bytes.
  • Likewise improved the name of the variable which was digits to be numberOfBytesAsUnit.
  • Used the intermediary variable roundedValue,
  • and used an interpolated string instead of string concatenation with the return value.
  • improved the way I initialised the units object in the tests, from being individual assignment statements to being a reduction of the units array.
This was off the back of a discussion about my implementation that I had with Brendan on the Slack channel. He asked me why I had the separate variable digits (now numberOfBytesAsUnit) instead of just using the argument x (bytes). This was because whilst the value passed to the function is indeed a number of bytes, once I start using it - specifically in that loop - it's no longer a value in bytes, so I use a new - more descriptively accurate - variable name instead. We also discussed my separate handling of bytes and the other units coming from the array, and I'm still not 100% happy with it, but in using the better variable names around it I mind it less than I did before.

Any other code review input is welcomed, btw.

I was a bit naughty as I tested it by hand whilst developing it, but then felt guilty and formalised a bunch of tests after the fact. I guess I still did TDD whilst throwing this together, but not as deliberately as perhaps I shoulda. Anyhow, here are the tests too:

"use strict";

let assert = require("chai").assert;

let numberToMemoryUnits = require("../src/numberToMemoryUnits.js");

let binaryFactor = 1024;

let units = ["kB", "MB", "GB", "TB", "PB"].reduce(function(units, unit, index){
    units[unit] = Math.pow(binaryFactor, index + 1);
    return units;
}, {});

describe("Tests for each unit", function(){
    it("should work for bytes", function(){
        let result = numberToMemoryUnits(123);
        let expectation = "123B";
        assert.equal(expectation, result);
    });
    it("should work for kB", function(){
        let result = numberToMemoryUnits(2345);
        let expectation = "2kB";
        assert.equal(expectation, result);
    });
    it("should work for MB", function(){
        let result = numberToMemoryUnits(3456789);
        let expectation = "3MB";
        assert.equal(expectation, result);
    });
    it("should work for GB", function(){
        let result = numberToMemoryUnits(4567890123);
        let expectation = "4GB";
        assert.equal(expectation, result);
    });
    it("should work for TB", function(){
        let result = numberToMemoryUnits(5678901234567);
        let expectation = "5TB";
        assert.equal(expectation, result);
    });
    it("should work for PB", function(){
        let result = numberToMemoryUnits(6789012345678901);
        let expectation = "6PB";
        assert.equal(expectation, result);
    });
    
});

describe("Test exact units", function(){
    it("should work for 1kB", function(){
        let result = numberToMemoryUnits(units.kB);
        let expectation = "1kB";
        assert.equal(expectation, result);
    });
    it("should work for 1MB", function(){
        let result = numberToMemoryUnits(units.MB);
        let expectation = "1MB";
        assert.equal(expectation, result);
    });
    it("should work for 1GB", function(){
        let result = numberToMemoryUnits(units.GB);
        let expectation = "1GB";
        assert.equal(expectation, result);
    });
    it("should work for 1TB", function(){
        let result = numberToMemoryUnits(units.TB);
        let expectation = "1TB";
        assert.equal(expectation, result);
    });
    it("should work for 1PB", function(){
        let result = numberToMemoryUnits(units.PB);
        let expectation = "1PB";
        assert.equal(expectation, result);
    });
});
describe("Test off by one", function(){
    it("should work for <1kB", function(){
        let result = numberToMemoryUnits(units.kB-1);
        let expectation = "1023B";
        assert.equal(expectation, result);
    });
    it("should work for >1kB", function(){
        let result = numberToMemoryUnits(units.kB+1);
        let expectation = "1kB";
        assert.equal(expectation, result);
    });
    it("should work for <1MB", function(){
        let result = numberToMemoryUnits(units.MB-1);
        let expectation = "1023kB";
        assert.equal(expectation, result);
    });
    it("should work for >1MB", function(){
        let result = numberToMemoryUnits(units.MB+1);
        let expectation = "1MB";
        assert.equal(expectation, result);
    });
    it("should work for <1GB", function(){
        let result = numberToMemoryUnits(units.GB-1);
        let expectation = "1023MB";
        assert.equal(expectation, result);
    });
    it("should work for >1GB", function(){
        let result = numberToMemoryUnits(units.GB+1);
        let expectation = "1GB";
        assert.equal(expectation, result);
    });
    it("should work for <1TB", function(){
        let result = numberToMemoryUnits(units.TB-1);
        let expectation = "1023GB";
        assert.equal(expectation, result);
    });
    it("should work for >1TB", function(){
        let result = numberToMemoryUnits(units.TB+1);
        let expectation = "1TB";
        assert.equal(expectation, result);
    });
    it("should work for <1PB", function(){
        let result = numberToMemoryUnits(units.PB-1);
        let expectation = "1023TB";
        assert.equal(expectation, result);
    });
    it("should work for >1PB", function(){
        let result = numberToMemoryUnits(units.PB+1);
        let expectation = "1PB";
        assert.equal(expectation, result);
    });
});

describe("Test boundaries", function(){
    it("should work for 0bytes", function(){
        let result = numberToMemoryUnits(0);
        let expectation = "0B";
        assert.equal(expectation, result);
    });
    it("should work for 1024PB", function(){
        let result = numberToMemoryUnits(units.PB*units.kB);
        let expectation = "1024PB";
        assert.equal(expectation, result);
    });
    it("should work for 1048576PB", function(){
        let result = numberToMemoryUnits(units.PB*units.MB);
        let expectation = "1048576PB";
        assert.equal(expectation, result);
    });
});


They're a bit long-winded in total, but the individual tests are simple enough. The key here is I test either side of each each unit, eg: 1023 is presented in bytes, 1024 is 1kB and so is 1025 etc. Also a test of zero for good measure, as well that it handles numbers beyond PB, and just uses PB thereafter (eg: 1025PB displays as such).

And they all pass:
C:\src\js\puzzle\20160903>mocha test\numberToMemoryUnitsTest.js


  Tests for each unit
    should work for bytes
    should work for kB
    should work for MB
    should work for GB
    should work for TB
    should work for PB

  Test exact units
    should work for 1kB
    should work for 1MB
    should work for 1GB
    should work for 1TB
    should work for 1PB

  Test off by one
    should work for <1kB
    should work for >1kB
    should work for <1MB
    should work for >1MB
    should work for <1GB
    should work for >1GB
    should work for <1TB
    should work for >1TB
    should work for <1PB
    should work for >1PB

  Test boundaries
    should work for 0bytes
    should work for 1024PB
    should work for 1048576PB


  24 passing (31ms)


C:\src\js\puzzle\20160903>


That's it. Nothing special today, but still required some thought, and also reminding myself how Mocha works.

Give it a go. Do it in some other language than CFML!

Righto.

--
Adam

Monday 16 May 2016

"There, and back again" (or: I suck at JavaScript sometimes)

G'day:
Sometimes I think I'm OK at JavaScript (well: I guess I'm OK at it [he says, whilst seesawing his hand]), and then sometimes I put this to the test and conclude I'm actually a bit shit. In this episode, I cover the journey from former to latter.

Important note:

Given the nature of this article, most of the code in it is wrong, and I am in no-way advocating its usage. The raison d'être of the article is demonstrating bad / wrong / pointless code. Do not copy this code. Do not think the code in here is a solution to any problem you might have.

We've been trying to get coherence in our approach to JavaScript for years. But it always just seems to be a series of false starts, for one reason or another. But I still chip away at trying to improve things where I can. This tends to involve a lot of experimentation, and not much ground made.

A number of years ago we had my mate Chris Kobrzak on the team, and he'd migrated from being a CFML developer to being a dedicated JavaScript developer. This was excellent as he say to it we had a formal JavaScript coding standard, and actively mentored it. During his tenure I think all of us improved our JS no end. However his tenure was short, and he moved onto other challenges after a while. For one reason or another the role was never re-filled (although we needed it more than ever), and our progress on that improvement has stalled. I think we're all still a bunch better than we were then, but this is more down to repetition and attrition more than anything else.

One legacy that Chris left us was a formalised approach to defining JavaScript "classes", wherein we unified on the standard "prototype-based" approach. This won't seem like news, but we had several different variations of inline-objects / kinda-classes / wodges-of-procedural-code going at once. Basically each dev started a new file however they felt like at the time. Or sometimes in the middle of a file, change tactic / style.

The basic rules of the coding standard were:
  • acknowledge that JavaScript is OO but not in the sense we were used to (coming from CFML), and don't bother trying to pretend it's traditional OO by trying to do inheritance or non-supported stuff like that.
  • Namespace everything.
  • We emulated classes using  the prototypical function approach, with a consistency rule that the there was only one class per file, the file was named the same as the class, and the namespace of the class was reflected in the directory hierarchy.
  • Separate the definition of the class from the usage of the class, ie: in different files, indeed different directory trees.
  • Object methods were always added to the main function's prototype.
  • "Class methods" were added to the function directly.
An example of this might be:

var ns = ns || {};

ns.Person = function(firstName, lastName, dob, status){
    this.firstName = firstName;
    this.lastName = lastName;
    this.dob = dob;
    this.status = status;
};

ns.Person.capitaliseName = function(name){
    return name.replace(/(^|\b)([a-z])/g, function(match){return match.toUpperCase();});
}

ns.Person.prototype.getFullName = function(){
    return ns.Person.capitaliseName(this.firstName)
        + " "
        + ns.Person.capitaliseName(this.lastName);
};

ns.Person.prototype.getAgeInYears = function () {
    return new Date().getFullYear() - this.dob.getFullYear();
};

ns.Person.prototype.setStatus = function (status) {
    this.status = status;
};

ns.Person.prototype.getStatus = function () {
    return this.status;
};

ns.Person.prototype.toJSON = function(){
    return {
        firstName : this.firstName,
        lastName : this.lastName,
        fullName : this.fullName,
        dob : this.dob,
        status : this.status
    };
};


Don't worry too much about the code, but here we have a class for a Person which has properties firstName, lastName, date-of-birth, and status. We have object methods to get the person's full name; their age in years; plus set and get a status; and also a method for JSON.stringify() to use when serialising the object. The class also exposes a class method capitaliseName() which can be used for out-of-object situations where one just needs to capitalise a name, but doesn't need an object. The whole thing is namespaced (unimaginatively "ns" in this case) to keep our application's stuff away from other JavaScript stuff.

This file is called Person.js and is at something like /webapp/public/js/lib/ns/Person.js.

It could be used like this:

var person = new ns.Person("kiri", "te kanawa", new Date("1944-03-06"));
console.dir(person);
console.log(person.firstName);
console.log(person.lastName);
console.log(person.getFullName());
console.log(JSON.stringify(person));

console.log(ns.Person.capitaliseName("jean batten"));

var secondPerson = new ns.Person("georgina", "beyer", new Date("1957-11-01"));
console.dir(secondPerson);
console.log(secondPerson.firstName);
console.log(secondPerson.lastName);
console.log(secondPerson.getFullName());

secondPerson.setStatus("Active");
console.log(secondPerson.getStatus());

console.log(JSON.stringify(secondPerson));

Now I hasten to add that all the JavaScript code we're talking about here is client-side JavaScript, I perhaps shoulda mentioned that before. That said, I'm running it via node as the output is easier to copy/paste here:

{ firstName: 'kiri',
  lastName: 'te kanawa',
  dob: Mon Mar 06 1944 00:00:00 GMT+0000 (GMT Standard Time),
  status: undefined }
kiri
te kanawa
Kiri Te Kanawa
{"firstName":"kiri","lastName":"te kanawa","dob":"1944-03-06T00:00:00.000Z"}
Jean Batten
{ firstName: 'georgina',
  lastName: 'beyer',
  dob: Fri Nov 01 1957 00:00:00 GMT+0000 (GMT Standard Time),
  status: undefined }
georgina
beyer
Georgina Beyer
Active
{"firstName":"georgina","lastName":"beyer","dob":"1957-11-01T00:00:00.000Z","status":"Active"}

C:\src>

OK, so that demonstrates everything works A-OK, but also demonstrates something that tends to irk me with JavaScript: for the object methods to access the state of the object, the state needs to be public (accessed via the object's this reference). There's no sense of privacy to the object's properties, and it means all (shared) methods are also really public. In reality this is not often an issue, other than the fact sometimes we find ill-thought-out code battering into a property directly instead of using the class's API. This is a non-theoretical issue: it has caused us problems in the past. Mostly with refactoring. It's always best to keep one's API "on point", as it reduces refactoring challenges.

And it just irks me. I also know I am not alone in this irkery.

One thing some of the bods at work have been looking at recently is YUI's Module Pattern, which kinda deals with the property privacy thing by leveraging closure. Outwardly this sounds all right actually, and when I only fleetingly looked at some example code it looked like the business for us. Here's a reimplementation of the above class using the traditional Module Pattern.

var ns = ns || {};

ns.person = (function(firstName, lastName, dob){
    var fullName = firstName + " " + lastName;
    var private = {
        status : undefined
    };

    var capitalise = function(name){
        return name.replace(/(^|\b)([a-z])/g, function(match){return match.toUpperCase();});
    };

    return {
        getFullName : function(){
            return capitalise(fullName);
        },
        capitaliseName : capitalise,
        getAgeInYears : function () {
            return new Date().getFullYear() - private.dob.getFullYear();
        },
        setStatus : function (status) {
            private.status = status;
        },
        getStatus : function(){
            return private.status;
        },
        toJSON : function() {
            return {
                firstName : firstName,
                lastName : lastName,
                fullName : fullName,
                dob : dob,
                status : private.status
            };
        }
    };
})("jerry", "mateparae", new Date("1954-11-14"));

And some calling code:

console.log(ns.person.getFullName());
console.dir(ns.person);

console.log(ns.person.capitaliseName("temuera morrison"));

ns.person.setStatus("governor general");
console.log(ns.person.getStatus());
console.log(JSON.stringify(ns.person));


And output:
Jerry Mateparae
{ getFullName: [Function],
  capitaliseName: [Function],
  getAgeInYears: [Function],
  setStatus: [Function],
  getStatus: [Function],
  toJSON: [Function] }
Temuera Morrison
governor general
{"firstName":"jerry","lastName":"mateparae","fullName":"jerry mateparae","dob":"1954-11-14T00:00:00.000Z","status":"governor general"}

C:\src>

OK, so this works, in that it hides the properties from the calling code. But - thanks to the IIFE approach - it just creates a one-off object. That's a pretty shit pattern, IMO: the "class" code is not at all reusable... if one wants a second person, one needs to repeat the code. It's also a wee bit shit cos if we want to change the state of any of the private values we need to call them something different from any argument name that might be used to pass-in the values. EG: one cannot do this:

setStatus : function (status) {
    status = status;
},

One needs to do this (or some variation on this, eg have an _status internal variable or something:

ns.Person = function(firstName, lastName, dob, status){
    var private = {
        // ...
        status : status
    };

    // ...

    return {
        // ...
        setStatus : function (status) {
            private.status = status;
        },
        // ...
    };
};

It's not the end of the world, but it's all getting a bit "let's pretend we can do some stuff that we actually can't".

I don't quite know what the thinking was here. We're not gonna be using this approach.

Wednesday 11 May 2016

JavaScript TDD: getting Mocha tests running on Bamboo

G'day:
We've finally having a chance to move forward with our JavaScript TDD / testing. You might have recalled that I'd lamented in the past we'd wanted to take this sort of thing more seriously with our JS, but it was taking an age to get it moving. Well now it's been placed in the hands of the devs, so it's... moving.

This came to us on Friday, and we had a wee faff around with it then, but didn't make a huge amount of progress. On the weekend I decided to get a proof of concept working, which I did on Saturday. It was kind of a bit of a mission, and I didn't write down what I did so I replicated it again on my other laptop this morning, and took some notes. Which I'll now pad out into an article.

We'd been looking at Jasmine for our JavaScript TDD, but this was based solely on that being the one JS testing framework I was aware of. Jasmine works really well, and superficially I like the approach it has (I say "superficially" cos that's the full extent of my exposure to it). However when I went to see how to get Bamboo running Jasmine tests, all I could really find was chatter about Mocha. I had a quick look at Mocha and it seemed much the same as Jasmine, and looked pretty good, so decided to shift my attention onto using that instead. It seems like Bamboo was kinda pushing me in that direction anyhow.

Installing Mocha is done via NPM:

C:\temp>npm install -g mocha
npm WARN deprecated jade@0.26.3: Jade has been renamed to pug, please install the latest version of pug instead of jade
npm WARN deprecated graceful-fs@2.0.3: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
C:\Users\adam.cameron\AppData\Roaming\npm\mocha -> C:\Users\adam.cameron\AppData\Roaming\npm\node_modules\mocha\bin\mocha
C:\Users\adam.cameron\AppData\Roaming\npm\_mocha -> C:\Users\adam.cameron\AppData\Roaming\npm\node_modules\mocha\bin\_mocha
mocha@2.4.5 C:\Users\adam.cameron\AppData\Roaming\npm\node_modules\mocha
├── escape-string-regexp@1.0.2
├── commander@2.3.0
├── diff@1.4.0
├── growl@1.8.1
├── supports-color@1.2.0
├── debug@2.2.0 (ms@0.7.1)
├── mkdirp@0.5.1 (minimist@0.0.8)
├── jade@0.26.3 (commander@0.6.1, mkdirp@0.3.0)
└── glob@3.2.3 (inherits@2.0.1, graceful-fs@2.0.3, minimatch@0.2.14)

(don't worry about the deprecation warnings, everything still works... this is all just part of a trademark dispute).

And similarly install Chai and bamboo-mocha-reporter.

Another thing I needed to do is to make sure I had my NODE_PATH environment variable set. I was kinda expecting the Node install to do this, but it didn't. For me that needs to point to C:\Users\adam.cameron\AppData\Roaming\npm\node_modules

So a basic Mocha test looks like this:

var assert = require('chai').assert;

describe('MyClass tests', function() {
    var MyClass = require("../myApp/MyClass");
    describe('Baseline', function () {
        it('should exist and be usable', function () {
            var myObj = new MyClass();
            assert.instanceOf(myObj, MyClass, "myObj should be an instance of MyClass");
        });
    });
});

This is a tweaking of the sample they had on their website's "getting started" section:

var assert = require('chai').assert;
describe('Array', function() {
  describe('#indexOf()', function () {
    it('should return -1 when the value is not present', function () {
      assert.equal(-1, [1,2,3].indexOf(5));
      assert.equal(-1, [1,2,3].indexOf(0));
    });
  });
});

Note that the instructions there don't actually implicitly mention the dependency on Chai here, which is a bit crap of them. They seem to kinda assume one knows this. This is not a safe assumption to make.

Another thing that is "interesting" about Mocha is that it doesn't come with any assertion capabilities at all, so one needs to install Chai (or some other assertions lib) for it to be of much use. I guess this is symptomatic of this mania Node apps have for making everything a "dependency" even if it's really more of a necessity. Equally I discovered that Mocha has no mocking framework either (which almost makes the name of the thing a misrepresentation! ;-), so I think I'll be using Sinon for that. But later.

But anyhow, one one installs all that crap, one can run a test.

I've created a basic GitHub repo, JavaScriptTesting, which has a basic class and a basic test of it:

var Person = function(firstName, lastName) {
    this.firstName = firstName;
    this.lastName = lastName;
};

Person.prototype.getFullName = function(){
    return this.firstName + " " +this.lastName;
};

module.exports = Person;


var assert = require('chai').assert;

describe('Person tests', function() {
    var Person = require("../myApp/Person");
    describe('Baseline', function () {
        it('should exist and be usable', function () {
            var person = new Person();
            assert.instanceOf(person, Person);
        });
    });
    describe('Method tests', function () {
        describe('getFullName tests', function () {
            it('should return a full name', function () {
                var person = new Person("FIRST_NAME", "LAST_NAME");
                var fullName = person.getFullName();
                assert.equal("FIRST_NAME LAST_NAME", fullName);
            });
        });
    });
});

And I can now run these from the command prompt:

C:\src\JavaScriptTesting>mocha


  Person tests
    Baseline
      V should exist and be usable
    Method tests
      getFullName tests
        V should return a full name


  2 passing (47ms)


C:\src\JavaScriptTesting>

Cool. So I know the tests work.

Now I go about getting it all to work on Bamboo. I'd not installed this before so I followed the instructions. One pitfall I fell foul of the first time is the requirements state:

Servlet container requirements
You will need a servlet container that supports the Servlet 2.4 specification. Most modern containers should comply with this.

This is from "Bamboo installation guide". So I messed around getting Tomcat installed (which granted is not much messing around), only to find out that's bullshit. Bamboo installs its own instance of Tomcat, and does that out of the box as part of the installer. Oh well: this is not much of a hardship.

I knew Bamboo will be needing to write stuff to disk and do other stuff that the local system account perhaps did not have permissions to do, so I created a specific account for Bamboo (I called it "bamboo" with a password of "bamboo" ;-), and gave that user full access permissions to the Bamboo home directory.

Oh... during install I let Bamboo create its "home" (/working) directory in the default location which - in hindsight was daft both on my part, and on part of the Bamboo installer, cos it created it in my own user account's user directory. That makes no sense for a server-oriented application. So I changed this to be c:\bamboo.tmp (I'll be blowing all this away on this machine once I've done with this article). So it was that dir I gave the Bamboo user full control permissions to.

After that I dropped down to a command prompt (running as administrator) and ran c:\apps\atlassian\bamboo\InstallAsService.bat to install the Windows service, then I edited that to login with the new bamboo account and started it.

Browsing to http://localhost:8085/ yielded success! (I can't show you a screen shot as I forgot to take one at the time, and I cannot get back to that first screen again now. Just tryst me ;-)

Next Bamboo needs some configuration, so I chose the Express object, which just requires me to enter a login and a password and an email address. Then I was in, and it was time to create a new job to run my tests.

I really have no idea what's going on in Bamboo, but I kinda followed my nose.

I created a "project" called "JavaScript App" which was assigned a short code of "JA". As far as I can tell this is just like in Jira when one creates a new project, and it gets that abbreviation which ends up as the prefix to all tickets. But I dunno. I just agreed with what Bamboo suggested. Next I created a "plan" called "JS TDD CI", and that got an abbrev. of "JTC" (f*** knows if any of this will ever be relevant again, but it's what was going on on the screen as I was thinking "yes, lovely, that's nice").



Then we started to do something that seemed useful: I linked the plan to a repository which is where I point Bamboo to my GitHub repository (that JavaScriptTesting one I mentioned further up). All good.




On the next screen I add the other steps to the plan. It's already added the "Source Code checkout" one for me. So running that much would get the source code checked out.

Next I need to tell Bamboo about Mocha and Chai (and its own mocha-bamboo-reporter), so I add three NPM tasks. Here's the screens for adding the mocha one: the others are all the same:





Having done that, I then add the task to run the tests, which is done via the Mocha Test Runner:



And that's it. I can now run the tests by selecting the run option from the top right of the main plan screen. Stuff whirs into action (and it logs an awful lot of stuff), and eventually:



Now I dart back to my other machine and add a new test to my code:

describe('getAgeInYears tests', function () {
    it('should return the correct age in years', function () {
        var person = new Person("FIRST_NAME", "LAST_NAME", new Date("1970-02-17"));
        var ageInYears = person.getAgeInYears();
        assert.equal(46, ageInYears); // obviously this is a poorly-written test, but it works for a few months yet ;-)
    });
});

This fails cos I have not added the getAgeInYears method yet, but for the sake of demonstration I'll push it up to GitHub.

Bamboo is polling GitHub every three minutes, so after a bit a test run starts and...




There's my latest changes breaking the build. I've never been so pleased to see a failing test before. Hang on whilst I implement that method...



Cool.

Now I did have some problems, and I'm not that happy with a coupla things. Firstly I've got all those NPM modules installed globally already, and I have them on my NODE_PATH. But Bamboo refuses to "see" them. It only seems interested in looking in the node_modules dir in its own working directory. I could not work out how to change that.

I did have some success initially with doing an npm link instead of an npm install in my test script, but when I came to test everything today I must have screwed something up as it had stopped working. Using the install approach requires dragging the stuff down off the 'net each time I do a test run, which seems a bit wasteful. I'm sure it's something I'm doing wrong, and when I work it out, I'll update this article accordingly.

Having got this proof of concept running, we are now in the position to give it a go in our actual work environment (which we've done... we've got our first test passing).

I'm pretty pleased about this. I am now in a place with my JavaScript development that I should have been (and wanted to be) about three years ago. But at least I've got there.

Righto.

--
Adam