Thursday 25 February 2021

Fine, Dara. Fine. This thing is now on HTTPS

G'day:

My mate Dara McGann has been pestering me for a while to get this thing onto HTTPS. I couldn't be arsed forking out the readies to get an SSL for this domain, and didn't think it really mattered, but he - rightly (grumble) - pointed out that Google etc could possibly penalise non-secure sites these days in their search rankings. And given this thing has such low readership these days (poss due to the quality of the content… ahem), it needs all the help it can get.

So I forked out the £££.

I had a bit of an outage before due to some DNS shenanigans or some such, but it seems to be stable now. HTTP requests ought to redirect to HTTPS, and most importantly the site should - you know - be up and working. Obviously this is a dumb place to say this, but if you encounter any issues let me know.

Grumble.

--
Adam

Wednesday 24 February 2021

Docker: using TDD to initialise my app's DB with some data

G'day:

I'll start by saying I am not convinced this exercise really ought to be a TDD-oriented one, but I'm gonna approach it that way anyhow because I suspect I'm going to need to mess around a bit to get this working. Secondly, this is very much a log of what I'm (trying to ~) work on today, and I doubt there will be any shrewd insights going on, given I'm basically googling and RTFMing, then doing what the docs say.

The exercise here is to take the MariaDB database that I already have in my Docker set up (see "Creating a web site with Vue.js, Nginx, Symfony on PHP8 & MariaDB running in Docker containers - Part 5: MariaDB"), which is currently empty and only accessible via the root login; and add some baseline tables and data to it. At the same time also create a user for code to connect to the DB with so I don't need code using root access. Another thing I want to do is stop storing the DB passwords in the Docker .env file like I am now:

COMPOSE_PROJECT_NAME=fullStackExercise
DATABASE_ROOT_PASSWORD=123

The data I need is to fulfil an exercise I have given myself (well: it wasn't me who gave me the original exercise, but I'm reinventing it a bit here) to construct an event registration form (personal details, a selection of workshops to register for), save the details to the DB and echo back a success page. Simple stuff. Less so for me given I'm using tooling I'm still only learning (Vue.js, Symfony, Docker, Kahlan, Mocha, MariaDB).

For my tests, I can already derive a bunch of test specs from those first few paragraphs above, so let's put them together now in spec/integration/baselineDatabase.spec.php:

<?php

namespace adamCameron\fullStackExercise\spec\integration;

describe('Tests for registration database', function () {
    describe('Connectivity tests', function () {
        it('can connect to the database with environment-based credentials', function () {
        });
    });

    describe('Schema tests', function () {
        it('has a workshops table with the required schema', function () {
        });

        it('has a registrations table with the required schema', function () {
        });

        it('has a registeredWorkshops table with the required schema', function () {
        });
    });

    describe('Data tests', function () {
        it('has the required baseline workshop data', function () {
        });
    });
});

And I can now run those to see them… not be implemented:

root@fde4be76c908:/usr/share/fullstackExercise# composer spec -- --spec=spec/integration/baselineDatabase.spec.php --reporter=verbose
> vendor/bin/kahlan '--spec=spec/integration/baselineDatabase.spec.php' '--reporter=verbose'


  Tests for registration database
    Connectivity tests
      ✓ it can connect to the database with environment-based credentials
    Schema tests
      ✓ it has a workshops table with the required schema
      ✓ it has a registrations table with the required schema
      ✓ it has a registeredWorkshops table with the required schema
    Data tests
      ✓ it has the required baseline workshop data


  Pending specifications: 5
  .spec/integration/baselineDatabase.spec.php, line 8
  .spec/integration/baselineDatabase.spec.php, line 13
  .spec/integration/baselineDatabase.spec.php, line 16
  .spec/integration/baselineDatabase.spec.php, line 19
  .spec/integration/baselineDatabase.spec.php, line 24


Expectations   : 0 Executed
Specifications : 5 Pending, 0 Excluded, 0 Skipped

Passed 0 of 0 PASS in 0.015 seconds (using 4MB)

root@fde4be76c908:/usr/share/fullstackExercise#

And now I can implement that first test:

describe('Tests for registration database', function () {

    $this->getConnectionDetailsFromEnvironment = function () {
        return (object) [
            'database' => $_ENV['MYSQL_DATABASE'],
            'user' => $_ENV['MYSQL_USER'],
            'password' => $_ENV['MYSQL_PASSWORD']
        ];
    };

    describe('Connectivity tests', function () {
        it('can connect to the database with environment-based credentials', function () {
            $connectionDetails = $this->getConnectionDetailsFromEnvironment();
            $connection = new PDO(
                "mysql:dbname=$connectionDetails->database;host=database.backend",
                $connectionDetails->user,
                $connectionDetails->password
            );
            $statement = $connection->query("SELECT 'OK' AS test FROM dual");
            $statement->execute();

            $testResult = $statement->fetch(PDO::FETCH_ASSOC);

            expect($testResult)->toContainKey('test');
            expect($testResult['test'])->toBe('OK');
        });
    });

There's not much to this. I'm reading the DB connectivity details from the environment variables Docker has set for me, and using those to do a simple DB query from the database, and just verify the DB is responding as expected. To be honest I don't think I need / ought to be using the environment variable for the database name here: that environment variable is just for MariaDB to create a DB of that name when it first starts up. In the app itself, we'll have a static value for the database name, because the app wants to use that exact database, not simply whatever DB is in that environment variable. Hopefully you see the subtle difference in intent there. Anyhow, we now run our tests:

> vendor/bin/kahlan '--spec=spec/integration/baselineDatabase.spec.php' '--reporter=verbose'


  Tests for registration database
    Connectivity tests
      ✖ it can connect to the database with environment-based credentials
        an uncaught exception has been thrown in `spec/integration/baselineDatabase.spec.php` line 11

        message:`Kahlan\PhpErrorException` Code(0) with message "`E_WARNING` Undefined array key \"MYSQL_DATABASE\""

          [NA] - spec/integration/baselineDatabase.spec.php, line 7 to 11
          […etc…]

Cool. Now we can sort those credentials out and watch that test pass. The first thing I have done is to update docker/.env (see above) to get rid of the root password, and add the other credentials MariaDB expects to initialise a database when its container is first built (see the "Environment Variables" section in mariadb - Docker Official Images for info about that):

COMPOSE_PROJECT_NAME=fullStackExercise
MYSQL_DATABASE=fullstackExercise
# the following are to be provided to `docker-compose up`
DATABASE_ROOT_PASSWORD=
MYSQL_USER=
MYSQL_PASSWORD=

Those empty entries are not necessary, I've just left them there for the sake of documentation. The bit I do actually need to do is in docker/docker-compose.yml. This is best shown with a diff I think:

$ git diff docker/docker-compose.yml
diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml
index bc399a0..0eec553 100644
--- a/docker/docker-compose.yml
+++ b/docker/docker-compose.yml
@@ -24,7 +24,9 @@ services:
       context: ../backend
       dockerfile: ../docker/php-fpm/Dockerfile
     environment:
-      - DATABASE_ROOT_PASSWORD=${DATABASE_ROOT_PASSWORD}
+      - MYSQL_DATABASE=${MYSQL_DATABASE}
+      - MYSQL_USER=${MYSQL_USER}
+      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
     volumes:
       - ../backend/config:/usr/share/fullstackExercise/config
       - ../backend/public:/usr/share/fullstackExercise/public
@@ -52,6 +54,9 @@ services:
       context: ./mariadb
     environment:
       - MYSQL_ROOT_PASSWORD=${DATABASE_ROOT_PASSWORD}
+      - MYSQL_DATABASE=${MYSQL_DATABASE}
+      - MYSQL_USER=${MYSQL_USER}
+      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
     ports:
       - "3306:3306"
     volumes:

I've taken out PHP's access to the DB root password as it doesn't need it any more. It has two tests that will now fail, but they were only ever temporary ones until I did this work anyhow, so I'll be deleting those when I verify they now fail. And I've also added in the three new environment variables to both the MariaDB service, and the PHP one. MariaDB uses it to create the fullstackExercise DB, and PHP will use the same credentials to connect to it. I now have no DB credentials anywhere in the codebase. Instead, I pass them in when I first bring the containers up:

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ DATABASE_ROOT_PASSWORD=123 MYSQL_USER=fullstackExercise MYSQL_PASSWORD=1234 docker-compose up --build --detach

This is not completely secure. One can still see the passwords if one terminals into the containers, eg:

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker exec --interactive --tty fullstackexercise_php-fpm_1 /bin/bash
root@ac3872091c8e:/usr/share/fullstackExercise# set | grep MYSQL
MYSQL_DATABASE=fullstackExercise
MYSQL_PASSWORD=1234
MYSQL_USER=fullstackExercise
root@ac3872091c8e:/usr/share/fullstackExercise#

A better way would perhaps be to use Docker Secrets, but I could not work out how to get the values from the files it creates into environment variables in the docker-compose.yml file. But will also admit I pretty much read the docs and went "yeah CBA with that right now". It might be dead easy (UPDATE: just now when linking to the MariaDB docker image page a coupla paragraphs up, I noticed it's all actually explained there, and it is dead easy. I might look at doing this later then).

Now I will run my tests again. My expectations are that that test that failed before will now be passing; and one each of the Kahlan and PHPUnit tests will start to fail because they are testing connecting to the DB using the root credentials, which I've removed.

root@ac3872091c8e:/usr/share/fullstackExercise# composer spec
> vendor/bin/kahlan

..........E.PPPP.                                                 17 / 17 (100%)


  Pending specifications: 4
  .spec/integration/baselineDatabase.spec.php, line 37
  .spec/integration/baselineDatabase.spec.php, line 40
  .spec/integration/baselineDatabase.spec.php, line 43
  .spec/integration/baselineDatabase.spec.php, line 48

Tests database availability
  ✖ it should return the expected database version
    an uncaught exception has been thrown in `spec/integration/database.spec.php` line 14

    message:`Kahlan\PhpErrorException` Code(0) with message "`E_WARNING` Undefined array key \"DATABASE_ROOT_PASSWORD\""

      [NA] - spec/integration/database.spec.php, line 11 to 14
      Kahlan\Filter\Filters::run() - vendor/kahlan/kahlan/src/Suite.php, line 236
      […etc…]


Expectations   : 18 Executed
Specifications : 4 Pending, 0 Excluded, 0 Skipped

Passed 12 of 13 FAIL (EXCEPTION: 1) in 0.491 seconds (using 6MB)

Script vendor/bin/kahlan handling the spec event returned with error code 255

This is good: only one failing test: the one we expect to fail, and it's failing for the right reason. And with PHPUnit:

PHPUnit 9.5.2 by Sebastian Bergmann and contributors.

.....E                                                              6 / 6 (100%)

Time: 00:00.268, Memory: 14.00 MB

There was 1 error:

1) adamCameron\fullStackExercise\tests\integration\DatabaseTest::testDatabaseVersion
Undefined array key "DATABASE_ROOT_PASSWORD"

/usr/share/fullstackExercise/tests/integration/DatabaseTest.php:16

ERRORS!
Tests: 6, Assertions: 13, Errors: 1.

Generating code coverage report in HTML format ... done [00:00.374]
Script vendor/bin/phpunit handling the test event returned with error code 2

I'll get rid of those failing tests. They are redundant now.

The next test cases we have to address are these ones:

    Schema tests
      ✓ it has a workshops table with the required schema
      ✓ it has a registrations table with the required schema
      ✓ it has a registeredWorkshops table with the required schema

Looking at the docs for MariaDB's Docker image ("Docker Official Images > mariadb > Initializing a fresh instance"), when the DB starts up, it looks for files in a docker-entrypoint-initdb.d directory, and runs any scripts it finds in there. This makes things easy.

However let's not get ahead of ourselves. We need tests first. But first… a bit of an aside. I'm actually questioning the merits of these tests. They are handy when I'm doing the initial DB setup though. Later as the application develops, we'll have more finely-tuned integration tests that will implicitly test the table schemata are correct; but I guess at the moment all we need to have is the schema (then some baseline data), so as transient tests I suppose the have some merit. I'm not sure. One one hand it might be overkill; on another hand we're supposed to be developing the application iteratively, and these are a first iteration. I guess the situation is similar to the DB tests I had that were using the root connectivity details, because for that iteration that's where we were at. Now we've moved on so those tests are redundant, and these new tests replace them. And these tests will likely be replaced in the next coupla iterations as we go. Anyhow: I'm writing them. Here we go.

describe('Schema tests', function () {
    $schemata = [
        [
            'tableName' => 'workshops',
            'schema' => [
                ['Field' => 'id', 'Type' => 'int(11)'],
                ['Field' => 'name', 'Type' => 'varchar(500)']
            ]
        ],
        [
            'tableName' => 'registrations',
            'schema' => [
                ['Field' => 'id', 'Type' => 'int(11)'],
                ['Field' => 'fullName', 'Type' => 'varchar(100)'],
                ['Field' => 'phoneNumber', 'Type' => 'varchar(50)'],
                ['Field' => 'emailAddress', 'Type' => 'varchar(320)'],
                ['Field' => 'password', 'Type' => 'varchar(255)'],
                ['Field' => 'ipAddress', 'Type' => 'varchar(15)'],
                ['Field' => 'uniqueCode', 'Type' => 'varchar(36)'],
                ['Field' => 'created', 'Type' => 'timestamp']
            ]
        ],
        [
            'tableName' => 'registeredWorkshops',
            'schema' => [
                ['Field' => 'id', 'Type' => 'int(11)'],
                ['Field' => 'registrationId', 'Type' => 'int(11)'],
                ['Field' => 'workshopId', 'Type' => 'int(11)']
            ]
        ]
    ];

    array_walk($schemata, function ($tableSchema) {
        $tableName = $tableSchema['tableName'];
        $expectedSchema = $tableSchema['schema'];

        it("has a $tableName table with the required schema", function () use ($tableName, $expectedSchema) {
            $statement = $this->connection->query("SHOW COLUMNS FROM $tableName");
            $statement->execute();

            $columns = $statement->fetchAll(PDO::FETCH_ASSOC);

            expect($columns)->toHaveLength(count($expectedSchema));
            foreach ($expectedSchema as $i => $column) {
                expect($columns[$i]['Field'])->toBe($expectedSchema[$i]['Field']);
                expect($columns[$i]['Type'])->toBe($expectedSchema[$i]['Type']);
            }
        });
    });
});

There was an intermediary refactoring here: initially I had three "hard-coded" cases, as listed further up. As I wrote the test for the second case I noticed I was duplicating everything from the first test except the table name and the details of the schema, so I extracted those as test data, and looped over them. All the test does here is to get the table columns description, and verify they match the name, type and length of my expectations. The expectations were taken directly from the requirement I had been given to implement.

If I now run the tests, those three cases fail, as we'd expect given the tables don't yet exist:

Tests for registration database
  Schema tests
    ✖ it has a workshops table with the required schema
      an uncaught exception has been thrown in `spec/integration/baselineDatabase.spec.php` line 74

      message:`PDOException` Code(42S02) with message "SQLSTATE[42S02]: Base table or view not found: 1146 Table 'fullstackexercise.workshops' doesn't exist"

        [NA] - spec/integration/baselineDatabase.spec.php, line 73 to 74
        […etc…]

    ✖ it has a registrations table with the required schema
      an uncaught exception has been thrown in `spec/integration/baselineDatabase.spec.php` line 74

      message:`PDOException` Code(42S02) with message "SQLSTATE[42S02]: Base table or view not found: 1146 Table 'fullstackexercise.registrations' doesn't exist"

        [NA] - spec/integration/baselineDatabase.spec.php, line 73 to 74
        […etc…]

    ✖ it has a registeredWorkshops table with the required schema
      an uncaught exception has been thrown in `spec/integration/baselineDatabase.spec.php` line 74

      message:`PDOException` Code(42S02) with message "SQLSTATE[42S02]: Base table or view not found: 1146 Table 'fullstackexercise.registeredWorkshops' doesn't exist"

        [NA] - spec/integration/baselineDatabase.spec.php, line 73 to 74
        […etc…]
[…etc…]

Now to add the tables.I've set up these files:

adam@DESKTOP-QV1A45U:/mnt/c/src/ttct$ tree docker/mariadb/do*
docker/mariadb/docker-entrypoint-initdb.d
├── 1.createAndPopulateWorkshops.sql
├── 2.createRegistrations.sql
└── 3.createRegisteredWorkshops.sql

Note: for now that first file name is slightly misnamed, as it'll only have the DDL statement in it at the moment, and the data-insertion will come in a subsequent step. The file contents are as follows:

/* docker/mariadb/docker-entrypoint-initdb.d/1.createAndPopulateWorkshops.sql */

USE fullstackExercise;

CREATE TABLE workshops (
    id INT NOT NULL AUTO_INCREMENT,
    name VARCHAR(500) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
    
    PRIMARY KEY (id)
) ENGINE=InnoDB;


/* docker/mariadb/docker-entrypoint-initdb.d/2.createRegistrations.sql */

USE fullstackExercise;

CREATE TABLE registrations (
   id INT NOT NULL AUTO_INCREMENT,
   fullName VARCHAR(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
   phoneNumber VARCHAR(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
   emailAddress VARCHAR(320) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
   password VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
   ipAddress VARCHAR(15) NOT NULL,
   uniqueCode VARCHAR(36) NOT NULL DEFAULT (UUID()),
   created TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
   
   PRIMARY KEY (id)
) ENGINE=InnoDB;


/* docker/mariadb/docker-entrypoint-initdb.d/3.createRegisteredWorkshops.sql */

USE fullstackExercise;

CREATE TABLE registeredWorkshops (
   id INT NOT NULL AUTO_INCREMENT,
   registrationId INT NOT NULL,
   workshopId INT NOT NULL,
   PRIMARY KEY (id),
   FOREIGN KEY (registrationId) REFERENCES registrations(id),
   FOREIGN KEY (workshopId) REFERENCES workshops(id)
);

And lastly I need to copy that directory into my MariaDB container when I build it (docker/mariadb/Dockerfile):

FROM mariadb:latest
COPY ./docker-entrypoint-initdb.d/ /docker-entrypoint-initdb.d/
CMD ["mysqld"]
EXPOSE 3306

After I rebuild my containers, I run the tests and we're all good:

    Schema tests
       it has a workshops table with the required schema
       it has a registrations table with the required schema
       it has a registeredWorkshops table with the required schema

Finally I need some seed data in the workshops table. First I'm going to write my test cases for this:

describe('Data tests', function () {
    it('has the required baseline workshop data', function () {
        $expectedWorkshops = [
            ['id' => '2', 'name' => 'TEST_WORKSHOP 1'],
            ['id' => '3', 'name' => 'TEST_WORKSHOP 2'],
            ['id' => '5', 'name' => 'TEST_WORKSHOP 3'],
            ['id' => '7', 'name' => 'TEST_WORKSHOP 4']
        ];

        $statement = $this->connection->query("SELECT id, name FROM workshops ORDER BY id");
        $statement->execute();
        $workshops = $statement->fetchAll(PDO::FETCH_ASSOC);

        expect($workshops)->toEqual($expectedWorkshops);
    });

    it('correctly auto-increments the ID on new insertions', function () {
        $$expectedWorkshopName = 'TEST_WORKSHOP 5';

        $this->connection->beginTransaction();

        $statement = $this->connection->prepare(query: "INSERT INTO workshops (name) VALUES (:name)");
        $statement->execute(['name' => $expectedWorkshopName]);
        $id = $this->connection->lastInsertId();

        $statement = $this->connection->prepare("SELECT id, name FROM workshops WHERE id = :id");
        $statement->execute(['id' => $id]);
        $workshops = $statement->fetchAll(PDO::FETCH_ASSOC);

        expect($workshops)->toHaveLength(1)
        expect($workshops[0])->toContainKey('name')
        expect($workshops[0]['name'])->toBe($expectedWorkshopName)

        $this->connection->rollback();
    });
});

Those are reasonably self-explanatory. I need to insert four baseline workshop records, and in the first case I just SELECT the data and check it's what I expect it to be. The second case only occurred to me when I went to look at the changes to the SQL I needed to make in 1.createAndPopulateWorkshops.sql to insert that data. I needed to take the auto-increment off the table-create statement so I could insert records with the specific IDs I need, then after doing that I altered the table to have the ID auto-increment. I figured I had better test that that worked too. So I insert a new record (just the name, letting the DB handle the ID), get the ID back and use that to get the whole record back for that ID, verifying it's also got the correct name. I do no want that data cluttering my DB so I put the whole thing in a transaction so that it rolls-back when I'm done or if there's an error.

Running those, only the first one errors:

> vendor/bin/kahlan '--spec=spec/integration/baselineDatabase.spec.php'

F.                                                                  2 / 2 (100%)


Tests for registration database
  Data tests
    ✖ it has the required baseline workshop data
      expect->toEqual() failed in `.spec/integration/baselineDatabase.spec.php` line 102

      It expect actual to be equal to expected (==).

      actual:
        (array) []
      expected:
        (array) [
            0 => [
                "id" => "2",
                "name" => "TEST_WORKSHOP 1"
            ],
            […etc…]


Expectations   : 4 Executed
Specifications : 0 Pending, 0 Excluded, 0 Skipped

Passed 1 of 2 FAIL (FAILURE: 1) in 0.025 seconds (using 4MB)

Focus Mode Detected in the following files:
fdescribe - spec/integration/baselineDatabase.spec.php, line 89 to 124
exit(-1)

Script vendor/bin/kahlan handling the spec event returned with error code 255
root@e7d6aa6cf839:/usr/share/fullstackExercise#

This puzzled me at first, but then it occurred to me that the auto-increment test case really ought to have been added when I did the first round of tests before creating the table, because that is when that functionality was added. All I'm doing with the changes I'm about to make is insert some data-insertion code into the script. It's already doing the auto-increment on the ID, and all I'm doing with that is changing when it's being applied: from the table-creation statement to its own statement after the inserts are done. See below for what I mean.

And now I'll now update that docker/mariadb/docker-entrypoint-initdb.d/1.createAndPopulateWorkshops.sql to also insert the baseline data:

USE fullstackExercise;

CREATE TABLE workshops (
    id INT NOT NULL /* AUTO_INCREMENT <- this has been removed from here */,
    name VARCHAR(500) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
    
    PRIMARY KEY (id)
) ENGINE=InnoDB;

INSERT INTO workshops (id, name)
VALUES
    (2, 'TEST_WORKSHOP 1'),
    (3, 'TEST_WORKSHOP 2'),
    (5, 'TEST_WORKSHOP 3'),
    (7, 'TEST_WORKSHOP 4')
;

ALTER TABLE workshops MODIFY COLUMN id INT auto_increment;

(Note I've moved the auto-increment on the ID field when I create the table now, so the seed data can have specific IDs. Once I do the insert, then I make the column auto-increment).

Once I rebuild my containers, all the tests now pass:

> vendor/bin/kahlan '--spec=spec/integration/baselineDatabase.spec.php' '--reporter=verbose'


  Tests for registration database
    Connectivity tests
       it can connect to the database with environment-based credentials
    Schema tests
       it has a workshops table with the required schema
       it has a registrations table with the required schema
       it has a registeredWorkshops table with the required schema
    Data tests
       it has the required baseline workshop data
       it correctly auto-increments the ID on new insertions



Expectations   : 35 Executed
Specifications : 0 Pending, 0 Excluded, 0 Skipped

Passed 6 of 6 PASS in 0.033 seconds (using 5MB)

root@44850303b17a:/usr/share/fullstackExercise#

And I think that's about it. I'm not doing anything with the data yet, but that'll start to be fleshed out in the next article (or maybe the following one. Not sure). This was just an exercise in doing some stuff with Docker and MariaDB, and thinking about the merits of TDDing exercises like this. I think it was worth it, especially during this early phase of working with these containers as I'm still reconfiguring stuff a lot, so it's good to know things don't get messed up when I'm monkeying with stuff.

Righto.

--
Adam

Monday 22 February 2021

Troubleshooting an issue with Kahlan

G'day:

I'm ramping-up to do some more development with this fullstackExercise codebase I've been working on (see "Creating a web site with Vue.js, Nginx, Symfony on PHP8 & MariaDB running in Docker containers", parts 1-12), and as part of the yak-shaving phase of things, I decided to port my PHP tests from PHPUnit to Kahlan. The reason for this is twofold. Firstly, I much prefer the describe / it syntax to writing and running tests over xUnit; and secondly - other than perpetually finding new ways to be frustrated by how PHPUnit handles/documents things - I am not learning a great deal about PHPUnit any more. So Kahlan fits in well with this learning exercise.

Porting the tests is easy, largely cos my tests are currently minimal in number and in complexity. But this PHPUnit class (test/functional/ExampleComTest.php):

namespace adamCameron\kahlanIssue\test\functional;

use GuzzleHttp\Client;
use PHPUnit\Framework\TestCase;
use Symfony\Component\HttpFoundation\Response;

class ExampleComTest extends TestCase
{
    /** @coversNothing */
    public function testExampleDotComReturnsExpectedContent()
    {
        $expectedContent = "Example Domain";


        $client = new Client([
            'base_uri' => 'http://example.com/'
        ]);

        $response = $client->get('index.html');

        $this->assertEquals(Response::HTTP_OK, $response->getStatusCode());

        $html = $response->getBody();
        $document = new \DOMDocument();
        $document->loadHTML($html);

        $xpathDocument = new \DOMXPath($document);

        $hasTitle = $xpathDocument->query('/html/head/title[text() = "' . $expectedContent . '"]');
        $this->assertCount(1, $hasTitle);

        $hasHeading = $xpathDocument->query('/html/body/div/h1[text() = "' . $expectedContent . '"]');
        $this->assertCount(1, $hasHeading);
    }
}

Becomes this Kahlan spec(spec/functional/example.com.spec.php):

namespace adamCameron\fullStackExercise\spec\functional;

use GuzzleHttp\Client;
use Symfony\Component\HttpFoundation\Response;

describe('Tests that example.com can be curled', function () {

    beforeAll(function () {
        $client = new Client([
            'base_uri' => 'http://example.com/'
        ]);

        $this->response = $client->get('index.html');

        $html = $this->response->getBody();
        $document = new \DOMDocument();
        $document->loadHTML($html);
        $this->xpathDocument = new \DOMXPath($document);
    });

    it("should have expected status code", function () {
        expect($this->response->getStatusCode())->toBe(Response::HTTP_OK);
    });

    it("should have expected title", function () {
        $hasTitle = $this->xpathDocument->query('/html/head/title[text() = "Example Domain"]');
        expect($hasTitle)->toHaveLength(1);
    });

    it("should have expected heading", function () {
        $hasHeading = $this->xpathDocument->query('/html/body/div/h1[text() = "Example Domain"]');
        expect($hasHeading)->toHaveLength(1);
    });
});

The problems for me started when I came to run that test in Kahlan. I got this:

            _     _
  /\ /\__ _| |__ | | __ _ _ __
 / //_/ _` | '_ \| |/ _` | '_ \
/ __ \ (_| | | | | | (_| | | | |
\/  \/\__,_|_| |_|_|\__,_|_| |_|

The PHP Test Framework for Freedom, Truth and Justice.

src directory  :
spec directory : /mnt/c/src/kahlanIssue/spec

                                                                    3 / 3 (100%)


Tests that example.com can be curled
  an uncaught exception has been thrown in `vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php` line 158

  message:`ParseError` Code(0) with message "Unclosed '{' on line 142 does not match ')'"

    [NA] - vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php, line  to 158
    Kahlan\Jit\ClassLoader::loadFile() - vendor/kahlan/kahlan/src/Jit/ClassLoader.php, line 759
    Kahlan\Jit\ClassLoader::loadClass() - vendor/guzzlehttp/guzzle/src/Utils.php, line 95
    [… etc …]


Expectations   : 0 Executed
Specifications : 0 Pending, 0 Excluded, 0 Skipped

Passed 0 of 1 FAIL (EXCEPTION: 1) in 0.015 seconds (using 7MB)

Eek, OK that's a compilation error, not a runtime exception, which is odd. I checked the code of CurlMultiHandler.php to see if anything was broken in there, and unsurprisingly it wasn't. I decided to chuck a try / catch around the code in question, and see if the actual issue was something other than being reported. I found it odd that Kahlan was reporting it as an Exception, when do me a code-parsing issue is an out-and-out Error. Two different things. So I wondered if Kahlan was doing something odd. I ran the code again and the error spewed out again. What the hell? Then I looked more closely. The error had changed slightly. Now it says this:

an uncaught exception has been thrown in `vendor/guzzlehttp/guzzle/src/Handler/StreamHandler.php` line 201

message:`ParseError` Code(0) with message "Unclosed '{' on line 192 does not match ')'"

Note before it was complaining about CurlMultiHandler.php. Now it's StreamHandler.php. OK. I'll play yer silly game. I stuck a try / catch around that code, and re-ran. Now it errors with this:

an uncaught exception has been thrown in `vendor/guzzlehttp/promises/src/Promise.php` line 151

message:`ParseError` Code(0) with message "Unclosed '{' on line 148 does not match ')'"

Grrr. OK, one last time with the try / catch:

            _     _
  /\ /\__ _| |__ | | __ _ _ __
 / //_/ _` | '_ \| |/ _` | '_ \
/ __ \ (_| | | | | | (_| | | | |
\/  \/\__,_|_| |_|_|\__,_|_| |_|

The PHP Test Framework for Freedom, Truth and Justice.

src directory  :
spec directory : /mnt/c/src/kahlanIssue/spec

...                                                                 3 / 3 (100%)



Expectations   : 3 Executed
Specifications : 0 Pending, 0 Excluded, 0 Skipped

Passed 3 of 3 PASS in 0.565 seconds (using 7MB)

Um. OK. Why's it working? Next I backed-out all my debug code and ran it again. And the tests worked still. Next I blew-away the entire vendor directory and re-did composer install, and the tests continued to work.

It's worth noting here that I know those are all Guzzle files, but If I run the equivalent code in PHPUnit, there is never any issue, hence me doubting it's anything to do with Guzzle specifically, it's how Kahlan is calling Guzzle. Probably with vendor/kahlan/kahlan/src/Jit/ClassLoader.php, which is the preceding file in the callstack, each time. I have to admit I did not look into that file to see what it is doing. I'll crack on with that shortly.

At this point I was thinking that it might be something to do with me running all this in a Docker container, and decided I should verify this in the native file system. At the same time I decided to create a simple repro case: my initial experiments at this stage had not been with the example spec above, they were with one of the fullstack-exercise test specs, and those required the rest of the Docker containers to be up and running, so too complicated to be portable (see "Short, Self Contained, Correct (Compilable), Example"). Accordingly I created the spec above, in a stand-alone application (as per the kahlan-issue repo on Github).

I sighed and installed PHP 8 and Composer on my PC - something I was hoping to avoid needing to do every again, now that I'm using Docker - and ran the test again. Again PHPUnit ran fine, but Kahlan errored-out. The act of putting debug code into the files erroring was enough to make that file now parse and compile, and the error moved on to the next file. Always the same three files. After touching the files once, the problem stayed away. Blowing away vendor and re-doing composer install did not make the erroring situation recur. However if I stuck the app in a different base directory - eg: I had it set up in C:\src\kahlanIssue initially, but moved it to C:\temp\kahlanIssue - then the error recurred again, until I tweakd the files. I noted that putting just whitespace into the file did not "fix" the issue; it needed to be something that PHP would need to parse. Even if I restarted the PC, once I had "touched" those three files once, even if I reverted the touches, the problem did not recur. That was weird.

I was also able to replicate the issue in all of:

  • on Ubuntu within a Docker container (host machine is a Windows PC);
  • on Ubuntu via WSL on that same Windows PC;
  • natively on that Windows PC;
  • natively on Windows on a different PC.

It was only when I downgraded from PHP 8.0.2 to 7.4.15 that I could no longer replicate the issue. So I guess something odd Kahlan is doing to load the PHP files worked fine in previous versions of PHP, but PHP 8 doesn't like it. Or something like that.

I think now I have enough to go on to raise an issue with the Kahlan bods, which I shall do now… done: 370.

All that kinda killed my Sunday afternoon (and now Monday morning writing this article). Hopefully I can get some actual work done this evening though…

Status update on this

The bod from the Kahlan project (specifically Simon Jaillet) jumped on this straight away. In the mean time I looked at what was going on in that vendor/kahlan/kahlan/src/Jit/ClassLoader.php file I mentioned earlier, which was the last touch point in Kahlan before the error. Part of that loading process was to take the original files, monkey-patch the hell out of them to make mocking easier, and then load the monkey-patched version of the file instead of the original one. It also cached the monkey-patched version so it only had to do that exercise once. When it was monkey-patching the erroring files on PHP8, it was ending up with this sort of thing. Original code:

// Step through the task queue which may add additional requests.
P\Utils::queue()->run();

And patching it to be this:

// Step through the task queue which may add additional requests.($__KMONKEY__396__?$__KMONKEY__396__:
$__KMONKEY__396::queue())->run();

Note how the first part of the patched code is stuck on the comment line, not the code line. It should be like this:

// Step through the task queue which may add additional requests.
($__KMONKEY__396__?$__KMONKEY__396__:$__KMONKEY__396::queue())->run();

And when I ran Kahlan on PHP 7.4, it was patching it correctly (as per the last block, above).

The bloke from Kahlan located the issue, which is down to a change in how PHP8 tokenises comments (see "Backward Incompatible Changes > Tokenizer":

T_COMMENT tokens will no longer include a trailing newline. The newline will instead be part of a following T_WHITESPACE token.

That explains that.

I raised the issue at 13:27, and it was fixed by 15:16. I've updated my Kahlan version, and it does indeed sort-out the issue. Brilliant work!

Righto.

--
Adam

Friday 19 February 2021

Part 12: unit testing Vue.js components

G'day

OKOK, another article in that bloody series I've been doing. Same caveats as usual: go have a breeze through the earlier articles if you feel so inclined. That said this is reasonably stand-alone.

  1. Intro / Nginx
  2. PHP
  3. PHPUnit
  4. Tweaks I made to my Bash environment in my Docker containers
  5. MariaDB
  6. Installing Symfony
  7. Using Symfony
  8. Testing a simple web page built with Vue.js using Mocha, Chai and Puppeteer
  9. I mess up how I configure my Docker containers
  10. An article about moving files and changing configuration
  11. Setting up a Vue.js project and integrating some existing code into it
  12. Unit testing Vue.js components (this article)

This is really very frustrating. You might recall I ended my previous article with this:

Now I just need to work out how to implement a test of just the GreetingMessage.vue component discretely, as opposed to the way I'm doing it now: curling a page it's within and checking the page's content. […]

Excuse me whilst I do some reading.

[Adam runs npm install of many combinations of NPM libs]

[Adam downgrades his version of Vue.js and does the npm install crap some more]

OK screw that. Seems to me - on initial examination - that getting all the libs together to make stand-alone components testable is going to take me longer to work out than I have patience for. I'll do it later. […] Sigh.

I have returned to this situation, and have got everything working fine. Along the way I had an issue with webpack that I eventually worked around, but once I circled back to replicate the work and write this article, I could no longer replicate the issue. Even rolling back to the previous version of the application code and step-by-step repeating the steps to get to where the problem was. This is very frustrating. However other people have had similar issues in the past so I'm going to include the steps to solve the problem here, even if I have to fake the issue to get error messages, etc.

Right so I'm back with the Vue.js docs regarding testing: "Vue.JS > Testing > Component Testing". The docs are incredibly content-lite, and pretty much just fob you off onto other people to work out what to do. Not that cool, and kinda suggests Vue.js considers the notion of unit testing pretty superficially. I did glean I needed to install a coupla Node modules.

First, @testing-library/vue, for which I ran into a glitch immediately:

root@00e2ea0a3109:/usr/share/fullstackExercise# npm install --save-dev @testing-library/vue
npm ERR! code ERESOLVE
npm ERR! ERESOLVE unable to resolve dependency tree
npm ERR!
npm ERR! While resolving: full-stack-exercise@2.13.0
npm ERR! Found: vue@3.0.5
npm ERR! node_modules/vue
npm ERR!   vue@"^3.0.0" from the root project
npm ERR!
npm ERR! Could not resolve dependency:
npm ERR! peer vue@"^2.6.10" from @testing-library/vue@5.6.1
npm ERR! node_modules/@testing-library/vue
npm ERR!   dev @testing-library/vue@"*" from the root project
npm ERR!
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force, or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
npm ERR!
npm ERR! See /var/cache/npm/eresolve-report.txt for a full report.

npm ERR! A complete log of this run can be found in:
npm ERR!     /var/cache/npm/_logs/2021-02-19T11_29_15_901Z-debug.log
root@00e2ea0a3109:/usr/share/fullstackExercise#

The current version of @testing-library/vue doesn't work with the current version of Vue.js. Neato. Nice one Vue team. After some googling of "wtf?", I landed on an issue someone else had raised already: "Support for Vue 3 #176": I need to use the next branch (npm install --save-dev @testing-library/vue@next). This worked OK.

The other module I needed was @vue/cli-plugin-unit-mocha. That installed with no problem. This all gives me the ability to run vue-cli-service test:unit, which will run call up Mocha and run some tests. The guidance is to set this up in package.json, thus:

  "scripts": {
    "test": "mocha test/**/*Test.js",
    "serve": "vue-cli-service serve --watch",
    "build": "vue-cli-service build",
    "lint": "vue-cli-service lint",
    "test:unit": "vue-cli-service test:unit test/unit/**/*.spec.js"
  },

Then one can jus run it as npm run test:unit.

I looked at the docs for how to test a component ("Unit Testing Vue Components" - for Vue v2, but there's no equivalent page in the v3 docs), and knocked together an initial first test which would just load the component and do nothing else: just to check everything was running (frontend/test/unit/GreetingMessage.spec.js (final version) on Github):

import GreetingMessage from "../../src/gdayWorld/components/GreetingMessage";

import { shallowMount } from "@vue/test-utils";

import {expect} from "chai";

describe("Tests of GreetingMessage component", () => {
    it("should successfully load the component", () => {
        let greetingMessage = shallowMount(GreetingMessage, {propsData: {message: "G'day world"}});

        expect(true).to.be.true;
    });
});

Here's where I got to the problem I now can't replicate. When I ran this, I got something like:

root@eba0490b453d:/usr/share/fullstackExercise# npm run test:unit

> full-stack-exercise@2.13.0 test:unit
> vue-cli-service test:unit test/unit/**/*.spec.js

 WEBPACK  Compiling...

  [=========================] 98% (after emitting)

 ERROR  Failed to compile with 1 error

error  in ./node_modules/mocha/lib/cli/cli.js

Module parse failed: Unexpected character '#' (1:0)
File was processed with these loaders:
* ./node_modules/cache-loader/dist/cjs.js
* ./node_modules/babel-loader/lib/index.js
* ./node_modules/eslint-loader/index.js
You may need an additional loader to handle the result of these loaders.
> #!/usr/bin/env node
| 'use strict';
|
[…etc…]

There's a JS file that has a shell-script shebang thing at the start of it, and the Babel transpiler doesn't like that. Fair enough, but I really don't understand why it's trying to transpile stuff in the node_modules directory, but at this point in time, I just thought "Hey-ho, it knows what it's doing so I'll take its word for it".

Googling about I found a bunch of other people having a similar issue with Webpack compilation, and the solution seemed to be to use a shebang-loader in the compilation process (see "How to keep my shebang in place using webpack?", "How to Configure Webpack with Shebang Loader to Ignore Hashbang…", "Webpack report an error about Unexpected character '#'"). All the guidance for this solution was oriented aroud sticking stuff in the webpack.config.js file, but of course Vue.js hides that away from you, and you need to do things in a special Vue.js way, but adding stuff with a special syntax to the vue.config.js file. The docs for this are at "Vue.js > Working with Webpack". The docs there showed how to do it using chainWebpack, but I never actually got this approach to actually solve the problem, so I mention this only because it's "something I tried".

From there I starting thinking, "no, seriously why is it trying to transpile stuff in the node_modules directory?" This does not seem right. I changed my googling tactic to try to find out what was going on there, and came across "Webpack not excluding node_modules", and that let me to update my vue.config.js file to actively exclude node_modules (copied from that answer):

var nodeExternals = require('webpack-node-externals');
...
module.exports = {
    ...
    target: 'node', // in order to ignore built-in modules like path, fs, etc. 
    externals: [nodeExternals()], // in order to ignore all modules in node_modules folder 
    ...
};

And that worked. Now when I ran the test I have made progress:

root@eba0490b453d:/usr/share/fullstackExercise# npm run test:unit

> full-stack-exercise@2.13.0 test:unit
> vue-cli-service test:unit test/unit/**/*.spec.js

 WEBPACK   Compiling...

  [=========================] 98% (after emitting)

 DONE   Compiled successfully in 1161ms

  [=========================] 100% (completed)

WEBPACK  Compiled successfully in 1161ms

MOCHA  Testing...



  Tests of GreetingMessage component
    ✓ should successfully load the component


  1 passing (17ms)

MOCHA  Tests completed successfully

root@eba0490b453d:/usr/share/fullstackExercise#

From there I rounded out the tests properly (frontend/test/unit/GreetingMessage.spec.js):

import GreetingMessage from "../../src/gdayWorld/components/GreetingMessage";

import { shallowMount } from "@vue/test-utils";

import {expect} from "chai";

describe("Tests of GreetingMessage component", () => {

    let greetingMessage;
    let expectedText = "TEST_MESSAGE";

    before("Load test GreetingMessage component", () => {
        greetingMessage = shallowMount(GreetingMessage, {propsData: {message: expectedText}});
    });

    it("should return the correct heading", () => {
        let heading = greetingMessage.find("h1");
        expect(heading.exists()).to.be.true;

        let headingText = heading.text();
        expect(headingText).to.equal(expectedText);
    });

    it("should return the correct content", () => {
        let contentParagraph = greetingMessage.find("h1+p");
        expect(contentParagraph.exists()).to.be.true;

        let contentParagraphText = contentParagraph.text();
        expect(contentParagraphText).to.equal(expectedText);
    });
});

Oh! Just a reminder of what the component is (frontend/src/gdayWorld/components/GreetingMessage.vue)! Very simple stuff, as the tests indicate:

<template>
    <h1>{{ message }}</h1>
    <p>{{ message }}</p>
</template>

<script>
export default {
  name: 'GreetingMessage',
  props : {
    message : {
      type: String,
      required: true
    }
  }
}
</script>

One thing I found was that every time I touched the test file, I was getting this compilation error:

> full-stack-exercise@2.13.0 test:unit
> vue-cli-service test:unit test/unit/**/*.spec.js

WEBPACK  Compiling...

  [=========================] 98% (after emitting)

ERROR  Failed to compile with 1 error

error  in ./test/unit/GreetingMessage.spec.js

Module Error (from ./node_modules/eslint-loader/index.js):

/usr/share/fullstackExercise/test/unit/GreetingMessage.spec.js
   7:1  error  'describe' is not defined  no-undef
  12:5  error  'before' is not defined    no-undef
  16:5  error  'it' is not defined        no-undef
  24:5  error  'it' is not defined        no-undef

error4 problems (4 errors, 0 warnings)

But if I ran it again, the problem went away. Somehow ESLint was getting confused by things; it only lints things when they've changed, and on the second - and subsequent - runs it doesn't run so the problem doesn't occur. More googling, and I found this: "JavaScript Standard Style does not recognize Mocha". The guidance here is to let the linter know I'm running Mocha, with the inference that there will be some global functions it can just assume are legit. This is just an entry in package.json:

  "eslintConfig": {
    "root": true,
    "env": {
      "node": true,
      "mocha": true
    },
    // etc

Having done that, everything works perfectly, and despite the fact that is a very very simple unit test… I'm quite pleased with myself that I got it running OK.

After sorting it all out, I reverted everything in source control back to how it was before I started the exercise, so as to replicate it and write it up here. This is when I was completely unable to reproduce that shebang issue at all. I cannot - for the life of me - work out why not. Weird. Anyway, I decided to not waste too much time trying to reproduce a bug I had solved, and just decided to mention it here as a possible "thing" that could happen, but otherwise move on with my life.

I have now got to where I wanted to get at the beginning of this series of articles, so I am gonna stop now. My next task with Vue.js testing is to work out how to mock things, because my next task will require me to make remote calls and/or hit a DB. And I don't want to be doing that when I'm running tests. But that was never an aim of this article series, so I'll revert to stand-alone articles now.

Righto.

--
Adam

Monday 15 February 2021

Part 11: setting up a Vue.js project and integrating some existing code into it

G'day:

OK so maybe this will be the last article in this series. But given what a monster it's become: who knows. As a recap, here are links to the earlier articles:

  1. Intro / Nginx
  2. PHP
  3. PHPUnit
  4. Tweaks I made to my Bash environment in my Docker containers
  5. MariaDB
  6. Installing Symfony
  7. Using Symfony
  8. Testing a simple web page built with Vue.js using Mocha, Chai and Puppeteer
  9. I mess up how I configure my Docker containers
  10. An article about moving files and changing configuration
  11. Setting up a Vue.js project and integrating some existing code into it (this article)
  12. Unit testing Vue.js components

It's up to you whether you read the rest of that lot, or just skim it, or whatever. Looking at the source code might help. This is where it's at as I am writing this sentence: Fullstack Exercise v 2.10. That has already got some of the code in it that is "new" to this article. Any other code links in here I'll link to the final version of the work. I think so far I've done the Vue.js project install, and reconfigured it and Nginx so that I'm using Nginx as the front-end web server, not the one that ships with the Vue.js project. Other than that the Vue code is just the "hello world" stuff the project starts with.

OK so the object of this exercise is to take my gdayWorldViaVue.html page which is is built with its Vue template embedded in just a JS file. I discuss the creation of this work in the article "Part 8: Testing a simple web page built with Vue.js using Mocha, Chai and Puppeteer". Here's the code:

frontend/public/gdayWorldViaVue.html:

<!doctype html>

<html lang="en">
<head>
    <meta charset="utf-8">

    <title id="title">{{ message }}</title>
</head>

<body>
<div id="app">
    <greeting :message="message"></greeting>
</div>

<script src="https://cdn.jsdelivr.net/npm/vue@2/dist/vue.js"></script>
<script src="assets/scripts/gdayWorldViaVue.js"></script>
</body>
</html>

And frontend/public/assets/scripts/gdayWorldViaVue.js:

Vue.component('greeting', {
    props : {
        message : {
            type: String,
            required: true
        }
    },
    template : `
    <div>
        <h1>{{ message }}</h1>
        <p>{{ message }}</p>
    </div>
    `
});


let appData = {message: "G'day world via Vue"};
new Vue({el: '#title', data: appData});
new Vue({el: '#app', data: appData});

Oh, an for the sake of completeness, the output:



Ah, and of course there's a test (frontend/test/functional/GdayWorldViaVueTest.js):

let puppeteer = require('puppeteer');

let chai = require("chai");
let chaiAsPromised = require("chai-as-promised");
chai.use(chaiAsPromised);
let should = chai.should();

describe("Baseline test of vue.js working", function () {
    let browser;
    let page;

    const expectedText = "G'day world via Vue";

    before("Load the test document", async function () {
        this.timeout(5000);

        browser = await puppeteer.launch( {args: ["--no-sandbox"]});
        page = await browser.newPage();

        await page.goto("http://fullstackexercise.frontend/gdayWorldViaVue.html");
    });

    after("Close down the browser", async function () {
        await page.close();
        await browser.close();
    });

    it("should return the correct page title", async function () {
        await page.title().should.eventually.equal(expectedText);
    });

    it("should return the correct page heading", async function () {
        let headingText = await page.$eval("h1", headingElement => headingElement.innerText);

        headingText.should.equal(expectedText);
    });

    it("should return the correct page content", async function () {
        let paragraphContent = await page.$eval("p", paragraphElement => paragraphElement.innerText);

        paragraphContent.should.equal(expectedText);
    });
});

Output:

root@18a88721ed2a:/usr/share/fullstackExercise# npm test

> full-stack-exercise@2.6.0 test
> mocha test/**/*.js



  Baseline test of vue.js working
     should return the correct page title
     should return the correct page heading
     should return the correct page content


  3 passing (535ms)

root@18a88721ed2a:/usr/share/fullstackExercise#

Having those test cases there are gold, because it means all this work is basically a refactoring exercise (we've done the red / green bit in the earlier article), so the object of this exercise is to separate-out the template from the JS file and into a .vue file, and know the work is done because those test cases still pass.

Right, so I have hit the Vue.JS website, and done some reading, and to use .vue files, I need to create a Vue project. First I just want to see what the project-creation does, to check if it'll be easier to integrate my work into the project, or the project into the work: you might remember a similar drama when I was installing Symphony ("Part 6: Installing Symfony"). Once-bitten, twice-shy, I'm just gonna see what Vue thinks it's doing first. I'm gonna create this project in /tmp.

Oh Adam… back-up! First of all I need to install Vue CLI. This is the app that deals with project creation and stuff like that. This is easy, I just integrate it into the node/Dockerfile:

FROM node
RUN apt-get update \
    && apt-get install -y wget gnupg \
    && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
    && apt-get update \
    && apt-get install -y google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf libxss1 \
      --no-install-recommends \
    && rm -rf /var/lib/apt/lists/*
WORKDIR  /usr/share/fullstackExercise/
COPY config/* ./
RUN npm install -g @vue/cli
RUN npm install
EXPOSE 8080

Just to be clear, the first npm install there is installing Vue CLI globally, the second one is installing the modules necessary for my app. Incidentally, it was getting this stuff working that was the inspiration for two of my recent articles: "Part 9: I mess up how I configure my Docker containers" and "Part 10: An article about moving files and changing configuration". But it's all working now.

Now I've got Vue CLI installed, I'll install a project:

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker exec --interactive --tty fullstackexercise_node_1 /bin/bash
root@18a88721ed2a:/usr/share/fullstackExercise# cd /tmp
root@18a88721ed2a:/tmp# ll
total 20
drwxrwxrwt 1 root root  4096 Feb 12 12:18 ./
drwxr-xr-x 1 root root  4096 Feb 10 16:43 ../
drwx------ 2 root root  4096 Feb  6 19:32 apt-key-gpghome.b8FchIBzPr/
-rw-r--r-- 1 root staff  543 Feb  9 12:36 core-js-banners
drwxr-xr-x 3 root root  4096 Feb  6 03:03 v8-compile-cache-0/
root@18a88721ed2a:/tmp# vue create hello-world


Vue CLI v4.5.11
? Please pick a preset:
  Default ([Vue 2] babel, eslint)
> Default (Vue 3 Preview) ([Vue 3] babel, eslint)
  Manually select features


? Please pick a preset: Default ([Vue 2] babel, eslint)
? Pick the package manager to use when installing dependencies:
  Use Yarn
> Use NPM


✨  Creating project in /tmp/hello-world.
🗃  Initializing git repository...
⚙️  Installing CLI plugins. This might take a while...


added 1269 packages, and audited 1270 packages in 57s

62 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
🚀  Invoking generators...
📦  Installing additional dependencies...


added 71 packages, and audited 1341 packages in 6s

68 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
⚓  Running completion hooks...

📄  Generating README.md...

🎉  Successfully created project hello-world.
👉  Get started with the following commands:

$ cd hello-world
$ npm run serve

 WARN  Skipped git commit due to missing username and email in git config, or failed to sign commit.
You will need to perform the initial commit yourself.


root@18a88721ed2a:/tmp#

Lovely. Let's do this run-serve thing:

root@18a88721ed2a:/tmp# cd hello-world/
root@18a88721ed2a:/tmp/hello-world# npm run serve

> hello-world@0.1.0 serve
> vue-cli-service serve

 INFO  Starting development server...
98% after emitting CopyPlugin

 DONE  Compiled successfully in 2221ms                                                                        1:27:33 PM

  App running at:
  - Local:   http://localhost:8081/

  It seems you are running Vue CLI inside a container.
  Access the dev server via http://localhost:<your container's external mapped port>/

  Note that the development build is not optimized.
  To create a production build, run npm run build.

This won't work because I need to poke a hole through from the container to the host machine on 8081, but I'll quickly do this (I'll spare you the detail). Gimme a minute.


Cool. My first challenge is that I've already got a web server, and I want to use it. I don't want to be using Vue's stub web server. Now for dev there's no real need for this: Vue's web server would be fine, but I've got Nginx there, so I'm going to use it. Also I don't want to use localhost to access it. I figured I needed to configure a proxy_pass "thing" (sorry for technical buzzword there), but I had no idea how to do it. I'm a noob with Nginx. Anyhow, I googled about and found this article "VueJS dev serve with reverse proxy" by someone called Marko Mitranić . I didn't understand all the settings it was suggesting, but I grabbed them and it all works (docker/nginx/sites/frontend.conf):

server {
    listen 80;
    listen [::]:80;

    server_name fullstackexercise.frontend;
    root /usr/share/nginx/html/frontend;
    index index.html;

    location / {
        #autoindex on;
        #try_files $uri $uri/ =404;

        # from https://medium.com/homullus/vuejs-dev-serve-with-reverse-proxy-cdc3c9756aeb
        proxy_pass  http://vuejs.backend:8080/;
        proxy_set_header Host vuejs.backend;
        proxy_set_header Origin vuejs.backend;
        proxy_hide_header Access-Control-Allow-Origin;
        add_header Access-Control-Allow-Origin "http://fullstackexercise.frontend";
    }

    # from https://medium.com/homullus/vuejs-dev-serve-with-reverse-proxy-cdc3c9756aeb
    location /sockjs-node/ {
        proxy_pass http://vuejs.backend:8080;
        proxy_redirect off;
        proxy_set_header Host vuejs.backend;
        proxy_set_header Origin vuejs.backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_hide_header Access-Control-Allow-Origin;
        add_header Access-Control-Allow-Origin "http://fullstackexercise.frontend";
    }

    location ~ /\.ht {
        deny all;
    }
}

The first block is for the website (on http://fullstackexercise.frontend); the second block is for the listener for changes in the source code. Vue does some magic with web sockets and ping up to the browser to reload any code changes that are back on the server. This is quite cool. Oh I also needed to put some settings in docker/node/config/vue.config.js:

module.exports = {
    devServer: {
        host: "vuejs.backend",
        disableHostCheck: false,
        port: 8080,
        watchOptions : {
            ignored: /node_modules/,
            poll: 1000
        }
    }
}

But I'm getting ahead of myself here. The Nginx stuff is for the fullstackExercise websites. I'm still with this one in /tmp/ directory. All I want to check is what it installs, which is this lot:

root@61c6341dc75c:/tmp/hello-world# tree -aF --dirsfirst -L 3 .
.
|-- node_modules/
|   `-- [… seemingly millions of files elided…]
|-- public/
|   |-- favicon.ico
|   `-- index.html
|-- src/
|   |-- assets/
|   |   `-- logo.png
|   |-- components/
|   |   `-- HelloWorld.vue
|   |-- App.vue
|   `-- main.js
|-- .gitignore
|-- README.md
|-- babel.config.js
|-- package-lock.json
`-- package.json

1760 directories, 5161 files

root@61c6341dc75c:/tmp/hello-world#

OK so it seems to be just a bunch of NPM libs and stuff in package.json:

root@61c6341dc75c:/tmp/hello-world# cat package.json
{
  "name": "hello-world",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "serve": "vue-cli-service serve",
    "build": "vue-cli-service build",
    "lint": "vue-cli-service lint"
  },
  "dependencies": {
    "core-js": "^3.6.5",
    "vue": "^3.0.0"
  },
  "devDependencies": {
    "@vue/cli-plugin-babel": "~4.5.0",
    "@vue/cli-plugin-eslint": "~4.5.0",
    "@vue/cli-service": "~4.5.0",
    "@vue/compiler-sfc": "^3.0.0",
    "babel-eslint": "^10.1.0",
    "eslint": "^6.7.2",
    "eslint-plugin-vue": "^7.0.0-0"
  },
  "eslintConfig": {
    "root": true,
    "env": {
      "node": true
    },
    "extends": [
      "plugin:vue/vue3-essential",
      "eslint:recommended"
    ],
    "parserOptions": {
      "parser": "babel-eslint"
    },
    "rules": {}
  },
  "browserslist": [
    "> 1%",
    "last 2 versions",
    "not dead"
  ]
}
root@61c6341dc75c:/tmp/hello-world#

All of which will merge into my existing package.json just fine. And some source code files and assets. I took a punt and just integrated all of that into my app, stuck CMD ["npm", "run", "serve"] in my docker/node/Dockerfile, and rebuilt my containers…

Cool. Plus I re-ran all my own tests on the other code in the app, and everything is still currently green. Also I can change my source code on the server, and almost instantly it's displayed in the browser without a reload:

Right. So now I have to try to integrate this gdayWorldViaVue.html page into my Vue app. It took a while to work out how to do this: I'm sure it's in the Vue.js docs somewhere, but I couldn't find it. Ultimately I found a Q&A on Stack Overflow that explained it: "multiple pages in Vue.js CLI" (and a handy Github repo with example code in it). Basically Vue CLI assumes everyone wants a single-page app, and doesn't really expose how to add additional pages into this SPA. But it's just a matter of defining the pages in vue.config.js (and I have the doc reference now: Configuration Reference > Pages. First I just shifted the project's own index page into a page definition:

module.exports = {
    pages : {
        index: {
            entry: "src/index/main.js",
            template: "public/index.html",
            filename: "index.html"
        }
    },
    devServer: {
        host: "vuejs.backend",
        disableHostCheck: false,
        port: 8080,
        watchOptions : {
            ignored: /node_modules/,
            poll: 1000
        }
    }
}

This also necessitated moving main.js, components/HelloWorld.vue and App.vue from the base of the src/ directory structure, into the src/index/ subdirectory, as well as changing a relative reference to ./assets/logo.png to ../assets/logo.png, given it's being referenced from a file in that index/ subdirectory now. I rebuilt the container, and restarted it, and the Hello World page still worked. Now to add the gdayWorldViaVue.html page into it.

This was pretty easy, and was just a matter of moving some stuff around in the file system and within files. Previously we had a single HTML file, gdayWorldViaVue.html, and a single JS file, gdayWorldViaVue.js, as per above in this article. First I implemented the page mapping for this page in vue.config.js, as per the index example above:

        gdayWorld: {
            entry: "src/gdayWorld/main.js",
            template: "public/gdayWorldViaVue.html",
            filename: "gdayWorldViaVue.html"
        }

And then I need to distribute the contents of the existing .html and .js file to their more Vue / page / component-centric equivalents. src/gdayWorld/main.js is identical to the one for the index page:

import { createApp } from 'vue'
import App from './App.vue'

createApp(App).mount('#app')

frontend/public/gdayWorldViaVue.html has had the reference to the JS files in the foot of the <body> removed as the application now handles those, it also has the body of the #app <div/> removed as this is handled by the App.vue file now, and also has placeholder text put in for the page title:

<!doctype html>

<html lang="en">
<head>
    <meta charset="utf-8">

    <title id="title">PAGE-TITLE</title>
</head>

<body>
<div id="app"></div>
</body>
</html>

I was handling the page title in quite an unorthodox fashion before, and I'm going to fix this now.

frontend/src/gdayWorld/App.vue defines what goes in the #app <div/>, and loads in the GreetingMessage component:

<template>
    <greeting-message :message="message"></greeting-message>
</template>

<script>
import GreetingMessage from './components/GreetingMessage.vue'
export default {
    name: 'App',
    components: {
        GreetingMessage
    },
    data () {
        return {
            message: "G'day world via Vue"
        };
    },
    created () {
        // document.title = this.message;
    }
}
</script>

Note it will also sets the document.title value now, from the app data. It's commented-out for now because I want to see the test for this fail initially, so I am sure what we're seeing is the refactored implementation.

And frontend/src/gdayWorld/components/GreetingMessage.vue defines the behaviour of itself. So this file is a stand-alone component now, which is what we were aiming to do all along.

<template>
  <div>
    <h1>{{ message }}</h1>
    <p>{{ message }}</p>
  </div>
</template>

<script>
export default {
  name: 'GreetingMessage',
  props : {
    message : {
      type: String,
      required: true
    }
  }
}
</script>

I also deleted frontend/public/assets/scripts/gdayWorldViaVue.js as it's now surplus to requirement. Because we've changed the Vue app's config, and that is copied over in Dockerfile, I need to rebuild the containers to see this "change". Once that's done I re-run my tests:

root@ebd7dc7b4a3d:/usr/share/fullstackExercise# npm test

> full-stack-exercise@2.6.0 test
> mocha test/**/*.js



  Baseline test of vue.js working
    1) should return the correct page title
     should return the correct page heading
     should return the correct page content


  2 passing (319ms)
  1 failing

  1) Baseline test of vue.js working
       should return the correct page title:

      AssertionError: expected 'PAGE-TITLE' to equal 'G\'day world via Vue'
      + expected - actual

      -PAGE-TITLE
      +G'day world via Vue

      at /usr/share/fullstackExercise/node_modules/chai-as-promised/lib/chai-as-promised.js:302:22
      at processTicksAndRejections (node:internal/process/task_queues:94:5)
      at async Context.<anonymous> (test/functional/GdayWorldViaVueTest.js:29:9)



npm ERR! code 1
npm ERR! path /usr/share/fullstackExercise
npm ERR! command failed
npm ERR! command sh -c mocha test/**/*.js

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2021-02-12T17_13_50_305Z-debug.log
root@ebd7dc7b4a3d:/usr/share/fullstackExercise#

Woohoo! Two test are passing, and the expected one is failing. If I now uncomment that line above (and as soon as I do that my page in the browser now gets its title too btw), the test is now passing:

  Baseline test of vue.js working
     should return the correct page title
     should return the correct page heading
     should return the correct page content

Similarly if I go and change the message in frontend/src/gdayWorld/App.vue the tests fail appropriately, so I'm happy the tests are testing the new implementation.


Now I just need to work out how to implement a test of just the GreetingMessage.vue component discretely, as opposed to the way I'm doing it now: curling a page it's within and checking the page's content. This is fine for an end-to-end test, but not so good for a unit test. TBH in this simple case the current approach is actually fine, but I want to know how to test components.

Excuse me whilst I do some reading.

[Adam runs npm install of many combinations of NPM libs]

[Adam downgrades his version of Vue.js and does the npm install crap some more]

OK screw that. Seems to me - on initial examination - that getting all the libs together to make stand-alone components testable is going to take me longer to work out than I have patience for. I'll do it later. So. I'm gonna leave this article here at the "and lo, I have .vue-file-based Vue components working now, and kinda understand how that side of things comes together, but - nor now at least - I'm gonna do my front-end testing via HTTP requests of the whole document, not via each component. For what I need to do right now this is fine anyhow, I think. Sigh.

Righto.

--
Adam