Showing posts with label Docker. Show all posts
Showing posts with label Docker. Show all posts

Tuesday 9 February 2021

Part 10: An article about moving files and changing configuration

G'day:

I hope I correctly set the excitement expectations with the title of this one. It's gonna be dead dull. In the previous article ("I mess up how I configure my Docker containers"), I detailed a fundamental flaw in how I was configuring my Dockerfiles and docker-compose.yml file, which pretty much had a logic-conflict in them. Instead of using the fullstack-exercise codebase I've been working on in this series, I used a cut down one that focused specifically on the issue. In this article I am detailing the file-system and config reorganisation I then performed on the fullstack-exercise codebase to fix the issue. TBH I'd probably not bother reading it if I was you (my fictitious reader), cos it's even more dry than my usual efforts. I'm pretty much only writing it out of a sense of completeistness (!), and also in case someone happens to be reading along with the rest of the series and - if they came to the next article - suddenly thought "hang on all the files have moved around? What subterfuge is this?". So it's a full-disclosure exercise I guess. If you do insist on reading this, read the previous article first though, eh? Good luck.

As per usual: I'll remind you that this is part 10 of an 11(?) part series, with the earlier articles linked below:

  1. Intro / Nginx
  2. PHP
  3. PHPUnit
  4. Tweaks I made to my Bash environment in my Docker containers
  5. MariaDB
  6. Installing Symfony
  7. Using Symfony
  8. Testing a simple web page built with Vue.js using Mocha, Chai and Puppeteer
  9. I mess up how I configure my Docker containers
  10. An article about moving files and changing configuration (this article)
  11. Setting up a Vue.js project and integrating some existing code into it
  12. Unit testing Vue.js components

The TL;DR of the previous article is kinda:

Don't map volumes in docker-compose.yml over the top of the working directory specified in Dockerfile, if the Dockerfile actually creates stuff you need in that working directory (like a node_modules subdirectory, for example). This is because a volume mapping replaces what's there, it does not merge with it.

Schoolkid dumbarsery from me there.

Now I'm gonna apply the lessons learned there to the main codeabse for this project. This is the directory structure I had previously (on github). Note some stuff not relevant to this exercise has been removed:

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise$ tree -F --dirsfirst -L 2
.
├── bin/
│   └── console*
├── config/
│   └── [… Symfony stuff …]
├── docker/ [… subdirectory contents elided for brevity …]
│   ├── mariadb/
│   ├── nginx/
│   ├── node/
│   ├── php-fpm/
│   └── docker-compose.yml*
├── public/
│   ├── button.html*
│   ├── gdayWorld.html*
│   ├── gdayWorld.php*
│   ├── gdayWorldViaVue.html*
│   ├── githubProfiles.html*
│   ├── index.php*
│   ├── invalidNotificationType.html*
│   └── notification.html*
├── src/
│   ├── Kernel.php*
│   └── MyClass.php*
├── tests/
│   ├── functional/
│   │   ├── public/
│   │   │   ├── ButtonTest.js*
│   │   │   ├── GdayWorldViaVueTest.js*
│   │   │   ├── GithubProfilesTest.js*
│   │   │   ├── NotificationTest.js*
│   │   │   ├── PhpTest.php*
│   │   │   └── WebServerTest.php*
│   │   └── SymfonyTest.php*
│   ├── integration/
│   │   └── DatabaseTest.php*
│   └── unit/
│       └── MyClassTest.php*
├── LICENSE*
├── README.md*
├── composer.json*
├── composer.lock*
├── package-lock.json*
├── package.json*
├── phpcs.xml.dist*
├── phpmd.xml*
├── phpunit.xml.dist*
└── symfony.lock*

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise$

The two points that make it most obvious that things are poorly-organised here are:

  • the root directory which has a mix of stuff intended for the PHP container and other stuff intended for the Node.js container.
  • And, similarly the tests subdirectory has a mix of back-end PHPUnit tests and front-end Mocha tests in the same substructure.

Where there's a mess or a mix of things intended for two different purposes in the same place, it's a flag that something's possibly not right. Now I will be honest and say that this decision originally was purposeful on my part. I'm not distinguishing between the front-end part of the app (Node.js, Vue and Mocha stuff), and the back-end running Symfony and PHP. The front-end stuff is the web site for this app; the back-end will be the web service to support the front-end. They are not two distinct apps in my view (or one way of looking at it). This is not to say they can't still be organised a bit more coherently than I have.

To separate my concerns, I've decided to move all the code-related stuff into one of backend or frontend subdirectories. First the backend directory (see backend on Github):

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise$ tree -F --dirsfirst
.
└── backend/
   ├── config/
   │   ├── packages/
   │   │   ├── prod/
   │   │   │   └── routing.yaml*
   │   │   ├── test/
   │   │   │   └── framework.yaml*
   │   │   ├── cache.yaml*
   │   │   ├── framework.yaml*
   │   │   └── routing.yaml*
   │   ├── routes/
   │   │   └── dev/
   │   │       └── framework.yaml*
   │   ├── bundles.php*
   │   ├── preload.php*
   │   ├── routes.yaml*
   │   └── services.yaml*
   ├── public/
   │   ├── test-coverage-report/
   │   ├── gdayWorld.html*
   │   ├── gdayWorld.php*
   │   └── index.php*
   ├── src/
   │   ├── Controller/
   │   │   └── GreetingsController.php*
   │   ├── Kernel.php*
   │   └── MyClass.php*
   └── tests/
       ├── functional/
       │   ├── public/
       │   │   ├── PhpTest.php*
       │   │   └── WebServerTest.php*
       │   └── SymfonyTest.php*
       ├── integration/
       │   └── DatabaseTest.php*
       ├── unit/
       │   └── MyClassTest.php*
       └── bootstrap.php*

In the backend subdirectory I have all the PHP / Symfony / PHPUnit stuff, plus a public directory that is purely for the back-end web root. And - below - the Docker Nginx config now has separate websites for back-end and front-end; and in the php-fpm section we now have all the PHP / Symfony config stuff moved out of the application root, and into its own specific root:

└── docker/
    ├── nginx/
    │   ├── sites/
    │   │   ├── backend.conf*
    │   │   └── frontend.conf*
    │   └── Dockerfile*
    ├── php-fpm/
    │   ├── app_root/
    │   │   ├── bin/
    │   │   │   └── console*
    │   │   ├── var/
    │   │   │   └── cache/
    │   │   ├── composer.json*
    │   │   ├── composer.lock*
    │   │   ├── phpcs.xml.dist*
    │   │   ├── phpmd.xml*
    │   │   ├── phpunit.xml.dist*
    │   │   └── symfony.lock*
    │   ├── root_home/
    │   ├── Dockerfile*
    │   └── phpunit-code-coverage-xdebug.ini*
    └── docker-compose.yml*

The key part of the Nginx configuration changes here is that the two sites now have distinct host names: fullstackexercise.backend (see backend.conf on Github) for the PHP-oriented stuff, and fullstackexercise.frontend (see frontend.config on Github) for the Vue- / Node-based stuff. Each website only serves the type of files appropriate for their purpose.

The Nginx Dockerfile (on Github) has not changed significantly, but now the PHP-FPM one (on Github) copies all the application-root stuff into the working directory, rather than docker-compose.yml file using a volume to do this:

WORKDIR  /usr/share/fullstackExercise/
COPY ./app_root/ /usr/share/fullstackExercise/

It's worth looking at the whole lot of the service definitions for these from docker-compose.yml:

services:
  nginx:
    build:
      context: ./nginx
    volumes:
      - ../frontend/public:/usr/share/nginx/html/frontend
      - ../backend/public:/usr/share/nginx/html/backend
      - ../log:/var/log
      - ./nginx/root_home:/root
    ports:
      - "80:80"
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    networks:
      backend:
        aliases:
          - fullstackexercise.frontend
          - fullstackexercise.backend

  php-fpm:
    build:
      context: ./php-fpm
    environment:
      - DATABASE_ROOT_PASSWORD=${DATABASE_ROOT_PASSWORD}
    volumes:
      - ../backend/config:/usr/share/fullstackExercise/config
      - ../backend/public:/usr/share/fullstackExercise/public
      - ../backend/src:/usr/share/fullstackExercise/src
      - ../backend/tests:/usr/share/fullstackExercise/tests
      - ./php-fpm/root_home:/root
    stdin_open: true
    tty: true
    networks:
      - backend

For Nginx we are mapping-in two separate volumes into the html directory: as per above, one for the back-end site, one for the front-end site. These are then used as the webroots in the site configuration for each website. We are also setting an alias for each website. This is just so the other containers can access the websites too.

In the PHP block, we now have separate volumes for each of the code directories in the application route (note that the config sub-directory there is Symfony app config, not like the composer.json, phpunit.xml.dist etc stuff that has been copied to the application root by (spoilers) php-fpm/Dockerfile. And, yeah, now the Dockerfile (on Github) for the PHP stuff. The only significant line is this one:

COPY ./app_root/ /usr/share/fullstackExercise/

That copies all the config files the PHP components need to run into the application root. One downside of this is that I can't make on-the-fly changes to things like the PHPUnit config from within PHPStorm, I need to use vi in the container, test it, then copy it back to the host machine. But that stuff changes so seldom it's fine by me.

The changes on the front-end side of things is along the same lines. Here's the file structure (and on Github):

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise$ tree -F --dirsfirst
. # a lot of stuff has been removed for the sake of brevity
├── docker/
│   └── node/
│       ├── config/
│       │   ├── babel.config.js*
│       │   ├── package-lock.json*
│       │   ├── package.json*
│       │   └── vue.config.js*
│       └── Dockerfile*
└── frontend/
    ├── public/
    │   ├── assets/
    │   │   └── scripts/
    │   │       ├── button.js*
    │   │       ├── gdayWorldViaVue.js*
    │   │       ├── githubProfiles.js*
    │   │       └── notification.js*
    │   ├── button.html*
    │   ├── gdayWorld.html*
    │   ├── gdayWorldViaVue.html*
    │   ├── githubProfiles.html*
    │   ├── invalidNotificationType.html*
    │   └── notification.html*
    ├── src/
    └── test/
        └── functional/
            ├── ButtonTest.js*
            ├── GdayWorldViaVueTest.js*
            ├── GithubProfilesTest.js*
            └── NotificationTest.js*

And the relevant bit of the node/Dockerfile (on Github):

WORKDIR  /usr/share/fullstackExercise/
COPY config/* ./

And docker-compose.yml (on Github):

  node:
    build:
      context: ./node
    environment:
      - GITHUB_PERSONAL_ACCESS_TOKEN=${GITHUB_PERSONAL_ACCESS_TOKEN}
    volumes:
      - ../frontend/public:/usr/share/fullstackExercise/public
      - ../frontend/src:/usr/share/fullstackExercise/src
      - ../frontend/test:/usr/share/fullstackExercise/test
      - ./node/root_home:/root
    stdin_open: true
    tty: true
    networks:
      backend:
        aliases:
          - vuejs.backend

Here we see how - same as with the PHP stuff - we copy the config files over in Dockerfile, and then map volumes for the code directories in docker-compose.yml.

That's pretty much it really. The good thing with all this is that because I have full test coverage of my code, and some functional and integration tests as well, I have testing for all the config and all the interactions between all the containers, so at any moment in time when I go to refactor something - because all this really is is an exercise in refactoring - at every step I can check that everything still works. Or spend time working out why something didn't work. But that safety net is always there.

OK. I promise the next article is actually gonna get around to looking at Vue.js components, testing thereof, and hopefully draw a line under this series. BTW if you read this article all the way to here, you're bloody weird. But well done ;-)

Righto.

--
Adam

Saturday 6 February 2021

Part 9: I mess up how I configure my Docker containers

G'day:

This is an article I am inserting into my ongoing series of setting up a web app using Docker containers, running Nginx, PHP (and Symfony within that), MariaDB, with a Vue.js front end being built in a Node.js container. The other articles can be found here:

  1. Intro / Nginx
  2. PHP
  3. PHPUnit
  4. Tweaks I made to my Bash environment in my Docker containers
  5. MariaDB
  6. Installing Symfony
  7. Using Symfony
  8. Testing a simple web page built with Vue.js using Mocha, Chai and Puppeteer
  9. I mess up how I configure my Docker containers (this article)
  10. An article about moving files and changing configuration
  11. Setting up a Vue.js project and integrating some existing code into it
  12. Unit testing Vue.js components

A week or so ago I wrote an article "Part 8: Testing a simple web page built with Vue.js using Mocha, Chai and Puppeteer". In there I detail how I had configured my Node.js container in Dockerfile and docker-compose.yml, and how it all worked nicely. A few days ago when prepping for the last article of that series ("Refactoring the simple web page into Vue components" - not written yet, so no link yet either, sorry), I discovered my Node.js container was only working by a coincidental quirk of how I built it: and as it stood, the configuration I cited didn't actually work as I described it. I had made a rather school-boy-ish error.

I'm going to explain what I did wrong using a stand-along Node.js container / app, because it'll be a bit more clear if I just focus on the one container, not the array of moving parts I have in that other application. Then I show how I applied a working solution to the that main app.

The requirement I am fulfilling here is to set up a simple Docker container running Node.js, and run some Mocha tests for a code puzzle I have. Normally-speaking one would not have both a Dockerfile and a docker-compose.yml file for this sort of minimal requirement, but the issue I had was with how Dockerfile and docker-compose.yml interact, so I'll include both.

Firstly my Dockerfile:

FROM node
WORKDIR  /usr/share/nodeJs/

This just grabs the Node.js image from Docker, sets the directory I'm gonna be using for my code. nothing exciting. In the docker-compose.yml file I've got this lot:

version: '3'

services:
  node:
    build:
      context: node
    volumes:
      - ..:/usr/share/nodeJs
      - ./node/root_home:/root
    stdin_open: true
    tty: true

This is all straight-forward seeming. So as I can develop my source code files as test as I go, I have mapped my host's nodeJs directory to the container's working directory /usr/share/nodeJs. In addition to that I'm mapping another directory I have in my project as the container's root user's home directory. This is so I can a) avail the container of a .bashrc file I've got some aliases in; b) the .bash_history will be stored on my host machine, so it will persist between rebuilds of the container. This stuff with the root_home stuff is not important to this article.

I can build this container and run it and dive inside:

adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs/docker$ docker-compose up --build --detach
Building node
Step 1/2 : FROM node
latest: Pulling from library/node
Digest: sha256:7543db0284b51d9bff8226cf8098f15fdf0a9ee6c4ca9bc0df4f11fa0e69e09d
Status: Downloaded newer image for node:latest
---> ea27efc47a35
Step 2/2 : WORKDIR /usr/share/nodeJs/
---> Using cache
---> 97812d1f94c8

Successfully built 97812d1f94c8
Successfully tagged nodejs_node:latest
Creating nodejs_node_1 ... done


adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs/docker$ docker exec --interactive --tty nodejs_node_1 /bin/bash
root@b01ee364c0cc:/usr/share/nodeJs# npm install

[...]

added 113 packages, and audited 114 packages in 14s

[...]

found 0 vulnerabilities
root@b01ee364c0cc:/usr/share/nodeJs#


root@b01ee364c0cc:/usr/share/nodeJs# node_modules/.bin/mocha test/**/*Test.js --reporter min
  36 passing (212ms)

root@f112a78fde91:/usr/share/nodeJs#

Here I am doing the following:

  • Building the container from Dockerfile and docker-compose.yml config;
  • shelling into it;
  • Remembering I need to actually install the npm modules;
  • Running some unit tests to demonstrate the npm stuff all installed OK, and the code is operational.

Oh, for good measure, npm install is installing this lot (from package.json):

  "devDependencies": {
    "chai": "^4.2.0",
    "chai-as-promised": "^7.1.1",
    "chai-datetime": "^1.7.0",
    "chai-match": "^1.1.1",
    "chai-string": "^1.5.0",
    "deep-eql": "^3.0.1",
    "mocha": "^8.2.1"
  },

So if everything had not worked A-OK, the tests would not have run, let alone passed.

The process for the app I'm writing in the blog article series is the same as this, just with a bunch more stuff in the docker-compose.yml file. And it all worked the same as above, so I'm happy a Larry, and off I go to write my blog article.

When I come to start prepping for the next article, I remember I had to do the npm install step manually, and that was wrong, so I went to move it into the Dockerfile. At the same time the next article called for me to install Vue CLI, so I chucked that into the Dockerfile too:

FROM node
WORKDIR /usr/share/nodeJs/
RUN npm install -g @vue/cli
RUN npm install

Because I wanted to test this from scratch, I removed the package-lock.json file and node-modules directory from the previous install, and completely rebuilt the container:

adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs/docker$ docker-compose down --remove-orphans && docker-compose up --build --detach
Removing nodejs_node_1 ... done
Removing network nodejs_default
Creating network "nodejs_default" with the default driver
Building node
Step 1/4 : FROM node
---> ea27efc47a35
Step 2/4 : WORKDIR /usr/share/nodeJs/
---> Using cache
---> 97812d1f94c8
Step 3/4 : RUN npm install -g @vue/cli
---> Running in 8a33bda430b9

[...]

added 1398 packages, and audited 1399 packages in 4m

[...]

found 0 vulnerabilities

[...]

Removing intermediate container 8a33bda430b9
---> f0752a8149f4
Step 4/4 : RUN npm install
---> Running in df849deb15ad

up to date, audited 1 package in 178ms

found 0 vulnerabilities
Removing intermediate container df849deb15ad
---> 5445fb6c455a

Successfully built 5445fb6c455a
Successfully tagged nodejs_node:latest
Creating nodejs_node_1 ... done
adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs/docker$

That all seems fine: I can see the two npm install runs happening. So I proceed into the container:

adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs/docker$ docker exec --interactive --tty nodejs_node_1 /bin/bash
root@691991a95337:/usr/share/nodeJs# vue --version
@vue/cli 4.5.11

So far… so good. Vue CLI is installed and running. And just to make sure the tests still run:

root@691991a95337:/usr/share/nodeJs# ls
LICENSE  README.md  docker  package.json  src  test
root@691991a95337:/usr/share/nodeJs#

Ummm… what? No node_modules, and no package-lock.json.

but the Vue CLI stuff is working, and all there OK:

root@691991a95337:/usr/share/nodeJs# npm root -g
/usr/local/lib/node_modules
root@691991a95337:/usr/share/nodeJs# ls /usr/local/lib/node_modules
@vue  npm
root@691991a95337:/usr/share/nodeJs#

I googled all over the place for someone / something to blame for this. I googled for npm install silectly failing. I removed the Vue CLI stuff and the npm install seemed to run, but did nothing. I noticed the Removing intermediate container df849deb15ad, and surmised Docker was doing the second npm install in a temporary container for some reason, and then dropping it so… no more node_modules or package-lock.json in my working directory. So I googled about the place to find ways to stop bloody Docker putting things into intermediate containers (without thinking to google why it might be doing that in the first place).

And this is something that really annoys me about myself sometimes. I also see it in other devs (and I've mentioned it here before, too). My first thought when something is going wrong is that it's someone else's fault. Someone else has buggered something up in Docker, Node, Vue, etc; and I'm just being caught out by it. This is so hubristic of me given this is my first time working with this stuff, and millions of other people get on just fine with it. My first thought should always be: what have I done wrong?

Then it dawned on me. Well actually someone said something in a Stack Overflow answer which was unrelated to this, but got me thinking. Here is what happens:

We have this in the Dockerfile:

WORKDIR /usr/share/nodeJs/
[...]
RUN npm install

What does this do? It created a directory /usr/share/nodeJs/ and then runs npm install. What's in the directory at that point? F*** all. Certainly not a package.json file. That doesn't come into play until the container is created, and we're mapping volumes into it, when DOcker looks at the docker-compose.yml stuff, which isn't until after the Dockerfile processing is done.

OK so I'll need the package.json file around in the Dockerfile. First I just try to copy it from the root of my app directory (where it currently is), and this yields and error when I try to build the container:

Step 3/5 : COPY ../../package.json .
ERROR: Service 'node' failed to build : COPY failed: forbidden path outside the build context: ../../package.json ()

Fair enough. it occurs to me I only need it in the root of the app in the container; as I'm not running any ofthis stuff on my host machine, it does not need to be in the root of the application there at all. It can be anywhere. So I shift it into a config subdirectory, within my docker subdirectory:

adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs/docker$ tree -aF --dirsfirst -L 2 node
node
├── config/
│   └── package.json*
├── root_home/
│   └── [...]
└── Dockerfile*

This time I can see it installing all the dependencies fine, but it's still in a intermediate container:

Step 5/5 : RUN npm install
---> Running in caeaa9d3b75c

added 113 packages, and audited 114 packages in 5s

16 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
Removing intermediate container caeaa9d3b75c
---> 3b0c15a86641

Successfully built 3b0c15a86641
Successfully tagged nodejs_node:latest
Creating nodejs_node_1 ... done

Unsurprisingly that being the case, the node_modules directory is not there when I check in the container file system. But also…neither is the package.json file! All that's in there is the contents of … [the penny has just dropped] … my host machine's application directory:

root@6e2d1c5d482f:/usr/share/nodeJs# ls
LICENSE  README.md  docker  src  test
root@6e2d1c5d482f:/usr/share/nodeJs# exit
exit

adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs/docker$ ls ..
LICENSE  README.md  docker  src  test
adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs/docker$

Now. Do you think that might be because that's exactly what I'm telling docker-compose.yml to do? Here:

volumes:
  - ..:/usr/share/nodeJs

I'm telling docker-compose to plaster that directory from my host machine over the top of the directory that Dockerfile has been working in.

Sigh.

And this makes total sense too.

Right. I have some reorganising to do. The problem really is that I'm voluming-in (to invent a phrase) the entire app directory into the container's working directory - which I know now will not work - I have to think about what I need in the working directory for the app to run:

  • I need the package.json file in there at the time Dockerfile is processed, because that's when npm install happens. Cool I've already done that.
  • After I've done a successful npm install I should also grab the package-lock.json file and make sure I also stick it in the working directory in the Dockerfile. But I'll deal with that later after everything is working.
  • If we have a look at what else is in that root application directory:
    camer@DESKTOP-QV1A45U MINGW64 /c/src/nodejs ((1.0.4))
    $ ll
    total 125
    drwxr-xr-x 1 camer 197609     0 Feb  6 10:14 docker/
    -rw-r--r-- 1 camer 197609 35149 Jan 29 14:47 LICENSE
    -rw-r--r-- 1 camer 197609   680 Feb  6 10:14 package.json
    -rw-r--r-- 1 camer 197609 79915 Feb  6 10:14 package-lock.json
    -rw-r--r-- 1 camer 197609    30 Jan 29 14:47 README.md
    drwxr-xr-x 1 camer 197609     0 Feb  1 12:37 src/
    drwxr-xr-x 1 camer 197609     0 Feb  1 23:30 test/

    camer@DESKTOP-QV1A45U MINGW64 /c/src/nodejs ((1.0.4))
    All we really need in the container is the src/ and test/ directories. We don't need to add the whole directory as a volume: just those two will do. Also I'm thinking ahead of my other app here, which has a container for PHP as well, and in that there's a bunch of stuff that lives in the app root directory for Symfony, PHPUnit, PHPMD, PHPCS etc. Currently messed in along with the Node.js stuff. That's a bit of a mess, and when those files are being used, they're never all in a given container: they're used in different ones. So now that I think about it, it's "wrong" that they are all in the same directory in source control. But I'm getting ahead of myself. Back to this simple Node.js container situation.

So the solution to this seems to be:

adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs$ tree -F --dirsfirst docker
docker
├── node/
│   ├── config/
│   │   ├── package-lock.json*
│   │   └── package.json*
│   └── Dockerfile*
└── docker-compose.yml*

2 directories, 4 files
adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs$
FROM node
WORKDIR  /usr/share/nodeJs/
COPY config/* ./
RUN npm install

As I said earlier, I move the config stuff that Dockerfile needs into a subdirectory of the docker config.

Next in the application root directory, I have no container-specific stuff, just the docker, src and test directories; and the source control project bumpf:

adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs$ tree -F --dirsfirst -L 1
.
├── docker/
├── src/
├── test/
├── LICENSE*
└── README.md*

3 directories, 2 files
adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs$

In docker-compose.yml we specifically make volumes for just the src/ and test/ directories:

version: '3'

services:
  node:
    build:
      context: node
    volumes:
      - ./node/root_home:/root
      - ../src:/usr/share/nodeJs/src
      - ../test:/usr/share/nodeJs/test
    stdin_open: true
    tty: true

This means everything is in the right place when it's needed, and there's no overlap in any of the file system stuff in Dockerfile and docker-compose.yml. And when I now build the container from scratch again…

adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs/docker$ docker-compose up --build --detach
Creating network "nodejs_default" with the default driver
Building node
Step 1/4 : FROM node
[...]
Step 2/4 : WORKDIR  /usr/share/nodeJs/
---> Running in 6790771b61ba
Removing intermediate container 6790771b61ba
[...]
Step 3/4 : COPY config/* ./
[...]
Step 4/4 : RUN npm install
[...]
Removing intermediate container de1de62c87e8

Successfully built c7d34c963f25
Successfully tagged nodejs_node:latest
Creating nodejs_node_1 ... done


adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs/docker$ docker exec --interactive --tty nodejs_node_1 /bin/bash
root@1b011f8852b1:/usr/share/nodeJs# ll
total 100
drwxr-xr-x  1 root root  4096 Feb  6 11:05 ./
drwxr-xr-x  1 root root  4096 Feb  6 11:05 ../
drwxr-xr-x 97 root root  4096 Feb  6 11:05 node_modules/
-rwxrwxrwx  1 root root 79915 Feb  6 11:05 package-lock.json*
-rwxrwxrwx  1 root root   680 Feb  6 11:05 package.json*
drwxrwxrwx  1 node node   512 Feb  6 10:52 src/
drwxrwxrwx  1 node node   512 Feb  6 10:52 test/


root@1b011f8852b1:/usr/share/nodeJs# ./node_modules/.bin/mocha test/**/*Test.js --reporter=min

  36 passing (212ms)


root@1b011f8852b1:/usr/share/nodeJs#

… it's all good.

I note with interest that the "Removing intermediate container" thing was a red herring in all this: it happens as a matter of course anyhow. The good news is that everything is there and working fine.

That's about all I have to say on that. I have already made equivalent changes to the codebase for the code for the long-running article series I'm writing. I'll summarise that in the next article.

It's just worth reflecting on this again though:

And this is something that really annoys me about myself sometimes. [...] My first thought when something is going wrong is that it's someone else's fault[...] and I'm just being caught out by it. This is so hubristic of me given this is my first time working with this stuff, and millions of other people get on just fine with it. My first thought should always be: what have I done wrong?

Righto.

--
Adam

Tuesday 12 January 2021

Part 5: MariaDB

G'day:

Please note that I initially intended this to be a part of a single article, but by the time I had finished the first two sections, it was way too long for a single read, so I've split it into the following sections, each as their own article:

  1. Intro / Nginx
  2. PHP
  3. PHPUnit
  4. Tweaks I made to my Bash environment in my Docker containers
  5. MariaDB (this article)
  6. Installing Symfony
  7. Using Symfony
  8. Testing a simple web page built with Vue.js
  9. I mess up how I configure my Docker containers
  10. An article about moving files and changing configuration
  11. Setting up a Vue.js project and integrating some existing code into it
  12. Unit testing Vue.js components

As indicated: this is the fifth article in the series, and follows on from Part 4: Tweaks I made to my Bash environment in my Docker containers. It's probably best to go have a breeze through the earlier articles first just to contextualise things. Also as indicated in earlier articles: I'm a noob with all this stuff so this is basically a log of me working out how to get things working, rather than any sort of informed tutorial on the subject.

OK, let's get on with this MariaDB stuff.

Firstly; why am I using MariaDB instead of MySQL? Initially I started pottering around with MySQL on Docker for another piece of work I was doing, and I ran up against a show-stopper that doesn't seem to have a resolution. It's detailed on GitHub at "MySQL docker 5.7.6 and later fails to initialize database", and demonstrated here:

adam@DESKTOP-QV1A45U:~$ docker pull mysql
Using default tag: latest
latest: Pulling from library/mysql
6ec7b7d162b2: Pull complete
[...]
a369b92bfc99: Pull complete
Digest: sha256:365e891b22abd3336d65baefc475b4a9a1e29a01a7b6b5be04367fcc9f373bb7
Status: Downloaded newer image for mysql:latest
docker.io/library/mysql:latest
adam@DESKTOP-QV1A45U:~$
adam@DESKTOP-QV1A45U:~$
adam@DESKTOP-QV1A45U:~$ docker create --name mysql --expose 3306 -p 3306:3306 --interactive --volume /var/lib/mysql:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=123 --tty mysql
dfabbf24a7b76831cdb95d20302cc46587cfeb2f7b9f63ac2d907fb8505b07b8
adam@DESKTOP-QV1A45U:~$
adam@DESKTOP-QV1A45U:~$
adam@DESKTOP-QV1A45U:~$ docker start --interactive mysql
2020-12-20 12:18:56+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.22-1debian10 started.
2020-12-20 12:18:56+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2020-12-20 12:18:56+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.22-1debian10 started.
2020-12-20 12:18:56+00:00 [Note] [Entrypoint]: Initializing database files
2020-12-20T12:18:56.174257Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.22) initializing of server in progress as process 46
2020-12-20T12:18:56.183150Z 0 [Warning] [MY-010159] [Server] Setting lower_case_table_names=2 because file system for /var/lib/mysql/ is case insensitive
2020-12-20T12:18:56.186621Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2020-12-20T12:18:58.227909Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
mysqld: Cannot change permissions of the file 'ca.pem' (OS errno 1 - Operation not permitted)
2020-12-20T12:19:00.524436Z 0 [ERROR] [MY-010295] [Server] Could not set file permission for ca.pem
2020-12-20T12:19:00.524927Z 0 [ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable. You can remove all files that the server added to it.
2020-12-20T12:19:00.525836Z 0 [ERROR] [MY-010119] [Server] Aborting
2020-12-20T12:19:02.414805Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.22) MySQL Community Server - GPL.
adam@DESKTOP-QV1A45U:~$

I tried everything suggested in that thread, everything else I could find, and nothing improved the situation. However this entry on that issue page above looked like good advice:



So I just decided to run with MariaDB instead, and that worked perfectly:

adam@DESKTOP-QV1A45U:~$ docker create --name mariadb --expose 3306 -p 3306:3306 --interactive --volume /var/lib/mysql:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=123 --tty mariadb
Unable to find image 'mariadb:latest' locally
latest: Pulling from library/mariadb
da7391352a9b: Pull complete
[...]
a33f860b4aa6: Pull complete
Digest: sha256:cdc553f0515a8d41264f0855120874e86761f7c69407b5cfbe49283dc195bea8
Status: Downloaded newer image for mariadb:latest
bb2a4128911e52f2b16a25c4f994fe12eeec3c36a7e9e188cba2758522785522
adam@DESKTOP-QV1A45U:~$
adam@DESKTOP-QV1A45U:~$
adam@DESKTOP-QV1A45U:~$ docker start mariadb
2020-12-20 12:27:15+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.8+maria~focal started.
2020-12-20 12:27:15+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2020-12-20 12:27:15+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.8+maria~focal started.
2020-12-20 12:27:15+00:00 [Note] [Entrypoint]: Initializing database files
[... bunch of stuff snipped ...]
mariadb
adam@DESKTOP-QV1A45U:~$
adam@DESKTOP-QV1A45U:~$
adam@DESKTOP-QV1A45U:~$ docker exec -it mariadb mariadb --user=root --password=123
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 10.5.8-MariaDB-1:10.5.8+maria~focal mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> SELECT @@VERSION;
+-------------------------------------+
| @@VERSION |
+-------------------------------------+
| 10.5.8-MariaDB-1:10.5.8+maria~focal |
+-------------------------------------+
1 row in set (0.000 sec)

MariaDB [(none)]>

Now I don't doubt it's possible to get MySQL working in my environment, but… erm… shrug. I don't care. I just need a DB running. I'm not here to fart-arse around with DBs any more than absolutely necessary.

We should actually now back-up a bit. All that stuff above was done a while ago - although I replicated it just now for the sake of the notes above - and for my current exercise we're getting ahead of ourselves. I'm gonna start this exercise with a failing test (test/integration/DatabaseTest.php):

namespace adamCameron\fullStackExercise\test\integration;

use PHPUnit\Framework\TestCase;
use \PDO;

class DatabaseTest extends TestCase
{
    /** @coversNothing */
    public function testDatabaseVersion()
    {
        $connection = new PDO(
            'mysql:dbname=mysql;host=database.backend',
            'root',
            '123'
        );

        $statement = $connection->query("show variables where variable_name = 'innodb_version'");
        $statement->execute();

        $version = $statement->fetchAll();

        $this->assertCount(1, $version);
        $this->assertSame('10.5.8', $version[0]['Value']);
    }
}

Note: in a better world, I'd never have my user and password hard-coded there (see further down for where I address this), even in a test. And I'd not be using root (I don't address this one quite yet though). Also checking the version right down to the patch level is egregious, I know. Just checking for 10 would perhaps be better there.

This fails as one would expect:

root@5962e5abd527:/usr/share/fullstackExercise# vendor/bin/phpunit test/integration/
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

E                                                                   1 / 1 (100%)

Time: 00:00.245, Memory: 6.00 MB

There was 1 error:

1) adamCameron\fullStackExercise\test\integration\DatabaseTest::testDatabaseVersion
PDOException: SQLSTATE[HY000] [2002] php_network_getaddresses: getaddrinfo failed: Name or service not known

/usr/share/fullstackExercise/test/integration/DatabaseTest.php:15

Caused by
PDOException: PDO::__construct(): php_network_getaddresses: getaddrinfo failed: Name or service not known

/usr/share/fullstackExercise/test/integration/DatabaseTest.php:15

ERRORS!
Tests: 1, Assertions: 0, Errors: 1.

Generating code coverage report in HTML format ... done [00:01.786]

Once we've installed the MariaDB container, got it up and running and networked it: this test should pass.

I'm back to taking Martin Pham's lead in the Docker config for all this (reminder, from his article "Symfony 5 development with Docker")

Here's the docker/mariadb/Dockerfile:

FROM mariadb:latest
CMD ["mysqld"]
EXPOSE 3306
Quick sidebar whilst I'm doing a final edit of this. If I'm testing for a specific version 10.5.8, should I perhaps be forcing that version here too? Hmmm… probably.

No surprises there. Next the stuff in docker/docker-compose.yml:

  mariadb:
    build:
      context: ./mariadb
    environment:
      - MYSQL_ROOT_PASSWORD=${DATABASE_ROOT_PASSWORD}
    ports:
      - "3306:3306"
    volumes:
      - ./mariadb/data:/var/lib/mysql
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    networks:
      backend:
        aliases:
          - database.backend

One new thing for me here is the ${DATABASE_ROOT_PASSWORD}. Looking at Martin's set-up, he's also got a file docker/.env. Seems to me one can sling environment variables in there, and docker-compose picks them up automatically. So I've just got this (docker/.env):

DATABASE_ROOT_PASSWORD=123

I'll come back to this in a bit.

Also note that I'm configuring MariaDB to put its data in a volume back on the host machine. This is so the data persists when I shut the container down.

And now I should be able to start everything up, and it'll just work. Right?

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker-compose up --build --detach
Creating network "docker_backend" with driver "bridge"
Building nginx
[...]
Successfully built 7cb155649c3b
Successfully tagged docker_nginx:latest

Building php-fpm
[...]
Successfully built e483795cc006
Successfully tagged docker_php-fpm:latest

Building mariadb
[...]
Successfully built 1f05ad3e3ad3
Successfully tagged docker_mariadb:latest

Creating docker_mariadb_1 ... done
Creating docker_nginx_1 ... done
Creating docker_php-fpm_1 ... done

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$

OK that's more promising than I expected. How about that integration test?

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker exec --interactive --tty docker_php-fpm_1 /bin/bash
root@4861480bcbad:/usr/share/fullstackExercise# vendor/bin/phpunit test/integration/
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

.                                                                   1 / 1 (100%)

Time: 00:00.200, Memory: 6.00 MB

OK (1 test, 2 assertions)

Generating code coverage report in HTML format ... done [00:01.215]
root@4861480bcbad:/usr/share/fullstackExercise#

Gasp (I actually did gasp a bit). Blimey. It only went and worked first time.

I'm intrigued by this .env stuff. I figured if I can set that root password in the .env file, then I can shift it out of my integration test, and just use the environment variable. So I've updated the php-fpm section of docker-compose.yml to also set than environment variable:

php-fpm:
  build:
    context: ./php-fpm
  environment:
    - DATABASE_ROOT_PASSWORD=${DATABASE_ROOT_PASSWORD}
  # etc

And update the integration test to use that:

$connection = new PDO(
    'mysql:dbname=mysql;host=database.backend',
    'root',
    $_ENV['DATABASE_ROOT_PASSWORD']
);

And test that's all OK:

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker-compose down --remove-orphans
[...] adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker-compose up --build --detach
[...] adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker exec --interactive --tty docker_php-fpm_1 /bin/bash
root@48dedafac625:/usr/share/fullstackExercise# env | grep DATABASE_ROOT_PASSWORD
DATABASE_ROOT_PASSWORD=123
root@48dedafac625:/usr/share/fullstackExercise# vendor/bin/phpunit test/integration/
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

.                                                                   1 / 1 (100%)

Time: 00:00.196, Memory: 6.00 MB

OK (1 test, 2 assertions)

Generating code coverage report in HTML format ... done [00:01.195]
root@48dedafac625:/usr/share/fullstackExercise#

Cool!

Right so that's all that done: it was very easy (thanks largely to Martin's guidance). Note that once I get underway with the app I'll have a specific database to use, and specific credentials to use against it; at that point I'll stop using the root credentials in that test. But for where we are now - just checking that the DB is up and networked and PHP can see it: the test I'm doing is fine.

That was quite a brief article. In the next one I'll get on to installing Symfony.

Righto.

--
Adam

Monday 11 January 2021

Part 4: Tweaks I made to my Bash environment in my Docker containers

G'day:

Intro

Please note that this is a sub-article of a larger body of work that is an exercise in setting up a Vue.js-driven website backed by PHP8 and MariaDB running in Docker containers. All of this is completely new to me, so is a learning exercise, rather than some exposition of my wisdom (which I have none of). I initially intended the whole exercise to be a single article, but by the time I had finished the first two sections, it was way too long for a single read, so I've split it into the following sections, each as their own article:

  1. Intro / Nginx
  2. PHP
  3. PHPUnit
  4. Tweaks I made to my Bash environment in my Docker containers (this article)
  5. MariaDB
  6. Installing Symfony
  7. Using Symfony
  8. Testing a simple web page built with Vue.js using Mocha, Chai and Puppeteer
  9. I mess up how I configure my Docker containers
  10. An article about moving files and changing configuration
  11. Setting up a Vue.js project and integrating some existing code into it
  12. Unit testing Vue.js components

As indicated: this is the fourth article in the series, and - chronologically - follows on from Part 3: PHPUnit. That said, this article is reasonably stand-alone, so not sure if it's really necessary to read everything else first. The only real "cross over" is that I'm experimenting within Docker containers that I created in the earlier articles. I guess you can refer back to the other articles if anything I say here seems to be making assumptions you can't make.

Also please note that there is very little that is earthshattering here. It's more an exercise of me - as a relative *nix noob - getting stuff working how I've become accustomed to it. I discuss nothing tricky or advanced here.

Today's exercise

Whilst doing all the crap to get Nginx, PHP and PHPUnit working, I was spending an awful lot of time in and out of Bash, running various bits of code and testing stuff and the like. I found there were a few annoying things about running Bash in these containers:

  • The Debian distro that the PHP container uses doesn't include ping and other network utils I was needing to use.
  • Bash history works for the life of the container, but does not - obviously - live between rebuilds of a container. This was a pain in the butt as I had a bunch of things to run every rebuild to check stuff.
  • I needed to add a coupla aliases: ll which everyone has; and one, cls, that I've got that is a bit nicer than just clear

In this article I'm gonna work through implementing those, and a coupla other things.

Adding the networking utils was a one-liner in the docker/php-fpm/Dockerfile:

FROM php:8.0-fpm
RUN apt-get update
RUN apt-get install git --yes
RUN apt-get install net-tools iputils-ping --yes
RUN docker-php-ext-install pdo_mysql
RUN pecl install xdebug-3.0.1 && docker-php-ext-enable xdebug
COPY --from=composer /usr/bin/composer /usr/bin/composer
#COPY ./.bashrc /root/.bashrc
ENV XDEBUG_MODE=coverage
WORKDIR  /usr/share/fullstackExercise/
CMD composer install ; php-fpm
EXPOSE 9000

That just installs both net-tools and iputils-ping. The --yes just suppresses confirmation the installation asks, which breaks the Docker build if I don't answer it in a batch fashion: the Docker build process is not interactive.

Sorting out the Bash history was slightly trickier. My initial plan was to just mount a stubbed file in the container (this from docker/docker-compose.yml):

  php-fpm:
    build:
      context: ./php-fpm
    volumes:
      - ..:/usr/share/fullstackExercise
      - ./php-fpm/.bash_history:/root/.bash_history
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    networks:
      - backend

This has some drawbacks. The build broke if ./php-fpm/.bash_history wasn't in the host file system. To guarantee it was there, I needed it in source control. But… by its very nature it's getting changed all the time, which gets annoying when I go to commit stuff. There's ways to work around this in git, but those were causing issues as well.

In the end I decided to take this approach:

    volumes:
      - ..:/usr/share/fullstackExercise
      - ./php-fpm/root_home:/root

That's the entire home directory for the root user. By default it has nothing in it, so I don't need to replicate much in the file system in source control: just make the directory exist, and .gitignore .bash_history:

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ tree -aF --dirsfirst -L 2 php-fpm/root_home/
php-fpm/root_home/
├── .bash_history*
└── .gitignore*

0 directories, 2 files

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ cat php-fpm/root_home/.gitignore
*
!.gitignore

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$

And this just works! Below I show rebuilding my containers, going into bash on the PHP container, showing the empty history, doing some stuff then exiting. Then I repeat the whole operation and you can see the history is sticking across containers now.

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker-compose down --remove-orphans
Stopping docker_nginx_1   ... done
Stopping docker_php-fpm_1 ... done
Removing docker_nginx_1   ... done
Removing docker_php-fpm_1 ... done
Removing network docker_backend
adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker-compose up --build --detach
Creating network "docker_backend" with driver "bridge"
Building nginx
[...]
Building php-fpm
[...]
Creating docker_nginx_1   ... done
Creating docker_php-fpm_1 ... done
adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker exec --interactive --tty docker_php-fpm_1 /bin/bash
root@fd43b546b8c0:/usr/share/fullstackExercise# history
    1  history
root@fd43b546b8c0:/usr/share/fullstackExercise# cat ~/.bash_history
root@fd43b546b8c0:/usr/share/fullstackExercise# ls
LICENSE    _public        composer.lock  log        phpmd.xml    public  test
README.md  composer.json  docker         phpcs.xml  phpunit.xml  src     vendor
root@fd43b546b8c0:/usr/share/fullstackExercise# exit
exit
adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker-compose down --remove-orphans
Stopping docker_php-fpm_1 ... done
Stopping docker_nginx_1   ... done
Removing docker_php-fpm_1 ... done
Removing docker_nginx_1   ... done
Removing network docker_backend
adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker-compose up --build --detach
Creating network "docker_backend" with driver "bridge"
Building nginx
[...]
Building php-fpm
[...]
Creating docker_nginx_1   ... done
Creating docker_php-fpm_1 ... done
adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker exec --interactive --tty docker_php-fpm_1 /bin/bash
root@9b7b60d6208a:/usr/share/fullstackExercise# history
    1  history
    2  cat ~/.bash_history
    3  ls
    4  exit
    5  history
root@9b7b60d6208a:/usr/share/fullstackExercise# cat ~/.bash_history
history
cat ~/.bash_history
ls
exit
root@9b7b60d6208a:/usr/share/fullstackExercise# exit
exit
adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$

Quite pleased with that, I am.

Given how I'm voluming-in the entire home directory, doing the aliases as easy enough, I just needed to drop a .bashrc into that docker/php-fpm/root_home directory. The image doesn't have a .bashrc by default, so this is fine.

alias ll='ls -alF'
alias cls='clear; printf "\033[3J"'

There's a coupla things to note here. Initially by accident I saved this file with CRLF line endings, and so the ll alias didn't work:

root@50fbd03160ef:/usr/share/fullstackExercise# ll
's: invalid option -- '
Try 'ls --help' for more information.
root@50fbd03160ef:/usr/share/fullstackExercise#

That's a trap for young players there (even old burnt-out ones like me, too). It was not immediately obvious what the issue is there, so it took a bit of googling and stack-overflow-ing to find the answer. Something to remember. But once one sorts that out, it works fine:

root@5962e5abd527:/usr/share/fullstackExercise# ll
total 173
drwxrwxrwx 1 1000 1000    512 Dec 14 20:14 ./
drwxr-xr-x 1 root root   4096 Dec 14 18:32 ../
drwxrwxrwx 1 1000 1000    512 Dec 14 19:03 .git/
-rwxrwxrwx 1 1000 1000     49 Dec 14 17:59 .gitignore*
drwxrwxrwx 1 1000 1000    512 Dec 14 20:14 .idea/
-rwxrwxrwx 1 1000 1000  35149 Dec 11 11:29 LICENSE*
-rwxrwxrwx 1 1000 1000    119 Dec 11 11:29 README.md*
-rwxrwxrwx 1 1000 1000    563 Dec 14 14:01 composer.json*
-rwxrwxrwx 1 1000 1000 123727 Dec 14 17:00 composer.lock*
drwxrwxrwx 1 1000 1000    512 Dec 14 19:10 docker/
drwxrwxrwx 1 1000 1000    512 Dec  5 13:19 log/
-rwxrwxrwx 1 1000 1000    479 Dec 14 17:00 phpcs.xml*
-rwxrwxrwx 1 1000 1000   1913 Dec 14 17:00 phpmd.xml*
-rwxrwxrwx 1 1000 1000    709 Dec 14 17:00 phpunit.xml*
drwxrwxrwx 1 1000 1000    512 Dec 14 17:00 public/
drwxrwxrwx 1 1000 1000    512 Dec 14 15:36 src/
drwxrwxrwx 1 1000 1000    512 Dec 14 17:00 test/
drwxrwxrwx 1 1000 1000    512 Dec 14 14:01 vendor/
root@5962e5abd527:/usr/share/fullstackExercise#

The cls alias I make just solves something that annoys me about clear is that it doesn't clear the scrollback history when SSHed into the box, so if one mouse-scrolls up, one can still move past where you cleared. This makes things hard to find sometimes, and defeats the purpose of clear IMO. The escape sequence in the alias alias cls='clear; printf "\033[3J"' just clears the scrollback as well as the screen. Or something that amounts to that anyhow. Shrug. I just copy & pasted it once upon a time, and still use it.

That's all I had for this one. Next stop: "Part 5: MariaDB".

Righto.

--
Adam

Friday 8 January 2021

Part 3: PHPUnit

G'day:

Please note that I initially intended this to be a single article, but by the time I had finished the first two sections, it was way too long for a single read, so I've split it into the following sections, each as their own article:

  1. Intro / Nginx
  2. PHP
  3. PHPUnit (this article)
  4. Tweaks I made to my Bash environment in my Docker containers
  5. MariaDB
  6. Installing Symfony
  7. Using Symfony
  8. Testing a simple web page built with Vue.js using Mocha, Chai and Puppeteer
  9. I mess up how I configure my Docker containers
  10. An article about moving files and changing configuration
  11. Setting up a Vue.js project and integrating some existing code into it
  12. Unit testing Vue.js components

As indicated: this is the third article in the series, and follows on from Part 2: PHP. It's probably best to go have a breeze through the earlier articles first.

PHPUnit

I got slightly ahead of myself and added PHPUnit into composer.json when I was working through the PHP configuration part of this exercise, so it's already installed. Before I run it though, I need a phpunit.xml file, so I'll chuck one of those in:

<?xml version="1.0" encoding="UTF-8"?>
<phpunit
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/9.5/phpunit.xsd"
        colors="true"
        forceCoversAnnotation="true"
        cacheResult="false"
>
    <coverage>
        <include>
            <directory suffix=".php">src</directory>
        </include>
        <report>
            <html outputDirectory="public/test-coverage-report/" />
        </report>
    </coverage>
    <testsuites>
        <testsuite name="Functional tests">
            <directory>test/functional/</directory>
        </testsuite>
        <testsuite name="Unit tests">
            <directory>test/unit/</directory>
        </testsuite>
    </testsuites>
</phpunit>

There's no real surprises here. I've got two separate test suits: one for functional tests in which I'll test those test web-browseable files I created in the previous article; and one for unit tests. To test the code coverage config here I'll need some actual code to test and cover.

I had some drama getting PHPUnit working with code coverage, and that in itself is covered in a separate article, PHPUnit: get code coverage reporting working on PHP 8 / PHPUnit 9.5. The stuff I've written there is very focused on PHPUnit and not so much on the Docker side of things, or the testing in the context of this notional application I'm putting together, hence splitting it out into its own article, and also so I can focus on the Docker side of things here.

First things first, I need some tests! I decided to functional-test the two web browseable files: gdayWorld.html and gdayWorld.php. As a reminder their contents are (respectively):

<!doctype html>

<html lang="en">
<head>
    <meta charset="utf-8">

    <title>G'day world</title>
</head>

<body>
<h1>G'day world</h1>
<p>G'day world</p>
</body>
</html>

And:

<?php

$message = "G'day World";
echo $message;

So for the tests I'm just gonna make sure I can hit them, and the HTML one has the expected title, heading and content; and the PHP one just has the expected string. I could horse around with PHP's native curl implementation to make these work, but its programming interface written like something out of 1995, so I tend to avoid it where I can. I'm gonna use Guzzle instead. Also, and this is slightly OTT I know, but I like using Symfony's Response constants when checking for HTTP status codes to make the code more clear, so I'm adding in symfony/http-foundation. Lastly as I'll be using PHP's DOM API for testing the HTML, I'm placating a warning in PHPStorm that says "ooh but ext-dom might not be installed!" So I'm forcing that too. Oh and I like keeping my code tidy so I'm also slinging PHPMD and PHPCS in there too. My composer.json file becomes:

{
    "name": "adamcameron/full-stack-exercise",
    "description": "Full Stack Exercise",
    "license": "GPL-3.0-or-later",
    "require-dev": {
        "phpunit/phpunit": "^9.5",
        "guzzlehttp/guzzle": "^7",
        "symfony/http-foundation": "^5.2",
        "ext-dom": "*",
        "phpmd/phpmd": "^2.9",
        "squizlabs/php_codesniffer": "^3.5"
    },
    "autoload": {
        "psr-4": {
            "adamCameron\\fullStackExercise\\": "src/"
        }
    },
    "autoload-dev": {
        "adamCameron\\fullStackExercise\\test\\": "test/"
    }
}

Now I can write some tests. I did all this incrementally, but you know how to do that, so here is the "final" (see below for why I put that in quotes) version of WebServerTest:

namespace adamCameron\fullStackExercise\test\functional\_public; // "public" is reserved

use GuzzleHttp\Client;
use PHPUnit\Framework\TestCase;
use Symfony\Component\HttpFoundation\Response;

class WebServerTest extends TestCase
{
    /** @coversNothing */
    public function testGdayWorldHtmlReturnsExpectedContent()
    {
        $expectedContent = "G'day world";


        $client = new Client([
            'base_uri' => 'http://localhost/'
        ]);

        $response = $client->get('gdayWorld.html');

        $this->assertEquals(Response::HTTP_OK, $response->getStatusCode());

        $html = $response->getBody();
        $document = new \DOMDocument();
        $document->loadHTML($html);

        $xpathDocument = new \DOMXPath($document);

        $hasTitle = $xpathDocument->query('/html/head/title[text() = "' . $expectedContent . '"]');
        $this->assertCount(1, $hasTitle);

        $hasHeading = $xpathDocument->query('/html/body/h1[text() = "' . $expectedContent . '"]');
        $this->assertCount(1, $hasHeading);

        $hasContent = $xpathDocument->query('/html/body/p[text() = "' . $expectedContent . '"]');
        $this->assertCount(1, $hasContent);
    }
}

And I triumphantly run this:

/usr/share/fullstackExercise # vendor/bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

Warning:       No code coverage driver available

E                                                                   1 / 1 (100%)

Time: 00:00.438, Memory: 6.00 MB

There was 1 error:

1) adamCameron\fullStackExercise\test\functional\_public\WebServerTest::testGdayWorldHtmlReturnsExpectedContent
GuzzleHttp\Exception\ConnectException: cURL error 7: Failed to connect to localhost port 80: Connection refused (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost/gdayWorld.html

Doh! But… but… but… I quickly went and hit http://localhost/gdayWorld.html in my browser and it was fine. Then the penny dropped. I'm not running the tests from my host machine. I'm running them from with the PHP container. I've told the host machine about the Nginx container's web server; but I've not told the PHP container about it. Reminder as to what the docker-compose.yml is like at the moment:

version: '3'

services:
  nginx:
    build:
      context: ./nginx
    volumes:
      - ../public:/usr/share/fullstackExercise/public
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/sites/:/etc/nginx/sites-available
      - ./nginx/conf.d/:/etc/nginx/conf.d
      - ../log:/var/log
    depends_on:
      - php-fpm
    ports:
      - "80:80"
    stdin_open: true # docker run -i
    tty: true        # docker run -t

  php-fpm:
    build:
      context: ./php-fpm
    volumes:
      - ..:/usr/share/fullstackExercise
    stdin_open: true # docker run -i
    tty: true        # docker run -t

I read a whole bunch of stuff on networking in Docker. I didn't find the Docker docs very useful at the time, but now I re-read them knowing how I'm supposed to interpret them, they seem clear. Not sure if that's an indictment of me or the docs. Or both. I also looked at a whole bunch of Stack Overflow Q&A and the answers were conflicting and divergent. However after distilling what I could from all these sources, it's really pretty easy. Here's the updated version:

version: '3'

services:
  nginx:
    build:
      context: ./nginx
    volumes:
      - ../public:/usr/share/fullstackExercise/public
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/sites/:/etc/nginx/sites-available
      - ./nginx/conf.d/:/etc/nginx/conf.d
      - ../log:/var/log
    depends_on:
      - php-fpm
    ports:
      - "80:80"
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    networks:
      - backend

  php-fpm:
    build:
      context: ./php-fpm
    volumes:
      - ..:/usr/share/fullstackExercise
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    networks:
      - backend
        
networks:
  backend:
    driver: "bridge"

I just added the networks section, and then told the Nginx and PHP containers to be on that backend network. NB: backend has no significance as a word here, it's just a label, and one used in the docs I was reading. After rebuild, I could now see the Nginx server from the PHP container:

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker exec --interactive --tty docker_php-fpm_1 /bin/sh
/usr/share/fullstackExercise # curl http://nginx/gdayWorld.php
G'day World


/usr/share/fullstackExercise #

Note that I'm using the Nginx services container name there as the host name, ie:

services:
  nginx:

Not that it matters, but that seems a bit manky to me, so I wanted to specify a hostname here. That's just a matter of giving the Nginx container a hostname:

services:
  nginx:
    build:
      context: ./nginx
    hostname: webserver.backend
    volumes:
      # etc
adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker exec --interactive --tty docker_php-fpm_1 /bin/sh
/usr/share/fullstackExercise # curl http://webserver.backend/gdayWorld.php
G'day World


/usr/share/fullstackExercise #

Now my functional tests should work, provided I use that new hostname:

/usr/share/fullstackExercise # vendor/bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

Warning:       No code coverage driver available

..                                                                   1 / 1 (100%)

Time: 00:00.462, Memory: 6.00 MB

OK (2 test, 6 assertions)
/usr/share/fullstackExercise #

Cool!

Oh there was a separate test for gdayWorld.php too:

namespace adamCameron\fullStackExercise\test\functional\_public; // "public" is reserved

use GuzzleHttp\Client;
use PHPUnit\Framework\TestCase;
use Symfony\Component\HttpFoundation\Response;

class PhpTest extends TestCase
{
    /** @coversNothing */
    public function testGdayWorldPhpReturnsExpectedContent()
    {
        $client = new Client([
            'base_uri' => 'http://webserver.backend/'
        ]);

        $response = $client->get('gdayWorld.php');

        $this->assertEquals(Response::HTTP_OK, $response->getStatusCode());

        $content = $response->getBody()->getContents();

        $this->assertSame("G'day world", $content);
    }
}

I'm glad I wrote these tests, because this one identified a small bug I had introduced into gdayWorld.php. I'd fixed it by the time I catpured that output above, but the first run was less positive:

/usr/share/fullstackExercise # vendor/bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

Warning:       No code coverage driver available

F.                                                                  2 / 2 (100%)

Time: 00:00.509, Memory: 6.00 MB

There was 1 failure:

1) adamCameron\fullStackExercise\test\functional\_public\PhpTest::testGdayWorldPhpReturnsExpectedContent
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'G'day world'
+'G'day World'

/usr/share/fullstackExercise/test/functional/PhpTest.php:24

FAILURES!
Tests: 2, Assertions: 6, Failures: 1.
/usr/share/fullstackExercise #

Yay for testing! I did not contrive this situation as an example of "always test first!", but there it is. This is hugely trivial code, but I still messed it up, and simply eyeballing it did not spot the bug.

Speaking of being observant… you will no-doubt have noticed the warning about code coverage driver above. I still need to install XDebug to make this work. This is a matter of adding this into the Dockerfile:

FROM php:fpm-alpine
RUN apk --update --no-cache add git
RUN docker-php-ext-install pdo_mysql
RUN pecl install xdebug-3.0.1 && docker-php-ext-enable xdebug
COPY --from=composer /usr/bin/composer /usr/bin/composer
WORKDIR  /usr/share/fullstackExercise/
CMD composer install ; php-fpm
EXPOSE 9000

Or at least that was the theory:

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker-compose up --build --detach
Creating network "docker_backend" with driver "bridge"
Building php-fpm
Step 1/8 : FROM php:fpm-alpine
---> 6bd7d9173974
Step 2/8 : RUN apk --update --no-cache add git
---> Using cache
---> 098d91282e3e
Step 3/8 : RUN docker-php-ext-install pdo_mysql
---> Using cache
---> 6f74a2ec5bb1
Step 4/8 : RUN pecl install xdebug-3.0.1 && docker-php-ext-enable xdebug
---> Running in 12444e0a094c
downloading xdebug-3.0.1.tgz ...
Starting to download xdebug-3.0.1.tgz (214,467 bytes)
.............................................done: 214,467 bytes
87 source files, building
running: phpize
Configuring for:
PHP Api Version: 20200930
Zend Module Api No: 20200930
Zend Extension Api No: 420200930
Cannot find autoconf. Please check your autoconf installation and the
$PHP_AUTOCONF environment variable. Then, rerun this script.

ERROR: `phpize' failed
ERROR: Service 'php-fpm' failed to build : The command '/bin/sh -c pecl install xdebug-3.0.1 && docker-php-ext-enable xdebug' returned a non-zero code: 1
adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$

Sigh. I have encountered this sort of thing before with other containers. Alpine is really really pared down, and doesn't include a bunch of packages that tools might need to run. This is fine for a lot of situations, but it's also a pain in the arse for others. In this case I added in autoconf, but then I needed to install a C compiler too. And then after that I think it wanted something else. Sod that. I just stopped using Alpine and wend back to the Debian version of the container, in the Dockerfile:

FROM FROM php:8.0-fpm
RUN apt-get update --yes && apt-get install git --yes
ENV XDEBUG_MODE=coverage
NB that changed from:
FROM :fpm-alpine
RUN apk --update --no-cache add git

Note: the apk / apt-get change is just the difference in package manager between Alpine and Debian. I won't show you the installation of this because it's 70kB of telemetry, all of which culminates in:

Creating docker_php-fpm_1 ... done Creating docker_nginx_1 ... done

And now we can run our functional tests, and the code coverage report should create (even if it hasn't got anything in it yet, cos I'm not code-covering those functional tests):

root@2e5f56af2f54:/usr/share/fullstackExercise# vendor/bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

Warning: Incorrect filter configuration, code coverage will not be processed
..                                                                  2 / 2 (100%)

Time: 00:00.493, Memory: 6.00 MB

OK (2 tests, 6 assertions)
root@2e5f56af2f54:/usr/share/fullstackExercise#

Grrrr… what now?. I reviewed the docs and my phpunit.xml was legit-looking, and validated fine, so I was flumoxed. But then I came across this issue with PHPUnit: Misleading error message when no files present in coverage path. That sums it up. I have this in my phpunit.xml file:

<coverage>
    <include>
        <directory suffix=".php">src</directory>
    </include>
    <report>
        <html outputDirectory="public/test-coverage-report/" />
    </report>
</coverage>

But I don't actually have a src/ directory yet. Once I added that, and also added the public/ directory into that <include> block, I get a report. At the same time, I will add a stub PHP class and test thereof into the src/ directory as well, to better test the reporting:

root@2e5f56af2f54:/usr/share/fullstackExercise# vendor/bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

...                                                                 3 / 3 (100%)

Time: 00:02.050, Memory: 12.00 MB

OK (3 tests, 7 assertions)

Generating code coverage report in HTML format ... done [00:00.780]
root@2e5f56af2f54:/usr/share/fullstackExercise#

And the coverage report generates fine too:
above



below

The test for MyClass's needsTesting method is as simple as you might imagine:

namespace adamCameron\fullStackExercise\test\unit;

use adamCameron\fullStackExercise\MyClass;
use PHPUnit\Framework\TestCase;

/** @coversDefaultClass adamCameron\fullStackExercise\MyClass */
class MyClassTest extends TestCase
{
    private $myClass;

    protected function setUp(): void
    {
        $this->myClass = new MyClass();
    }

    /** @covers ::needsTesting */
    public function testNeedsTesting()
    {

        $needsTesting = $this->myClass->needsTesting();
        $this->assertTrue($needsTesting);
    }
}

OK, so I'm in a good place to be able to TDD some PHP code now. But before I do that, I want to get a MariaDB container added into the mix as well, and write an intergration test for its connectivity. Before I get to that though, as I've been doing the work on this project up until now - especially when troubleshooting the networking and the PHPUnit issue I had - I've tweaked some stuff with my Bash environment, and I wanted to rip that out into a separate wee article. So before we get onto MariaDB, I'll write that up: Tweaks I made to my Bash environment in my Docker containers.

Righto.

--
Adam