G'day:
This is an article I am inserting into my ongoing series of setting up a web app using Docker containers, running Nginx, PHP (and Symfony within that), MariaDB, with a Vue.js front end being built in a Node.js container. The other articles can be found here:
- Intro / Nginx
- PHP
- PHPUnit
- Tweaks I made to my Bash environment in my Docker containers
- MariaDB
- Installing Symfony
- Using Symfony
- Testing a simple web page built with Vue.js using Mocha, Chai and Puppeteer
- I mess up how I configure my Docker containers (this article)
- An article about moving files and changing configuration
- Setting up a Vue.js project and integrating some existing code into it
- Unit testing Vue.js components
A week or so ago I wrote an article "Part 8: Testing a simple web page built with Vue.js using Mocha, Chai and Puppeteer". In there I detail how I had configured my Node.js container in Dockerfile and docker-compose.yml, and how it all worked nicely. A few days ago when prepping for the last article of that series ("Refactoring the simple web page into Vue components" - not written yet, so no link yet either, sorry), I discovered my Node.js container was only working by a coincidental quirk of how I built it: and as it stood, the configuration I cited didn't actually work as I described it. I had made a rather school-boy-ish error.
I'm going to explain what I did wrong using a stand-along Node.js container / app, because it'll be a bit more clear if I just focus on the one container, not the array of moving parts I have in that other application. Then I show how I applied a working solution to the that main app.
The requirement I am fulfilling here is to set up a simple Docker container running Node.js, and run some Mocha tests for a code puzzle I have. Normally-speaking one would not have both a Dockerfile and a docker-compose.yml file for this sort of minimal requirement, but the issue I had was with how Dockerfile and docker-compose.yml interact, so I'll include both.
Firstly my Dockerfile:
FROM node
WORKDIR /usr/share/nodeJs/
This just grabs the Node.js image from Docker, sets the directory I'm gonna be using for my code. nothing exciting. In the docker-compose.yml file I've got this lot:
version: '3'
services:
node:
build:
context: node
volumes:
- ..:/usr/share/nodeJs
- ./node/root_home:/root
stdin_open: true
tty: true
This is all straight-forward seeming. So as I can develop my source code files as test as I go, I have mapped my host's nodeJs directory to the container's working directory /usr/share/nodeJs. In addition to that I'm mapping another directory I have in my project as the container's root user's home directory. This is so I can a) avail the container of a .bashrc file I've got some aliases in; b) the .bash_history will be stored on my host machine, so it will persist between rebuilds of the container. This stuff with the root_home stuff is not important to this article.
I can build this container and run it and dive inside:
Building node
Step 1/2 : FROM node
latest: Pulling from library/node
Digest: sha256:7543db0284b51d9bff8226cf8098f15fdf0a9ee6c4ca9bc0df4f11fa0e69e09d
Status: Downloaded newer image for node:latest
---> ea27efc47a35
Step 2/2 : WORKDIR /usr/share/nodeJs/
---> Using cache
---> 97812d1f94c8
Successfully built 97812d1f94c8
Successfully tagged nodejs_node:latest
Creating nodejs_node_1 ... done
adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs/docker$ docker exec --interactive --tty nodejs_node_1 /bin/bash
root@b01ee364c0cc:/usr/share/nodeJs# npm install
[...]
added 113 packages, and audited 114 packages in 14s
[...]
found 0 vulnerabilities
root@b01ee364c0cc:/usr/share/nodeJs#
root@b01ee364c0cc:/usr/share/nodeJs# node_modules/.bin/mocha test/**/*Test.js --reporter min
36 passing (212ms)
root@f112a78fde91:/usr/share/nodeJs#
Here I am doing the following:
- Building the container from Dockerfile and docker-compose.yml config;
- shelling into it;
- Remembering I need to actually install the npm modules;
- Running some unit tests to demonstrate the npm stuff all installed OK, and the code is operational.
Oh, for good measure, npm install is installing this lot (from package.json):
"devDependencies": {
"chai": "^4.2.0",
"chai-as-promised": "^7.1.1",
"chai-datetime": "^1.7.0",
"chai-match": "^1.1.1",
"chai-string": "^1.5.0",
"deep-eql": "^3.0.1",
"mocha": "^8.2.1"
},
So if everything had not worked A-OK, the tests would not have run, let alone passed.
The process for the app I'm writing in the blog article series is the same as this, just with a bunch more stuff in the docker-compose.yml file. And it all worked the same as above, so I'm happy a Larry, and off I go to write my blog article.
…
…
When I come to start prepping for the next article, I remember I had to do the npm install step manually, and that was wrong, so I went to move it into the Dockerfile. At the same time the next article called for me to install Vue CLI, so I chucked that into the Dockerfile too:
FROM node
WORKDIR /usr/share/nodeJs/
RUN npm install -g @vue/cli
RUN npm install
Because I wanted to test this from scratch, I removed the package-lock.json file and node-modules directory from the previous install, and completely rebuilt the container:
Removing nodejs_node_1 ... done
Removing network nodejs_default
Creating network "nodejs_default" with the default driver
Building node
Step 1/4 : FROM node
---> ea27efc47a35
Step 2/4 : WORKDIR /usr/share/nodeJs/
---> Using cache
---> 97812d1f94c8
Step 3/4 : RUN npm install -g @vue/cli
---> Running in 8a33bda430b9
[...]
added 1398 packages, and audited 1399 packages in 4m
[...]
found 0 vulnerabilities
[...]
Removing intermediate container 8a33bda430b9
---> f0752a8149f4
Step 4/4 : RUN npm install
---> Running in df849deb15ad
up to date, audited 1 package in 178ms
found 0 vulnerabilities
Removing intermediate container df849deb15ad
---> 5445fb6c455a
Successfully built 5445fb6c455a
Successfully tagged nodejs_node:latest
Creating nodejs_node_1 ... done
adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs/docker$
That all seems fine: I can see the two npm install runs happening. So I proceed into the container:
root@691991a95337:/usr/share/nodeJs# vue --version
@vue/cli 4.5.11
So far… so good. Vue CLI is installed and running. And just to make sure the tests still run:
LICENSE README.md docker package.json src test
root@691991a95337:/usr/share/nodeJs#
Ummm… what? No node_modules, and no package-lock.json.
but the Vue CLI stuff is working, and all there OK:
/usr/local/lib/node_modules
root@691991a95337:/usr/share/nodeJs# ls /usr/local/lib/node_modules
@vue npm
root@691991a95337:/usr/share/nodeJs#
I googled all over the place for someone / something to blame for this. I googled for npm install silectly failing. I removed the Vue CLI stuff and the npm install seemed to run, but did nothing. I noticed the Removing intermediate container df849deb15ad, and surmised Docker was doing the second npm install in a temporary container for some reason, and then dropping it so… no more node_modules or package-lock.json in my working directory. So I googled about the place to find ways to stop bloody Docker putting things into intermediate containers (without thinking to google why it might be doing that in the first place).
And this is something that really annoys me about myself sometimes. I also see it in other devs (and I've mentioned it here before, too). My first thought when something is going wrong is that it's someone else's fault. Someone else has buggered something up in Docker, Node, Vue, etc; and I'm just being caught out by it. This is so hubristic of me given this is my first time working with this stuff, and millions of other people get on just fine with it. My first thought should always be: what have I done wrong?
Then it dawned on me. Well actually someone said something in a Stack Overflow answer which was unrelated to this, but got me thinking. Here is what happens:
We have this in the Dockerfile:
WORKDIR /usr/share/nodeJs/
[...]
RUN npm install
What does this do? It created a directory /usr/share/nodeJs/ and then runs npm install. What's in the directory at that point? F*** all. Certainly not a package.json file. That doesn't come into play until the container is created, and we're mapping volumes into it, when DOcker looks at the docker-compose.yml stuff, which isn't until after the Dockerfile processing is done.
OK so I'll need the package.json file around in the Dockerfile. First I just try to copy it from the root of my app directory (where it currently is), and this yields and error when I try to build the container:
ERROR: Service 'node' failed to build : COPY failed: forbidden path outside the build context: ../../package.json ()
Fair enough. it occurs to me I only need it in the root of the app in the container; as I'm not running any ofthis stuff on my host machine, it does not need to be in the root of the application there at all. It can be anywhere. So I shift it into a config subdirectory, within my docker subdirectory:
node
├── config/
│ └── package.json*
├── root_home/
│ └── [...]
└── Dockerfile*
This time I can see it installing all the dependencies fine, but it's still in a intermediate container:
---> Running in caeaa9d3b75c
added 113 packages, and audited 114 packages in 5s
16 packages are looking for funding
run `npm fund` for details
found 0 vulnerabilities
Removing intermediate container caeaa9d3b75c
---> 3b0c15a86641
Successfully built 3b0c15a86641
Successfully tagged nodejs_node:latest
Creating nodejs_node_1 ... done
Unsurprisingly that being the case, the node_modules directory is not there when I check in the container file system. But also…neither is the package.json file! All that's in there is the contents of … [the penny has just dropped] … my host machine's application directory:
LICENSE README.md docker src test
root@6e2d1c5d482f:/usr/share/nodeJs# exit
exit
adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs/docker$ ls ..
LICENSE README.md docker src test
adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs/docker$
Now. Do you think that might be because that's exactly what I'm telling docker-compose.yml to do? Here:
volumes:
- ..:/usr/share/nodeJs
I'm telling docker-compose to plaster that directory from my host machine over the top of the directory that Dockerfile has been working in.
Sigh.
And this makes total sense too.
Right. I have some reorganising to do. The problem really is that I'm voluming-in (to invent a phrase) the entire app directory into the container's working directory - which I know now will not work - I have to think about what I need in the working directory for the app to run:
- I need the package.json file in there at the time Dockerfile is processed, because that's when npm install happens. Cool I've already done that.
- After I've done a successful npm install I should also grab the package-lock.json file and make sure I also stick it in the working directory in the Dockerfile. But I'll deal with that later after everything is working.
-
If we have a look at what else is in that root application directory:
camer@DESKTOP-QV1A45U MINGW64 /c/src/nodejs ((1.0.4))All we really need in the container is the src/ and test/ directories. We don't need to add the whole directory as a volume: just those two will do. Also I'm thinking ahead of my other app here, which has a container for PHP as well, and in that there's a bunch of stuff that lives in the app root directory for Symfony, PHPUnit, PHPMD, PHPCS etc. Currently messed in along with the Node.js stuff. That's a bit of a mess, and when those files are being used, they're never all in a given container: they're used in different ones. So now that I think about it, it's "wrong" that they are all in the same directory in source control. But I'm getting ahead of myself. Back to this simple Node.js container situation.
$ ll
total 125
drwxr-xr-x 1 camer 197609 0 Feb 6 10:14 docker/
-rw-r--r-- 1 camer 197609 35149 Jan 29 14:47 LICENSE
-rw-r--r-- 1 camer 197609 680 Feb 6 10:14 package.json
-rw-r--r-- 1 camer 197609 79915 Feb 6 10:14 package-lock.json
-rw-r--r-- 1 camer 197609 30 Jan 29 14:47 README.md
drwxr-xr-x 1 camer 197609 0 Feb 1 12:37 src/
drwxr-xr-x 1 camer 197609 0 Feb 1 23:30 test/
camer@DESKTOP-QV1A45U MINGW64 /c/src/nodejs ((1.0.4))
So the solution to this seems to be:
docker
├── node/
│ ├── config/
│ │ ├── package-lock.json*
│ │ └── package.json*
│ └── Dockerfile*
└── docker-compose.yml*
2 directories, 4 files
adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs$
FROM node
WORKDIR /usr/share/nodeJs/
COPY config/* ./
RUN npm install
As I said earlier, I move the config stuff that Dockerfile needs into a subdirectory of the docker config.
Next in the application root directory, I have no container-specific stuff, just the docker, src and test directories; and the source control project bumpf:
.
├── docker/
├── src/
├── test/
├── LICENSE*
└── README.md*
3 directories, 2 files
adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs$
In docker-compose.yml we specifically make volumes for just the src/ and test/ directories:
version: '3'
services:
node:
build:
context: node
volumes:
- ./node/root_home:/root
- ../src:/usr/share/nodeJs/src
- ../test:/usr/share/nodeJs/test
stdin_open: true
tty: true
This means everything is in the right place when it's needed, and there's no overlap in any of the file system stuff in Dockerfile and docker-compose.yml. And when I now build the container from scratch again…
Creating network "nodejs_default" with the default driver
Building node
Step 1/4 : FROM node
[...]
Step 2/4 : WORKDIR /usr/share/nodeJs/
---> Running in 6790771b61ba
Removing intermediate container 6790771b61ba
[...]
Step 3/4 : COPY config/* ./
[...]
Step 4/4 : RUN npm install
[...]
Removing intermediate container de1de62c87e8
Successfully built c7d34c963f25
Successfully tagged nodejs_node:latest
Creating nodejs_node_1 ... done
adam@DESKTOP-QV1A45U:/mnt/c/src/nodejs/docker$ docker exec --interactive --tty nodejs_node_1 /bin/bash
root@1b011f8852b1:/usr/share/nodeJs# ll
total 100
drwxr-xr-x 1 root root 4096 Feb 6 11:05 ./
drwxr-xr-x 1 root root 4096 Feb 6 11:05 ../
drwxr-xr-x 97 root root 4096 Feb 6 11:05 node_modules/
-rwxrwxrwx 1 root root 79915 Feb 6 11:05 package-lock.json*
-rwxrwxrwx 1 root root 680 Feb 6 11:05 package.json*
drwxrwxrwx 1 node node 512 Feb 6 10:52 src/
drwxrwxrwx 1 node node 512 Feb 6 10:52 test/
root@1b011f8852b1:/usr/share/nodeJs# ./node_modules/.bin/mocha test/**/*Test.js --reporter=min
36 passing (212ms)
root@1b011f8852b1:/usr/share/nodeJs#
… it's all good.
I note with interest that the "Removing intermediate container" thing was a red herring in all this: it happens as a matter of course anyhow. The good news is that everything is there and working fine.
That's about all I have to say on that. I have already made equivalent changes to the codebase for the code for the long-running article series I'm writing. I'll summarise that in the next article.
It's just worth reflecting on this again though:
And this is something that really annoys me about myself sometimes. [...] My first thought when something is going wrong is that it's someone else's fault[...] and I'm just being caught out by it. This is so hubristic of me given this is my first time working with this stuff, and millions of other people get on just fine with it. My first thought should always be: what have I done wrong?
Righto.
--
Adam
