G'day:
The subtitle is a serious question there: whilst I like doing programming stuff, I really could do without all the rest of the shite we need to do these days to get apps running. In the past "we've had people for that" (SysOps / SRE Team / etc), so I'd never needed to care about this sort of thing. However now I find myself in smaller outfits that don't have dedicated people for this, and I've had to pick up some knowledge of Docker and that sort of thing. It's fine, but… right at the border of my comfort zone. And it's a "needs must" thing, not something I think is really how I ought to be spending my time.
Over the last four years, I've been "leading" our efforts with dockerising our apps, streamlining them etc. At a point we needed to improve our uptime to be largely seamless for our users, and for reasons that were above my pay grade (literally), we weren't given budget to put Docker Swarm in; instead I heath-robinson-ed an implementation of blue/green deployment leveraging how quickly Nginx could reload config to point to different PHP-FPM upstreams. Way less than ideal, but it worked, and it's about all I could manage with my limited knowledge / time / interest in the subject matter. That said, I was actually quite pleased with myself that I managed to work out how to get it working. But still: it's not how things should be done.
I've left that role now, and during some time I have available to do some R&D to fill some gaps in my tech knowledge, I've decided I had better find out how to do that blue/green shiz in a more polished fashion.
Kubernetes has been something that's perplexed me for years. Mostly cos I heard ppl talk about it, and my eyes started to glaze over. All I could take on board was "[something something] multiple containers", "largely impenetrable to understand", "way more complicated than it ought to be". Great. Anyhow, I decided to see if I could perform this experiment:
- Set up a small "G'day world" app with a Nginx, PHP and MariaDB containers, dockerised, orchestrated with docker compose.
- Leave the Nginx and MariaDB containers as is.
- For a production-ish environment, orchestrate the PHP container via Kubernetes; leaving dev to be orchestrated via Docker-only.
- Obviously comms from Nginx->PHP and PHP->MariaDB still need to work (in both dev and prod).
Baseline / control
First things first, I need a stable Nginx-PHP8-MariaDB-Docker environment. This is pretty familiar to me, and I have some code lying around that just needs a bit of polish (adamcameron/php8). I've written about its initial config before, in PHP: returning to PHP and setting up a PHP8 dev environment. The Dockerfiles for this (Nginx, PHP, MariaDB) are routine, and nothing comment-worthy. The PHP part of the docker-compose.yml file is as follows:
php:
build:
context: php
dockerfile: Dockerfile
env_file:
- envVars.public
- php/envVars.public
- envVars.private
- php/envVars.private
stdin_open: true
tty: true
volumes:
- ..:/var/www
There's four environment variables files there, two public, two private:
# docker/envVars.public
MARIADB_HOST=mariadb
MARIADB_PORT=3306
MARIADB_DATABASE=db1
MARIADB_USER=user1
# docker/php/envVars.public
APP_CACHE_DIR=/var/cache/symfony
APP_LOG_DIR=/var/log/symfony
# docker/envVars.private
MARIADB_PASSWORD=123
docker/php/envVars.private
APP_SECRET=does_not_matter
That mostly speaks for itself. The different directories being used is just cos the MariaDB container also uses the files with the MariaDB stuff in them. The ones in the docker/php directory are specific to the PHP container. The private files are not in source control.
Note also there is that one volume used to mount the source code from the host machine into the container file system. Standard stuff.
I bring this lot up:
$ pwd
~/src/php8-on-k8s
$ docker compose -f docker/docker-compose.yml down --volumes
[output snipped]
$ docker compose -f docker/docker-compose.yml build --no-cache
[output snipped]
$ docker compose -f docker/docker-compose.yml up --detach
[output snipped]
# wait for a bit for composer install to finish in the PHP container;
# and the MariaDB container to run its /docker-entrypoint-initdb.d scripts
$ docker container ls --format "table {{.Names}}\t{{.Status}}"
NAMES STATUS
php8-on-k8s-nginx-1 Up 2 minutes (healthy)
php8-on-k8s-php-1 Up 2 minutes (healthy)
php8-on-k8s-mariadb-1 Up 2 minutes
$ docker exec php8-on-k8s-php-1 composer test-all
./composer.json is valid
PHPUnit 12.2.6 by Sebastian Bergmann and contributors.
Runtime: PHP 8.4.10 with Xdebug 3.4.4
Configuration: /var/www/phpunit.xml
............... 15 / 15 (100%)
Time: 00:02.428, Memory: 28.00 MB
Tests Database objects
✔ It has a proc called sleep_and_return which idles for n seconds before returning its parameter value
Tests of Symfony installation
✔ It serves the home page
✔ It can run the console in a shell
Tests of Symfony testing
✔ It serves the default welcome page after installation
Tests of environment variables
✔ The expected environment variables exist with data set "APP_CACHE_DIR"
✔ The expected environment variables exist with data set "APP_LOG_DIR"
✔ The expected environment variables exist with data set "APP_SECRET"
✔ The expected environment variables exist with data set "MARIADB_DATABASE"
✔ The expected environment variables exist with data set "MARIADB_HOST"
✔ The expected environment variables exist with data set "MARIADB_PASSWORD"
✔ The expected environment variables exist with data set "MARIADB_PORT"
✔ The expected environment variables exist with data set "MARIADB_USER"
Tests of the Greeter class
✔ It greets formally by default
✔ It greets informally
Tests of the PHP installation
✔ It has the expected PHP version
OK (15 tests, 20 assertions)
Generating code coverage report in HTML format ... done [00:00.010]
And, of course: does it actually serve its homepage:
(Not very exciting, I know).
But it's up and healthy. I note I forgot to do a health-check on the MariaDB container. Also note in general I have automated tests on the DB connection, Symfony install, environment, and some stub code. This all just checks the moving parts are moving together and in the right direction. This is basically the "control" for my experiment: via Docker, everything is connected and working.
I realise this is pretty meaningless without looking at the code, but it's all on Github, linked-to above, and for completeness it's @ https://github.com/adamcameron/php8-on-k8s/tree/0.41. That it's the 0.41 tag is significant. The main branch has moved on, and is quite different. The 0.41 tag is what I used to get as far as we are currently: Docker containers up, running, tested.
WTF is Kubernetes?
I seriously didn't really know at this point. I knew it was something to do with orchestrating clusters of containers. But that was about it. I mean to frame my ignorance I didn't even know that it still uses Docker containers. I had ass-u-me`d that Kubernetes had its own container mechanics. But no: it's all and only about the orchestration. So that is something at least: I don't need to learn a new container-creation lingo.
I read the Wikipedia page (Kubernetes), and most of it just had me going "huh?". The most interesting thing was from the criticism section:
A common criticism of Kubernetes is that it is too complex. Google admitted this as well.
https://en.wikipedia.org/wiki/Kubernetes#Criticism
Great.
I thought I'd have a look at some stuff on YT to see if it was any help. I watched these two:
Both were informative, and had me begin to realise that I can probably ignore most of the complexity of Kubernetes, and really only had to focus on "deployments" and "services" (and config thereof). Everything else will "just work", and I guess only needs tinkering with if yer a Kubernetes admin, which I will never need to be. So that's heartening.
The first thing I needed to do was to get Kubernetes installed. My mate Brendan has said "you'll need Minikube: it's like the dev/test/tinkering version of Kubernetes". This was re-iterated on the first page of the Learn Kubernetes Basics. So let's do that (I worked through all the tutorials in that section of the docs. I'll just include the highlights here).
From minikube start (Linux » Debian approach for WSL running on Windows).
$ curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
[output snipped]
$ sudo dpkg -i minikube_latest_amd64.deb
[output snipped]
$ minikube start
😄 minikube v1.36.0 on Ubuntu 22.04 (amd64)
✨ Automatically selected the docker driver
📌 Using Docker driver with root privileges
❗ For an improved experience it's recommended to use Docker Engine instead of Docker Desktop.
Docker Engine installation instructions: https://docs.docker.com/engine/install/#server
👍 Starting "minikube" primary control-plane node in "minikube" cluster
🚜 Pulling base image v0.0.47 ...
💾 Downloading Kubernetes v1.33.1 preload ...
> preloaded-images-k8s-v18-v1...: 347.04 MiB / 347.04 MiB 100.00% 11.77 M
🔥 Creating docker container (CPUs=2, Memory=2200MB) ...
🐳 Preparing Kubernetes v1.33.1 on Docker 28.1.1 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring bridge CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
# checking that things work (as per the tutorial):
$ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-674b8bbfcf-5gm2v 1/1 Running 0 33s
kube-system etcd-minikube 1/1 Running 0 38s
kube-system kube-apiserver-minikube 1/1 Running 0 38s
kube-system kube-controller-manager-minikube 1/1 Running 0 38s
kube-system kube-proxy-szq8k 1/1 Running 0 33s
kube-system kube-scheduler-minikube 1/1 Running 0 38s
kube-system storage-provisioner 1/1 Running 1 (12s ago) 36s
Minikube also comes with a great web-based dashboard. This needs to be run from its own terminal window, as it needs to stay running:
$ minikube dashboard --port 40000
🔌 Enabling dashboard ...
▪ Using image docker.io/kubernetesui/metrics-scraper:v1.0.8
▪ Using image docker.io/kubernetesui/dashboard:v2.7.0
💡 Some dashboard features require the metrics-server addon. To enable all features please run:
minikube addons enable metrics-server
🤔 Verifying dashboard health ...
🚀 Launching proxy ...
🤔 Verifying proxy health ...
🎉 Opening http://127.0.0.1:40000/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ in your default browser...
👉 http://127.0.0.1:40000/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/
I learned after letting it use random ports for a while that one can force it to stay on a port with that port setting. Browsing to that URL:
There's not much to see, but clearly something is running OK.
Back to my docker containers
I'd forgotten one step in all this. My PHP container is only dev-ready: I need to build a prod-ready version. Plus currently all the Docker containers are using the internal Docker network provided by docker compose, whereas I will need them to communicate-out via the host machine, as the PHP container won't be managed by docker compose any more. I'll just run through the changes here:
# docker/nginx/etc/nginx/conf.d/default.conf
upstream php-fpm {
server php:9000;
server host.docker.internal:9000;
keepalive 32;
}
As per above: this needs to forward the requests to the host machine; there will be no PHP container on the Docker network, in "production mode". This will also work fine in "dev mode".
# .env
APP_ENV=dev
# stick stuff in here that all envs need
# docker/php/envVars.dev.public
APP_ENV=dev
docker/php/envVars.prod.public
APP_ENV=prod
This was just wrong. One cannot set the APP_ENV environment variable in Symfony's .env system, as Symfony needs to know the environment before the .env files are loaded (which seems very obvious now that I say it). So now docker compose needs to load that env file as well (see below).
# docker/docker-compose.yml
services:
nginx:
container_name: nginx
build:
context: nginx
dockerfile: Dockerfile
ports:
- "8080:80"
stdin_open: true
tty: true
volumes:
- ../public:/usr/share/nginx/html/
depends_on:
- php
php:
container_name: php
build:
context: php
dockerfile: Dockerfile
dockerfile: Dockerfile.dev
env_file:
- envVars.public
- php/envVars.public
- php/envVars.dev.public
- envVars.private
- php/envVars.private
ports:
- "9000:9000"
stdin_open: true
tty: true
volumes:
- ..:/var/www
mariadb:
container_name: db
build:
context: mariadb
dockerfile: Dockerfile
env_file:
- envVars.public
- envVars.private
- mariadb/envVars.private
ports:
- "3380:3306"
stdin_open: true
tty: true
volumes:
- mariadb-data:/var/lib/mariadb
volumes:
mariadb-data:
Top to bottom here:
- Unrelated to this exercise, but I never remember to give my containers names. Adding this in.
- The Nginx container can't rely on the PHP container being up before it starts any more, as the latter won't be being orchestrated by docker when in prod mode. There's no harm here really.
- We need a specific .dev Dockerfile now, as the requirements for dev and prod are slightly different. See below.
- And we need to expose the PHP-FPM port, so Nginx can talk to it still, given Nginx is now forwarding via the host network, not the Docker network.
The main changes need to be made to the Dockerfile. Here's the original (docker/php/Dockerfile):
FROM php:8.4.10-fpm-bookworm
RUN ["apt-get", "update"]
RUN ["apt-get", "install", "-y", "zip", "unzip", "git", "vim"]
RUN echo "alias ll='ls -alF'" >> ~/.bashrc
RUN echo "alias cls='clear; printf \"\033[3J\"'" >> ~/.bashrc
RUN cp /usr/local/etc/php/php.ini-development /usr/local/etc/php/php.ini
COPY usr/local/etc/php/conf.d/php8-on-k8s.ini /usr/local/etc/php/conf.d/php8-on-k8s.ini
COPY --from=composer:latest /usr/bin/composer /usr/local/bin/composer
RUN pecl install xdebug && docker-php-ext-enable xdebug
COPY usr/local/etc/php/conf.d/xdebug.ini /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini
COPY usr/local/etc/php/conf.d/error_reporting.ini /usr/local/etc/php/conf.d/error_reporting.ini
RUN apt-get install -y libicu-dev && docker-php-ext-configure intl && docker-php-ext-install intl
RUN [ \
"apt-get", "install", "-y", \
"libz-dev", \
"libzip-dev", \
"libfcgi0ldbl" \
]
RUN docker-php-ext-configure zip && docker-php-ext-install zip
RUN docker-php-ext-configure pdo_mysql && docker-php-ext-install pdo_mysql
RUN docker-php-ext-configure opcache && docker-php-ext-install opcache
RUN curl -1sLf 'https://dl.cloudsmith.io/public/symfony/stable/setup.deb.sh' | bash
RUN ["apt-get", "install", "-y", "symfony-cli"]
RUN git config --global --add safe.directory /var/www
RUN mkdir -p /var/cache && chown www-data:www-data /var/cache && chmod 775 /var/cache
RUN mkdir -p /var/log && chown www-data:www-data /var/log && chmod 775 /var/log
HEALTHCHECK \
--start-period=30s \
--interval=30s \
--timeout=3s \
--retries=3 \
CMD SCRIPT_FILENAME=/var/www/bin/healthCheck.php REQUEST_METHOD=GET cgi-fcgi -bind -connect localhost:9000 | grep 'pong' || exit 1
WORKDIR /var/www
ENV COMPOSER_ALLOW_SUPERUSER 1
COPY --chmod=755 usr/local/bin/entrypoint.sh /usr/local/bin/
ENTRYPOINT ["entrypoint.sh"]
EXPOSE 9000
I've highlighted all the dev-specific bits in this:
- We need to use the php.ini-production when in production mode.
- We do not need to run xdebug in production mode.
- This is slightly wrong. I need to create and give perms to the dev/prod subdirectories of these paths.
- The entry point runs composer install, but we'll not need to do this in production mode, as we'll do it at build-time in the image.
What I've ended up doing is to push all the common stuff up into a Dockerfile.base, and building that separately:
docker build -f docker/php/Dockerfile.base -t adamcameron/php8-on-k8s-base .
From there I have two files:
# docker/php/Dockerfile.dev
FROM adamcameron/php8-on-k8s-base
RUN cp /usr/local/etc/php/php.ini-development /usr/local/etc/php/php.ini
COPY usr/local/etc/php/conf.d/php8-on-k8s.ini /usr/local/etc/php/conf.d/php8-on-k8s.ini
RUN pecl install xdebug && docker-php-ext-enable xdebug
COPY usr/local/etc/php/conf.d/xdebug.ini /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini
# need to use 777 as both php-fpm and php-cli will write to these directories
RUN mkdir -p /var/cache/symfony/dev && chown www-data:www-data /var/cache/symfony/dev && chmod 777 /var/cache/symfony/dev
RUN git config --global --add safe.directory /var/www
COPY --chmod=755 usr/local/bin/entrypoint.sh /usr/local/bin/
ENTRYPOINT ["entrypoint.sh"]
EXPOSE 9000
This is the dev-specific stuff from the original file we had. Note how it's using FROM adamcameron/php8-on-k8s-base as its base image, which has all the common stuff abstracted away; this just focuses on what it is to be a dev container image.
This is the prod version:
# docker/php/Dockerfile.prod
FROM adamcameron/php8-on-k8s-base
WORKDIR /var/www
RUN cp /usr/local/etc/php/php.ini-production /usr/local/etc/php/php.ini
COPY docker/php/usr/local/etc/php/conf.d/php8-on-k8s.ini /usr/local/etc/php/conf.d/php8-on-k8s.ini
# need to use 777 as both php-fpm and php-cli will write to these directories
RUN mkdir -p /var/cache/symfony/prod && chown www-data:www-data /var/cache/symfony/prod && chmod 777 /var/cache/symfony/prod
COPY .env .env
COPY .env.prod .env.prod
COPY config config
COPY src src
COPY public public
COPY templates templates
COPY composer.json composer.lock .
RUN composer install --no-interaction --no-ansi --no-progress --prefer-dist --no-dev
EXPOSE 9000
Key points:
- As per the dev version: most stuff is now abstracted away in the adamcameron/php8-on-k8s-base image, and this just focuses on what it is to be a production container image.
- We use the php.ini-production file as the basis of our php.ini settings.
- We creat the prod subdirectories of Symfony's /var/cache directory.
- We aren't wanting to maintain the app files from the host machine with the production container, so we copy all the app's files into the image's file system. Note we're not copying the tests into the image's filesystem. We won't be needing these in the production environment.
- And we don't install any of the dev dependencies when we do the composer install. Also note we are doing composer install at build time, not run time. This makes container start-up faster.
- Unlike the dev version of this file, we have no special entrypoint directive. All the container needs to do to finish start-up is to run php-pfm, which is what the underlying PHP image already does.
We need to re-test both the updated dev set-up (via docker compose), and the new prod set-up… for now, still via Docker, but via docker run. I want to know two things:
- The dev tweaks don't break the dev environment.
- The prod version works, as a baseline before we start trying to make it work via Kubernetes
$ docker compose -f docker/docker-compose.yml down --volumes
[success output snipped]
$ docker compose -f docker/docker-compose.yml build --no-cache
[success output snipped]
$ docker compose -f docker/docker-compose.yml up --detach
[success output snipped]
$ docker exec php composer test-all
[success output snipped]
OK (15 tests, 23 assertions)
[success output snipped]
$ curl http://php8-on-k8s.local:8080/
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>Welcome!</title>
<link rel="icon" type="image/x-icon" href="/favicon.ico">
</head>
<body>
Hello world from Symfony<br>
Mode: dev<br>
Pod name: unknown<br>
DB version: 10.11.13-MariaDB-ubu2204<br>
</body>
</html>
(Full disclousre, I accidentally broke two tests with the changes to the homepage output. So much for me and TDD eh?)
OK, so the new dev config works. Now to build and run the container in prod mode:
# get rid of the existing dev container
$ docker compose -f docker/docker-compose.yml down php
[success output snipped]
$ docker build -f docker/php/Dockerfile.prod -t adamcameron/php8-on-k8s .
[success output snipped]
$ docker run \
--name php \
--restart unless-stopped \
-p 9000:9000 \
--env-file docker/envVars.public \
--env-file docker/php/envVars.public \
--env-file docker/php/envVars.prod.public \
--env-file docker/envVars.private \
--env-file docker/php/envVars.private \
--add-host=host.docker.internal:host-gateway \
--detach \
-it \
adamcameron/php8-on-k8s:latest
[success output snipped]
Looks promising. We can't run the tests to verify the install / config this time as we did not install them in the prod container, so we'll just check the homepage:
$ curl http://php8-on-k8s.local:8080/
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>Welcome!</title>
<link rel="icon" type="image/x-icon" href="/favicon.ico">
</head>
<body>
Hello world from Symfony<br>
Mode: prod<br>
Pod name: unknown<br>
DB version: 10.11.13-MariaDB-ubu2204<br>
</body>
</html>
The key point here is Mode: prod. We can also check some Symfony config:
$ docker exec php symfony console about | grep -b3 -a4 "Environment"
469- -------------------- -------------------------------------------
536- Kernel
603- -------------------- -------------------------------------------
670- Type App\Kernel
737: Environment prod
804- Debug false
871- Charset UTF-8
938- Cache directory /var/cache/symfony/prod (668 KiB)
1005- Build directory /var/cache/symfony/prod (668 KiB)
All good.
Converting all that to something Kubernetes can work with
OK. The distillation of my reading of the Kubernetes docs (and some discussion with Co-pilot, who has been very helpful in all this) is that I need to look at four things:
- Deployment
- A deployment is a set of pods, where "pod" is Kubernetes lingo for "some containers". It's kinda the Kubernetes equivalent of a bunch of containers defined in a docker-compose.yml file. In my case I have just the one container per pod. The pod is the "definition mechanism" for what is clustered. I'm gonna get Kubernetes to run three pods, each with my PHP container in it. It's the "cattle" version of my container's "pet", in a way I guess. Ref: Pets vs Cattle
- ConfigMap
- A config map is what Kubernetes uses where Docker would use an env-file. True to form for Kubernetes, they're more complicated. But I have been able to ignore most of the complication.
- Secrets
- And secrets are what one uses in Kubernetes instead of what I do with non-source-controlled env-files. Again, more complicated than they need to be, but most of that can swerved.
- Service
- A service is the bit of the Kubernetes system that exposes the pods in the deployment over the network.
Full disclosure, I asked Co-pilot to do the work here, based on the docker-compose.yaml file. It nailed it. There are also other tools out there to perform these conversions, so it's a legit approach I think. Rather than slavishly reading the docs and understanding the minutiae of what's going on in here.
apiVersion: apps/v1
kind: Deployment
metadata:
name: php
spec:
replicas: 3
selector:
matchLabels:
app: php
template:
metadata:
labels:
app: php
spec:
containers:
- name: php
image: adamcameron/php8-on-k8s:latest
ports:
- containerPort: 9000
envFrom:
- configMapRef:
name: php-config
- secretRef:
name: php-secret
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: CONTAINER_NAME
value: php
restartPolicy: Always
Notes:
- replicas: 3 is telling Kubernetes I want three pods built from this template. So: three containers running my app.
- image: adamcameron/php8-on-k8s:latest: This needs to come from Docker Hub; it won't pick the image up locally. It's fine, but it caught me out initially.
- containerPort: 9000: this is telling the pod what port the container expects to be listening on. Exposing that to downstream services is handled by the service config (see below).
- The envFrom bit is the equiv of the env-file bit of docker-compose.yaml.
- And the env bit is more like the inline env part of a docker-compose file. Here I'm exposing an env var POD_NAME with value of the unique reference to the pod containing the container. I'm using this on the home page to demonstrate that each request could be serviced by any of the pods.
- And CONTAINER_NAME is the equiv of the similar-named value in docker-compose.yaml.
apiVersion: v1
data:
APP_CACHE_DIR: /var/cache/symfony
APP_ENV: prod
APP_LOG_DIR: /var/log/symfony
MARIADB_DATABASE: db1
MARIADB_PORT: "3380"
MARIADB_USER: user1
kind: ConfigMap
metadata:
creationTimestamp: null
name: php-config
One thing to observe here is that I am not setting MARIADB_HOST. This is because I need to use the host's network, and in Docker I'd use host.docker.internal, but Kubernetes does not have this, so I have to set it separately. See the section on k8s/bin/createDeployment.sh, below.
k8s/php.secret.yaml
NB: this file is not source-controlled.
apiVersion: v1
kind: Secret
metadata:
name: php-secret
type: Opaque
data:
MARIADB_PASSWORD: MTIz
APP_SECRET: ZG9lc19ub3RfbWF0dGVy
The chief difference here is that the values are base-64 encoded. This is easy to do in bash:
$ echo -n '123' | base64
MTIz
Again the name is to what the deployment.yaml is referring.
apiVersion: v1
kind: Service
metadata:
name: php
spec:
type: LoadBalancer
selector:
app: php
ports:
- protocol: TCP
port: 9000
targetPort: 9000
nodePort: 31000
Notes:
- type: LoadBalancer. With Minikube, this is the easiest way to expose container (via their pods) to the outside world. Kubernetes itself expects other kit to sit in front of it to be the load balancer, but Minikube comes with one out of the box.
- The port / targetPort bits are the equivalent of exposing ports in docker-compose.yaml. This only exposes the port within the Kubernetes network, it does not in itself expose the port - and therefore the container - to the "outside world": that's the nodePort bit, which is what Minikube's "load-balancer" exposes the service as. There is a twist to that statement in my Windows environment, but I'll get to that.
One thing different in Kubernetes config compared to how Docker works is these files are not what it actually uses to configure things, they're just the source data which we need to "apply" to the Kubernetes config:
$ kubectl apply -f k8s/deployment.yaml
deployment.apps/php created
And having done that, there some action in the Dashboard:
I mean red is not good, but at least it's seen the deployment config, and I can see the three pods I asked for, etc. What's the problem though?
This is entirely fair enough: I've said it needs some config, and I have not given it yet. OK so let's do that:
$ kubectl apply -f k8s/php-config.yaml
configmap/php-config created
I can then restart my deployment:
And the news is not quite what I expected:
The pod seems to have it right: it's missing its secret (cos I ain't applied it yet!); I'd expect the deployment error to be the same. Ah well. Let's crack on:
$ kubectl apply -f k8s/php.secret.yaml
secret/php-secret created
Woohoo!
Look at that green :-)
I can now also access the containers:
$ kubectl exec -it php-8dc55f46d-875lw -- bash
# printenv | grep -i mariadb
MARIADB_PORT=3380
MARIADB_USER=user1
MARIADB_DATABASE=db1
MARIADB_PASSWORD=123
Oh right: I've not fixed that missing MARIADB_HOST issue yet. Let's do that now.
#!/bin/bash
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
kubectl apply -f "$SCRIPT_DIR/php-config.yaml"
kubectl apply -f "$SCRIPT_DIR/php.secret.yaml"
kubectl apply -f "$SCRIPT_DIR/deployment.yaml"
kubectl apply -f "$SCRIPT_DIR/service.yaml"
HOST_IP=$(ip route | awk '/default/ { print $3 }')
if [ -z "$HOST_IP" ]; then
echo "Could not determine host IP"
exit 1
fi
kubectl set env deployment/php MARIADB_HOST=$HOST_IP
There's a coupla things going on here, but we only need to look at the highlighted bit. I have to work out what the MariaDB address is dynamically, because there's no guarantee what the host's IP address will be. Co-pilot wrote this for me, and it seems to do the trick.
This file also demonstrates that one can apply new config to a deployment after the fact. This is what's going in in that last line there: adding a new env-var for the MARIADB_HOST value that we skipped in the initial configMap
I'll run the bit to fix the env var directly now:
$ HOST_IP=$(ip route | awk '/default/ { print $3 }')
if [ -z "$HOST_IP" ]; then
echo "Could not determine host IP"
exit 1
fi
$ echo $HOST_IP
172.24.176.1
$ kubectl set env deployment/php MARIADB_HOST=$HOST_IP
deployment.apps/php env updated
Wait for a bit so Kubernetes can restart all the containers with the new config.
$ kubectl exec -it php-7b57f5f6b9-bklkb -- bash
# printenv | grep -i mariadb
MARIADB_PORT=3380
MARIADB_HOST=172.24.176.1
MARIADB_USER=user1
MARIADB_DATABASE=db1
MARIADB_PASSWORD=123
Cool: it's there now. And we can test connectivity to it:
# apt update
[success output snipped]
# apt install mariadb-client -y
[success output snipped]
# mysql --host=$MARIADB_HOST --port=$MARIADB_PORT --database=$MARIADB_DATABASE -u$MARIADB_USER -p$MARIADB_PASSWORD
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 453
Server version: 10.11.13-MariaDB-ubu2204 mariadb.org binary distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [db1]>
Nice.
The observant might notice th first time I exec-ed into the container, the container name was php-8dc55f46d-875lw. This time it was php-7b57f5f6b9-bklkb. When I updated the deployment config, Kubernetes shut down and removed the containers in my app, and rebuilt them from scratch with the new environment variable. The best thing is, it does this in a controlled fashion: rebuilding one pod at a time, and only continuing if the pod rebuild was successful. So the app was never down during that exercise.
OK the next step is to configure the service:
$ kubectl apply -f k8s/php.secret.yaml
secret/php-secret created
Having run that, I saw the service config appear in Kubernetes dashboard, but it was sitting on orange (I'll spare you the screencap: you get the idea - there was orange instead of green). This was because I'm running the service in loadBalancer mode, with Minikube providing the load-balancer. Forthis to work, I also need to run minikube tunnel:
$ minikube tunnel
✅ Tunnel successfully started
📌 NOTE: Please do not close this terminal as this process must stay alive for the tunnel to be accessible ...
🏃 Starting tunnel for service php.
Note that it specifically sees my php service. Having run that, the board is green (screepcap this time):
OK so look at what port it's opened to the outside world. 9000. Not the 31000 I had set. Now: this saves me some work getting Nginx to use 31000 in prod mode and 9000 in dev mode so it's cool; but I don't understand what is going on here. But I don't care enough to drill into it.
Having done that, everything is working. If I hit the site in a browser I get the expected results:
Hello world from Symfony
Mode: prod
Pod name: php-7b57f5f6b9-sbqxk
DB version: 10.11.13-MariaDB-ubu2204
And if I reload, I can see different pods being used:
Hello world from Symfony
Mode: prod
Pod name: php-7b57f5f6b9-t8lgj
DB version: 10.11.13-MariaDB-ubu2204
Cool!
This all took a while - I started reading docs a week ago, and also worked on this yesterday (Sunday). Four days getting it to work; a solid day to write up. Doing the write-up (which basically required me to do everything a second time) really helped cement my knowledge. Now, I am not claiming any real expertise with Kubernetes, but I do think I have the starting-point knowledge I need as a dev to get my Dockerised app into Kubernetes in a prod environment. Provided I have someone who knows what they are doing to do the management of that prod environment.
I have some tidy-up to do with the source code, so I don't have the most up-to-date tag for it as yet, but the current state of the app is on Github at php8-on-k8s (tag 0.8 ATM).
Righto.
--
Adam
