Showing posts with label Nginx. Show all posts
Showing posts with label Nginx. Show all posts

Saturday, 21 January 2023

PHP: returning to PHP and setting up a PHP8 dev environment

G'day:

I need to do some PHP work, and for that I need to have a PHP dev environment. I'm very rusty when it comes to PHP - I've not touched it for 2-3 years or so and my old brain doesn't hold on to things very well - and since that time I have shifted to using Docker for my environments anyhow. I've never used PHP in Docker before. So there's a challenge. And what's this? PHP is now up to version 8.2, with 8.3 being tested. The last time I touched PHP 7.2 was the new thing (we were still mostly on 5.5 at that time, that said).

Therefore I have a mini project ahead of me:

  • Get PHP8.2 running in a Docker container.
  • Get Nginx running in a different container, proxying requests to the PHP one.
  • Have Composer up and running.
  • So I can install PHPUnit.
  • And run some basic tests of the installation.
  • With code-coverage reporting on the tests (code coverage requires a debug module to be installed and running too).
  • Also get PHPMD and PHPCS running too.
  • Bonus: be able to run the tests from my IDE, on my host machine.

Success here will be to be able to view the HTML code coverage report, served by Nginx, showing code being covered by testing.

Full disclosure: I did all this a few nights ago, and I am repeating the exercise now for the purposes of this article.

Application file structure

This shows the file system layout I'm aiming for:

/var/www# tree -L 1
.
|-- docker
|-- html
|-- src
|-- test
`-- vendor

5 directories, 0 files
/var/www#
  • docker.Docker stuff like docker-compose.yml and sub-directories for the various containers' Dockerfiles and other config / assets are in here.
  • html. Files that will be served by Nginx go in here.
  • src. Application code goes here.
  • test. Test code goes here.
  • vendor. The app's Composer dependencies go in here.

This is all standard PHP-app stuff, except my personal decision of how to organise the Docker files.


PHP in a container

docker-compose.yml

The docker-compose.yml service definition is pretty simple:

version: "3"

services:
  php:
    build:
      context: php
      dockerfile: Dockerfile

    stdin_open: true
    tty: true

    volumes:
      - ..:/var/www

/var/www is the directory the container expected to see PHP stuff in, so I ran with it. It doesn't really matter.

.env

Oh I have a wee .env file too:

COMPOSE_PROJECT_NAME=php8

Just so the container names are a bit more on-point when they get created.

Dockerfile

The Dockerfile, on the other hand, is a bit complicated, and took me ages to google all the crap I needed to get together to make PHP 8.2 work in a container with a real-world set of extensions loaded, etc. Deep breath…

FROM php:8.2.1-fpm

RUN ["apt-get", "update"]
RUN ["apt-get", "install", "-y", "zip", "unzip", "git", "vim"]

COPY php.ini /usr/local/etc/php/php.ini

COPY --from=composer:latest /usr/bin/composer /usr/local/bin/composer

RUN pecl install xdebug && docker-php-ext-enable xdebug
COPY conf.d/xdebug.ini /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini
COPY conf.d/error_reporting.ini /usr/local/etc/php/conf.d/error_reporting.ini

RUN apt-get install -y libicu-dev && docker-php-ext-configure intl && docker-php-ext-install intl

RUN ["apt-get", "install", "-y", "libz-dev", "libzip-dev"]
RUN docker-php-ext-configure zip && docker-php-ext-install zip
RUN docker-php-ext-configure bcmath && docker-php-ext-install bcmath
RUN docker-php-ext-configure pdo_mysql && docker-php-ext-install pdo_mysql
RUN docker-php-ext-configure opcache && docker-php-ext-install opcache

RUN curl -1sLf 'https://dl.cloudsmith.io/public/symfony/stable/setup.deb.sh' | bash
RUN ["apt-get", "install", "-y", "symfony-cli"]

WORKDIR /var/www
ENV COMPOSER_ALLOW_SUPERUSER 1

I'll go line-by-line, where there is anything noteworthy, or to explain my decisions.

  • php-fpm. I readily concede I have no idea what all the tag variants of PHP images are on Docker Hub. But I have used php-fpm in the past and know it to work. So: running with it. I am specifically not using the Alpine variant as it doesn't come with BASH, and life is too short to negotiate ASH instead. And I am not trying to economise on disk space for this container anyhow.
  • Baseline APT stuff. Composer needs zip, unzip and git (I learned this by trial and error). I need vim.
  • PHP doesn't have a php.ini file by default although needs it. It ships with php.ini-development and php.ini-production, and it's down to the dev to pick which to use. This file is base on php.ini-development, with the following changes (mostly from recommendations from Symfony › Performance › Use the OPcache Byte Code Cache):
    • realpath_cache_size = 4096k
    • realpath_cache_ttl = 600
    • date.timezone = Europe/London
    • opcache.enable=1
    • opcache.memory_consumption=256
    • opcache.max_accelerated_files=20000

    These are all just a matter of "uncommenting the example setting and tweak its value": normal php.ini stuff. No doubt I will further tweak that as I go, but that's a start.
  • Install Composer. It seems odd that Composer doesn't have the ubiquity that there's an APT package for it.
  • Install Xdebug. PHPUnit needs this for code coverage analysis. Plus at some stage I might start coding like a grown-up and use line debugging. Maybe.
  • I was following along the instructions on "Setup Step Debugging in PHP with Xdebug 3 and Docker Compose" to install Xdebug, and it suggested these settings to use. Yeah cool. I do not know any better. See below this list for the file contents.
  • All of this lot are just libraries that the cited PHP extensions need to be able to run.
  • The ultimate object of the exercise (well: the next part of the exercise) is to get Symfony installed in this app. Whilst setting up the PHP extensions I knew I would be wanting, I thought to look-up what Symfony would need too, and its guidance was to install the Symfony CLI and it would tell me (via symfony check:requirements. Hence installing this now. It was Symfony that reminded me to install all the highlighted extensions. handy. It's also handy that the PHP Docker image comes with those docker-php-ext-configure and docker-php-ext-configure utils, as it makes it a lot easier. There's also a bit of dependency-heck (not quite bad enough to use the work "hell" here) going on installing them, cos sometimes - like with libz-dev and libzip-dev - there are upsteam dependencies needed too. I think I got off pretty lightly here, just needing those two.
  • /var/www is where I want to land when I start a shell on the container.
  • I need to set this otherwise Composer complains about installing stuff as root. This'd be an issue in a production container, maybe. But it's not an issue on dev IMO.

Here are those PHP config files I mentioned above in the Xdebug bit:

# docker/php/conf.d/xdebug.ini

zend_extension=xdebug

[xdebug]
xdebug.mode=develop,debug,coverage
xdebug.client_host=host.docker.internal
xdebug.start_with_request=no
  • I added coverage to this, for PHPUnit.
  • The suggested setting for start_with_request was yes for this, but this meant IntelliJ would interrupt PHPUnit every time I ran my tests from the shell, so I've switched it off.
# docker/php/conf.d/error_reporting.ini
error_reporting=E_ALL

Normally I'd set this directly in php.ini, but I actually did this part of the config before I remembered about php.ini needing to be configured, so stuck with it.


After doing that lot I could build the container and bring it up, and run composer install:

/mnt/c/src/containers/php8/docker$ docker-compose build
[+] Building 4.0s (25/25) FINISHED
[...]
 => exporting to image                                                                                                                                                                                      0.1s
 => => exporting layers                                                                                                                                                                                     0.0s
 => => writing image sha256:3e9dfd6aca1527f8d0906a0d9f2b2ec2c74ddc0bd9ea9d2b8f0d8b1dce773951                                                                                                                0.0s
 => => naming to docker.io/library/php8-php                                                                                                                                                                 0.0s

/mnt/c/src/containers/php8/docker$ docker compose up --detach
[+] Running 2/2
 ⠿ Network php8_default  Created                                                                                                                                                                            0.0s
 ⠿ Container php8-php-1  Started                                                                                                                                                                            0.5s

/mnt/c/src/containers/php8/docker$ docker exec -it php8-php-1 /bin/bash

/var/www# composer install
Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
Package operations: 49 installs, 0 updates, 0 removals
  - Downloading 
    [...]
Generating autoload files
41 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
/var/www#
/var/www# composer validate
./composer.json is valid

composer.json

Speaking of Composer, here's the composer.json file thusfar (don't worry too much about it: I'm including it here for completeness):

{
    "name" : "adamcameron/php8",
    "description" : "PHP8 containers",
    "type" : "project",
    "license" : "proprietary",
    "require": {
        "php" : "^8.2",
        "ext-iconv": "*",
        "ext-pdo_mysql": "*",
        "ext-mbstring": "*",
        "ext-intl": "*",
        "ext-json": "*",
        "ext-curl": "*",
        "ext-simplexml": "*",
        "ext-zip": "*",
        "ext-pcre": "*",
        "ext-ctype": "*",
        "ext-session": "*",
        "ext-tokenizer": "*",
        "ext-bcmath": "*",
        "ext-zend-opcache": "*",
        "monolog/monolog": "^3.2.0",
        "doctrine/dbal": "^3.5.3"
    },
    "require-dev": {
        "phpunit/phpunit": "^9.5.28",
        "phpmd/phpmd": "^2.13.0",
        "squizlabs/php_codesniffer": "^3.7.1"
    },
    "autoload": {
        "psr-4": {
            "adamcameron\\php8\\": "src/"
        }
    },
    "autoload-dev": {
        "psr-4": {
            "adamcameron\\php8\\test\\": "test/"
        }
    },
    "scripts" : {
        "test": "phpunit --testdox test",
        "phpmd": "phpmd src,test text phpmd.xml",
        "phpcs": "phpcs src test",
        "test-all": [
            "@test",
            "@phpmd",
            "@phpcs"
        ]
    }
}

The require section there is not stuff I needed for the install, it's also a bunch of baseline stuff I know I will need for the app I'm heading towards. I don't think there's anything surprising there.

I also already have some PHPUnit, phpmd and phpcs scripts in there. I'll get to those next…

Testing the PHP container

It would not be me if I didn't test stuff. I will admit I did not TDD the bits above, cos that would just be mad. However I wanted to test things worked, so have put a few tests in. Plus part of this is testing the debug module and PHPUnit work together as well.

PHPUnit

Here's the phpunit.xml.dist file:

<?xml version="1.0" encoding="UTF-8"?>
<phpunit
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/9.5/phpunit.xsd"
        colors="true"
        cacheResult="false"
        testdox="true"
        stopOnFailure="true"
        stopOnError="true"
        stopOnWarning="true"
        failOnWarning="true"
>
    <coverage>
        <include>
            <directory suffix=".php">src</directory>
        </include>
        <report>
            <html outputDirectory="html/test-coverage-report/" />
        </report>
    </coverage>
    <testsuites>
        <testsuite name="Integration tests">
            <directory>test/integration/</directory>
        </testsuite>
        <testsuite name="Unit tests">
            <directory>test/unit/</directory>
        </testsuite>
    </testsuites>
</phpunit>

All standard stuff, and there's nothing I can say about it that the docs don't already say.

Tests

A a few minimal tests, which hopefully are self-explanatory:

// test/integration/PhpTest.php

namespace adamcameron\php8\test\integration;

use PHPUnit\Framework\TestCase;

/** @testdox Tests of the PHP installation */
class PhpTest extends TestCase
{
    /** @testdox It has the expected PHP version */
    public function testPhpVersion()
    {
        $expectedPhpVersion = "8.2";
        $actualPhpVersion = phpversion();
        $this->assertStringStartsWith(
            $expectedPhpVersion,
            $actualPhpVersion,
            "Expected PHP version to start with $expectedPhpVersion, but got $actualPhpVersion"
        );
    }
}
// test/integration/ComposerTest.php

namespace adamcameron\php8\test\integration;

use PHPUnit\Framework\TestCase;

/** @testdox Tests of the Composer installation */
class ComposerTest extends TestCase
{
    /** @testdox It passes composer validate */
    public function testComposerValidates()
    {
        exec("composer validate 2> /dev/null", $output, $returnCode);
        $this->assertEquals(
            0,
            $returnCode,
            "Composer validate failed: " . implode("\n", $output)
        );
    }
}

These two test three things: PHP is running the version I expect; Composer is happy it's configured properly; and PHPUnit itself is running otherwise all this would go splat.

For the code coverage testing I need some source code to test:

// src/Greeter.php

namespace adamcameron\php8;

class Greeter
{
    public const FORMAL = 1;
    public const INFORMAL = 2;

    public static function greet(string $name, int $style = self::FORMAL): string
    {
        if ($style === self::FORMAL) {
            return "Hello, $name";
        }
        return "Hi, $name";
    }
    
}

And a test:

// test/unit/GreeterTest.php

namespace adamcameron\php8\test\unit;

use adamcameron\php8\Greeter;

use PHPUnit\Framework\TestCase;

/** @testdox Tests of the Greeter class */
class GreeterTest extends TestCase
{
    /** @testdox It greets formally */
    public function testFormalGreeting()
    {
        $name = "Zachary";
        $expectedGreeting = "Hello, $name";
        $actualGreeting = Greeter::greet($name, Greeter::FORMAL);
        $this->assertEquals(
            $expectedGreeting,
            $actualGreeting,
            "Expected greeting to be $expectedGreeting, but got $actualGreeting"
        );
    }

    /** @testdox It greets informally */
    public function testInformalGreeting()
    {
        $this->markTestSkipped("skipping this so the coverage report is more interesting");
        $name = "Zachary";
        $expectedGreeting = "Hey, $name";
        $actualGreeting = Greeter::greet($name, Greeter::INFORMAL);
        $this->assertEquals(
            $expectedGreeting,
            $actualGreeting,
            "Expected greeting to be $expectedGreeting, but got $actualGreeting"
        );
    }
}

Note how I am skipping one of the tests. This is so code coverage is not 100%.

Running the tests

root@e8896f5d5bd6:/var/www# composer test
> phpunit --testdox test
PHPUnit 9.5.28 by Sebastian Bergmann and contributors.

Tests of the Composer installation
  It passes composer validate

Tests of the PHP installation
  It has the expected PHP version

Tests of the Greeter class
  It greets formally
  It greets informally

Time: 00:05.675, Memory: 10.00 MB

Summary of non-successful tests:

Tests of the Greeter class
  It greets informally
OK, but incomplete, skipped, or risky tests!
Tests: 4, Assertions: 3, Skipped: 1.

Generating code coverage report in HTML format ... done [00:01.492]
root@e8896f5d5bd6:/var/www#

Cool! It all worked. Let's have a look at the code coverage report. Because I don't have Nginx installed yet I can't browse to it, but I can just open the file in a browser:

I've drilled down the report slightly to show the file I was testing. It's correctly showing that I have only tested one path in the logic. Excellent. This proves that the Xdebug extension is running.

phpmd and phpcs

I've installed these too, and have used a fairly stock config file for each (see: phpmd.xml and phpcs.xml). Running them is dead boring as there's hardly any code, and IntelliJ makes sure it's formatted well:

/var/www# composer phpmd
> phpmd src,test text phpmd.xml
/var/www# composer phpcs
> phpcs src test

FILE: /var/www/src/Greeter.php
------------------------------------------------------------------------------------------
FOUND 1 ERROR AFFECTING 1 LINE
------------------------------------------------------------------------------------------
 18 | ERROR | [x] The closing brace for the class must go on the next line after the body
------------------------------------------------------------------------------------------
PHPCBF CAN FIX THE 1 MARKED SNIFF VIOLATIONS AUTOMATICALLY
------------------------------------------------------------------------------------------

Time: 2.51 secs; Memory: 6MB

Script phpcs src test handling the phpcs event returned with error code 2
/var/www#

Ha! I didn't actually expect that. I clearly didn't run this before I did my final commit. If you scroll up to the Greeter.php file, it's complaining that there's an empty line between the last method closing brace and the class's closing brace:

That breaks one of PRS-12's rules. Fair cop. And hey: a good test that it's working!


Nginx in a container

docker-compose.yml

The relevant bit is this:

nginx:
  build:
    context: nginx
    dockerfile: Dockerfile

  ports:
    - "8008:80"

  stdin_open: true
  tty: true

  volumes:
    - ../html:/usr/share/nginx/html/

Nothing interesting there.

Dockerfile

FROM nginx:alpine
WORKDIR /usr/share/nginx/
COPY ./nginx.conf /etc/nginx/nginx.conf
COPY ./sites/ /etc/nginx/sites-available/
COPY ./conf.d/ /etc/nginx/conf.d/
CMD ["nginx"]
EXPOSE 80

Also nothing noteworthy here. Everything is in the config files.

Nginx config files

I freely admin to pretty much lifting these from other projects I already had. I don't really know what I'm doing with Nginx. I learn enough to achieve some goal, then I forget it all within about 5min.

// docker/nginx/nginx.conf
user  nginx;
worker_processes  4;
daemon off;

error_log  /var/log/nginx/error.log debug;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    access_log  /var/log/nginx/access.log;
    sendfile        on;
    keepalive_timeout  65;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-available/*.conf;
}
// docker/nginx/conf.d/default.conf
upstream php-upstream {
    server php:9000;
}
// docker/nginx/sites/default.conf
server {
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;

    server_name localhost;
    root /usr/share/nginx/html;
    index index.html index.php;

    location / {
        autoindex on;
        try_files $uri $uri/ /index.php$is_args$args;
    }

    location ~ \.php$ {
        try_files $uri /index.php =404;
        fastcgi_pass php-upstream;
        fastcgi_index index.php;
        fastcgi_buffers 16 16k;
        fastcgi_buffer_size 32k;
        fastcgi_param SCRIPT_FILENAME /var/www/html/$fastcgi_script_name;
        fastcgi_read_timeout 600;
        include fastcgi_params;
    }

    location ~ /\.ht {
        deny all;
    }
}

I hope it doesn't seem dismissive or that I'm tired of writing in that I add nothing here. I literally don't know what most of that stuff does, other than where it's obvious.

A test PHP file to browse to

I need to be able to test that Nginx is passing stuff to PHP:

<?php
// html/test.php
phpinfo();

Having done all that, I rebuild the containers (and note I now have an Nginx one as well), and bring them up.

If I browse to http://localhost:8008/test.php, I get this:

And if I run the PHPUnit tests again, I can now browse to the report via http://localhost:8008/test-coverage-report/. Cool.


Success

OK so I'm gonna consider that a victory. I did some tinkering around in IntelliJ and I can run the unit tests from in there as well, all via drilling into the Docker container and running it from in there, and presenting the results in IntelliJ:

However that took more dicking around than I can be arsed with re-doing right now. I needed a coupla extensions installed, and set the PHP interpreter to be locatable via the config in docker-compose.yml. It's handy anyhow. I need my team to set all this up, so I'll get the first one of them I dump all this on to work it out and write it down, and I'll report back.

I've linked to all the individual files as I reference them, but you could also clone the repo, checkout the 1.0 tag, and you should be able to set this up locally and have a look if you so choose. There are full instructions in the readme.md file. I have had one of my team test the instructions out on their own PC, and they seemed to have worked.

Righto.

--
Adam

Sunday, 2 May 2021

How TDD and automated testing helped me solve an Nginx config problem I had created for myself

G'day:

I have a "website" I'm building on as part of a series of articles I'm writing about Lucee / CFWheels / Docker. I have a Docker container running Nginx which proxies requests for CFML code to a Docker container running Lucee.

Due to the nature of web applications, I have my web-accessible assets - JS, CSS, image and "entry point" index.cfm and Application.cfc files - in a public directory off my app root; and adjacent to that I have a src directory for my code, and vendor directory for third-party code (like stuff I install from ForgeBox via CommandBox).

This /public abstraction needs to be hidden from the end user: they're going to want to be browsing to - for example - http://example.com/, not http://example.com/public. So this means the proxy from Nginx to Lucee also needs to deal with that. It is imperative that no CFML files are publicly exposed other than the aforementioned index.cfm and Application.cfc

I'm terrible at configuring Nginx, and mak a lot of mistakes. Some mistakes are obvious because Nginx flat-out refuses to start. Others are less obvious, and bleed out as "unexpected behaviour" later in the piece. Knowing this, I approached the exercise in a TDD fashion; identifying what cases need addressing and writing tests for them, and then doing the config work to make each pass. Note: my wording is ambiguous there: I did not write more than one test at a time. I wrote a test, then got the config to make that test pass. Then I wrote the next test and reconfigured to make that pass (whilst also keeping the earlier ones passing too). I've already detailed some of this in my earleir article "Adding TestBox, some tests and CFConfig into my Lucee container".

As a result of these efforts to get Nginx working how I expected it to, I ended up with these green tests:

(Ignore how I'm skipping some of these tests. This is down to a bug in CFWheels that doesn't report 404 situations with an actual 404 status code when in dev mode. This does demonstrate though how I identified a shortcoming in behaviour whilst TDDing the work, that said).

And that's great. At every step as I was configuring more and more of Nginx's handling of requests, I could see that any change I made had a) addressed the new requirement; b) not broken any previous requirement. And it was indeed the reality at times that something I tried fixed one issue, but broke something else.

Back to those tests I'm skipping. This showed to me that I was short some cases: CFWheels handles things differently from how I expect it to in some situations, so I figured I had better have two flavours of test: one set for proxying to non-CFWheel-handled URLs; another set for when the URLs are within CFWheels domain. And I'm glad I did make this call, because it showed-up a bug in my Nginx config:

And when I checked those URLs, I saw the problem:

expectedParamValue = "expectedValue"
// ... rest of test not relevant here
expect(response.fileContent).toInclude(
    "Expected query param value: [#expectedParamValue#]",
    "Query parameter value was incorrect (URL: #testUrl#)"
)

But what I was seeing at that URL was this:

Expected query param value: [expectedValue?testParam=expectedValue]

The query string part (including the ?) was being appended to the URL sent to Lucee twice (same problem for both those failing tests). I was pretty puzzled how my previous non-Wheels test was passing, and it still seemed legit. Bemusing. However I was actively appending the query string in my proxy_pass URL, so that was "clearly wrong":

proxy_pass  http://cfml-in-docker.lucee:8888/public$fastcgi_script_name$is_args$args;

I got rid of that, and figured "I had better go and check why those other tests are passing after I re-run the tests here, to check that change:


Dammit. Now the CFWheels-specific tests are passing, but the non-Wheels ones are failing. Time for me to RTFM, cos I'm clearly doing something wrong here.

The first thing I'm going wrong is here:

proxy_pass  http://cfml-in-docker.lucee:8888/public$fastcgi_script_name;

$fastcgi_script_name is a PHP thing, and whilst it coincidentally holds the URI I want, it's the wrong thing to have here. So I put $request_uri back in there.

Right and that broke all the path_info CFWheels tests, so wasn't right. I decided to read more closely. It turns out that request_uri is the whole original URI (including the path_info and query string), and thus is ignoring my rewrites, and it was the reason that the query params were getting doubled up. In my rewrite I had this:

location @rewrite {
    rewrite ^/(.*)? /index.cfm$request_uri last;
    rewrite ^ /index.cfm last;
    return 404;
}

I just wanted $uri, which is just the document URI part of the requested URI, and it also reflects any changes made to that URI during rewrites and what-have-you. So once I used that in my rewrite and for my proxy_pass URL, the tests now look better:

I've abbreviated how long it took me to work this out, and how many cycles of trial end error it too me. Having those automated tests in place were gold because after each iteration I knew how wrong I was - for all cases - in half a second. I didn't need to manually go "OK, did it work for this?" "did it break this other one?" etc, for 20-odd tests.

It was also a big help to me to take the TDD path here, and stop and think & reason about exactly what my expectations ought to be for each of the cases I had. It also lead me to add more cases, such as the combinations of "it has both path_info and query parameters", as well as realising the path through the Nginx config was different for URLs aimed at Wheels (which are completely rewritten), and the ones directly to files in the public directory. I could easily cover both cases by duplicating the tests and changing the URLs sligthly.

Things seem to be working now, but if I find something else wrong, I will first work out what my expectations of it to be right are, and write a quick test for it. Then I'll fix it (without breaking anything else).

For now though: I'm fed-up with Nginx & CFML & CFWheel and I'm gonna do something els for a while. But I'll be back to it later this afternoon: I'm wll behind wehre I want to be with this stuff, and using the bank-holiday weekend to catch up a bit.

The "final" state of my Nginx site config is (docker/nginx/sites/default.conf):

server {
    listen 80;
    listen [::]:80;

    #rewrite_log on;

    server_name cfml-in-docker.frontend;
    root /usr/share/nginx/html;
    index index.html index.cfm;

    resolver 127.0.0.11;

    location / {
        try_files $uri $uri/ @rewrite;
    }

    location @rewrite {
        rewrite ^/(.*)? /index.cfm$uri last;
        rewrite ^ /index.cfm last;
        return 404;
    }

    location ~ \.(?:cfm|cfc)\b {
        proxy_http_version  1.1;
        proxy_set_header    Connection "";
        proxy_set_header    Host                $host;
        proxy_set_header    X-Forwarded-Host    $host;
        proxy_set_header    X-Forwarded-Server  $host;
        proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;     ## CGI.REMOTE_ADDR
        proxy_set_header    X-Forwarded-Proto   $scheme;                        ## CGI.SERVER_PORT_SECURE
        proxy_set_header    X-Real-IP           $remote_addr;
        expires             epoch;

        proxy_pass  http://cfml-in-docker.lucee:8888/public$uri$is_args$args;
    }

    location ~ /\.ht {
        deny all;
    }
}

Righto, where's me shooting game?

--
Adam

Saturday, 17 April 2021

Using Docker to strum up an Nginx website serving CFML via Lucee

G'day

OK so this is not the blog article I expected to be writing, had you asked me two weeks ago. But here we are. I'll go into the reason why I'm doing this a bit later.

This will be a CFML-oriented version of the "VueJs/Symfony/Docker/TDD series", and has its own tag: "Lucee/CFWheels/Docker series":

  • Nginx website.
  • Proxying for Lucee as the CFML-processing application layer.
  • Running inside Docker containers.
  • TDD the whole enterprise.

If I have time (and any will-to-live remaining), I will add this lot into the mix:

  • Work out how Forgebox works, which seems to be CFML's equivalent of Composer / NPM
  • Use that to install Testbox (CFML-based Jasmine-ish testing framework)
  • And also install CFWheels, a CFML-based framework akin to Ruby on Rails.

I'll also be returning to SublimeText for the first time in seven-or-so years. Apparently it's still a reasonable text editor to use for CFML code.

For those few of you that have started paying attention to me more recently: CFML is not new to me. I spent over a decade as a CFML developer (2001-2013). I shifted to PHP because my erstwhile employer (HostelBookers, CFML shop), was bought by Hostelworld (PHP shop) back then. I've been doing PHP since. That said, I am very rusty with CFML, and - well, hopefully - they CFML landscape has moved on since then too. So whilst I'm not a newbie with CFML stuff, getting Lucee running in a container, Forgebox and CFWheels is entirely new to me.

I'm still gonna be using PHP to do the initial testing of things, because I won't have Testbox running for the first while. So I'll need a PHP container too. I'll refactor this out once I get Testbox in.

It needs a PHP container for running tests

There's nothing new here, and what I've done is largely irrelevant to this exercise, so I'll just list the files and link through to that current state of the files in source control:

adam@DESKTOP-QV1A45U:/mnt/c/src/cfml-in-docker$ tree -a --dirsfirst -I "vendor|.git|.idea"
.
├── docker
│   ├── php-cli
│   │   ├── root_home
│   │   │   ├── .bash_history
│   │   │   ├── .bashrc
│   │   │   ├── .gitignore
│   │   │   └── .vimrc
│   │   └── Dockerfile
│   ├── .env
│   └── docker-compose.yml
├── test
│   └── php
│       └── SelfTest.php
├── .gitignore
├── LICENSE
├── README.md
├── composer.json
├── composer.lock
└── phpunit.xml.dist

5 directories, 14 files
adam@DESKTOP-QV1A45U:/mnt/c/src/cfml-in-docker$

The test this just this:

/** @testdox Tests PHPUnit install */
class SelfTest extends TestCase
{
    /** @testdox it self-tests PHPUnit */
    public function testSelf()
    {
        $this->assertTrue(true);
    }
}

And it passes:

root@18c5eabeb9f2:/usr/share/cfml-in-docker# composer test
> vendor/bin/phpunit --testdox
PHPUnit 9.5.4 by Sebastian Bergmann and contributors.

Tests PHPUnit install
it self-tests PHPUnit

Time: 00:00.002, Memory: 6.00 MB

OK (1 test, 1 assertion)
root@18c5eabeb9f2:/usr/share/cfml-in-docker#

In this instance I could not actually run the test before I implemented the work, for what should seem obvious reasons. However I followed the TDD mindset of just doing the least amount of work possible to make the test pass. I also monkeyed around with the test itself to see it fail if I had an assertion that was no good (I changed the argument to that assertion to false, basically).

The TDD lesson here is: I've set myself a case - "It needs a PHP container for running tests" - and only resolved that case before pausing and assessing the situation. I also didn't move any further forward than I needed to to address that case.

It returns a 200-OK from requests to /gdayWorld.html

Next I need an Nginx container running, and serving a test file. Well: I need the test for that.

/** @testdox Tests Nginx is serving html */
class NginxTest extends TestCase
{
    /** @testdox It serves gdayWorld.html as 200-OK */
    public function testReturns200OK()
    {
        $client = new Client(['base_uri' => 'http://cfml-in-docker.backend/']);

        $response = $client->get('gdayWorld.html');

        $this->assertEquals(200, $response->getStatusCode());
        $content = $response->getBody()->getContents();
        $this->assertMatchesRegularExpression("/^\\s*G'day world!\\s*$/", $content);
    }
}

Once again, I'll largely just list the added files here, and link through to source control:

adam@DESKTOP-QV1A45U:/mnt/c/src/cfml-in-docker$ tree -a --dirsfirst -I "vendor|.git|.idea"
.
├── docker
│   ├── nginx
│   │   ├── root_home
│   │   │   ├── .gitignore
│   │   │   ├── .profile
│   │   │   └── .vimrc
│   │   ├── sites
│   │   │   └── default.conf
│   │   ├── Dockerfile
│   │   └── nginx.conf
│   └── [...]
├── public
│   └── gdayWorld.html
├── test
│   └── php
│       ├── NginxTest.php
│       └── [...]
├── var
│   └── log
│       └── nginx
│           ├── .gitkeep
│           ├── access.log
│           └── error.log
└── [...]

12 directories, 25 files
adam@DESKTOP-QV1A45U:/mnt/c/src/cfml-in-docker$

The contents of gdayWorld.html should be obvious from the test, but it's just:

G'day world!

OK so that was all stuff I've done a few times before now. Next… Lucee

It has a Lucee container which serves CFML code via its internal web server

I'm kinda guessing at this next case. I'm gonna need to have a Lucee container, this is a cert. And I recollect Adobe's ColdFusion CFML engine ships with an wee stubbed web server for dev use. I can't recall if Lucee does too. I'm assuming it does. You can see how prepared I am for all this: I've not even RTFMed about the Lucee Docker image on DockerHub yet (I did at least make sure there was one though ;-). The idea is that there's a two-step here: getting the Lucee container up and doing "something", and after that, wire it through from Nginx. But that's a separate case.

Right so this is all new to me, so I'll actually list the files I've created. First the test:

/** @testdox Tests Lucee is serving cfml */
class LuceeTest extends TestCase
{
    /** @testdox It serves gdayWorld.cfm as 200-OK on Lucee's internal web server */
    public function testReturns200OK()
    {
        $client = new Client(['base_uri' => 'http://cfml-in-docker.lucee:8888/']);

        $response = $client->get('gdayWorld.cfm');

        $this->assertEquals(200, $response->getStatusCode());
        $content = $response->getBody()->getContents();
        $this->assertMatchesRegularExpression("/^\\s*G'day world!\\s*$/", $content);
    }
}

It's the same as the HTML one except I'm hitting a different host, and on port 8888 (I have now done that RTFM I mentioned, and found the port Lucee serves on by default).

The Dockerfile is simple:

FROM lucee/lucee:5.3

RUN apt-get update
RUN apt-get install vim --yes

COPY ./root_home/.bashrc /root/.bashrc
COPY ./root_home/.vimrc /root/.vimrc

WORKDIR  /var/www

EXPOSE 8888

It's more complex than it needs to be as I always like vi installed in my containers because I inevitably need it (this is prescient as it turns out: I definitely did need it).

And the relevant bit from docker-compose.yml:

lucee:
    build:
        context: ./lucee
    volumes:
        - ../public:/var/www
        - ../var/log/tomcat:/usr/local/tomcat/log
        - ../var/log/lucee:/opt/lucee/web/logs
        - ./lucee/root_home:/root
    ports:
        - "8888:8888"
    stdin_open: true
    tty: true
    networks:
        backend:
            aliases:
                - cfml-in-docker.lucee

That's mostly just me mapping logging directories back to my host for convenience-sake.

Currently my test file - gdayWorld.cfm - is just plonked in the web root, which is not where one would normally put CFML files (except the application entry point file I mean), but it'll do for now:

<cfset message="G'day world!">
<cfoutput>#message#</cfoutput>

And that's it. After rebuilding my containers and running the tests, everything passes now:

root@a034afe670d4:/usr/share/cfml-in-docker# composer test
> vendor/bin/phpunit --testdox
PHPUnit 9.5.4 by Sebastian Bergmann and contributors.

Tests Lucee is serving cfml
It serves gdayWorld.cfm as 200-OK on Lucee's internal web server

Tests Nginx is serving html
It serves gdayWorld.html as 200-OK

Tests PHPUnit install
it self-tests PHPUnit

Time: 00:00.028, Memory: 6.00 MB

OK (3 tests, 5 assertions)
root@a034afe670d4:/usr/share/cfml-in-docker#

It proxies .cfm requests from Nginx to Lucee

OK so Lucee is working. Painless. Now I need to tell Nginx about it. I have NFI how to do that… I hope Google and/or Stack Overflow does.

After some googling, my recollection that some sort of connector was needed to run between the web server and the application server seems outdated, and all I need to do is use proxy_pass from Nginx to the address Lucee has configured Tomcat to listen on (Lucee runs atop of Tomcat: it's basically a Java Servlet). I can never remember the syntax for this, but fortunately Nando Breiter has documented it in article "Using Nginx With ColdFusion or Lucee". It's also reminded me a few other cases I need to test for, but first the baseline. Well actually first the test:

/** @testdox It proxies a CFM request to Lucee */
public function testCfmReturns200OK()
{
    $client = new Client(['base_uri' => 'http://cfml-in-docker.frontend/']);

    $response = $client->get('gdayWorld.cfm');

    $this->assertEquals(200, $response->getStatusCode());
    $content = $response->getBody()->getContents();
    $this->assertMatchesRegularExpression("/^\\s*G'day world!\\s*$/", $content);
}

This is the same as the previous one except I'm using the Nginx website's host, and on port 80. Also note I've changed the name of the host to be cfml-in-docker.frontend not cfml-in-docker.backend. This is cosmetic, and just to distinguish between references to stuff happening on the network within the containers (called backend), and addresses browsed from the public-facing websites.

The implementation for this case is simply this, in the website config default.conf:

location ~ \.(?:cfm|cfc) {
    proxy_pass  http://cfml-in-docker.lucee:8888$fastcgi_script_name;
}

Adding this and restarting Nginx has that test passing, as well as not interfering with any non-CFML requests (ie: the other Nginx tests still pass).

This config has some shortfalls though. Well I say "shortfalls". Basically I mean it doesn't work properly for a real-world situation. More test cases…

It passes query values to Lucee

The test demonstrates this:

/** @testdox It passes query values to Lucee */
public function testCfmReceivesQueryParameters()
{
    $client = new Client([
        'base_uri' => 'http://cfml-in-docker.frontend/',
        'http_errors' => false
    ]);

    $response = $client->get('queryTest.cfm?testParam=expectedValue');

    $this->assertEquals(200, $response->getStatusCode());
    $content = $response->getBody()->getContents();
    $this->assertSame("expectedValue", trim($content));
}

and queryTest.cfm is just this:

<cfoutput>#URL.testParam#</cfoutput>

If I run this test I get a failure because the 500 INTERNAL SERVER ERROR response from Lucee doesn't match the expected 200. This happens because Lucee can't see that param value. Because Nginx is not passing it. Easily fixed.

location ~ \.(?:cfm|cfc) {
    proxy_pass  http://cfml-in-docker.lucee:8888$fastcgi_script_name$is_args$args;
}

It passes the upstream remote address to Lucee

As it currently stands, Lucee will be receiving all requests as it they came from Nginx, rather than from whoever requested them. This is the nature of proxying, but we can work around this. First the test to set expectations:

/** @testdox It passes the upstream remote address to Lucee */
public function testLuceeReceivesCorrectRemoteAddr()
{
    $directClient = new Client([
        'base_uri' => 'http://cfml-in-docker.lucee:8888/',
        'http_errors' => false
    ]);
    $response = $directClient->get('remoteAddrTest.cfm');
    $expectedRemoteAddr = $response->getBody()->getContents();

    $proxiedClient = new Client([
        'base_uri' => 'http://cfml-in-docker.frontend/',
        'http_errors' => false
    ]);

    $testResponse = $proxiedClient->get('remoteAddrTest.cfm');

    $this->assertEquals(200, $testResponse->getStatusCode());
    $actualRemoteAddr = $testResponse->getBody()->getContents();
    $this->assertSame($expectedRemoteAddr, $actualRemoteAddr);
}

And remoteAddrTest.cfm is just this:

<cfoutput>#CGI.remote_addr#</cfoutput>

This is slightly more complicated than the previous tests, but only in that I can't know what the remote address is of the service running the test, because it could be "anything" (in reality inside these Docker containers, if they're brought up in the same order with the default bridging network, then it'll always be the same, but we don't want to break these tests if unrelated config should happen to change). The best way is to just check what the remote address is if we make the call directly to Lucee, and then expect that value if we make the same call via the Nginx proxy. As of now it fails because Lucee correctly sees the request as coming from the PHP container when we hit Lucee directly; but it sees the request as coming from the Nginx container when using Nginx's proxy. No surprise there. Fortunately Nando had the solution to this baked into his blog article already, so I can just copy and paste his work:

location ~ \.(?:cfm|cfc) {
    proxy_http_version  1.1;
    proxy_set_header    Connection "";
    proxy_set_header    Host                $host;
    proxy_set_header    X-Forwarded-Host    $host;
    proxy_set_header    X-Forwarded-Server  $host;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;     ## CGI.REMOTE_ADDR
    proxy_set_header    X-Forwarded-Proto   $scheme;                        ## CGI.SERVER_PORT_SECURE
    proxy_set_header    X-Real-IP           $remote_addr;
    expires             epoch;

    proxy_pass  http://cfml-in-docker.lucee:8888$fastcgi_script_name$is_args$args;
}

And if I restart Nginx: all good. One more issue to deal with…

It passes URL path_info to Lucee correctly

Something too few people know about, is there's an optional part of a URL between the script name and the query: path info. An example is: http://example.com/script/name/path/document.html/additional/path/info?queryParam=paramValue. That path is nothing to do with the script to be executed or where it's located, it's just… some extra pathing information for the script to do something with. It's seldom used, but it's part of the spec (RFC-3875, section 4.1.5). The spec says this:

The PATH_INFO variable specifies a path to be interpreted by the CGI script. It identifies the resource or sub-resource to be returned by the CGI script, and is derived from the portion of the URI path hierarchy following the part that identifies the script itself.

Anyway, from what I could see of what I have in the Nginx config, I suspected that we're not passing that on to Lucee, so its CGI.path_info value would be blank. A test for this is easy, and much the same as the earlier ones:

/** @testdox It passes URL path_info to Lucee correctly */
public function testLuceeReceivesPathInfo()
{
    $client = new Client([
        'base_uri' => 'http://cfml-in-docker.frontend/',
        'http_errors' => false
    ]);

    $response = $client->get('pathInfoTest.cfm/additional/path/info/');

    $this->assertEquals(200, $response->getStatusCode());
    $content = $response->getBody()->getContents();
    $this->assertSame("/additional/path/info/", trim($content));
}

And pathInfoTest.cfm is similarly familiar:

<cfoutput>#CGI.path_info#</cfoutput>

And as I predicted (although as we'll see below, not for the reasons I thought!) the test errors:

> vendor/bin/phpunit --testdox '--filter=testLuceeReceivesPathInfo'
PHPUnit 9.5.4 by Sebastian Bergmann and contributors.

Tests Nginx proxies CFML requests to Lucee
It passes URL path_info to Lucee correctly
  
   Failed asserting that 404 matches expected 200.
  
   /usr/share/cfml-in-docker/test/php/NginxProxyToLuceeTest.php:71
  

Time: 00:00.090, Memory: 8.00 MB


FAILURES!
Tests: 1, Assertions: 1, Failures: 1.
Script vendor/bin/phpunit --testdox handling the test event returned with error code 1
root@29840662fdf9:/usr/share/cfml-in-docker#

At this point I disappeared down a rabbit hole of irritation, as detailed in article "Repro for Lucee weirdness". There are two bottom lines to this:

  1. For reasons best known to [someone other than me], Lucee only handles path_info on requests to index.cfm, but not to any other .cfm file! This can be shown by changing that test by renaming pathInfoTest.cfm to index.cfm, and calling that instead.
  2. Actually Nginx already handles it correctly anyhow. In that the value is passed on already, and I don't need to do anything extra to make it work (as far as Nginx is concerned, anyhow).

I can fix the situation for pathInfoTest.cfm if I hack Lucee's web.xml file (this is down at line 4643):

<servlet-mapping>
    <servlet-name>CFMLServlet</servlet-name>
    <url-pattern>*.cfm</url-pattern>
    <url-pattern>*.cfml</url-pattern>
    <url-pattern>*.cfc</url-pattern>
    <url-pattern>/index.cfm/*</url-pattern>
    <url-pattern>/index.cfc/*</url-pattern>
    <url-pattern>/index.cfml/*</url-pattern>
</servlet-mapping>

I could slap a special mapping for it in there. But that's a daft way to deal with this. I'm going to just mark that test as "incomplete", and move on.

Thanks to Pete Freitag, Adam Tuttle, Zac Spitzer and Sean Corfield for putting me on the right direction for working out this particular "WTF, Lucee?" episode.


Speaking of "moving on", I said I'd get the code this far, but only progress onto the more CFML-oriented stuff if I still had will to live. Well Lucee has eroded that for now, so I'll get back to that part later, when I've stopped shaking my fist at the screen.

NB: this has become part of a series of articles, as things get more complicated, and require more effort on my part to achieve my end goal: Lucee/CFWheels/Docker series.

Righto.

--
Adam

Wednesday, 6 January 2021

Creating a web site with Vue.js, Nginx, Symfony on PHP8 & MariaDB running in Docker containers - Part 1: Intro & Nginx

G'day:

Please note that I initially intended this to be a single article, but by the time I had finished the first two sections, it was way too long for a single read, so I've split it into the following sections, each as their own article:

  1. Intro / Nginx (this article)
  2. PHP
  3. PHPUnit
  4. Tweaks I made to my Bash environment in my Docker containers
  5. MariaDB
  6. Installing Symfony
  7. Using Symfony
  8. Testing a simple web page built with Vue.js using Mocha, Chai and Puppeteer
  9. I mess up how I configure my Docker containers
  10. An article about moving files and changing configuration
  11. Setting up a Vue.js project and integrating some existing code into it
  12. Unit testing Vue.js components

I've already written everything down to and including "Using Symfony", and will release those over the coming days. Originally I was gonna finish the whole series before releasing anything, because I know what I'm like: there's a chance I'll get bored and not finish the series, so would be a bit rubbish to release the first n parts but not finish it. But then it occurred to me that that approach is not very agile, plus if I actually release what I've done, I'm probably more likely to see it through to completion. Each article is stand-alone anyhow, for the most part. I appreciate my mates Dara McGann and Brian Sadler also giving me a nudge on this.

Intro

This "Creating a web site with Vue.js, Nginx, Symfony on PHP8 & MariaDB running in Docker containers" exercise should be hilarious (at my expense, I mean), cos I've never touched Vue.js, only know enough about Nginx to get myself into trouble (and sometimes - but not always - back out of trouble again), never touched Symfony really. or PHP8 yet. And still a noob with Docker, and all this requires me to work with docker-compose, which I have not ever touched until about 10min ago. So I'll warn you up front that these articles just logs my journey through all this stuff, and any knowledge I am applying is coming to me via googling and reading stuff on Stack Overflow (etc), about 5min before I put it into practice. Nothing here is an expert's view, and no promises as to whether I'm doing anything correctly. Not sure why yer reading it, actually.

The motivation behind this work is multi-fold:

  • my client-side exposure is very out of date. I have not written any front-end production code for… blimey five years probably.
  • Accordingly I have not touched any of the new fancy JS frameworks, so figured I should take a look at Vue.js.
  • It'll need some sort of back-end. I'd usually use Silex for this, but it's end-of-life now, and the recommendation is to use Symfony instead. I've done the smallest amount on Symfony (supervising other devs whilst they do the work, basically), so this is an excuse to do some actual work with it.
  • PHP8's just been released so I'll use that, although I can't see it'll make any difference from 7.x for the purposes of what I'll be doing.
  • I've messed around with individual containers a bit in Docker now, but it's all been very ad hoc. This is a good excuse to tie some stuff together with Dockerfiles and docker-compose.yml.

To be completely frank, I am unenthused about this concept of "full stack" development. In my exposure to full-stack devs, what it means is instead of being good at one thing; they're very bloody ordinary at a bunch of things ("jack of all trades; masters of none"). I understand it might be useful in smaller shops, I guess. I've always been in the position that wherever I've worked there's been enough bods to have dedicated people in each role. I'm not saying there're no people who are good at an entire stack, but the bulk of people seem not to be. It strikes me as being the enterprise of people without the discipline to stick to one thing. The flipside of this is that I've been overlooked for two roles recently because they want people who do both client-side and back-end dev, and I will always admit to being weak on the front end. However I'm also pragmatic… if I need to improve my exposure here... so be it.

Right. Off we go. I'm gonna work through this in a stepped fashion; getting each component working (to a "G'day World" degree) before integrating the next component.

Before I start actually, I need to point out I am basing my approach on the excellent article "Symfony 5 development with Docker", by Martin Pham. I just found it on Google, but it's covering much of the same territory I need to cover.

Directory structure

I'm following Martin's lead here for the most part. The key bit is putting all the docker-related files into their own directory in my app directory:

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise$ tree -F --dirsfirst -L 2 .
.
├── docker/
│   ├── nginx/
│   └── docker-compose.yml*
├── log/
│   └── nginx/
├── public/
│   └── gdayWorld.html*
├── LICENSE*
└── README.md*

5 directories, 4 files
adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise$

(This was taken after I did the Nginx config, below, but it better shows what goes where if I include the Nginx stuff now).

Oh Martin also has the log directory as logs on the host and log in the containers. That bit me on the bum the first time I tried this (see further down), so I'm sticking with a uniform log throughout.

Nginx

Firstly I'm just going to get Nginx serving gdayWorld.html on localhost:80. I'm not yet going to worry about PHP or Symfony or anything like that.

I've largely copied Martin's docker/nginx/Dockerfile:

FROM nginx:alpine
WORKDIR /usr/share/nginx/
CMD ["nginx"]
EXPOSE 80

The only difference is I've changed the WORKDIR value from /var/www to /usr/share/nginx/ which seems to be where Nginx would naturally put its files. Martin's using /var/www because "the same directory with PHP-FPM since we’re gonna share this with a mounted volume" (that's from his article). However I will specifically not be doing that. I will be keeping the website directory and the application source code separate, as one usually would with a modern PHP app. Only the "entry-point" index.php will be in the web root; the rest will be in the adjacent src directory. In the context of this file, it doesn't matter here, but the same directory is referenced later where it does matter, and I want to keep things consistent. Also this seems to be a reference to the Nginx app directory really, not anything related to the web root, per se?

And again, the docker/nginx/nginx.conf file is lifted from Martin's article:

user  nginx;
worker_processes  4;
daemon off;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    access_log  /var/log/nginx/access.log;
    sendfile        on;
    keepalive_timeout  65;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-available/*.conf;
}

And the separate site config file, docker/nginx/sites/default.conf:

server {
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;

    server_name localhost;
    root /usr/share/fullstackExercise/public;
    index index.html;

    location / {
         try_files $uri $uri/;
    }

    location ~ /\.ht {
        deny all;
    }
}

One difference here is that I'm setting the web root to be /usr/share/fullstackExercise/public, rather than a subdirectory of the Nginx application directory as Martin had it. /public is where the app's web-browseable files will be home (Symfony's location for these is that directory, relative to the application root).

I don't need the PHP stuff in this file yet, so I've removed it for now. We're only interested in serving HTML at the moment.

Lastly the docker-compose.yml file is cut down from the one in the article to just handle Nginx for now:

version: '3'

services:
    nginx:
        build:
            context: ./nginx
        volumes:
            - ../public:/usr/share/fullstackExercise/public
            - ./nginx/nginx.conf:/etc/nginx/nginx.conf
            - ./nginx/sites/:/etc/nginx/sites-available
            - ./nginx/conf.d/:/etc/nginx/conf.d
            - ../log:/var/log
        ports:
            - "80:80"

As mentioned above, the only significant change here is that that first volume has been changed from being ../src:/var/www (PHP src directory) to be ../public:/usr/share/fullstackExercise/public again, as per above.

Oh! One last file! public/gdayWorld.html:

<!doctype html>

<html lang="en">
<head>
    <meta charset="utf-8">

    <title>G'day world</title>
</head>

<body>
<h1>G'day world</h1>
<p>G'day world</p>
</body>
</html>

We should now be… good to go. Let's try it…

adam@DESKTOP-QV1A45U:~$ cd /mnt/c/src/fullstackExercise/
adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise$ cd docker/
adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker-compose up
Building nginx
Step 1/4 : FROM nginx:alpine
---> 98ab35023fd6
Step 2/4 : WORKDIR /usr/share/nginx/
---> Running in 2223848549f7
Removing intermediate container 2223848549f7
---> 6f9bba05771d

Step 3/4 : CMD ["nginx"] ---> Running in 46c32d7862a7
Removing intermediate container 46c32d7862a7
---> 99cd0a9bb3fe
Step 4/4 : EXPOSE 80
---> Running in 62cb63572cab
Removing intermediate container 62cb63572cab
---> 7ae273be9176

Successfully built 7ae273be9176
Successfully tagged docker_nginx:latest
WARNING: Image for service nginx was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating docker_nginx_1 ... done
Attaching to docker_nginx_1
nginx_1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
nginx_1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
nginx_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
nginx_1 | 10-listen-on-ipv6-by-default.sh: error: /etc/nginx/conf.d/default.conf is not a file or does not exist
nginx_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
nginx_1 | /docker-entrypoint.sh: Configuration complete; ready for start up

This is promising. So over to a web browser on my host machine:

Cool!

Umm… OK… that didn't actually go as smooth for me as I make it sound there. The first time around - because I did not notice that discrepancy between log/logs in Martin's article - I ended up referencing a non-existent path in docker-compose.yml, and the docker compose call failed:

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker-compose up --build Creating network "docker_default" with the default driver Building nginx Step 1/4 : FROM nginx:alpine [...etc...] Successfully built 7ae273be9176 [...etc...] nginx_1 | /docker-entrypoint.sh: Configuration complete; ready for start up nginx_1 | nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (2: No such file or directory) nginx_1 | 2020/12/05 19:10:50 [emerg] 1#1: open() "/var/log/nginx/error.log" failed (2: No such file or directory) docker_nginx_1 exited with code 1 adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$

That was simply cos I had this in my docker-compose.yml:

- ../logs:/var/log

Instead of this:

- ../log:/var/log

(In the file system, the directory is log). I very quickly spotted this and fixed it, and tried to back-out what I had done:

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker-compose down
Traceback (most recent call last):
File "bin/docker-compose", line 3, in <module>
File "compose/cli/main.py", line 67, in main
File "compose/cli/main.py", line 123, in perform_command
File "compose/cli/command.py", line 69, in project_from_options
File "compose/cli/command.py", line 125, in get_project
File "compose/cli/command.py", line 184, in get_project_name
File "posixpath.py", line 383, in abspath
FileNotFoundError: [Errno 2] No such file or directory
[2052] Failed to execute script docker-compose
adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$

Try as I might I could not work out what file was missing. Nice one, btw, docker-compose for not actually saying what file-read didn't work. Amateur hour there.

After way too long of reading bug reports on Docker and various Stack Overflow Q&A, I spotted a solution ("Failed to execute script docker-compose"). The person answering just suggested "make sure Docker Desktop is running". That sounded off to me (of course it's running), but a comment on the answer made me look again: "I read this, thought what an answer for dummies, and then realised a few minutes later it was the answer I needed! DOH". I I went "oh all right then", and restarted Docker Desktop, dropped out of my bash instance and into a new one, and now:

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker-compose down
Removing docker_nginx_1 ... done
Removing network docker_default
adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$

Then I could just go docker-compose up --build, and the thing worked properly.

It's perhaps important to note I can replicate this at will, so this is just a workaround for some issue with Docker, but at least it got me moving forward. As I've been working through the rest of the articles in this series, I am getting this issue a lot: at least once a day. So it's not just caused by paths being wrong as per above. It seems to kick off quite often if I have changed docker-compose.yml to add/remove sections of config, or sometimes if I change files via the host machine file system (ie: my Windows desktop environment), rather than via the Ubuntu/WSL file system. I've not been able to tie it down to one particular thing. Something to look out for though.

Now to integrate PHP into the mix, see the next part of this series: Part 2: PHP.

Righto.

--
Adam

Saturday, 14 January 2017

Incredibly: a reader asks me for help with PHP & Nginx

G'day:
Jesus fuck, you lot. Don't ask me shit about systems support / server application config / all that godawful shit that should be consigned to the Systems Support Team (sorry to my mates in this role, but I fucking hate it, and became a dev so I didn't have to do it any more).

Right so ages ago I wrote an article "PHP: getting PHP 5 and PHP 7 running side by side on the same machine (a better way)", and someone recently asked me how do do the same on Nginx. Well it's "slow news night" here in my life: I'm just at the pub in Galway passing time by getting pissed on Guinness (I'm on me fifth pint, and even I can tell the writing here is reflecting that) and writing blog articles, so I had a look at it. It's pretty easy, as it turns out.

Firstly: I am no expert on Nginx. I don't like web servers. I try not to ever have to use them. I know enough about IIS to know it's a pain in the arse but I can do what I need if I have to; and I know enough about Apache to get things working. I don't even know why Nginx exists. I'm guessing the reason is "to annoy Adam, cos it's just one more bloody thing he'll need to know about at some point". Sigh. I got PHP working on Nginx once before ("PHP: getting my dev environment running on Nginx instead of Apache"), and other than that have converted some rewrite rules from Apache to Nginx (man: does Nginx suck compared to Apache for those!).

This is a different laptop from the one I did that other exercise on, so whilst I had Nginx on here (for the rewrite exercise, which was work-related and this is my work laptop), I decided to start from scratch. I deleted what I had installed (where for Nginx "installed" means "unzipped").

Here's what I did:
  1. grabbed the latest Windows Nginx download from their site. I just googled "nginx windows download" to find that.
  2. Unzipped it. Relocated it to my apps dir: c:\apps\nginx
  3. Stopped Apache (which listens on port 80), and instead started Nginx (just run nginx.exe from the dir above). By default it listens on localhost and port 80.
  4. Browse to http://localhost/ and verified the Nginx default index.html page was served. OK, so it works as a baseline. Always test the baseline before proceeding with any customisations of things.
  5. Set-up a coupla test hosts in my hosts file (C:\windows\system32\drivers\etc\hosts... make sure to start yer text editor as an admin, otherwise you won't be able to save it):
    127.0.0.1 php5.nginx.local
    127.0.0.1 php7.nginx.local

    I foresaw that to run both PHP5 and PHP7, Nginx is gonna need to differentiate between which one wants to use, and using different host names seemed to make sense and be easy.
  6. I edited my nginx.conf file (in the conf subdir of the one above) to know about these two hosts:
    http {
        # [...]
    
        server {
            listen       8800;
            server_name  php5.nginx.local;
            root   c:/src/php/php.local/www;
    
            # [...]
        }
    
        server {
            listen       8800;
            server_name  php7.nginx.local;
            root   c:/src/php/php.local/www;
    
            # [...]
        }
    }
    

    Where I've elided stuff, it's the same as it was before. The chief considerations here are:
    • having a server for both PHP5 and PHP7;
    • having them listen on each of those two new hosts I set up;
    • setting the root to be where my PHP code is. This is the same for both in this case, as I want to serve exactly the same code via both 5 and 7;
    • oh and I'm listening on port 8800 as I don't want Nginx to interfere with my normal Apache install (I'll be sticking with Apache after this experiment, thanks).
    Note that this config will still not serve PHP, but it'll at least run.
  7. When one runs Nginx from the console it hogs the prompt, so to stop it one needs to run another console and call nginx -s stop. We need to do this to test the config changes. So I did that, and used the other console to start it again.
  8. I browse to each of http://php5.nginx.local:8800 and http://php7.nginx.local:8800 to test they were working. They were running, but giving a 403 cos I didn't have an index.html in that directory, nor did I have directory browsing switched on (which for my test code I do, as it makes finding stuff easier).
  9. I told Nginx to allow directory browsing for each of the server configs:
    location / {
        index  index.html index.htm;
        autoindex on;
    }
    

    Do not do this in production. Well: don't do anything I say in production.
  10. I cycled Nginx again and tested both hosts:

    Index of /


    ../
    code-coverage-reports/                             15-Nov-2016 13:33                   -
    community/                                         30-Jun-2016 11:47                   -
    experiment/                                        15-Nov-2016 08:47                   -
    library/                                           30-Jun-2016 11:47                   -
    stackoverflow/                                     30-Jun-2016 11:47                   -
    deleteme.php                                       30-Jun-2016 11:47                 233
    gdayWorld.html                                     30-Jun-2016 11:47                  11
    gdayWorld.php                                      30-Jun-2016 11:47                  50
    phpinfo.php                                        30-Jun-2016 11:47                  21
    utf8.html                                          21-Dec-2016 08:21                 133
    


    So that's all good except for me not having deleted that file that perhaps I meant to, a while back ;-)
  11. Next I just followed the instructions from me other blog article, getting PHP to listen out to traffic coming in from Nginx (the code below is in a batch file):
    start C:\bin\RunHiddenConsole.exe C:\apps\php\5\5\php-cgi.exe -b 127.0.0.1:8550 start C:\bin\RunHiddenConsole.exe C:\apps\php\7\1\php-cgi.exe -b 127.0.0.1:8710
    Note that each of them is listening on a different port.
  12. And then tell Nginx to pass PHP requests across to PHP:
    server {
        # [...]
        location ~ \.php$ {
            fastcgi_pass   127.0.0.1:8710;
            fastcgi_index  index.php;
            fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
            include        fastcgi_params;
        }
        # [...]
    }
    

    That's the one for PHP7, but the PHP5 one is the same except for the port.
  13. I restart Nginx again, and this time hit phpinfo.php on each host:
    {i'd show you proof, but BlogSpot is refusing to let me put two images inline in an <ol> list, it seems. Trust me... it works).
  14. Hurrah!
  15. The last thing I tried to do is to wrap my batch file that starts Nginx into a service, using RunAsService.exe, but this is my work laptop and it's locked-down too tight for me to run that. But... well... the batch file works.
So, anyway... that's it. That's what one needs to do to get Nginx serving two different versions of PHP. My will to live has been sapped by even having to think about this sort of shit, so I'm going back to my Guinness (two pint further ahead than I was when I started this exercise).

Sorry this one is a bit scrappy but... well... I'm drunk.

Righto.

--
Adam