Showing posts with label Frameworks. Show all posts
Showing posts with label Frameworks. Show all posts

Tuesday 28 March 2023

PHP / Symfony: working through "Symfony: The Fast Track", part 2: creating a controller (eventually)

G'day:

Sit. rep.

OK so last time (PHP / Symfony: working through "Symfony: The Fast Track", part 1: preparation and pre-requisites (and not actually any Symfony!)) I didn't make much Symfony progress, but I got my ducks (and my Docker containers) in a row, ready to start. I had worked through "Symfony: The Fast Track › Checking your Work Environment", and prepped as much as I could. Running symfony book:check-requirements yields this:

root:/var/www# symfony book:check-requirements
[OK] Git installed
[OK] PHP installed version 8.2.4 (/usr/local/bin/php)
[OK] PHP extension "xsl" installed - required
[OK] PHP extension "tokenizer" installed - required
[OK] PHP extension "xml" installed - required
[OK] PHP extension "redis" installed - optional - needed only for chapter 31
[OK] PHP extension "amqp" installed - optional - needed only for chapter 32
[OK] PHP extension "json" installed - required
[OK] PHP extension "session" installed - required
[OK] PHP extension "curl" installed - optional - needed only for chapter 17 (Panther)
[OK] PHP extension "pdo_pgsql" installed - required
[OK] PHP extension "mbstring" installed - required
[OK] PHP extension "openssl" installed - required
[OK] PHP extension "sodium" installed - required
[OK] PHP extension "zip" installed - optional - needed only for chapter 17 (Panther)
[OK] PHP extension "gd" installed - optional - needed only for chapter 23 (Imagine)
[OK] PHP extension "ctype" installed - required
[OK] PHP extension "intl" installed - required
[OK] Composer installed
[KO] Cannot find Docker, please install it https://www.docker.com/get-started
[KO] Cannot find Docker Compose, please install it https://docs.docker.com/compose/install/
[KO] Cannot find the npm package manager, please install it https://www.npmjs.com/
  
You should fix the reported issues before starting reading the book.
root:/var/www#

I'm not bothered about the last three as I'm already in a Docker container, and Node.js is running in a separate container, pointing at the same application directory. Thinking about it, it's quite odd they require using Docker, but it doesn't occur to them to run the app itself in a container. I hope my approach doesn't some how unstick me somewhere along the way.

Right. Let's crack on with it. Next page…


Introducing the project

Here they discuss "learning is doing", whilst not actually doing anything. But there's a diagram of where we get to in this project:

(this is no doubt © Symfony. I am using it in the context of critically evaluating their book, so I think this is "fair use". If someone from Symfony wishes to disagree: hit me up).

That's ambitious! Cool.

The also list how little code is needed to achieve this:

  • 20 PHP classes under src/ for the website;
  • 550 PHP Logical Lines of Code (LLOC) as reported by PHPLOC;
  • 40 lines of configuration tweaks in 3 files (via attributes and YAML), mainly to configure the backend design;
  • 20 lines of development infrastructure configuration (Docker);
  • 100 lines of production infrastructure configuration (Platform.sh);
  • 5 explicit environment variables.

ibid.

What's missing from that list? No tests. That's disappointing in 2023. I realise this book is about Symfony, but no code exercise in 2023 should normalise "no tests" as an approach. I will try to take a TDD approach with everything, if it seems viable (which it should be?).

There's instructions how to download / install / start the final working app. This seems a bit pre to me. I want to write the thing, not just look at it. "Learning is doing", I read somewhere, recently. I hope this step is not integral to the rest of the process cos I ain't doing it.

(Ah OK, a few paras on the author alludes to DIYing the code, so I guess this is just for reference. Good-o).


Going from Zero to Production

OK, so I now need to create a new Symfony app. This is done via a Symfony helper:

symfony new guestbook --version=6.2 --php=8.1 --webapp --docker --cloud

I know from past experience ("Part 6: Installing Symfony" and "Symfony: installing in my PHP8 container (for a second time, as it turns out)") that Symfony whines if I try to create a new project in a directory that already has stuff in it, so I'll do what I did last time: install it in a temp directory then migrate stuff back into my project directory. I will also lift the test I used last time to verify it's working OK. The code is in that second article I linked to above, I'll not repeat it here, but will link to it in the repo once I've tagged it.

I also know that Symfony uses a bad namespace for its apps (Symfony: getting rid of App namespace and using a well-formed one), so I'm going to sort that out straight away too. I'll not repeat any of the steps to make these changes and get that test passing, unless there's anything different I need to do this time. It's well-documented in that earlier article.

Bear with me for a bit. Back soon …

I ran this in a shell within my PHP container:

root:/var/www# cd /var/tmp
root:/var/tmp# symfony new SymfonyFastTrack --version=6.2 --php=8.2 --webapp --docker --cloud

I then copied the ensuing /var/tmp/SymfonyFastTrack (source) directory down to my host machine and did a file compare on that directory and my existing SymfonyFastTrack (destination) directory, doing the following:

  • Where the source directory had files not in destination, I simply copied them across.
  • Where there were clear additions to a file for the new Symfony app, I just merged it (sometimes by hand) into my own version of the file. EG this diff of the two versions of composer.json:

  • There was a docker-compose.yml and docker-compose-override.yml plonked into the root directory. These added containers for a stub/fake STMP server and a "BlackFire" server (whatever that is). I lifted the relevant bits of that out and put it in my own docker/docker-compose.yml file and deleted the ones in the root.
  • I followed the instructions about setting up a Blackfire account, got the various keys I needed and stuck them into a docker/blackfire/envVars.private (not source controlled) file which I got the docker/docker-compose.yml to load into the Blackfire container.
  • Symfony hard-codes its APP_SECRET value into the codebase. If it's supposed to be a secret then treat it that way: I lifted it out and set an env var for it (via docker/php/envVars.private (again, not source controlled)).
  • I set Symfony's DATABASE_URL as an environment variable instead of Symfony's config files. No reason to really, except every component of it (host, user, password, DB etc) came from adjacent environment variables, so I figured I'd keep it all in one place.
  • The /config/packages/doctrine.yaml file was setting a db_suffix value which had the effect of the Connection object trying to connect to a DB called db1test (it's just db1. I have no reason to add suffixes to my DB name because of the environment). I'm not sure why this would be out-of-the-box behaviour but… I suspect this is just one more example of me going "please stop trying to be helpful, Symfony", and I should get used to it. Oh yeah, this surfaced because the tests put the APP_ENV into test mode, and I have integration tests (see below) hitting the DB. Yay for tests.

I also wrote a few tests; or, as I said above: lifted them from another project ([during proofreading]: I also said I wasn't going to repeat them here… hrm…):

  • To test Symfony came up OK:

    /** @testdox It serves the Symfony welcome page after installation */
    public function testSymfonyWelcomeScreenDisplays()
    {
        $ch = curl_init();
        curl_setopt_array($ch, [
            CURLOPT_URL => "http://host.docker.internal:8062/",
            CURLOPT_RETURNTRANSFER => 1
        ]);
        $response = curl_exec($ch);
    
        if (curl_error($ch)) {
            $this->fail(sprintf("curl failed with [%s]", curl_error($ch)));
        }
        $this->assertEquals(Response::HTTP_NOT_FOUND, curl_getinfo($ch, CURLINFO_HTTP_CODE));
        $document = new \DOMDocument();
    
        $document->loadHTML($response, LIBXML_NOWARNING | LIBXML_NOERROR);
    
        $xpathDocument = new \DOMXPath($document);
    
        $hasTitle = $xpathDocument->query('/html/head/title[text() = "Welcome to Symfony!"]');
        $this->assertCount(1, $hasTitle, "Could not find title 'Welcome to Symfony!'");
    }
    
  • To test its console also runs OK:

    /** @testdox It can run the Symfony console in a shell */
    public function testSymfonyConsoleRuns()
    {
        $appRootDir = dirname(__DIR__, 3);
    
        exec("{$appRootDir}/bin/console --help", $output, $returnCode);
    
        $this->assertEquals(0, $returnCode);
        $this->assertNotEmpty($output);
    }
    
  • And to test that DATABASE_URL got loaded into the Connection object in Symfony's DI container:
    /** @testdox It has configured the Connection in the container with the correct DATABASE_URL */
    public function testContainerConnection()
    {
        $container = $this->getContainer();
        $connection = $container->get("database_connection");
        $result = $connection->executeQuery("SELECT version() AS version");
    
        $this->assertStringStartsWith("PostgreSQL 15", $result->fetchOne());
    }
    

I had the first two tests in there before I started the app installation process - a nod to TDD - but the third one I wrote after the fact to make sure my messing around had worked.

All my tests still/now pass, so I'm ready for the next step…

The rest of the page discussing pushing the "nothing" I so far have to production, under the guise of "deploy early and often". I have no aspirations to ship this to production, it's just a learning exercise. I guess there could be vagaries of deploying in "Production mode", but there's not even enough here yet to worry about that I think. I'll skip this for now.


Adopting a Methodology

A page dedicated to reminding one to commit one's work. I'm beginning to think the author was being paid by the word ;-)


Troubleshooting Problems

This page has some stuff about the debugging bumpf Symfony has at the bottom of the page (in dev mode wth debug on):

Handy.

It also discusses different environment modes. EG in dev mode the default first-install index page 404s with:

Switch to prod mode, and one gets this instead:

This is most easily done in .env:

# In all environments, the following files are loaded if they exist,
# the latter taking precedence over the former:
#
#  * .env                contains default values for the environment variables needed by the app
#  * .env.local          uncommitted file with local overrides
#  * .env.$APP_ENV       committed environment-specific defaults
#  * .env.$APP_ENV.local uncommitted environment-specific overrides
#
# Real environment variables win over .env files.
#
# …
APP_ENV=devprod

I left the guidance about the order of env file loading there, and I like how real env vars trump them all.

I tried to write functional tests loading the app in different modes and checking their output, but this won't work:

/** @testdox Testing variations of APP_ENV */
class AppEnvTest extends WebTestCase
{
    /** @testdox it shows the Welcome page for the / 404 if in dev mode */
    public function testDevMode()
    {
        $client = static::createClient([
            'environment' => 'dev',
            'debug' => false
        ]);

        $client->request('GET', '/');

        $this->assertResponseStatusCodeSame(Response::HTTP_NOT_FOUND);
        $this->assertSelectorTextContains('h1', 'Welcome to Symfony');
    }

    /** @testdox it shows a plain error for / 404 if in prod mode */
    public function testProdMode()
    {
        $client = static::createClient([
            'environment' => 'prod',
            'debug' => false
        ]);

        $client->request('GET', '/');

        $this->assertResponseStatusCodeSame(Response::HTTP_NOT_FOUND);
        $this->assertSelectorTextContains('h1', 'Oops! An Error Occurred');
    }
}

I got an initially enigmatic error:

LogicException: You cannot create the client used in functional tests if the "framework.test" config is not set to true.

But I tracked it back through the code and found this:

protected static function getContainer(): ContainerInterface
{
    if (!static::$booted) {
        static::bootKernel();
    }

    try {
        return self::$kernel->getContainer()->get('test.service_container');
    } catch (ServiceNotFoundException $e) {
        throw new \LogicException('Could not find service "test.service_container". Try updating the "framework.test" config to "true".', 0, $e);
    }
}

This is called as part of my createClient call in the test. Bottom line: the only mode one can do functional tests in is in test mode. I'm not entirely on board with this, if I'm honest. This is not Symfony application code; this is code they have specifically written for testing. If I wanna create a test client in other modes, that ought to be up to me, not Symfony. A problem I have with these large frameworks that tout themselves as being "opinionated" (as if this is a good thing) is that they start second-guessing stuff, and every second-guess they make closes a door on a situation that didn't occur to them. This is why I much preferred simpler frameworks like Silex (RIP), which just did routing, DI and exposed a coupla other things, but otherwise got out of my way. Never mind though, it's not a biggie.


Creating a Controller

Cool. Maybe I'll be able to write some code now.

Not yet.

They show some of the symfony console stuff which is very… thorough. Here's the list function showing all the options under the make functioanlity:

root:/var/www# symfony console list make
Symfony 6.2.7 (env: dev, debug: true) #StandWithUkraine https://sf.to/ukraine

Usage:
  command [options] [arguments]

Options:
  -h, --help            Display help for the given command. When no command is given display help for the list command
  -q, --quiet           Do not output any message
  -V, --version         Display this application version
      --ansi|--no-ansi  Force (or disable --no-ansi) ANSI output
  -n, --no-interaction  Do not ask any interactive question
  -e, --env=ENV         The Environment name. [default: "dev"]
      --no-debug        Switch off debug mode.
  -v|vv|vvv, --verbose  Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

Available commands for the "make" namespace:
  make:auth                   Creates a Guard authenticator of different flavors
  make:command                Creates a new console command class
  make:controller             Creates a new controller class
  make:crud                   Creates CRUD for Doctrine entity class
  make:docker:database        Adds a database container to your docker-compose.yaml file
  make:entity                 Creates or updates a Doctrine entity class, and optionally an API Platform resource
  make:fixtures               Creates a new class to load Doctrine fixtures
  make:form                   Creates a new form class
  make:functional-test        Creates a new test class
  make:message                Creates a new message and handler
  make:messenger-middleware   Creates a new messenger middleware
  make:migration              Creates a new migration based on database changes
  make:registration-form      Creates a new registration form system
  make:reset-password         Create controller, entity, and repositories for use with symfonycasts/reset-password-bundle
  make:serializer:encoder     Creates a new serializer encoder class
  make:serializer:normalizer  Creates a new serializer normalizer class
  make:stimulus-controller    Creates a new Stimulus controller
  make:subscriber             Creates a new event subscriber class
  make:test                   [make:unit-test|make:functional-test] Creates a new test class
  make:twig-component         Creates a twig (or live) component
  make:twig-extension         Creates a new Twig extension with its runtime class
  make:unit-test              Creates a new test class
  make:user                   Creates a new security user class
  make:validator              Creates a new validator and constraint class
  make:voter                  Creates a new security voter class
root:/var/www#
  

I'm buggered if I know what any of that means, but… yeah cool. All that stuff.

It looks like I'm going to be creating my controller with a wziard instead of just… creating/editing some files. I guess it saves me copying and pasting some boilerplate. Let's see what it does:

root:/var/www# symfony console make:controller ConferenceController
 ! [NOTE] It looks like your app may be using a namespace other than "App".
 !
 !        To configure this and make your life easier,
          see: https://symfony.com/doc/current/bundles/SymfonyMakerBundle/index.html#configuration


In Generator.php line 62:

Could not determine where to locate the new class "App\Controller\ConferenceController", maybe try with a full namespace like "\My\Full\Namespace\ConferenceController"
make:controller [--no-template] [--] [<controller-class>] root:/var/www#

That's actually quite helpful, cool.

Looking at those docs, I need to create config/packages/dev/maker.yaml:

# config/packages/dev/maker.yaml
# create this file if you need to configure anything
maker:
    # tell MakerBundle that all of your classes live in an
    # Acme namespace, instead of the default App
    # (e.g. Acme\Entity\Article, Acme\Command\MyCommand, etc)
    root_namespace: 'adamcameron\symfonythefasttrack'

Let's see if that helps.

After I saw a post on Stack Overflow advising that after adding the file I also needed to clear the cache, it all worked fine:

root@4b91bad4b422:/var/www# symfony console make:controller ConferenceController

 created: src/Controller/ConferenceController.php
 created: templates/conference/index.html.twig


Success!
Next: Open your new controller class and add some pages! root@4b91bad4b422:/var/www#

It just occurs to me that I've let down Team TDD a bit here. I shoulda had a test ready for this lot! I should have read ahead to see what I was gonna end up with, when running this code. I've written a test before I ran the code though:

/** @testdox Tests the functionality of ConferenceController */
class ConferenceControllerTest extends WebTestCase
{
    /** @testdox The index action returns a successful response */
    public function testIndex()
    {
        $client = static::createClient();
        $client->request("GET", "/conference");

        $this->assertResponseIsSuccessful();
        $this->assertSelectorTextContains("h1", "Hello ConferenceController!");
    }
}

(I checked the twig file for the file content).

This doesn't help with the "red" part of red-green-refactor, but it's something at least.

Looking at the generated controller:

class ConferenceController extends AbstractController
{
    #[Route('/conference', name: 'app_conference')]
    public function index(): Response
    {
        return $this->render('conference/index.html.twig', [
            'controller_name' => 'ConferenceController',
        ]);
    }
}

I'm not a fan of munging routing into controllers, so I'm gonna move that back into the routing config (routes.yaml):

controllers:
    resource:
        path: ../src/Controller/
        namespace: adamcameron\symfonythefasttrack\Controller
    type: attribute

conference:
    resource: routes/conference.yaml
    prefix: /conference/
conference:
    path: /
    methods: [GET]
    controller: adamcameron\symfonythefasttrack\Controller\ConferenceController::index

And here the test helps: I can refactor that routing config, and my test stays green!

Everything else seems fine.

The rest of the page messes with that controller to show using inline HTML in the controller (just why? would you even show that??), and output some debugging stuff. The debug stuff was interesting. One can put a dump in one's code and it will send the dump down to the browser, but different from var_dump it's not part of the main response, it's part of the debug bar (which I have only just clocked is a separate request, done via AJAX from the main response). Example (no link to this code as I'm not keeping it):

public function index(Request $request): Response
{
    dump($request);
    return new Response("<html><body></body></html>");
}

(note that for the debug bar to show up, it needs to be an "HTML" response with a body element)

Cool.


Right, that was a lot of effort to make not much progress. There was quite a bit of googling in places when things didn't quite go how I expected, which I just summarised in the text. Anyway, I'm nackered so am gonna draw a line under this one.

All the code for this lot is tagged as 1.5.

Oh and I've done the next article in this series: PHP / Symfony: working through "Symfony: The Fast Track", part 3: doing some ORM / DB config.

Righto.

--
Adam

Saturday 25 March 2023

PHP / Symfony: working through "Symfony: The Fast Track", part 1: preparation and pre-requisites (and not actually any Symfony!)

G'day:

This new PHP app we're shifting in at work is gonna be running Symfony. The other bods on the dev team have been working through "Symfony: The Fast Track", and I figured I pretty much ought to as well. So… here goes. It's a big "book", so am not sure how much of it I'll be able to get through in one go, so this will be a multi-part series. Or just one part if I get bored and move on to something else, as if often my wont on this blog. Hopefully it'll keep me motivated. The stuff I've seen from the other peeps on the team is interesting though, so who knows.


Pre-requisites

I need PHP and Nginx running so I'll install the config for those first. I've actually already done this, tested it, and tagged it up on GitHub as adamcameron/SymfonyFastTrack. The baseline for this was version 1.0 of my PHP 8 repo. This repo has moved on a bit since it was a bare baseline, but that v1.0 tag was round about right and with a minimum of unnecessary crap, so I started with that: cloned it, and slapped the files in my SymfonyFastTrack repo. I fiddled around with some stuff, but nothing of consequence. Anyhoo, there's an installation set of two Docker containers there: Nginx and PHP. They expose the website on http://localhost:8062/ on yer host machine. All the instructions are in the README.md file for the repo.

The only other thing I've done thusfar is set up a project in IntelliJ to talk to my PHP container so it can run tests and code quality tooling, etc.

There is nothing to do with Symfony in this codebase yet. This is next. When I start reading that book. Which'll be now.

Symfony: The Fast Track

I'm gonna start from the front and work my way to the back.

First coupla sections

There are "Acknowledgements" and "What is it about?" up front. Necessary I suppose, but no real value beyond the author's obligation to write them.

Checking your Work Environment

The interesting things in here is that they say up front I'm gonna need PostgreSQL and Node.js installed. Oh and the Symfony CLI (which I actually have installed in the PHP container already, as it turns out. I forgot about that).

Other than that the requirements for this - to start with - are:

  • An IDE - IntelliJ in my case.
  • A terminal. Well, um, yeah obviously (isn't it?).
  • Git. Yup, done.
  • PHP. The book is written for 8,1, and I have 8.2 installed.
  • Some PHP extensions, some of which I do not have installed yet.
  • Composer. Yup, obviously.
  • Docker.

One good thing the Symfony CLI does is have a check to see if I meet the book's prerequisites:

root:/var/www# symfony book:check-requirements
[OK] Git installed
[OK] PHP installed version 8.2.4 (/usr/local/bin/php)
[KO] PHP extension "xsl" not found, please install it - not found
[OK] PHP extension "tokenizer" installed - required
[OK] PHP extension "xml" installed - required
[KO] PHP extension "redis" not found - optional - needed only for chapter 31
[KO] PHP extension "amqp" not found - optional - needed only for chapter 32
[OK] PHP extension "json" installed - required
[OK] PHP extension "session" installed - required
[OK] PHP extension "curl" installed - optional - needed only for chapter 17 (Panther)
[KO] PHP extension "pdo_pgsql" not found, please install it - required
[OK] PHP extension "mbstring" installed - required
[OK] PHP extension "openssl" installed - required
[OK] PHP extension "sodium" installed - required
[OK] PHP extension "zip" installed - optional - needed only for chapter 17 (Panther)
[KO] PHP extension "gd" not found - optional - needed only for chapter 23 (Imagine)
[OK] PHP extension "ctype" installed - required
[OK] PHP extension "intl" installed - required
[OK] Composer installed
[KO] Cannot find Docker, please install it https://www.docker.com/get-started
[KO] Cannot find Docker Compose, please install it https://docs.docker.com/compose/install/
[KO] Cannot find the npm package manager, please install it https://www.npmjs.com/
  
You should fix the reported issues before starting reading the book.
root:/var/www#

It's not seeing Docker as I'm in a shell of a container already. But I'll get all that other stuff installed now.


PHP extensions

That was mostly googling how to install the various extensions that were missing, doing it manually in the running container to test they were working, then adding the steps to docker/php/Dockerfile so they're there when I rebuild the containers. These were the additions to the Dockerfile:

FROM php:8.2.4-fpm

RUN ["apt-get", "update"]
RUN ["apt-get", "install", "-y", "zip", "unzip", "git", "vim"]

COPY php.ini /usr/local/etc/php/php.ini
COPY home/.bash_history /root/.bash_history
COPY home/.bashrc /root/.bashrc

COPY --from=composer:latest /usr/bin/composer /usr/local/bin/composer

RUN pecl install xdebug && docker-php-ext-enable xdebug
COPY conf.d/xdebug.ini /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini
COPY conf.d/error_reporting.ini /usr/local/etc/php/conf.d/error_reporting.ini

RUN apt-get install -y libicu-dev && docker-php-ext-configure intl && docker-php-ext-install intl

RUN ["apt-get", "install", "-y", "libpng-dev", "libpq-dev", "librabbitmq-dev", "libxslt1-dev", "libz-dev", "libzip-dev"]
RUN docker-php-ext-configure bcmath && docker-php-ext-install bcmath
RUN docker-php-ext-configure gd && docker-php-ext-install gd
RUN docker-php-ext-configure opcache && docker-php-ext-install opcache
RUN docker-php-ext-configure pdo_mysql && docker-php-ext-install pdo_mysql
RUN docker-php-ext-configure pdo_pgsql && docker-php-ext-install pdo_pgsql
RUN docker-php-ext-configure xsl && docker-php-ext-install xsl
RUN docker-php-ext-configure zip && docker-php-ext-install zip

RUN pear config-set php_ini /usr/local/etc/php/php.ini && pecl install amqp && docker-php-ext-enable amqp
RUN pear config-set php_ini /usr/local/etc/php/php.ini && pecl install redis && docker-php-ext-enable redis

RUN curl -1sLf 'https://dl.cloudsmith.io/public/symfony/stable/setup.deb.sh' | bash
RUN ["apt-get", "install", "-y", "symfony-cli"]

WORKDIR /var/www
ENV COMPOSER_ALLOW_SUPERUSER 1

I also enforced as many of them as I could in composer.json, eg:

{
    "name" : "adamcameron/symfonythefasttrack",
    "description" : "Symfony: The Fast Track",
    "type" : "project",
    "license" : "LGPL-3.0-only",
    "require": {
        "php" : "^8.2",
        "ext-bcmath": "*",
        "ext-ctype": "*",
        "ext-curl": "*",
        "ext-dom": "*",
        "ext-gd": "*",
        "ext-iconv": "*",
        …

Which worked fine for all of them except the amqp: composer validate just went "what the hell is that?" I could not find out why so I omitted that one. It is installed though:

I googled around for what's different about the AMPQ extension compared to the others, and how I might handle it in composer.json, but drew a blank. I thought it might be because it's a PECL extension, not a "normal" one, but then I'd also expect to get the same grief with the redis one, which I didn't. It's no big thing, so I've just moved on.

Symfony is much happier now:

root:/var/www# symfony book:check-requirements
[OK] Git installed
[OK] PHP installed version 8.2.4 (/usr/local/bin/php)
[OK] PHP extension "xsl" installed - required
[OK] PHP extension "tokenizer" installed - required
[OK] PHP extension "xml" installed - required
[OK] PHP extension "redis" installed - optional - needed only for chapter 31
[OK] PHP extension "amqp" installed - optional - needed only for chapter 32
[OK] PHP extension "json" installed - required
[OK] PHP extension "session" installed - required
[OK] PHP extension "curl" installed - optional - needed only for chapter 17 (Panther)
[OK] PHP extension "pdo_pgsql" installed - required
[OK] PHP extension "mbstring" installed - required
[OK] PHP extension "openssl" installed - required
[OK] PHP extension "sodium" installed - required
[OK] PHP extension "zip" installed - optional - needed only for chapter 17 (Panther)
[OK] PHP extension "gd" installed - optional - needed only for chapter 23 (Imagine)
[OK] PHP extension "ctype" installed - required
[OK] PHP extension "intl" installed - required
[OK] Composer installed
[KO] Cannot find Docker, please install it https://www.docker.com/get-started
[KO] Cannot find Docker Compose, please install it https://docs.docker.com/compose/install/
[KO] Cannot find the npm package manager, please install it https://www.npmjs.com/
  
You should fix the reported issues before starting reading the book.
root:/var/www#

The changes to get this far are tagged as 1.1. I'll crack on with adding a Node.js and a PostgreSQL container to the mix now, before I turn the next page. I don't need the PostgreSQL one yet, but I might as well do them both now, whilst I'm dicking around with Docker.


Node.js

There was not much do doing this. I think. I had an old scratch Node.js repo I created a while back, and I know that works, so I co-opted the bits I wanted from that and that was it really.

Dockerfile:

FROM node:19-bullseye

RUN ["apt-get", "update"]
RUN ["apt-get", "install", "-y", "vim"]

COPY home/.bashrc /root/.bashrc
COPY home/.bash_history /root/.bash_history
COPY home/.vimrc /root/.vimrc

WORKDIR  /usr/share/nodejs/

Most of that is just puff to make my life easier when I'm ferretting around in the container shell. The only important bit is the FROM bit. Which seems to be the latest Debian-based image.

docker-compose.yml (just the relevant bit):

nodejs:
    build:
        context: nodejs
        dockerfile: Dockerfile

    stdin_open: true
    tty: true

    volumes:
        - ..:/usr/share/nodejs/

Nowt interesting there either.

package.json:

{
    "name": "symfony-the-fast-track",
    "description": "Symfony: The Fast Track requires Node.js for some reason. So here it is.",
    "version": "1.0.0",
    "author": "Adam Cameron",
    "license": "GPL-3.0-or-later",
    "devDependencies": {
        "chai": "^4.3.7",
        "mocha": "^10.2.0"
    },
    "scripts": {
        "test": "mocha tests/nodejs/**/*.js"
    }
}

And it wouldn't be me if I didn't have a test (NodeTest.js):

let chai = require("chai");
let expect = chai.expect;

describe("Test Node.js", () => {
    it("has the expected version",  () => {
        expect(process.version).to.be.match(/v19\.\d+\.\d+/);
    })
});

And having rebuild the containers and shelled into the nodejs one, I just did npm install, and my test was runnable:

root:/usr/share/nodejs# npm test

> symfony-the-fast-track@1.0.0 test
> mocha tests/nodejs/**/*.js



  Test Node.js
    ✔ has the expected version


  1 passing (4ms)

root:/usr/share/nodejs#

I'm content that's doing its job.

One other thing I needed to do was to rearrgange the tests subdirectory hierarchy a bit:

/usr/share/nodejs# tree tests
tests
|-- nodejs
|   `-- integration
|       `-- NodeTest.js
`-- php
    |-- Integration
    |   |-- InstallationTest.php
    |   `-- PhpTest.php
    |-- Unit
    `-- bootstrap.php

5 directories, 4 files
root:/usr/share/nodejs#

I did not want the PHP tests munged in with the JS tests. Obvs I re-ran the PHP tests after I rehomed them. They were fine.

I've tagged all that as 1.2


PostgreSQL

Oddly, I have another project with a PostgreSQL container in it. So I'm copying that too.

Dockerfile:

FROM postgres:15-bullseye

RUN ["apt-get", "update"]
RUN ["apt-get", "install", "-y", "vim"]

COPY postgres/home/.bash_history /root/.bash_history
COPY shared/home/.bashrc /root/.bashrc
COPY shared/home/.vimrc /root/.vimrc

COPY postgres/docker-entrypoint-initdb.d/ /docker-entrypoint-initdb.d/

EXPOSE 5432

As with the Node.js one: a bunch of housekeeping there. I'm running a script (1.createAndPopulateTestTable.sql) to install some data at least:

CREATE TABLE test (
    id int GENERATED ALWAYS AS IDENTITY (START WITH 101),
    value VARCHAR(50) NOT NULL
);

INSERT INTO test (value)
VALUES
    ('Test row 1'),
    ('Test row 2')
;

The docker-compose.yml bit for this one is more interesting:

    postgres:
        build:
            context: .
            dockerfile: postgres/Dockerfile

        env_file:
            - shared/envVars.public
            - shared/envVars.private

        ports:
            - "5432:5432"
        volumes:
            - postgresData:/var/lib/postgresql/data

        stdin_open: true
        tty: true

        networks:
            backend:
                aliases:
                    - database.backend

volumes:
    postgresData:

networks:
    backend:
        driver: "bridge"

I need to pass some env vars into the containers now, including a password. I've put that in shared/envVars.private, and have excluded it from source control (docker/shared/.gitignore):

envVars.private

The other env vars don't need securing (docker/shared/envVars.public)

POSTGRES_DB=db1
POSTGRES_USER=user1

I've also added an explicit network now (not for any specific reason).

You might note I've also changed the build context here. I've done this across the board. This is because all th containers have the same .bashrc and .vimrc requirements, so instead of having image-specific ones (eg: php/home/.bashrc and nginx/home/.bashrc having the same content), I've put them in that shared directory adjacent to the service-specific directories (I'm just showing the PHP and Postgres ones here; the Nginx and Node.js ones are equivalent):

root:/var/www/docker# tree -a
.
// …
|-- php
|   |-- Dockerfile
|   |-- conf.d
|   |   |-- error_reporting.ini
|   |   `-- xdebug.ini
|   |-- home
|   |   `-- .bash_history
|   `-- php.ini
|
|-- postgres
|   |-- Dockerfile
|   |-- docker-entrypoint-initdb.d
|   |   `-- 1.createAndPopulateTestTable.sql
|   `-- home
|       `-- .bash_history
`-- shared
    |-- .gitignore
    |-- envVars.private
    |-- envVars.public
    `-- home
        |-- .bashrc
        `-- .vimrc

I've also added a test in the PHP container to make sure it can reach the DB:

namespace adamcameron\symfonythefasttrack\tests\Integration;

use Doctrine\DBAL\Connection;
use Doctrine\DBAL\DriverManager;
use PHPUnit\Framework\TestCase;
use stdClass;

/** @testdox DB tests */
class DbTest extends TestCase
{

    /** @testdox It can connect to the DB using DBAL */
    public function testDbalConnection()
    {
        $connection = $this->getDbalConnection();
        $result = $connection->executeQuery("SELECT version() AS version");

        $this->assertStringStartsWith("PostgreSQL 15", $result->fetchOne());
    }

    private function getConnectionParameters() : stdClass
    {
        return (object) [
            "host" => "database.backend",
            "port" => "5432",
            "database" => getenv("POSTGRES_DB"),
            "username" => getenv("POSTGRES_USER"),
            "password" => getenv("POSTGRES_PASSWORD")
        ];
    }

    private function getDbalConnection() : Connection
    {
        $parameters = $this->getConnectionParameters();
        return DriverManager::getConnection([
            "dbname" => $parameters->database,
            "user" => $parameters->username,
            "password" => $parameters->password,
            "host" => $parameters->host,
            "port" => $parameters->port,
            "driver" => "pdo_pgsql"
        ]);
    }
}

It simply checks the major version of the DB. If it can get that: it's connected enough.

I did not do a similar test in the Node.js container as I don't know that it will need to talk to the DB. I am guessing it's only going to be used to do client-side code building and crap like that. If I need this later: I'll do it later.

Once I rebuild the containers, that new test runs:

root:/var/www# composer test
> phpunit tests --display-deprecations
PHPUnit 10.0.18 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.2.4 with Xdebug 3.2.1
Configuration: /var/www/phpunit.xml.dist

...                                                                 3 / 3 (100%)

Time: 00:05.572, Memory: 10.00 MB

DB tests
  It can connect to the DB using DBAL

Tests of the whole installation
  Nginx is serving PHP content on the host system

Tests of the PHP installation
  It has the expected PHP version

OK (3 tests, 5 assertions)

Generating code coverage report in HTML format ... done [00:01.721]

And now… I'm sick of doing this for the day! I'm got a busy day tomorrow during the day, but a quiet Sunday evening might be a good time to start doing some actual Symfony stuff.

I've linked to all the files as I mention them, and there's four different tags in play in this article, but the current state of everything is here: 1.3.

I continue in the next part: PHP / Symfony: working through "Symfony: The Fast Track", part 2: creating a controller (eventually).

Righto.

--
Adam

Tuesday 31 January 2023

TDD & Symfony & Monolog: adding some logging to that endpoint from last time

G'day:

Hopefully this one is shorter than the previous monster (TDD & Symfony: creating a small web service end point). I should perhaps have split that one in two, in hindsight (the Adapter, then the Controller). This article builds on that code.

I have create a wee webservice that I call like this: http://localhost:8008/postcode-lookup/XX200X, and that goes off and hits getaddress.io with much the same request. This web service of mine is two components:

Simple. Except I managed to write 4000-ish words on it on Sunday, somehow.

One shortfall I identified with the initial implementation is that it was just swallowing some failure situations that - whilst should not cause a problem for the consuming client of my web service - should be something I pay attention to if they occur. A quick fix for this is to chuck some logging in. And this is a good exercise as it will require me to revisit Monolog, and also I'll need to work out a) how to wire it into Symfony; b) and test my integration.

In the previous article I TDDed the first part of the exercise, and then backfilled the testing on the second part as I didn't even know how to wire things together in Symfony when I started, so I decided to spike that first. Today I'm taking a hybrid approach. I'm going to try to TDD it, but there will be some points at which I need to wrestle with Symfony/Monolog config, and I am just gonna go do that when I need to. NB: this is not to suggest any of this config is arduous: I've just forgotten how to do it, so I need to work it out again.

Here goes.

This unit test in tests/Unit/Controller/PostcodeLookupControllerTest.php describes the problem I'm trying to solve:

/**
 * @testdox It logs any issues we might need to deal with
 * @dataProvider provideCasesForLoggingTests
 */
public function testLogging(
    int $statusCode,
    string $expectedMessage,
    Level $expectedLogLevel
) {
    $testHandler = new TestHandler();

    $this->configureControllerWithTestLoggingHandler(
        $statusCode,
        $expectedMessage,
        $testHandler
    );

    $this->client->request(
        "GET",
        sprintf("/postcode-lookup/%s", TestConstants::POSTCODE_OK)
    );

    $this->assertLogEntryIsCorrect(
        $testHandler,
        $expectedLogLevel,
        $statusCode,
        $expectedMessage
    );
}

Especially with the data-provider method:

public function provideCasesForLoggingTests() : array
{
    return [
        "Unauthorized should log critical" => [
            Response::HTTP_UNAUTHORIZED,
            "Unauthorized",
            Level::Critical
        ],
        "Forbidden should log critical" => [
            Response::HTTP_FORBIDDEN,
            "Forbidden",
            Level::Critical
        ],
        "Too many requests should log critical" => [
            Response::HTTP_TOO_MANY_REQUESTS,
            "Too Many Requests",
            Level::Warning
        ],
        "Server error should log critical" => [
            Response::HTTP_INTERNAL_SERVER_ERROR,
            "Internal Server Error",
            Level::Warning
        ]
    ];
}

Basically the getaddress.io call could return each of those failures, and I wanna log when they occur. I don't care about "the client app passed an invalid postcode", but I do care if I'm using the wrong API key, or if I haven't paid my account (those're both bad, so: critical); and I also kinda wanna keen an eye on throttling issues, and unexpected server errors on their end ("good to know", so just warnings). If any of these occur, I'm still returning a usable response to the client so they don't need to care, but I keep an eye on issues I am having with getaddress.io

There's nothing interesting in that test method, but there's some stuff in the helper methods:

private function configureControllerWithTestLoggingHandler(
    int $statusCode,
    string $expectedMessage,
    TestHandler $testHandler
): void {
    $container = self::getContainer();
    $mockedAddressServiceAdapter = $this
        ->getMockBuilder(AddressService\Adapter::class)
        ->disableOriginalConstructor()
        ->onlyMethods(['get'])
        ->getMock();
    $mockedAddressServiceAdapter
        ->expects($this->once())
        ->method('get')
        ->willReturn(new AddressService\Response(
            [],
            $statusCode,
            $expectedMessage
        ));
    $container->set(AddressService\Adapter::class, $mockedAddressServiceAdapter);

    $logger = $container->get("monolog.logger.address_service");
    $logger->setHandlers([$testHandler]);
}

This shows how to grab the DI container and replace my AddressService/Adapter with a mock that returns the values I need to exercise my controller code. Remember: I will be logging in the controller here, as it's a reaction to the response it's returning. I am not changing any Adapter logic here, I am adding some logging to the controller. TBH thinking about it now, maybe this should be in Service/Address instead of the controller. Hrm. Anyhow, I can refactor later if I want (100% test coverage so I'm safe to do that).

I'm also replacing the "live" logging handler in the container with a test one. This is so I don't actually log to the file system, it instead exposes an array of log entries, which I then check out in the custom assertion function:

public function assertLogEntryIsCorrect(
    TestHandler $testHandler,
    Level $expectedLogLevel,
    int $statusCode,
    string $expectedMessage
): void {
    $logRecords = $testHandler->getRecords();
    $this->assertCount(1, $logRecords);
    $this->assertEquals($expectedLogLevel->getName(), $logRecords[0]["level_name"]);
    $this->assertEquals(
        AddressService\Adapter::ERROR_MESSAGES[$statusCode],
        $logRecords[0]["message"]
    );
    $this->assertEquals(
        [
            "postcode" => TestConstants::POSTCODE_OK,
            "message" => $expectedMessage
        ],
        $logRecords[0]["context"]
    );
}

(Full disclosure, I am writing this after I have done the full implementation so like there's that ERROR_MESSAGES const array that came out of some refactoring I did after everything was working).

This assertion is simple enough: look for one log entry, and it needs to be the level, message and context that I should expect.

Before I can run that, I need to wire in Monolog, and before I do that, I need to install it. So I'm gonna have a quick functional test for that too:

/** @testdox It writes AddressService entries to the expected log file */
public function testAddressServiceLogFile()
{
    $kernel = new Kernel("test", false);
    $kernel->boot();
    $container = $kernel->getContainer();
    $logFile = $container->getParameter("kernel.logs_dir") . "/address_service.log";

    $logger = $container->get("monolog.logger.address_service");

    $this->assertEquals($logFile, $logger->getHandlers()[0]->getUrl());
}

This doesn't test any actual writing of data to a file: I figure that's Monolog's job to look after. I'm just verifying my config stays the same as I expect it. This is not a unit test, it's just an functional test: testing I've done the config right, and no-one monkeys with it later.

Now I will permit myself to actually install Monolog; or as it is in this case: symfony/monolog-bundle.

That needs a monolog.yaml file:

monolog:
    handlers:
        address_service_log:
            type: stream
            path: '%kernel.logs_dir%/address_service.log'
            level: debug
            channels: [address_service]

    channels: [address_service]

Having added that: my functional test works, so I'm happy I've configured my log.

Now I can do my implementation of the logging in the controller. I'll show you this in parts:

public function __construct(
    AddressService\Adapter $addressServiceAdapter,
    LoggerInterface $addressServiceLogger
) {
    $this->addressServiceAdapter = $addressServiceAdapter;
    $this->logger = $addressServiceLogger;
}

I need to add the logger parameter here, but I don't need to do anything to wire it in to the DI container. Symfony works out that if I ask for a LoggerInterface, then it'll take the parameter name, lop off "Logger" and look for a channel in my monolog.yaml file that is the snake-case version of that. So the paramter name here - $addressServiceLogger will find the address_service channel in my Monolog config. That's quite cool.

public function doGet(string $postcode) : JsonResponse
{
    try {
        $response = $this->addressServiceAdapter->get($postcode);

        $this->logUnexpectedFailures($response, $postcode);

        return new JsonResponse(
            [
                'postcode' => $postcode,
                'addresses' => $response->getAddresses(),
                'message' => $response->getMessage()
            ],
            $response->getHttpStatus()
        );
    } catch (\Exception $e) {
        return new JsonResponse(
            [
                'postcode' => $postcode,
                'addresses' => [],
                'message' => $e->getMessage()
            ],
            HttpStatusCode::HTTP_INTERNAL_SERVER_ERROR
        );
    }
}

There's just that one insertion into the controller logic, and that function is also pretty simple:

private function logUnexpectedFailures(
    AddressService\Response $response,
    string $postcode
): void {
    $statusCode = $response->getHttpStatus();

    if (array_key_exists($statusCode, self::RESPONSES_TO_LOG)) {
        $this->logger->log(
            self::RESPONSES_TO_LOG[$statusCode],
            AddressService\Adapter::ERROR_MESSAGES[$statusCode],
            ['postcode' => $postcode, 'message' => $response->getMessage()]
        );
    }
}

That also refers to this lot:

private const RESPONSES_TO_LOG = [
    HttpStatusCode::HTTP_UNAUTHORIZED => Level::Critical,
    HttpStatusCode::HTTP_FORBIDDEN => Level::Critical,
    HttpStatusCode::HTTP_TOO_MANY_REQUESTS => Level::Warning,
    HttpStatusCode::HTTP_INTERNAL_SERVER_ERROR => Level::Warning
];

Level is an enum, which are new to PHP since the last time I used it. I like. I'll need to look into those in another article maybe (not least of all cos the docs are not as good as they could be).

All the code here is doing is checking if there's a case that we want to log via comparing the returned HTTP status code in that RESPONSES_TO_LOG array, and if one is there, log a message with the defined log level.

What's logged comes from the Adapter:

public const ERROR_MESSAGES = [
    HttpFoundationResponse::HTTP_UNAUTHORIZED => "API key is not valid",
    HttpFoundationResponse::HTTP_FORBIDDEN => "Permission denied",
    HttpFoundationResponse::HTTP_TOO_MANY_REQUESTS  => "Too many requests",
    HttpFoundationResponse::HTTP_INTERNAL_SERVER_ERROR => "Server error"
];

We don't really need too much detail here, we just need to know it's happened.

Again, I wonder if this is an adapter's job to define these. I think I do need a skinny wee service in between the adapter and controller here. I will do that refactor.

And that's it. I mean there are a few use statements about the place I didn't show you, but I'll link through to the code and you can look at everything, and there's really not much to it. It did take quite a while to dig out the docs for all this, given I was working with Symfony and Monolog, and testing of each, and being a newbie didn't help because some of the docs seem to assume whilst I was a n00b at (for example) Monolog then that's fine we'll document it slowly, but not thinking about the fact I also don't know the Symfony side of things either, I found the docs assume a level of knowledge that they shouldn't (at times). Being rusty with PHP (eg: not even knowing PHP did enums!) did not help. But I got there.

All the code is in tag 1.9 of this project on Github.

Righto.

--
Adam

Sunday 29 January 2023

TDD & Symfony: creating a small web service end point

G'day:

Background

I have a real-world requirement to get a small web service written: one that wraps up calls to the getaddress.io UK postcode look-up service. For those not familiar with UK post codes, they look something like "SW1W 0NY" and the chief conceit is that each postcode "generally represents a street, part of a street…". It's very common on an address-filling form for the first step to be trying to use the postcode to return a range of correct addresses, from which the user selects their own. This reduces keying errors or vagueness on their part. We have a few forms in our web apps that hit this webservice, and we need to migrate them over to PHP.

Yesterday I took a stab at implementing this in my own PHP8 project (the one I've been building in my recent articles), and got so focused I forgot to document what I was doing and why. So today I'm revisiting the code again, and writing this thing up. Warning: it's ended up being a monster.


Implementation plan

I need to produce an endpoint along the lines of /postcode-lookup/XX200X, and from the response from that, derive a reliable list of matching addresses to pass to the UI. I'm going to implement this in two parts:

  • An adapter to sit in front of / around the HTTP call to the getaddress.io webs service. I mean "adapter" in the sense of the "Adapter Pattern"; I'll build a PHP interface to getaddress.io's HTTP one.
  • A controller handler which receives a post-code, calls the adapter, gets a result from it and determines how best to respond to the client based on that result.

There's two approaches I could take with this:

  • Top down: start with the interface of my own endpoint, nail that - mocking the adapter to start with - and then move downwards to the adapter and sort out the HTTP calls to getaddress.io.
  • Bottom up: start with the HTTP calls to getaddress.io, build and adapter around it, finish that and then create a controller that is driven by how the adapter works.

Either would work. I've chosen the latter for a coupla reasons:

Not in themselves great reasons to take that approach, but I doubt I would have bothered to do the work had I started with the Symfony side of things (reminding myself how routing, DI and controllers work). I was also only intending to do the adapter part for this exercise, even though I ultimately wired it up to the controller too (and it was dead easy as it turns out).

So. Time for some PHP code, some TDD via good old unit tests.


getaddress.io's web service

I had never used this web service before, but it's straight forward and well documented (again: documentation.getaddress.io). I clearly need an API key to access this thing, so signed-up for one of those (free, very limited usage), but they give some testing postcodes one can use to emulate the various responses one can get from their end point:

These postcodes yield both successful and unsuccessful responses to your request.

  • XX2 00X or TR19 7AA or KW1 4YT Returns a 'successful' response 200. Your request was successful.
  • XX4 00X Returns 'bad request' response400. Your postcode is not valid.
  • XX4 01X Returns 'unauthorized' response 401. Your api-key is not valid.
  • XX4 03X Returns 'forbidden' response 403. Your api-key is valid but you do not have permission to access to the resource.
  • XX4 29X Returns 'too many requests' response 429. You have made more requests than your allowed limit.
  • XX5 00X Returns 'server error' response 500. Server error, you should never see this.

Request made with these postcodes will not affect your usage.

(ibid.)

Oh, incidentally, they offer far more services than I need to use. I only need to deal with this call: https://api.getAddress.io/find/{postcode}?api-key={API key}.

Adapter

The adapter will need to deal with all those variants in some fashion, and returning the data in PHP rather than JSON; just return a PHP object that reflects success / and relevant failure states, along with a native-PHP representation of the returned address data. It can also through an exception in situations where it couldn't make sense of the response from getaddress.io. As it's only an adapter, it should not be performing any business logic; just interfacing between my app and the getaddress.io service. This is also why it's still returning HTTPish concepts rather than removing that entirely from the mix. That would be the job of a repository/service/some-other-domain-model-object sitting between it and the controller, if I chose to have one. I think an intermediary layer is overkill here, so have not bothered.

Routing and Controller

As mentioned, the public interface to my app needs to be to support GET requests to /postcode-lookup/{postcode}. The controller will be initialised with an adapter instance, the handler method will call its get method, and that will return a Response object, which will have properties for the returned addresses (if any), a message explaining potentially why no addresses were fetchable, and the HTTP status code of the upstream request. Or it - the adapter call - could throw an exception that will also need to be dealt with. The client won't be needing to know about those.


Code

API key

I've added a "missing" integration test here:

public function testEnvironmentVariables($expectedEnvironmentVariable)
{
    $this->assertNotFalse(
        getenv($expectedEnvironmentVariable),
        "Expected environment variable $expectedEnvironmentVariable to exist"
    );
}

public function expectedEnvironmentVariablesProvider() : array
{
    return [
        ["MARIADB_HOST"],
        ["MARIADB_PORT"],
        ["MARIADB_USER"],
        ["MARIADB_DATABASE"],
        ["MARIADB_ROOT_PASSWORD"],
        ["MARIADB_PASSWORD"],
        ["ADDRESS_SERVICE_API_KEY"]
    ];
}

I need all these environment variables to actually exist and be reachable by PHP, so I ought to have an integration test to make sure they do exist. I've added in the ADDRESS_SERVICE_API_KEY on there as this is what I'm about to add, and I want a failing test. You might or might not remember I'm loading my environment variables like this (in docker-compose.yml):

php:
  build:
    context: php
    dockerfile: Dockerfile

  env_file:
    - envVars.public
    - envVars.private

I have two files: the envVars.public one is in source control. It's got stuff like this in it:

MARIADB_HOST=mariadb
MARIADB_PORT=3306
MARIADB_DATABASE=db1
MARIADB_USER=user1

In contrast the envVars.private one is not in source control, as it has stuff like this in it:

# do not commit this file to your repository
MARIADB_ROOT_PASSWORD=[redacted]
MARIADB_PASSWORD=[redacted]
ADDRESS_SERVICE_API_KEY=[redacted]

You can see I've slung the ADDRESS_SERVICE_API_KEY in there. After cycling my containers, that test passes.


Adapter testing

Unit tests

I'm gonna eschew the "lets walk through the TDD steps" here as there's 10 unit tests, two integration tests, and 250 lines of test code that went through a chunk of refactoring after I got it all working. I'm just gonna show the tests - all of them - and discuss them.

/** @testdox It throws an AddressService\Exception if the getaddress.io call returns an unexpected status */
public function testThrowsExceptionOnUnexpectedStatus()
{
    $statusToReturn = Response::HTTP_NOT_IMPLEMENTED;

    $this->assertCorrectExceptionThrown(
        AddressServiceException::class,
        "Unexpected status code returned: $statusToReturn"
    );

    $adapter = $this->getTestAdapter($statusToReturn, "CONTENT_NOT_TESTED");

    $adapter->get("POSTCODE_NOT_TESTED");
}

If you refer back to the list of test postcodes above, and the HTTP response statuses they result in, 501 / not implemented is not one of them. So I am testing my handling of this situation throws an exception, because we have NFI what's going on with getaddress.io if we get one of these back from them, so there's no point continuing processing. The relevant fragment of the Adapter code is:

private const SUPPORTED_SERVICE_RESPONSES = [
    HttpFoundationResponse::HTTP_OK,
    HttpFoundationResponse::HTTP_BAD_REQUEST,
    HttpFoundationResponse::HTTP_UNAUTHORIZED,
    HttpFoundationResponse::HTTP_FORBIDDEN,
    HttpFoundationResponse::HTTP_TOO_MANY_REQUESTS,
    HttpFoundationResponse::HTTP_INTERNAL_SERVER_ERROR
];

// ...

if (!in_array($statusCode, self::SUPPORTED_SERVICE_RESPONSES)) {
    throw new AddressService\Exception("Unexpected status code returned: $statusCode");
}

Don't worry, I'll list the whole thing further down, I just wanna focus on which bits are being tested for now.

The test code above has a few helper methods I've extracted out during refactor:

private function assertCorrectExceptionThrown(string $type, string $message): void
{
    $this->expectException($type);
    $this->expectExceptionMessage($message);
}
private function getTestAdapter(int $statusToReturn, string $content): AddressServiceAdapter
{
    $client = $this->getMockedClient($statusToReturn, $content);

    return new AddressServiceAdapter("NOT_TESTED", $client);
}

private function getMockedClient(int $statusToReturn, string $content): MockObject
{
    $response = $this->getMockedResponse($statusToReturn, $content);

    $client = $this
        ->getMockBuilder(HttpClientInterface::class)
        ->disableOriginalConstructor()
        ->getMock();

    $client
        ->expects($this->once())
        ->method("request")
        ->willReturn($response);

    return $client;
}

private function getMockedResponse(int $status, string $content): MockObject
{
    $response = $this
        ->getMockBuilder(ResponseInterface::class)
        ->disableOriginalConstructor()
        ->getMock();
    $response
        ->expects($this->atLeastOnce())
        ->method("getStatusCode")
        ->willReturn($status);
    $response
        ->expects($this->any())
        ->method("getContent")
        ->willReturn($content);

    return $response;
}

The strategy here is that I am testing the Adapter's logic, and how it deals with different responses from the HTTP call. So I mock the HTTP client to return the various responses I need to exercise the Adapter logic I need to write. As you'll see from the rest of the tests, they all use the mocked client, and all the exception handling tests use that assertCorrectExceptionThrown custom assertion. This refactoring keeps the tests simple and clear, and just removes necessary boilerplate machinery.

It's just as important to refactor one's tests as it is to refactor one's source code. Test code still needs to be read by humans.

/** @testdox It throws an AddressService\Exception if the body is not JSON */
public function testThrowsExceptionOnBodyNotJson()
{
    $this->assertCorrectExceptionThrown(
        AddressServiceException::class,
        "json_decode returned [Syntax error]"
    );

    $adapter = $this->getTestAdapter(Response::HTTP_OK, "NOT_JSON");

    $adapter->get("NOT_TESTED");
}

You can see how simple this test is, and following the same approach as the previous one. Just testing a slightly different facet of the logic.

And the implementation code for this:

$body = $response->getContent(false);
$lookupResult = json_decode($body, JSON_OBJECT_AS_ARRAY);
if (json_last_error() != JSON_ERROR_NONE) {
    throw new AddressService\Exception(
        sprintf("json_decode returned [%s]", json_last_error_msg())
    );
}

I'll just give you the @testdox line from the other exception-handling tests, followed by the implementation code. The test implementations are all similar: just different return codes and body, so no point reproducing them.

/** @testdox Throws an AddressService\Exception if the body is not an array */
if (!is_array($lookupResult)) {
    throw new AddressService\Exception("Response JSON schema is not valid");
}
/** @testdox it throws an AddressService\Exception if there is no address data in the response json */
// …

/** @testdox it throws an AddressService\Exception if the addresses data is not an array */
// …

/** @testdox it throws an AddressService\Exception if the addresses data is not an array of strings */
// …

These three test different subexpressions in the one if expression:

if (
    !array_key_exists("addresses", $lookupResult)
    || !is_array($lookupResult["addresses"])
    || count(array_filter($lookupResult["addresses"], fn($address) => !is_string($address)))
) {
    throw new AddressService\Exception("Response JSON schema is not valid");
}

I'm not normally crazy about compound if expressions like that, but they're all testing variants of garbage that we don't want to pass back to the calling code.

Those are all the exception-handling tests, now for the situations where we can actually return something:

/** @testdox returns empty addresses with status code on a non-200-OK response */
public function testReturnsEmptyAddressesOnNon200Response()
{
    $statusToReturn = Response::HTTP_BAD_REQUEST;

    $adapter = $this->getTestAdapter(
        $statusToReturn,
        '{"Message": "Bad Request: Invalid postcode."}'
    );

    $result = $adapter->get("NOT_TESTED");

    $this->assertEquals($statusToReturn, $result->getHttpStatus());
    $this->assertEquals([], $result->getAddresses());
}
// …

/** @testdox it returns the message on a non-200 response */
// …

/** @testdox it returns a standard message if the non-200 response doesn't include a valid one */
// …

These are pretty similar to the exception-handling tests, just different assertions. And the implementation being tested for all of those is in the one function:

private function handleFailureResponse(
    array $lookupResult,
    int $statusCode
): AddressService\Response {
    if (array_key_exists("Message", $lookupResult) && is_string($lookupResult["Message"])) {
        return new AddressService\Response([], $statusCode, $lookupResult["Message"]);
    }
    return new AddressService\Response([], $statusCode, "No failure message returned from service");
}

And finally the happy path:

/** @testdox it returns a Response object if the response is valid */
public function testReturnsResponseObject()
{
    $statusToReturn = Response::HTTP_OK;
    $expectedAddresses = [
        "TEST_ADDRESS_1",
        "TEST_ADDRESS_2"
    ];

    $adapter = $this->getTestAdapter(
        $statusToReturn,
        sprintf('{"addresses": %s}', json_encode($expectedAddresses))
    );

    $result = $adapter->get("NOT_TESTED");

    $this->assertEquals($statusToReturn, $result->getHttpStatus());
    $this->assertEquals($expectedAddresses, $result->getAddresses());
}

Implemenation:

private function handleSuccessResponse(array $lookupResult): AddressService\Response
{
    if (
        !array_key_exists("addresses", $lookupResult)
        || !is_array($lookupResult["addresses"])
        || count(array_filter($lookupResult["addresses"], fn($address) => !is_string($address)))
    ) {
        throw new AddressService\Exception("Response JSON schema is not valid");
    }

    return new AddressService\Response(
        $lookupResult["addresses"],
        HttpFoundationResponse::HTTP_OK
    );
}

Integration tests

All of these unit tests only test the logic in the Adapter, having mocked-out the HTTP call. They don't actually test it will actually do what it's supposed to, which requires a call to the actual getaddress.io. This is where some integration tests come in.

/** @testdox Tests of the Adapter */
class AdapterTest extends TestCase
{
    private $adapter;

    protected function setUp(): void
    {
        $client = HttpClient::create();
        $this->adapter = new AddressServiceAdapter(getenv("ADDRESS_SERVICE_API_KEY"), $client);
    }

    /** @testdox It can get addresses from a valid postcode */
    public function testCanGetAddress()
    {
        $response = $this->adapter->get(TestConstants::POSTCODE_OK);

        $this->assertEquals(Response::HTTP_OK, $response->getHttpStatus());
        $this->assertGreaterThanOrEqual(1, count($response->getAddresses()));
        $this->assertEmpty($response->getMessage());
    }

    public function provideErrorTestCases(): array
    {
        return [
            [TestConstants::POSTCODE_INVALID, Response::HTTP_BAD_REQUEST],
            [TestConstants::POSTCODE_UNAUTHORIZED, Response::HTTP_UNAUTHORIZED],
            [TestConstants::POSTCODE_FORBIDDEN, Response::HTTP_FORBIDDEN],
            [TestConstants::POSTCODE_OVER_LIMIT, Response::HTTP_TOO_MANY_REQUESTS],
            [TestConstants::POSTCODE_SERVER_ERROR, Response::HTTP_INTERNAL_SERVER_ERROR]
        ];
    }

    /**
     * @testdox It returns the expected HTTP status code and a message but no addresses on an error
     * @dataProvider provideErrorTestCases
     */
    public function testReturnsExpectedHttpStatusAndMessageButNoAddressesOnError($postcode, $expectedHttpStatus)
    {
        $response = $this->adapter->get($postcode);

        $this->assertEquals($expectedHttpStatus, $response->getHttpStatus());
        $this->assertNotEmpty($response->getMessage());
        $this->assertEquals(0, count($response->getAddresses()));
    }
}

These test a happy-path response, as well as how the other "failure" responses from getaddress.io are handled. One might wonder why I don't test the "utter failure" cases here, where I throw an exception. I almost did, but then I thought that's nothing to do with the integration with getaddress.io, it's all down to how I handle that integration, and that's my code, and covered in the unit tests.

The only interesting / non-standard thing in that lot is the code to get the API key:

$this->adapter = new AddressServiceAdapter(getenv("ADDRESS_SERVICE_API_KEY"), $client);

The app never needs to know the key. Just the environment does.

Oh this code references some TestConstants:

<?php

namespace adamcameron\php8\tests\Fixtures\AddressService;

class TestConstants
{
    // provided by https://documentation.getaddress.io/ (these do not impact look-up usage)
    public const POSTCODE_OK = "XX2 00X";
    public const POSTCODE_INVALID = "XX4 00X";
    public const POSTCODE_UNAUTHORIZED = "XX4 01X";
    public const POSTCODE_FORBIDDEN = "XX4 03X";
    public const POSTCODE_OVER_LIMIT = "XX4 29X";
    public const POSTCODE_SERVER_ERROR = "XX5 00X";
}

These are separate from the integration test class as the controller tests need them too. I'll get to that.


Adapter implementation

namespace adamcameron\php8\Adapter\AddressService;

use adamcameron\php8\Adapter\AddressService;
use Symfony\Component\HttpFoundation\Response as HttpFoundationResponse;
use Symfony\Contracts\HttpClient\HttpClientInterface;
use Symfony\Contracts\HttpClient\ResponseInterface;

class Adapter
{

These are the details we need from getaddress.io:

    private const SUPPORTED_SERVICE_RESPONSES = [
        HttpFoundationResponse::HTTP_OK,
        HttpFoundationResponse::HTTP_BAD_REQUEST,
        HttpFoundationResponse::HTTP_UNAUTHORIZED,
        HttpFoundationResponse::HTTP_FORBIDDEN,
        HttpFoundationResponse::HTTP_TOO_MANY_REQUESTS,
        HttpFoundationResponse::HTTP_INTERNAL_SERVER_ERROR
    ];

    private const SERVICE_URL_TEMPLATE = "https://api.getAddress.io/find/{postcode}?api-key={api-key}";
    private string $apiKey;
    private HttpClientInterface $client;

    public function __construct(string $apiKey, HttpClientInterface $client)
    {
        $this->apiKey = $apiKey;
        $this->client = $client;
    }

This is the only public functions in here:

    public function get(string $postCode) : AddressService\Response
    {
        $response = $this->makeRequest($postCode);
        $lookupResult = $this->extractValidLookupResult($response);

        return $this->handleValidatedResponse($response, $lookupResult);
    }

Everything else is just refactoring to keep each task (ie: function) separate:

    private function makeRequest(string $postCode): ResponseInterface
    {
        $url = strtr(
            self::SERVICE_URL_TEMPLATE,
            ["{postcode}" => $postCode, "{api-key}" => $this->apiKey]
        );

        return $this->client->request("GET", $url);
    }

    private function extractValidLookupResult(ResponseInterface $response): array
    {
        $statusCode = $response->getStatusCode();

        if (!in_array($statusCode, self::SUPPORTED_SERVICE_RESPONSES)) {
            throw new AddressService\Exception("Unexpected status code returned: $statusCode");
        }

        $body = $response->getContent(false);
        $lookupResult = json_decode($body, JSON_OBJECT_AS_ARRAY);
        if (json_last_error() != JSON_ERROR_NONE) {
            throw new AddressService\Exception(
                sprintf("json_decode returned [%s]", json_last_error_msg())
            );
        }

        if (!is_array($lookupResult)) {
            throw new AddressService\Exception("Response JSON schema is not valid");
        }
        return $lookupResult;
    }

    private function handleValidatedResponse(
        ResponseInterface $response,
        array $lookupResult
    ): AddressService\Response {
        $statusCode = $response->getStatusCode();

        if ($statusCode == HttpFoundationResponse::HTTP_OK) {
            return $this->handleSuccessResponse($lookupResult);
        }
        return $this->handleFailureResponse($lookupResult, $statusCode);
    }

    private function handleSuccessResponse(array $lookupResult): AddressService\Response
    {
        if (
            !array_key_exists("addresses", $lookupResult)
            || !is_array($lookupResult["addresses"])
            || count(array_filter($lookupResult["addresses"], fn($address) => !is_string($address)))
        ) {
            throw new AddressService\Exception("Response JSON schema is not valid");
        }

        return new AddressService\Response(
            $lookupResult["addresses"],
            HttpFoundationResponse::HTTP_OK
        );
    }

    private function handleFailureResponse(
        array $lookupResult,
        int $statusCode
    ): AddressService\Response {
        if (array_key_exists("Message", $lookupResult) && is_string($lookupResult["Message"])) {
            return new AddressService\Response([], $statusCode, $lookupResult["Message"]);
        }
        return new AddressService\Response([], $statusCode, "No failure message returned from service");
    }
}

Hopefully given the explanations of all that in the tests, that's all clear. The code is pretty straight forward.

I'm using a custom exception so any consuming code can seprate out out "stuff this Adapter might throw" from "anything else that went wrong" (say for example the Symfony HttpClient spat the dummy for some reason):

namespace adamcameron\php8\Adapter\AddressService;

class Exception extends \RuntimeException
{
}

I will get murderous looks from some ppl for having a class that only holds a coupla data points and some accessors for same, but the adapter does simply return data, but it's a specific sort of data, and the consuming code ought only be reading it, not writing it. I think this is legit usage:

namespace adamcameron\php8\Adapter\AddressService;

class Response
{
    private array $addresses;
    private int $httpStatus;
    private string $message;

    public function __construct(array $addresses, int $httpStatus, string $message = "")
    {
        $this->addresses = $addresses;
        $this->httpStatus = $httpStatus;
        $this->message = $message;
    }

    public function getAddresses()
    {
        return $this->addresses;
    }

    public function getHttpStatus()
    {
        return $this->httpStatus;
    }

    public function getMessage()
    {
        return $this->message;
    }
}

If PHP allowed it, I'd probably make this an inner class of the Adapter as well. But PHP can't so just having it in the same package will have to do. I realise PHP now allows for class expressions, or anonymous classes or whatever they are, but I don't think that's quite right for this application. I just need to return an object from the adapter for the calling code to get predictable / save / checked values from.


Controller integration

I cheated with this bit and did not TDD it. I could not remember how controllers in Symfony worked, so I got the controller operational first, and then backfilled the tests. Soz.

Routing and service config

Firstly the routing (in routes.yaml):

controllers:
    resource:
        path: ../src/Controller/
        namespace: adamcameron\php8\Controller
    type: attribute

PostcodeLookup:
    path: /postcode-lookup/{postcode}
    controller: adamcameron\php8\Controller\PostcodeLookupController::doGet
    methods: GET

Then some DI config (in services.yaml):

# …
adamcameron\php8\Adapter\AddressService\Adapter:
    public: true
    arguments:
        $apiKey: '%env(ADDRESS_SERVICE_API_KEY)%'
        $client: '@Symfony\Component\HttpClient\HttpClient'

Symfony\Component\HttpClient\HttpClient:
    factory: ['\Symfony\Component\HttpClient\HttpClient','create']
# …

I only need to put the services that have non-obvious constructor argument requirements, like how I need to specify to use the environment variable for its $apiKey argument, and I need to specify to use the HttpClient::create factory method to create the $client value. Everything else is auto-wired on the basis of the type in the constructor's method signature.

Accordingly, the controller will be taking an instance of the Adapter, but Symfony can work that out from the type specification in its constructor method signature, and know to get the one I have configured here. Dead clever.


Controller implementation
namespace adamcameron\php8\Controller;

use adamcameron\php8\Adapter\AddressService\Adapter as AddressServiceAdapter;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\JsonResponse;
use Symfony\Component\HttpFoundation\Response;

class PostcodeLookupController extends AbstractController
{
    private AddressServiceAdapter $addressServiceAdapter;

    public function __construct(AddressServiceAdapter $addressServiceAdapter)
    {
        $this->addressServiceAdapter = $addressServiceAdapter;
    }

    public function doGet(string $postcode)
    {
        try {
            $response = $this->addressServiceAdapter->get($postcode);

            return new JsonResponse(
                [
                    'postcode' => $postcode,
                    'addresses' => $response->getAddresses(),
                    'message' => $response->getMessage()
                ],
                $response->getHttpStatus()
            );
        } catch (\Exception $e) {
            return new JsonResponse(
                [
                    'postcode' => $postcode,
                    'addresses' => [],
                    'message' => $e->getMessage()
                ],
                Response::HTTP_INTERNAL_SERVER_ERROR
            );
        }
    }
}

As all good controller methods ought to be: simple. It receives a value from the request… passes it to the model tier to get some data… based on vagaries of the data, works out how to build the response… and sends it.

Here the conceit is that I really don't care what exceptions the Adapter might throw: I don't want to break the response, so I just swallow them and return an emptyish response, with the appropriate server error.


Controller testing

These are all in PostcodeLookupControllerTest.

private KernelBrowser $client;

protected function setUp(): void
{
    $this->client = static::createClient();
}

/** @testdox It retrieves addresses when the post code is valid */
public function testRetrievesAddressesWhenPostCodeIsValid()
{
    $this->client->request(
        "GET",
        sprintf("/postcode-lookup/%s", TestConstants::POSTCODE_OK)
    );
    $response = $this->client->getResponse();

    $this->assertEquals(
        Response::HTTP_OK,
        $response->getStatusCode()
    );

    $result = json_decode($response->getContent(), false);
    $this->assertObjectHasAttribute('addresses', $result);
    $this->assertGreaterThanOrEqual(1, count($result->addresses));
}

Happy path: it returns addresses. Hurrah. I don't check the contents of the addresses as I am happy that the Adapter code will only let legit ones through, and the content of the addresses is outwith my control: that's on getaddress.io.

/**
 * @testdox It returns an error status code and no addresses when the postcode is invalid
 * @dataProvider provideCasesForClientErrorTests
 */
public function testReturnsErrorStatusCodeAndNoAddressesWhenPostCodeIsInvalid(
    string $postcode,
    int $statusCode
) {
    $this->client->request(
        "GET",
        sprintf("/postcode-lookup/%s", $postcode)
    );
    $response = $this->client->getResponse();

    $this->assertEquals($statusCode, $response->getStatusCode());

    $result = json_decode($response->getContent(), false);
    $this->assertObjectHasAttribute('addresses', $result);
    $this->assertEmpty($result->addresses);
}

public function provideCasesForClientErrorTests() : array
{
    return [
        [TestConstants::POSTCODE_INVALID, Response::HTTP_BAD_REQUEST],
        [TestConstants::POSTCODE_UNAUTHORIZED, Response::HTTP_UNAUTHORIZED],
        [TestConstants::POSTCODE_FORBIDDEN, Response::HTTP_FORBIDDEN],
        [TestConstants::POSTCODE_OVER_LIMIT, Response::HTTP_TOO_MANY_REQUESTS],
        [TestConstants::POSTCODE_SERVER_ERROR, Response::HTTP_INTERNAL_SERVER_ERROR]
    ];
}

Superficial tests of all the expected failure situations, making sure they "work".

This next one is more interesting:

/** @testdox it returns an error status and no addresses when there's been a server error */
public function testReturnsErrorStatusCodeAndNoAddressesWhenServerError()
{
    $container = self::getContainer();
    $mockedAddressServiceAdapter = $this
        ->getMockBuilder(AddressService\Adapter::class)
        ->disableOriginalConstructor()
        ->onlyMethods(['get'])
        ->getMock();
    $mockedAddressServiceAdapter
        ->expects($this->once())
        ->method('get')
        ->willThrowException(new AddressService\Exception("TEST_ERROR_MESSAGE"));
    $container->set(AddressService\Adapter::class, $mockedAddressServiceAdapter);

    $this->client->request(
        "GET",
        sprintf("/postcode-lookup/%s", TestConstants::POSTCODE_OK)
    );

    $response = $this->client->getResponse();

    $this->assertEquals(
        Response::HTTP_INTERNAL_SERVER_ERROR,
        $response->getStatusCode()
    );

    $result = json_decode($response->getContent(), false);
    $this->assertObjectHasAttribute('addresses', $result);
    $this->assertEmpty($result->addresses);
    $this->assertEquals("TEST_ERROR_MESSAGE", $result->message);
}

Here I'm mocking the dependency injection container to mock-out the HttpClient to fake a server error on "their" end. It's good to know how to do that, and it's fairly straight forward (he says, after googling for over an hour to find out how to do it!). I'm actually thinking now that this is a functional test, not an integration test, as it's not actually hitting the external service. Hrm. I shall move it I think. Oh: I think "functional test" not "unit test" because it's cutting across a couple of concerns: Adapter logic and Controller logic, and that's where I make the (admittedly fairly arbitrary) distinction: if the "unit" is in one object: unit test. If it goes across one object into a dependent object: functional test.

I'm pretty sure that's it. Just as well. This is a monster of an article. Sorry.


Conclusions, and "next…"

I need to improve this by paying attention to a couple of those statuses back from getaddress.io, namely the 401, 403 and 429 responses. These might need remediation on our end, so I want to log something if we get those responses. And also log something in the "catch all" catch there. If there's been an unexpected error, whilst the client might not care, we should.

I'm gonna need to mess around with Monolog to do that, so I'll take that as a separate refresher exercise.

Well done, if you got this far. All the code for this is in the 1.7.3 tag on GitHub (I've linked everything relevant directly as well).

Righto.

--
Adam