Showing posts with label PHP. Show all posts
Showing posts with label PHP. Show all posts

Thursday, 2 February 2023

PHP / Doctrine/DBAL: adding a logger to an connection

G'day:

There's plenty of stuff online about how to do this the old deprecated way, but I couldn't find any example of how to do it the "current" way. So. Here's an example.

NB: the old way is some variant of this sort of thing:

$logger = new \Doctrine\DBAL\Logging\DebugStack();
$config = new \Doctrine\DBAL\Configuration();
$config->setSQLLogger($logger);

$connection = DriverManager::getConnection(
    $connectionParams,
    $config
);

However both DebugStack and setSQLLogger are deprecated. The DebugStack thing is just a logger instance and I didn't want to use that anyhow, but being able to set the logger I did want to use would be nice.

I dug into setSQLLogger, and it says this:

@deprecated Use {@see setMiddlewares()} and {@see \Doctrine\DBAL\Logging\Middleware} instead.

OK we'll do that then.

I'll spare you the code, but initially I was trying to set the middlewares (I'm not sure(*) that's the plural of "middleware", guys) on an existing connection object, and that was not working at all. Nothing was being logged. Then I discovered someone else with the same woes as me @ https://github.com/doctrine/dbal/issues/5366, and apparently one cannot actually do this, as the middlewares are only paid-attention-to when the connection is first created, so I needed to pass the config in during connection-creation. This is not well-documented either, but looking at the source code answered that.

Here's what I ended up with in my test:

namespace adamcameron\php8\tests\Functional\Database;

use adamcameron\php8\tests\Integration\Fixtures\Database as DB;
use Doctrine\DBAL\Configuration;
use Doctrine\DBAL\Logging\Middleware;
use Monolog\Handler\TestHandler;
use Monolog\Logger;
use PHPUnit\Framework\TestCase;

/** @testdox Tests Database objects */
class DbLoggingTest extends TestCase
{
    /** @testdox It can log SQL traffic */
    public function testLogging()
    {
        $testLogger = new Logger("test");
        $testHandler = new TestHandler();
        $testLogger->setHandlers([$testHandler]);
        $middleware = new Middleware($testLogger);

        $config = new Configuration();
        $config->setMiddlewares([$middleware]);

        $connection = DB::getDbalConnection($config);

        $result = $connection->executeQuery("SELECT * FROM numbers WHERE id = ?", [5]);

        $this->assertEquals([5, 'five', 'rima'], $result->fetchNumeric());

        $withSql = array_filter($testHandler->getRecords(), function ($record) {
            return in_array('SELECT * FROM numbers WHERE id = ?', $record->context);
        });
        $this->assertCount(1, $withSql);
    }
}

Given Middleware only supports Loggers, I'm not sure this is a great architectural win for DBAL: moving from a clear and on-point description of the functionality to a nebulous, vague, and actually inaccurate term like "Middleware". Oh well.

For the sake of completeness (and an explanation of why I needed to dick around filtering the log to get the one I was looking for, this is what is logged for that call:

array(2) {
  [0] =>
  class Monolog\LogRecord#363 (7) {
    public readonly DateTimeImmutable $datetime =>
    class Monolog\DateTimeImmutable#357 (4) {
      private $useMicroseconds =>
      bool(true)
      public $date =>
      string(26) "2023-02-02 23:27:14.519171"
      public $timezone_type =>
      int(3)
      public $timezone =>
      string(13) "Europe/London"
    }
    public readonly string $channel =>
    string(4) "test"
    public readonly Monolog\Level $level =>
    enum Monolog\Level::Info : int(200);
    public readonly string $message =>
    string(35) "Connecting with parameters {params}"
    public readonly array $context =>
    array(1) {
      'params' =>
      array(6) {
        ...
      }
    }
    public array $extra =>
    array(0) {
    }
    public mixed $formatted =>
    string(204) "
        [2023-02-02T23:27:14.519171+00:00]
        test.INFO:
        Connecting with parameters {params}
        {
            "params":{
                "dbname":"db1",
                "user":"user1",
                "password":"<redacted>",
                "host":"mariadb",
                "port":"3306",
                "driver":"pdo_mysql"
            }
        }
        []
    "
  }
  [1] =>
  class Monolog\LogRecord#349 (7) {
    public readonly DateTimeImmutable $datetime =>
    class Monolog\DateTimeImmutable#348 (4) {
      private $useMicroseconds =>
      bool(true)
      public $date =>
      string(26) "2023-02-02 23:27:14.780537"
      public $timezone_type =>
      int(3)
      public $timezone =>
      string(13) "Europe/London"
    }
    public readonly string $channel =>
    string(4) "test"
    public readonly Monolog\Level $level =>
    enum Monolog\Level::Debug : int(100);
    public readonly string $message =>
    string(65) "Executing statement: {sql} (parameters: {params}, types: {types})"
    public readonly array $context =>
    array(3) {
      'sql' =>
      string(34) "SELECT * FROM numbers WHERE id = ?"
      'params' =>
      array(1) {
        ...
      }
      'types' =>
      array(1) {
        ...
      }
    }
    public array $extra =>
    array(0) {
    }
    public mixed $formatted =>
    string(194) "
        [2023-02-02T23:27:14.780537+00:00]
        test.DEBUG:
        Executing statement:
            {sql}
            (parameters: {params}, types: {types})
            {
                "sql":"SELECT * FROM numbers WHERE id = ?",
                "params":{
                    "1":5
                },
                "types":{
                    "1":2
                }
            }
        []
    "
  }
}

(Well that pads the word count…)

I like how it's had the presence of mind to redact the password from the connection params.

Anyway, that's it. I just wanted that code written down somewhere were it can be found.

Righto.

--
Adam


(*) In fact I've very bloody sure it oughtn't be (and, yesyesyes, I know it's been adopted as such, but this is LCD thinking at work. "-ware" already makes it a mass noun, so… plural).

Wednesday, 1 February 2023

PHP: refactoring logic out of a controller and into a service class

G'day:

Context

In yesterday's article - TDD & Symfony & Monolog: adding some logging to that endpoint from last time - I added some logging logic into this address-lookup web service endpoint I'm putting together. At the end of it, I thought the logic - which I had slung into the controller - was in the wrong place, and I needed to shimmy it into an intermediary service class between the controller and the upstream web service adapter (this will make more sense once you see the code below). Today I've done that refactoring.

Where I was

Here's the controller code:

public function doGet(string $postcode) : JsonResponse
{
    try {
        $response = $this->addressServiceAdapter->get($postcode);

        $this->logUnexpectedFailures($response, $postcode);

        return new JsonResponse(
            [
                'postcode' => $postcode,
                'addresses' => $response->getAddresses(),
                'message' => $response->getMessage()
            ],
            $response->getHttpStatus()
        );
    } catch (\Exception $e) {
        return new JsonResponse(
            [
                'postcode' => $postcode,
                'addresses' => [],
                'message' => $e->getMessage()
            ],
            HttpStatusCode::HTTP_INTERNAL_SERVER_ERROR
        );
    }
}
private const RESPONSES_TO_LOG = [
    HttpStatusCode::HTTP_UNAUTHORIZED => Level::Critical,
    HttpStatusCode::HTTP_FORBIDDEN => Level::Critical,
    HttpStatusCode::HTTP_TOO_MANY_REQUESTS => Level::Warning,
    HttpStatusCode::HTTP_INTERNAL_SERVER_ERROR => Level::Warning
];

// …

private function logUnexpectedFailures(
    AddressService\Response $response,
    string $postcode
): void {
    $statusCode = $response->getHttpStatus();

    if (array_key_exists($statusCode, self::RESPONSES_TO_LOG)) {
        $this->logger->log(
            self::RESPONSES_TO_LOG[$statusCode],
            AddressService\Adapter::ERROR_MESSAGES[$statusCode],
            ['postcode' => $postcode, 'message' => $response->getMessage()]
        );
    }
}

What's going on here is that the upstream web service could return with a number of cautionary HTTP status codes which we really need to know about if they happen, so I've added some logging in to make sure we know.

And the adapter code it calls:

public function get(string $postCode) : AddressService\Response
{
    $response = $this->makeRequest($postCode);
    $lookupResult = $this->extractValidLookupResult($response);

    return $this->handleValidatedResponse($response, $lookupResult);
}

After getting the response from the upstream third-party web service, I validate what comes back (valid JSON with the schema I expect, etc; this could throw an exception if the data is no good), I pass it to a handler that returns well-formed data:

private function handleValidatedResponse(
    ResponseInterface $response,
    array $lookupResult
): AddressService\Response {
    $statusCode = $response->getStatusCode();

    if ($statusCode == HttpFoundationResponse::HTTP_OK) {
        return $this->handleSuccessResponse($lookupResult);
    }
    return $this->handleFailureResponse($lookupResult, $statusCode);
}

private function handleSuccessResponse(array $lookupResult): AddressService\Response
{
    if (
        !array_key_exists("addresses", $lookupResult)
        || !is_array($lookupResult["addresses"])
        || count(array_filter($lookupResult["addresses"], fn($address) => !is_string($address)))
    ) {
        throw new AddressService\Exception("Response JSON schema is not valid");
    }

    return new AddressService\Response(
        $lookupResult["addresses"],
        HttpFoundationResponse::HTTP_OK
    );
}

private function handleFailureResponse(
    array $lookupResult,
    int $statusCode
): AddressService\Response {
    if (array_key_exists("Message", $lookupResult) && is_string($lookupResult["Message"])) {
        return new AddressService\Response([], $statusCode, $lookupResult["Message"]);
    }
    return new AddressService\Response([], $statusCode, "No failure message returned from service");
}

There's no need to worry too much about the details there. Just that we're returning data from a third-party web service, and the results could be good, bad or ugly.

This approach of having a controller calling the adapter directly was fine for the first iteration of this code, where it was basically getting the data from the adapter and returning it. Once I added in the logging - and the conditionality (business logic) around it - it occurred to me that the controller was not the right place for this. Logging is nothing to do with servicing an HTTP request, after all. And that's all a controller is supposed to do. I also couldn't push the logging into the adapter, as that's not an adapter's job either. It's just to provide an application-friendly interface to something that doesn't have such an interface. And nothing else.

So I'm shifting that logging logic into a service.

Where I am now

There's the controller now:

class PostcodeLookupController extends AbstractController
{

    public function __construct(private readonly PostcodeLookupService $postcodeLookupService)
    {
    }

    public function doGet(string $postcode) : JsonResponse
    {
        $response = $this->postcodeLookupService->lookup($postcode);
        return new JsonResponse(
            [
                'postcode' => $postcode,
                'addresses' => $response->getAddresses(),
                'message' => $response->getMessage()
            ],
            $response->getHttpStatus()
        );
    }
}

Note how it is receiving a PostcodeLookupService now. It was receiving the Adapter and the Logger before. And also note how it's really not doing much any more. Asking for some data from the service, returning it. Simple. Excellent.

And the service:

class PostcodeLookupService
{
    private const RESPONSES_TO_LOG = [
        HttpStatusCode::HTTP_UNAUTHORIZED => Level::Critical,
        HttpStatusCode::HTTP_FORBIDDEN => Level::Critical,
        HttpStatusCode::HTTP_TOO_MANY_REQUESTS => Level::Warning,
        HttpStatusCode::HTTP_INTERNAL_SERVER_ERROR => Level::Warning
    ];

    public function __construct(
        private readonly GetAddress\Adapter $adapter,
        private readonly LoggerInterface $addressServiceLogger
    ) {
    }

    public function lookup(string $postcode): GetAddress\Response
    {
        try {
            return $this->getAddresses($postcode);
        } catch (GetAddress\Exception $e) {
            return $this->handleAdapterException($e, $postcode);
        }
    }

    private function getAddresses(string $postcode): GetAddress\Response
    {
        $response = $this->adapter->get($postcode);

        $this->logUnexpectedFailures($response, $postcode);

        return $response;
    }

    private function logUnexpectedFailures(
        GetAddress\Response $response,
        string $postcode
    ): void {
        $statusCode = $response->getHttpStatus();

        if (array_key_exists($statusCode, self::RESPONSES_TO_LOG)) {
            $this->addressServiceLogger->log(
                self::RESPONSES_TO_LOG[$statusCode],
                GetAddress\Adapter::ERROR_MESSAGES[$statusCode],
                ['postcode' => $postcode, 'message' => $response->getMessage()]
            );
        }
    }

    private function handleAdapterException(
        GetAddress\Exception|\Exception $e,
        string $postcode
    ): GetAddress\Response {
        $this->addressServiceLogger->log(
            Level::Error,
            $e->getMessage(),
            ['postcode' => $postcode, 'message' => $e->getMessage()]
        );

        return new GetAddress\Response(
            [],
            HttpStatusCode::HTTP_INTERNAL_SERVER_ERROR,
            $e->getMessage()
        );
    }
}

All the logic around calling the Adapter for the data, the error handling and logging is now here instead. I've also made a point of refactoring the main try / catch so that the "what it's doing" and "handling errors" is nicely separate.

All the code for this iteration of this work is tagged as 1.10 on Github.

Righto.

--
Adam

Sunday, 22 January 2023

Docker: adding a MariaDB container to my PHP & Nginx ones

G'day:

I'm pretty much just noting down how I've progressed my PHP8 test app in this one (see PHP: returning to PHP and setting up a PHP8 dev environment and other articles around this date tagged with the PHP8 label, around this date). I need a DB added to the PHP8 and Nginx containers I already have, for the next bit of stuff I want to look at.


docker/docker-compose.yml

I've added a mariadb service, and set some environment variables in the PHP8 service as well:

version: "3"
services:
  nginx:
  	# […]

  php:
    build:
      context: php
      dockerfile: Dockerfile

    environment:
      - MARIADB_DATABASE=${MARIADB_DATABASE}
      - MARIADB_USER=${MARIADB_USER}
      - MARIADB_PASSWORD=${MARIADB_PASSWORD}

    stdin_open: true
    tty: true

    volumes:
      - ..:/var/www

  mariadb:
    build:
      context: mariadb
      dockerfile: Dockerfile

    environment:
      - MARIADB_ROOT_PASSWORD=${MARIADB_ROOT_PASSWORD}
      - MARIADB_DATABASE=${MARIADB_DATABASE}
      - MARIADB_USER=${MARIADB_USER}
      - MARIADB_PASSWORD=${MARIADB_PASSWORD}

    ports:
      - "3382:3306"

    stdin_open: true
    tty: true

    volumes:
      - mariaDbData:/var/lib/mariadb

volumes:
  mariaDbData:

Those env vars are ones the MariaDB image docs on Dockerhub mandate. I'm also passing them into the PHP container so that it doeesn't need them recorded anywhere.


docker/.env

COMPOSE_PROJECT_NAME=php8

MARIADB_DATABASE=db1
MARIADB_USER=user1

# the following are to be provided to `docker-compose up`
MARIADB_ROOT_PASSWORD=
MARIADB_PASSWORD=

The only notable thing here is that - because this file is going into source control - I am not specifying the passwords; I'm just signifiying they need to exist.


docker/mariadb/Dockerfile

FROM mariadb:latest

COPY ./docker-entrypoint-initdb.d/ /docker-entrypoint-initdb.d/

CMD ["mysqld"]

EXPOSE 3306

And in ./docker-entrypoint-initdb.d/ I have these:

# docker/mariadb/docker-entrypoint-initdb.d/1.createAndPopulateTestTable.sql

USE db1;

CREATE TABLE test (
    id INT NOT NULL,
    value VARCHAR(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,

    PRIMARY KEY (id)
) ENGINE=InnoDB;

INSERT INTO test (id, value)
VALUES
    (101, 'Test row 1'),
    (102, 'Test row 2')
;

ALTER TABLE test MODIFY COLUMN id INT auto_increment;
# docker/mariadb/docker-entrypoint-initdb.d/2.createAndPopulateNumbersTable.sql
USE db1;

CREATE TABLE numbers (
    id INT NOT NULL,
    en VARCHAR(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
    mi VARCHAR(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,

    PRIMARY KEY (id)
) ENGINE=InnoDB;

INSERT INTO numbers (id, en, mi)
VALUES
    (1, 'one', 'tahi'),
    (2, 'two', 'rua'),
    (3, 'three', 'toru'),
    (4, 'four', 'wha'),
    (5, 'five', 'rima'),
    (6, 'rima', 'ono')
;

ALTER TABLE numbers MODIFY COLUMN id INT auto_increment;

I'm seeding the DB with some test data. Any files dropped into that /docker-entrypoint-initdb.d/ directory in the image will be picked up by the MariaDB process when it first creates the DB (see the docs for the image again: Docker hub › MariaDB › Initializing a fresh instance).


Shell scripts

Because this rig requires one to pass passwords to docker-compose up, I've created a coupla shell scripts to remind me to do it right:

#!/bin/bash
# docker/bin/rebuildContainers.sh

# usage
# cd to directory containing docker-compose.yml
# bin/rebuildContainers.sh [DB root password] [DB user password]
# EG:
# cd ~/src/php8/docker
# bin/rebuildContainers.sh 123 1234

clear; printf "\033[3J"
docker-compose down --remove-orphans --volumes
docker-compose build --no-cache
MARIADB_ROOT_PASSWORD=$1 MARIADB_PASSWORD=$2 docker-compose up --force-recreate --detach
#!/bin/bash
# docker/bin/restartContainers.sh

# usage
# cd to directory containing docker-compose.yml
# bin/restartContainers.sh [DB root password] [DB user password]
# use same passwords as when initially calling rebuildContainers.sh

# EG:
# cd ~/src/php8/docker
# bin/restartContainers.sh 123 1234

clear; printf "\033[3J"
docker-compose stop
docker-compose up --detach nginx
MARIADB_PASSWORD=$2 docker-compose up --detach php
MARIADB_ROOT_PASSWORD=$1 MARIADB_PASSWORD=$2 docker-compose up --detach mariadb

This just saves some typing. In all honesty I am using 123 and 1234 for the respective passwords, but it doesn't matter. It's a good practice to not have passwords anywhere in source code, and this seems a reasonable way to me to make sure the values end up being where they need to be.


readme.md

I'll spare you the content here (the heading there is linked to the file), all I did to that was update my installation instructions to use the docker/bin/rebuildContainers.sh script instead of individual statements, and added a section about docker/bin/restartContainers.sh.


Test

Where would I be without a test. I've thrown a quick one together to test that the test data is there. And I will run this in conjunction with the rest of the tests, to make sure I have not caused any regressions.

// test/integration/DbTest.php

namespace adamcameron\php8\test\integration;

use Doctrine\DBAL\Connection;
use Doctrine\DBAL\DriverManager;
use PHPUnit\Framework\TestCase;
use \stdClass;

/** @testdox Tests the stub DB */
class DbTest extends TestCase
{

    /** @testdox it can fetch records from the test table */
    public function testFetchRecords()
    {
        $expectedRecords = [
            ["id" => 101, "value" => "Test row 1"],
            ["id" => 102, "value" => "Test row 2"]
        ];

        $connection = $this->getDbalConnection();
        $result = $connection->executeQuery("SELECT id, value FROM test ORDER BY id");

        $actualRecords = $result->fetchAllAssociative();

        $this->assertEquals($expectedRecords, $actualRecords);
    }

    private function getDbalConnection() : Connection
    {
        $parameters = $this->getConnectionParameters();
        return DriverManager::getConnection([
            'dbname' => $parameters->database,
            'user' => $parameters->username,
            'password' => $parameters->password,
            'host' => $parameters->host,
            'port' => $parameters->port,
            'driver' => 'pdo_mysql'
        ]);
    }

    private function getConnectionParameters() : stdClass
    {
        return (object) [
            "host" => "mariadb",
            "port" => "3306",
            "database" => getenv("MARIADB_DATABASE"),
            "username" => getenv("MARIADB_USER"),
            "password" => getenv("MARIADB_PASSWORD")
        ];
    }
}

All pretty straight forward. Note how I'm reading the DB info from the environment variables in getConnectionParameters. Take my word for it for now on the DB-handling code. I'll get to that in a different article.


That's it. Nothing insightful. I'm just documenting what I've done and why.

Righto.

--
Adam

Saturday, 21 January 2023

PHP: looking at ways of making HTTP requests

G'day:

I'm reacquainting myself with PHP, and part of this process is chucking some tests together to demonstrate to myself how bits and pieces of it works. This has the added bonus of being able to put the code in front of my team, to help provide learning info for them. This article is pretty much just showing sample code, and it's for the reader to compare and contrast. There's likely not gonna be too much exposition from me once we get to the code. I'm sure I can pad things out by a few hundred words before we get there though. I am me after all.

This time, I've decided to revisit how to make HTTP requests.

I've got four candidate solutions to look at:

I am aware of PHP's curl extension always being available, but its API is a bit of a mess (it's been part of PHP since the bad old days).

I've also used Guzzle in the past, with mixed success. It started out being simple and handy, and I liked it. But then between a major version bump (I can't remember which versions this occurred between), the old API was basically dumped in favour of a new, non-backwards-compatible, and largely (and pointlessly IMO) overly complex promise-based approach. To provide asynchronicity in HTTP requests. Which was something I never needed and seemed like an odd addition to an HTTP library. I suspect the author had started to look at Node.jS with all its async HTTP shiz, and went "I know… I'll ruin something perfectly useful by adding this crap into it as well". Ugh. However I note Guzzle is still around, so - armed with an open mind - I'll look at that too.

During my googling I have also spied that Symfony has an HTTP client too. It probably always did, but in my last gig we went the Guzzle route, so I had not looked further afield.

Also during my googling (and reading the Guzzle docs), I discovered PHP's own streams extension can be used to make HTTP requests. That sounds interesting, so have decided to give that a go too.

My approach is to create a test class, and add a test for each of those four platforms, to do each of a GET and a POST. They are not complicated tests, it's just a case of getting the thing to do something simple that I can expect results from.

I will also concede that I used Copilot to do probably 80% of the work here, with my polishing that last 20%.

Installation

  • curl needs ext-curl installed. It ships with the Docker image, so I didn't have to do anything for this.
  • Installing Guzzle is a matter of adding it as a dependency in composer.json: "guzzlehttp/guzzle": "^7.5.0" at time of writing.
  • Similarly with Symfony's HTTP client: "symfony/http-client": "^6.2.2"
  • And PHP's streams lib is native to PHP. No installation necessary.

Curl

/** @testdox it can make a GET request */
public function testGet()
{
    $ch = curl_init();
    curl_setopt_array($ch, [
        CURLOPT_URL => 'https://api.github.com/users/adamcameron',
        CURLOPT_USERAGENT => $this->getUserAgentForCurl(),
        CURLOPT_RETURNTRANSFER => 1
    ]);
    $response = curl_exec($ch);
    curl_close($ch);

    $this->assertEquals(200, curl_getinfo($ch, CURLINFO_HTTP_CODE));
    $this->assertJson($response);
    $this->assertGitInfoIsCorrect($response);
}

It also uses these two helper methods:

private function getUserAgentForCurl(): string
{
    return sprintf("curl/%s", curl_version()['version']);
}
protected function assertGitInfoIsCorrect(string $response): void
{
    $myGitMetadata = json_decode($response);
    $this->assertEquals('adamcameron', $myGitMetadata->login);
    $this->assertEquals('Adam Cameron', $myGitMetadata->name);
}

(A bunch of the other tests below also use that one above).

The GET test in each case will be to get my own GitHub profile and to superficially check it's been fetched properly. BTW I needed that getUserAgentForCurl carry-on because curl by itself does notsent a user agent, and Github says "nuh-uh" if it doesn't get one. So I've just contrived a user agent that is the one that the underlying curl implementation would use (eg: ike if one was doing a curl from bash).

The POST test will post to https://httpbin.org/post. I only discovered httpbin.org when I was doing this exercise, and I wanted a simple (public) way of echoing back a post request. Handy.

/** @testdox it can make a POST request */
public function testPost()
{
    $ch = curl_init();
    curl_setopt_array($ch, [
        CURLOPT_URL => 'https://httpbin.org/post',
        CURLOPT_USERAGENT => $this->getUserAgentForCurl(),
        CURLOPT_RETURNTRANSFER => 1,
        CURLOPT_POST => 1,
        CURLOPT_POSTFIELDS => ['foo' => 'bar']
    ]);
    $response = curl_exec($ch);
    curl_close($ch);

    $this->assertEquals(200, curl_getinfo($ch, CURLINFO_HTTP_CODE));
    $this->assertJson($response);
    $httpBinResponse = json_decode($response);

    $this->assertEquals('bar', $httpBinResponse->form->foo);
}

An example of what https://httpbin.org/post returns is:

{
    "args":{
        
    },
    "data":"",
    "files":{
        
    },
    "form":{
        "foo":"bar"
    },
    "headers":{
        "Accept":"*/*",
        "Content-Length":"141",
        "Content-Type":"multipart/form-data; boundary=------------------------b0133bb008e6829b",
        "Host":"httpbin.org",
        "User-Agent":"curl/7.74.0",
        "X-Amzn-Trace-Id":"Root=1-63cc3251-4a3f92c809ad00f75261466a"
    },
    "json":null,
    "origin":"82.8.81.31",
    "url":"https://httpbin.org/post"
}

Guzzle

First up: I'm really pleased how compact and straight-forward Guzzle's code is for these exercises. And also that one is not forced to write async code for a non-async situation.

/** it can make a GET request */
public function testGet()
{
    $client = new Client();
    $response = $client->request('GET', 'https://api.github.com/users/adamcameron');
    $this->assertEquals(200, $response->getStatusCode());
    $this->assertJson($response->getBody());
    $this->assertGitInfoIsCorrect($response->getBody());
}
/** @testdox it can make a POST request */
public function testPost()
{
    $client = new Client();
    $response = $client->request(
        'POST',
        'https://httpbin.org/post',
        ['form_params' => ['foo' => 'bar']]
    );
    $this->assertEquals(200, $response->getStatusCode());
    $this->assertJson($response->getBody());
    $httpBinResponse = json_decode($response->getBody());
    $this->assertEquals('bar', $httpBinResponse->form->foo);
}

I also decided to revisit the async side of things:

/** it can make an asynchronous GET request */
public function testAsyncGet()
{
    $client = new Client();
    $promise = $client->requestAsync('GET', 'https://api.github.com/users/adamcameron');
    $response = $promise->wait();
    $this->assertEquals(200, $response->getStatusCode());
    $this->assertJson($response->getBody());
    $this->assertGitInfoIsCorrect($response->getBody());
}

Simple. It seems the current implementation is taking the "async-await" approach with these things like JS has these days.

That was not much of a test though. This time I am gonna make a bunch of requests (which are artificially slow) and make sure they do seem to run asynchronously. I've slung this in my web directory:

// html/test-fixtures/slow.php
$timeToWait = $_GET['timeToWait'] ?? 0;
sleep($timeToWait);
echo "waited $timeToWait seconds";

And calling that with varying delays:

/** it can make multiple asynchronous GET requests */
public function testMultipleAsyncGet()
{
    $client = new Client();
    $requestsToMakeConcurrently = [
        $client->getAsync('http://nginx/test-fixtures/slow.php?timeToWait=1'),
        $client->getAsync('http://nginx/test-fixtures/slow.php?timeToWait=2'),
        $client->getAsync('http://nginx/test-fixtures/slow.php?timeToWait=3')
    ];
    $startTime = microtime(true);
    $responses = Promise\Utils::unwrap($requestsToMakeConcurrently);
    $endTime = microtime(true);

    $totalTime = $endTime - $startTime;
    $this->assertGreaterThan(3, $totalTime);
    $this->assertLessThan(4, $totalTime);

    array_walk($responses, function ($response, $i) {
        $this->assertEquals(200, $response->getStatusCode());
        $this->assertEquals(sprintf("waited %d seconds", $i+1), $response->getBody());
    });
}

The assertions there are a bit woolly. I figured it should def take longer than 3sec cos at least one of the requests will take 3sec. Plus there'll be a wee bit of overhead. That overhead ought not be more than a second, so if the whole lot finishes in less than 4sec, it's a pretty good indicator that all three requests were being made simultaneously. It occurs to me now I could perhaps look @ the Nginx activity logs for when the requests come in. Please hold…

172.31.0.4 - - [21/Jan/2023:18:57:49 +0000] "GET /test-fixtures/slow.php?timeToWait=1 HTTP/1.1" 200 27 "-" "GuzzleHttp/7"
172.31.0.4 - - [21/Jan/2023:18:57:50 +0000] "GET /test-fixtures/slow.php?timeToWait=2 HTTP/1.1" 200 27 "-" "GuzzleHttp/7"
172.31.0.4 - - [21/Jan/2023:18:57:51 +0000] "GET /test-fixtures/slow.php?timeToWait=3 HTTP/1.1" 200 27 "-" "GuzzleHttp/7"

Now Nginx is logging when it responds to the request, not when it receives it, so what we can infer from this is that the requests all arrived at 18:57:48, and the 1sec request finished after 1sec at 18:57:49; the 2sec request finished after 2sec @ 18:57:50, and similarly the third one, 3sec, finished after 3 seconds at 18:57:51.

It's easier to see if I make the requests hang on for different periods of time. Here's an example where they take 1sec, 12sec and 23sec respectively:

172.31.0.4 - - [21/Jan/2023:18:59:07 +0000] "GET /test-fixtures/slow.php?timeToWait=1 HTTP/1.1" 200 27 "-" "GuzzleHttp/7"
172.31.0.4 - - [21/Jan/2023:18:59:18 +0000] "GET /test-fixtures/slow.php?timeToWait=12 HTTP/1.1" 200 28 "-" "GuzzleHttp/7"
172.31.0.4 - - [21/Jan/2023:18:59:29 +0000] "GET /test-fixtures/slow.php?timeToWait=23 HTTP/1.1" 200 28 "-" "GuzzleHttp/7"

We can infer they all arrived at 18:59:06. 1sec later at 18:59:07 the first request completed; 12sec later the second one completed at 18:59:18 (18:59:18 - 18:59:06 is 12sec); and lastly the third request - which will take 23sec to run - indeed finishes at 18:59:29 - 18:59:06 = 23sec later.

Excellent. Working as expected.

For completeness I also tested an async POST request:

/** @testdox it can make an asynchronous POST request */
public function testAsyncPost()
{
    $client = new Client();
    $promise = $client->requestAsync(
        'POST',
        'https://httpbin.org/post',
        ['form_params' => ['foo' => 'bar']]
    );
    $response = $promise->wait();
    $this->assertEquals(200, $response->getStatusCode());
    $this->assertJson($response->getBody());
    $httpBinResponse = json_decode($response->getBody());
    $this->assertEquals('bar', $httpBinResponse->form->foo);
}

No surprises.


Symfony

/** @testdox it can make a GET request */
public function testGet()
{
    $client = HttpClient::create();
    $response = $client->request('GET', 'https://api.github.com/users/adamcameron');
    $this->assertEquals(200, $response->getStatusCode());
    $this->assertJson($response->getContent());
    $this->assertGitInfoIsCorrect($response->getContent());
}

This is identical to the Guzzle example except Symfony uses a factory method to create the client object compared Guzzle just using new; and Guzzle uses getBody instead of Symfony's getContent.

/** @testdox it can make a POST request */
public function testPost()
{
    $client = HttpClient::create();
    $response = $client->request(
        'POST',
        'https://httpbin.org/post',
        ['body' => ['foo' => 'bar']]
    );
    $this->assertEquals(200, $response->getStatusCode());
    $this->assertJson($response->getContent());
    $httpBinResponse = json_decode($response->getContent());
    $this->assertEquals('bar', $httpBinResponse->form->foo);
}

It's just occurred to me that knowing Symfony, it can likely do async request collections too. And after some googling: sure enough I've found a way ("Symfony › HTTP Client › Concurrent Requests"):

/** it can make multiple asynchronous GET requests */
public function testMultipleAsyncGet()
{
    $client = HttpClient::create();
    $requestsToMakeConcurrently = [
        $client->request('GET', 'http://nginx/test-fixtures/slow.php?timeToWait=1'),
        $client->request('GET', 'http://nginx/test-fixtures/slow.php?timeToWait=2'),
        $client->request('GET', 'http://nginx/test-fixtures/slow.php?timeToWait=3')
    ];
    $stream = $client->stream($requestsToMakeConcurrently);

    $i = 1;
    $startTime = microtime(true);
    foreach ($stream as $response => $chunk) {
        if ($chunk->isLast()) {
            $this->assertEquals(200, $response->getStatusCode());
            $this->assertEquals("waited $i seconds", $response->getContent());
            $i++;
        }
    }
    $endTime = microtime(true);
    $totalTime = $endTime - $startTime;
    $this->assertGreaterThan(3, $totalTime);
    $this->assertLessThan(4, $totalTime);
}

This is analogous to the Guzzle version. It's implementation is not as nice though IMO.


PHP Streams

/** @testdox it can make a GET request */
public function testGet()
{
    $context = stream_context_create([
        'http' => [
            'method' => 'GET',
            'header' => ['User-Agent: ' . $this->getUserAgentForCurl()]
        ]
    ]);
    $response = file_get_contents('https://api.github.com/users/adamcameron', false, $context);
    $this->assertJson($response);
    $this->assertGitInfoIsCorrect($response);
}
/** @testdox it can make a POST request */
public function testPost()
{
    $context = stream_context_create([
        'http' => [
            'method' => 'POST',
            'header' => [
                'User-Agent: ' . $this->getUserAgentForCurl(),
                'Content-Type: application/x-www-form-urlencoded'
            ],
            'content' => http_build_query(['foo' => 'bar'])
        ]
    ]);
    $response = file_get_contents('https://httpbin.org/post', false, $context);
    $this->assertJson($response);
    $httpBinResponse = json_decode($response);
    $this->assertEquals('bar', $httpBinResponse->form->foo);
}

OK. It's poss just me being pedantic, but I get a bit itchy looking at file_get_contents on a URL. I mean I know an HTTP request is fetching a file - so semantically that's fine - but it still seems odd.


Conclusion

For these superficial test cases, I prefer Guzzle. Doubtless there more one can do with Symfony's HTTP client, because there's always more one can do with Symfony's stuff; but the same will apply with Guzzle too no doubt. I did not know about PHP's streams before, and whilst this might not be a good use of it, there'll likely be other situations to use them.

I'm mostly pleased that Guzzle seems easy to use again, and for both sync and async stuff. Cool.

All the code shown in here is @ /test/integration/http on Github.

Righto.

--
Adam

PHP: returning to PHP and setting up a PHP8 dev environment

G'day:

I need to do some PHP work, and for that I need to have a PHP dev environment. I'm very rusty when it comes to PHP - I've not touched it for 2-3 years or so and my old brain doesn't hold on to things very well - and since that time I have shifted to using Docker for my environments anyhow. I've never used PHP in Docker before. So there's a challenge. And what's this? PHP is now up to version 8.2, with 8.3 being tested. The last time I touched PHP 7.2 was the new thing (we were still mostly on 5.5 at that time, that said).

Therefore I have a mini project ahead of me:

  • Get PHP8.2 running in a Docker container.
  • Get Nginx running in a different container, proxying requests to the PHP one.
  • Have Composer up and running.
  • So I can install PHPUnit.
  • And run some basic tests of the installation.
  • With code-coverage reporting on the tests (code coverage requires a debug module to be installed and running too).
  • Also get PHPMD and PHPCS running too.
  • Bonus: be able to run the tests from my IDE, on my host machine.

Success here will be to be able to view the HTML code coverage report, served by Nginx, showing code being covered by testing.

Full disclosure: I did all this a few nights ago, and I am repeating the exercise now for the purposes of this article.

Application file structure

This shows the file system layout I'm aiming for:

/var/www# tree -L 1
.
|-- docker
|-- html
|-- src
|-- test
`-- vendor

5 directories, 0 files
/var/www#
  • docker.Docker stuff like docker-compose.yml and sub-directories for the various containers' Dockerfiles and other config / assets are in here.
  • html. Files that will be served by Nginx go in here.
  • src. Application code goes here.
  • test. Test code goes here.
  • vendor. The app's Composer dependencies go in here.

This is all standard PHP-app stuff, except my personal decision of how to organise the Docker files.


PHP in a container

docker-compose.yml

The docker-compose.yml service definition is pretty simple:

version: "3"

services:
  php:
    build:
      context: php
      dockerfile: Dockerfile

    stdin_open: true
    tty: true

    volumes:
      - ..:/var/www

/var/www is the directory the container expected to see PHP stuff in, so I ran with it. It doesn't really matter.

.env

Oh I have a wee .env file too:

COMPOSE_PROJECT_NAME=php8

Just so the container names are a bit more on-point when they get created.

Dockerfile

The Dockerfile, on the other hand, is a bit complicated, and took me ages to google all the crap I needed to get together to make PHP 8.2 work in a container with a real-world set of extensions loaded, etc. Deep breath…

FROM php:8.2.1-fpm

RUN ["apt-get", "update"]
RUN ["apt-get", "install", "-y", "zip", "unzip", "git", "vim"]

COPY php.ini /usr/local/etc/php/php.ini

COPY --from=composer:latest /usr/bin/composer /usr/local/bin/composer

RUN pecl install xdebug && docker-php-ext-enable xdebug
COPY conf.d/xdebug.ini /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini
COPY conf.d/error_reporting.ini /usr/local/etc/php/conf.d/error_reporting.ini

RUN apt-get install -y libicu-dev && docker-php-ext-configure intl && docker-php-ext-install intl

RUN ["apt-get", "install", "-y", "libz-dev", "libzip-dev"]
RUN docker-php-ext-configure zip && docker-php-ext-install zip
RUN docker-php-ext-configure bcmath && docker-php-ext-install bcmath
RUN docker-php-ext-configure pdo_mysql && docker-php-ext-install pdo_mysql
RUN docker-php-ext-configure opcache && docker-php-ext-install opcache

RUN curl -1sLf 'https://dl.cloudsmith.io/public/symfony/stable/setup.deb.sh' | bash
RUN ["apt-get", "install", "-y", "symfony-cli"]

WORKDIR /var/www
ENV COMPOSER_ALLOW_SUPERUSER 1

I'll go line-by-line, where there is anything noteworthy, or to explain my decisions.

  • php-fpm. I readily concede I have no idea what all the tag variants of PHP images are on Docker Hub. But I have used php-fpm in the past and know it to work. So: running with it. I am specifically not using the Alpine variant as it doesn't come with BASH, and life is too short to negotiate ASH instead. And I am not trying to economise on disk space for this container anyhow.
  • Baseline APT stuff. Composer needs zip, unzip and git (I learned this by trial and error). I need vim.
  • PHP doesn't have a php.ini file by default although needs it. It ships with php.ini-development and php.ini-production, and it's down to the dev to pick which to use. This file is base on php.ini-development, with the following changes (mostly from recommendations from Symfony › Performance › Use the OPcache Byte Code Cache):
    • realpath_cache_size = 4096k
    • realpath_cache_ttl = 600
    • date.timezone = Europe/London
    • opcache.enable=1
    • opcache.memory_consumption=256
    • opcache.max_accelerated_files=20000

    These are all just a matter of "uncommenting the example setting and tweak its value": normal php.ini stuff. No doubt I will further tweak that as I go, but that's a start.
  • Install Composer. It seems odd that Composer doesn't have the ubiquity that there's an APT package for it.
  • Install Xdebug. PHPUnit needs this for code coverage analysis. Plus at some stage I might start coding like a grown-up and use line debugging. Maybe.
  • I was following along the instructions on "Setup Step Debugging in PHP with Xdebug 3 and Docker Compose" to install Xdebug, and it suggested these settings to use. Yeah cool. I do not know any better. See below this list for the file contents.
  • All of this lot are just libraries that the cited PHP extensions need to be able to run.
  • The ultimate object of the exercise (well: the next part of the exercise) is to get Symfony installed in this app. Whilst setting up the PHP extensions I knew I would be wanting, I thought to look-up what Symfony would need too, and its guidance was to install the Symfony CLI and it would tell me (via symfony check:requirements. Hence installing this now. It was Symfony that reminded me to install all the highlighted extensions. handy. It's also handy that the PHP Docker image comes with those docker-php-ext-configure and docker-php-ext-configure utils, as it makes it a lot easier. There's also a bit of dependency-heck (not quite bad enough to use the work "hell" here) going on installing them, cos sometimes - like with libz-dev and libzip-dev - there are upsteam dependencies needed too. I think I got off pretty lightly here, just needing those two.
  • /var/www is where I want to land when I start a shell on the container.
  • I need to set this otherwise Composer complains about installing stuff as root. This'd be an issue in a production container, maybe. But it's not an issue on dev IMO.

Here are those PHP config files I mentioned above in the Xdebug bit:

# docker/php/conf.d/xdebug.ini

zend_extension=xdebug

[xdebug]
xdebug.mode=develop,debug,coverage
xdebug.client_host=host.docker.internal
xdebug.start_with_request=no
  • I added coverage to this, for PHPUnit.
  • The suggested setting for start_with_request was yes for this, but this meant IntelliJ would interrupt PHPUnit every time I ran my tests from the shell, so I've switched it off.
# docker/php/conf.d/error_reporting.ini
error_reporting=E_ALL

Normally I'd set this directly in php.ini, but I actually did this part of the config before I remembered about php.ini needing to be configured, so stuck with it.


After doing that lot I could build the container and bring it up, and run composer install:

/mnt/c/src/containers/php8/docker$ docker-compose build
[+] Building 4.0s (25/25) FINISHED
[...]
 => exporting to image                                                                                                                                                                                      0.1s
 => => exporting layers                                                                                                                                                                                     0.0s
 => => writing image sha256:3e9dfd6aca1527f8d0906a0d9f2b2ec2c74ddc0bd9ea9d2b8f0d8b1dce773951                                                                                                                0.0s
 => => naming to docker.io/library/php8-php                                                                                                                                                                 0.0s

/mnt/c/src/containers/php8/docker$ docker compose up --detach
[+] Running 2/2
 ⠿ Network php8_default  Created                                                                                                                                                                            0.0s
 ⠿ Container php8-php-1  Started                                                                                                                                                                            0.5s

/mnt/c/src/containers/php8/docker$ docker exec -it php8-php-1 /bin/bash

/var/www# composer install
Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
Package operations: 49 installs, 0 updates, 0 removals
  - Downloading 
    [...]
Generating autoload files
41 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
/var/www#
/var/www# composer validate
./composer.json is valid

composer.json

Speaking of Composer, here's the composer.json file thusfar (don't worry too much about it: I'm including it here for completeness):

{
    "name" : "adamcameron/php8",
    "description" : "PHP8 containers",
    "type" : "project",
    "license" : "proprietary",
    "require": {
        "php" : "^8.2",
        "ext-iconv": "*",
        "ext-pdo_mysql": "*",
        "ext-mbstring": "*",
        "ext-intl": "*",
        "ext-json": "*",
        "ext-curl": "*",
        "ext-simplexml": "*",
        "ext-zip": "*",
        "ext-pcre": "*",
        "ext-ctype": "*",
        "ext-session": "*",
        "ext-tokenizer": "*",
        "ext-bcmath": "*",
        "ext-zend-opcache": "*",
        "monolog/monolog": "^3.2.0",
        "doctrine/dbal": "^3.5.3"
    },
    "require-dev": {
        "phpunit/phpunit": "^9.5.28",
        "phpmd/phpmd": "^2.13.0",
        "squizlabs/php_codesniffer": "^3.7.1"
    },
    "autoload": {
        "psr-4": {
            "adamcameron\\php8\\": "src/"
        }
    },
    "autoload-dev": {
        "psr-4": {
            "adamcameron\\php8\\test\\": "test/"
        }
    },
    "scripts" : {
        "test": "phpunit --testdox test",
        "phpmd": "phpmd src,test text phpmd.xml",
        "phpcs": "phpcs src test",
        "test-all": [
            "@test",
            "@phpmd",
            "@phpcs"
        ]
    }
}

The require section there is not stuff I needed for the install, it's also a bunch of baseline stuff I know I will need for the app I'm heading towards. I don't think there's anything surprising there.

I also already have some PHPUnit, phpmd and phpcs scripts in there. I'll get to those next…

Testing the PHP container

It would not be me if I didn't test stuff. I will admit I did not TDD the bits above, cos that would just be mad. However I wanted to test things worked, so have put a few tests in. Plus part of this is testing the debug module and PHPUnit work together as well.

PHPUnit

Here's the phpunit.xml.dist file:

<?xml version="1.0" encoding="UTF-8"?>
<phpunit
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/9.5/phpunit.xsd"
        colors="true"
        cacheResult="false"
        testdox="true"
        stopOnFailure="true"
        stopOnError="true"
        stopOnWarning="true"
        failOnWarning="true"
>
    <coverage>
        <include>
            <directory suffix=".php">src</directory>
        </include>
        <report>
            <html outputDirectory="html/test-coverage-report/" />
        </report>
    </coverage>
    <testsuites>
        <testsuite name="Integration tests">
            <directory>test/integration/</directory>
        </testsuite>
        <testsuite name="Unit tests">
            <directory>test/unit/</directory>
        </testsuite>
    </testsuites>
</phpunit>

All standard stuff, and there's nothing I can say about it that the docs don't already say.

Tests

A a few minimal tests, which hopefully are self-explanatory:

// test/integration/PhpTest.php

namespace adamcameron\php8\test\integration;

use PHPUnit\Framework\TestCase;

/** @testdox Tests of the PHP installation */
class PhpTest extends TestCase
{
    /** @testdox It has the expected PHP version */
    public function testPhpVersion()
    {
        $expectedPhpVersion = "8.2";
        $actualPhpVersion = phpversion();
        $this->assertStringStartsWith(
            $expectedPhpVersion,
            $actualPhpVersion,
            "Expected PHP version to start with $expectedPhpVersion, but got $actualPhpVersion"
        );
    }
}
// test/integration/ComposerTest.php

namespace adamcameron\php8\test\integration;

use PHPUnit\Framework\TestCase;

/** @testdox Tests of the Composer installation */
class ComposerTest extends TestCase
{
    /** @testdox It passes composer validate */
    public function testComposerValidates()
    {
        exec("composer validate 2> /dev/null", $output, $returnCode);
        $this->assertEquals(
            0,
            $returnCode,
            "Composer validate failed: " . implode("\n", $output)
        );
    }
}

These two test three things: PHP is running the version I expect; Composer is happy it's configured properly; and PHPUnit itself is running otherwise all this would go splat.

For the code coverage testing I need some source code to test:

// src/Greeter.php

namespace adamcameron\php8;

class Greeter
{
    public const FORMAL = 1;
    public const INFORMAL = 2;

    public static function greet(string $name, int $style = self::FORMAL): string
    {
        if ($style === self::FORMAL) {
            return "Hello, $name";
        }
        return "Hi, $name";
    }
    
}

And a test:

// test/unit/GreeterTest.php

namespace adamcameron\php8\test\unit;

use adamcameron\php8\Greeter;

use PHPUnit\Framework\TestCase;

/** @testdox Tests of the Greeter class */
class GreeterTest extends TestCase
{
    /** @testdox It greets formally */
    public function testFormalGreeting()
    {
        $name = "Zachary";
        $expectedGreeting = "Hello, $name";
        $actualGreeting = Greeter::greet($name, Greeter::FORMAL);
        $this->assertEquals(
            $expectedGreeting,
            $actualGreeting,
            "Expected greeting to be $expectedGreeting, but got $actualGreeting"
        );
    }

    /** @testdox It greets informally */
    public function testInformalGreeting()
    {
        $this->markTestSkipped("skipping this so the coverage report is more interesting");
        $name = "Zachary";
        $expectedGreeting = "Hey, $name";
        $actualGreeting = Greeter::greet($name, Greeter::INFORMAL);
        $this->assertEquals(
            $expectedGreeting,
            $actualGreeting,
            "Expected greeting to be $expectedGreeting, but got $actualGreeting"
        );
    }
}

Note how I am skipping one of the tests. This is so code coverage is not 100%.

Running the tests

root@e8896f5d5bd6:/var/www# composer test
> phpunit --testdox test
PHPUnit 9.5.28 by Sebastian Bergmann and contributors.

Tests of the Composer installation
  It passes composer validate

Tests of the PHP installation
  It has the expected PHP version

Tests of the Greeter class
  It greets formally
  It greets informally

Time: 00:05.675, Memory: 10.00 MB

Summary of non-successful tests:

Tests of the Greeter class
  It greets informally
OK, but incomplete, skipped, or risky tests!
Tests: 4, Assertions: 3, Skipped: 1.

Generating code coverage report in HTML format ... done [00:01.492]
root@e8896f5d5bd6:/var/www#

Cool! It all worked. Let's have a look at the code coverage report. Because I don't have Nginx installed yet I can't browse to it, but I can just open the file in a browser:

I've drilled down the report slightly to show the file I was testing. It's correctly showing that I have only tested one path in the logic. Excellent. This proves that the Xdebug extension is running.

phpmd and phpcs

I've installed these too, and have used a fair stock config file for each (see: phpmd.xml and phpcs.xml). Running them is dead boring as there's hardly any code, and IntelliJ makes sure it's formatted well:

/var/www# composer phpmd
> phpmd src,test text phpmd.xml
/var/www# composer phpcs
> phpcs src test

FILE: /var/www/src/Greeter.php
------------------------------------------------------------------------------------------
FOUND 1 ERROR AFFECTING 1 LINE
------------------------------------------------------------------------------------------
 18 | ERROR | [x] The closing brace for the class must go on the next line after the body
------------------------------------------------------------------------------------------
PHPCBF CAN FIX THE 1 MARKED SNIFF VIOLATIONS AUTOMATICALLY
------------------------------------------------------------------------------------------

Time: 2.51 secs; Memory: 6MB

Script phpcs src test handling the phpcs event returned with error code 2
/var/www#

Ha! I didn't actually expect that. I clearly didn't run this before I did my final commit. If you scroll up to the Greeter.php file, it's complaining that there's an empty line between the last method closing brace and the class's closing brace:

That breaks one of PRS-12's rules. Fair cop. And hey: a good test that it's working!


Nginx in a container

docker-compose.yml

The relevant bit is this:

nginx:
  build:
    context: nginx
    dockerfile: Dockerfile

  ports:
    - "8008:80"

  stdin_open: true
  tty: true

  volumes:
    - ../html:/usr/share/nginx/html/

Nothing interesting there.

Dockerfile

FROM nginx:alpine
WORKDIR /usr/share/nginx/
COPY ./nginx.conf /etc/nginx/nginx.conf
COPY ./sites/ /etc/nginx/sites-available/
COPY ./conf.d/ /etc/nginx/conf.d/
CMD ["nginx"]
EXPOSE 80

Also nothing noteworthy here. Everything is in the config files.

Nginx config files

I freely admin to pretty much lifting these from other projects I already had. I don't really know what I'm doing with Nginx. I learn enough to achieve some goal, then I forget it all within about 5min.

// docker/nginx/nginx.conf
user  nginx;
worker_processes  4;
daemon off;

error_log  /var/log/nginx/error.log debug;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    access_log  /var/log/nginx/access.log;
    sendfile        on;
    keepalive_timeout  65;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-available/*.conf;
}
// docker/nginx/conf.d/default.conf
upstream php-upstream {
    server php:9000;
}
// docker/nginx/sites/default.conf
server {
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;

    server_name localhost;
    root /usr/share/nginx/html;
    index index.html index.php;

    location / {
        autoindex on;
        try_files $uri $uri/ /index.php$is_args$args;
    }

    location ~ \.php$ {
        try_files $uri /index.php =404;
        fastcgi_pass php-upstream;
        fastcgi_index index.php;
        fastcgi_buffers 16 16k;
        fastcgi_buffer_size 32k;
        fastcgi_param SCRIPT_FILENAME /var/www/html/$fastcgi_script_name;
        fastcgi_read_timeout 600;
        include fastcgi_params;
    }

    location ~ /\.ht {
        deny all;
    }
}

I hope it doesn't seem dismissive or that I'm tired of writing in that I add nothing here. I literally don't know what most of that stuff does, other than where it's obvious.

A test PHP file to browse to

I need to be able to test that Nginx is passing stuff to PHP:

<?php
// html/test.php
phpinfo();

Having done all that, I rebuild the containers (and note I now have an Nginx one as well), and bring them up.

If I browse to http://localhost:8008/test.php, I get this:

And if I run the PHPUnit tests again, I can now browse to the report via http://localhost:8008/test-coverage-report/. Cool.


Success

OK so I'm gonna consider that a victory. I did some tinkering around in IntelliJ and I can run the unit tests from in there as well, all via drilling into the Docker container and running it from in there, and presenting the results in IntelliJ:

However that took more dicking around than I can be arsed with re-doing right now. I needed a coupla extensions installed, and set the PHP interpreter to be locatable via the config in docker-compose.yml. It's handy anyhow. I need my team to set all this up, so I'll get the first one of them I dump all this on to work it out and write it down, and I'll report back.

I've linked to all the individual files as I reference them, but you could also clone the repo, checkout the 1.0 tag, and you should be able to set this up locally and have a look if you so chose. There are full instructions in the readme.md file. I have had one of my team test the instructions out on their own PC, and they seemed to have worked.

Righto.

--
Adam

Friday, 20 January 2023

PHP: PrimaryReadReplicaConnection - configuration / usage example

G'day:

I've been dusting off my out-of-date PHP skills (such as they are), and I had a right arse of a time finding any documentation for PrimaryReadReplicaConnections via Google, so I'm adding something here for the next time I need to find it. Because knowing me I won't remember.

Back when I was last doing PHP for a living, DBAL had a MasterSlaveConnection class. This was used to create a single composite connnection object that comprising one primary connection - likely to a read/write DB user - and an array of secondary connections - likely to read-replicas of the same. The way it works is that for read-only operations (SELECTs basically). The reasoning being that this would ease up traffic on the primary DB for queries that don't need to be done on it, using the read replicas instead. But for writes, it uses the primary. It's a good idea.

I'm returning to PHP and I need to mentor my team, so I've been writing some code examples to show them how things work in PHP. I googled "MasterSlaveConnection" to RTFM, and found this issue on Github: "RFC: Rename MasterSlaveConnection". OK so MasterSlaveConnection connection has been renamed to PrimaryReadReplicaConnection. Makes sense. So I googled that instead. Frickin nothing. Nothing useful anyhow. Some stuff on setting up a connection in a Doctrine YAML file, and some other references, but I can't find actual docs with an actual useful example of "do it like this".

Before I start, here's a vanilla DBAL connection example for context:

private function getDbalConnection() : Connection
{
    $parameters = $this->getConnectionParameters();
    return DriverManager::getConnection([
        'dbname' => $parameters->database,
        'user' => $parameters->username,
        'password' => $parameters->password,
        'host' => $parameters->host,
        'port' => $parameters->port,
        'driver' => 'pdo_mysql'
    ]);
}

And here's the equivalent for a PrimaryReadReplicaConnection:

private function getPrimaryReadReplicaConnection() : PrimaryReadReplicaConnection
{
    $parameters = $this->getConnectionParameters();
    return DriverManager::getConnection([
        'wrapperClass' => PrimaryReadReplicaConnection::class,
        'driver' => 'pdo_mysql',
        'primary' => [
            'host' => $parameters->host,
            'port' => $parameters->port,
            'user' => $parameters->username,
            'password' => $parameters->password,
            'dbname' => $parameters->database
        ],
        'replica' => [[
            'host' => $parameters->readOnlyHost,
            'port' => $parameters->port,
            'user' => $parameters->username,
            'password' => $parameters->password,
            'dbname' => $parameters->database
        ]]
    ]);
}

It's pretty basic really. As with a normal DBAL connection one still uses the DriverManager::getConnection factory method to create a connection.

The difference is with the PrimaryReadReplicaConnection case is one needs to pass the deets of the primary and all the replicas instead of just the one set of connection config:

  • wrapperClass: the class name of the PrimaryReadReplicaConnection class.
  • driver: which DB I'm using.
  • primary; an associative array of the usual connection params for the connection to the primary (read/write) DB connection.
  • replica; an indexed array of associative arrays of connection params to the read replicas. One can pass as many of these as one likes, and I believe the connection wrapper controls how to pick one to sue for a given query. I'm only using one here because I only have one read replica.

I think that replica key should be plural, right? Oh well: can't always get it right I guess.

Here are some tests describing its operations / behaviour:

/** @testdox it will use a read-only connection if it can */
public function testPrimaryReadReplicaConnectionReadOnlyConnection()
{
    $connection = $this->getPrimaryReadReplicaConnection();

    $result = $connection->executeQuery("SELECT @@VERSION");

    $this->assertFalse($connection->isConnectedToPrimary());
}

false because it's not connected to the primary: it's connected to a replica because it's only doing reading.

/** @testdox a connection will start on a replica then change to the primary and stay there after a write operation */
public function testPrimaryReadReplicaConnectionSwitchToPrimary()
{
    $testName = "TEST_NAME_" . uniqid();

    $connection = $this->getPrimaryReadReplicaConnection();

    $this->assertFalse($connection->isConnectedToPrimary(), "Should start on a replica");
    $connection->beginTransaction();
    try {
        $this->assertTrue($connection->isConnectedToPrimary(), "Should be on the primary in a transaction");
        $connection->executeStatement(
            "INSERT INTO features (name,value) VALUES (:name, :value)", [
            "name" => $testName,
            "value" => "TEST_VALUE",
        ]);

        $newRow = $connection->executeQuery("SELECT value FROM features WHERE name = ?", [$testName]);
        $this->assertEquals("TEST_VALUE", $newRow->fetchOne());
        $this->assertTrue($connection->isConnectedToPrimary(), "Should still be on the primary after a read following a write");
    } finally {
        $connection->rollBack();
    }
    $this->assertTrue($connection->isConnectedToPrimary(), "Should still be on the primary after a rollback");

    $connection = $this->getPrimaryReadReplicaConnection();
    $this->assertFalse($connection->isConnectedToPrimary(), "A new connection should start on a replica");
}

This second one shows something important. A given connection might start on a read replica, but once there's been a first write operation (in this case the transaction singles some writing about to happen, intrinsically) - so the connection switches to the primary writeable connection - it stays on that primary connection. This is for the sake of data reliability: if one writes some data to primary and then tries to read it back later (for whatever reason), there'd be no guarantee the read replica(s) will have received the updates themselves yet, so it's safer to stay on the primary. Good thinking.

And a new connection within the same code will start afresh using a replica if it can.

OK, that's it. I just wanted this noted down some place.

Righto.

--
Adam

Friday, 20 December 2019

PHPUnit: trying to be clever with data providers and failing

G'day:
Yesterday we had a seemingly strange situation with a test one of my colleagues had written. It was behaving as if PHPUnit's exactly matcher was not working. The developer concerned was in the process of concluding that that is exactly what the problem is: PHPUnit. I suggested we pause and back-up a bit: given thousands of people uses PHPUnit and the exactly matcher every day (including ourselves) and have not reported this issue, whereas the code we are running it with was freshly written and not used by anyone yet... almost certainly the problem lay with our code.

Before I get on with what the problem was, this bears repeating:

If you are doing something new and it's going wrong, the problem will almost certainly not be related to any established third-party tools you are using. It'll be down to how you're using them.

I find it's a really helpful mindset when troubleshooting problems to just own the issue from the outset, rather than looking around for something else to blame. And if you've looked and looked and can't see the problem with your own code, this probably just means you haven't spotted the problem yet. Absence of evidence is not evidence of absence.

Anyway, the issue at hand. Here's a pared-down repro of the situation. The code is meaningless as I've removed all business context from it, but we have this:

class MyService
{
    private $dependency;

    function __construct(MyDependency $dependency){
        $this->dependency = $dependency;
    }

    function myMethod() {
        return $this->dependency->someMethod();
    }
}


And MyDependency is this:

class MyDependency
{
    function someMethod(){
        return "ORIGINAL VALUE";
    }
}

myMethod is new, so the dev had written a test for this:

class MyServiceTest extends TestCase
{

    /** @dataProvider provideTestCasesForMyMethodTests */
    public function testMyMethodViaDataProvider($dependency)
    {
        $dependency
            ->expects($this->once())
            ->method('someMethod')
            ->willReturn("MOCKED VALUE");

        $service = new MyService($dependency);

        $result = $service->myMethod();

        $this->assertEquals("MOCKED VALUE", $result);
    }

    public function provideTestCasesForMyMethodTests(){
        return [
            'it should call someMethod once' => [
                'dependency' => $this->createMock(MyDependency::class)
            ]
        ];
    }
}

I hasten to add that it's completely daft to have the data provider in this example. However in the real-world example there were valid test variations, and given the method being tested was a factory method, part of the test was to check that the relevant method had been called on the relevant object being returned by the factory method. The data provider was legit.

This test passes:

PS C:\src\php\general> vendor\bin\phpunit .\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php
PHPUnit 8.5.0 by Sebastian Bergmann and contributors.

.                                                                   1 / 1 (100%)

Time: 77 ms, Memory: 4.00 MB

OK (1 test, 1 assertion)
PS C:\src\php\general>

However when we were trouble shooting something, we ended up with this assertion:

$dependency
    ->expects($this->exactly(10))
    ->method('someMethod')
    ->willReturn("MOCKED VALUE");


And the test still passed:

PS C:\src\php\general> vendor\bin\phpunit .\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php
PHPUnit 8.5.0 by Sebastian Bergmann and contributors.

.                                                                   1 / 1 (100%)

Time: 77 ms, Memory: 4.00 MB

OK (1 test, 1 assertion)
PS C:\src\php\general>

Whut the?

When we were looking at this I asked my colleague to simplify their tests a bit. The IRL version was testing this case via a data provider and test that was testing a bunch of things in a method that had some complexity issues, and I wanted to pare the situation back to just one simple test, testing this one thing. We ended up with this test:

public function testMyMethodViaInlineMock()
{
    $dependency = $this->createMock(MyDependency::class);
    $dependency
        ->expects($this->once())
        ->method('someMethod')
        ->willReturn("MOCKED VALUE");

    $service = new MyService($dependency);

    $result = $service->myMethod();

    $this->assertEquals("MOCKED VALUE", $result);
}


The difference here being we are not using a data provider any more, the mocking of the dependency is in the test. And this passes:

PS C:\src\php\general> vendor\bin\phpunit .\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php                 PHPUnit 8.5.0 by Sebastian Bergmann and contributors.

.                                                                   1 / 1 (100%)

Time: 61 ms, Memory: 4.00 MB

OK (1 test, 2 assertions)
PS C:\src\php\general>                                                                                                                              

Good, fine, but now we tested out the ->expects($this->exactly(10)) issue again:

PS C:\src\php\general> vendor\bin\phpunit .\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php                 PHPUnit 8.5.0 by Sebastian Bergmann and contributors.

F                                                                   1 / 1 (100%)

Time: 102 ms, Memory: 4.00 MB

There was 1 failure:

1) me\adamcameron\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest::testMyMethodViaInlineMock
Expectation failed for method name is "someMethod" when invoked 10 time(s).
Method was expected to be called 10 times, actually called 1 times.

FAILURES!
Tests: 1, Assertions: 2, Failures: 1.
PS C:\src\php\general>                                                                                                                                                    

This is more like it. But... erm... why?

I'd had "unexpected" behaviour previously when referring to $this inside a data provider method, because the data provider method is not called on the same instance of the test class that the tests are run with (why? Buggered if I know, but it is), and had my suspicions that this could be contributing to it. This is when we just shrugged and left the test as it was - working now - and I undertook to investigate some more later. The code I've been sharing here is the start of my investigation. I was at this point none the wiser though.

On a whim I tried another test variation, to see if there was any change in behaviour across the threshold of the count (so checking "off by one" situations). Here I have a new method that calls the same dependency method five times:
function myOtherMethod(){
     $this->dependency->someOtherMethod();
     $this->dependency->someOtherMethod();
     $this->dependency->someOtherMethod();
     $this->dependency->someOtherMethod();
    return  $this->dependency->someOtherMethod();
}
And a test for it that just checks what happens when we expect it to be called 3-7 times:

/** @dataProvider provideTestCasesForMyOtherMethodTests */
public function testMyOtherMethodCallsDependencyMethodEnoughTimes($dependency, $expectedCount) {
    $dependency
        ->expects($this->exactly($expectedCount))
        ->method('someOtherMethod')
        ->willReturn("MOCKED VALUE");

    $service = new MyService($dependency);

    $result = $service->myOtherMethod();

    $this->assertEquals("MOCKED VALUE", $result);
}

public function provideTestCasesForMyOtherMethodTests(){
    return [
        'it expects myOtherMethod to be called 3 times' => [
            'dependency' => $this->createMock(MyDependency::class),
            'expectedCount' => 3
        ],
        'it expects myOtherMethod to be called 4 times' => [
            'dependency' => $this->createMock(MyDependency::class),
            'expectedCount' => 4
        ],
        'it expects myOtherMethod to be called 5 times' => [
            'dependency' => $this->createMock(MyDependency::class),
            'expectedCount' => 5
        ],
        'it expects myOtherMethod to be called 6 times' => [
            'dependency' => $this->createMock(MyDependency::class),
            'expectedCount' => 6
        ],
        'it expects myOtherMethod to be called 7 times' => [
            'dependency' => $this->createMock(MyDependency::class),
            'expectedCount' => 7
        ]
    ];
}


All things being equal, I was expecting no failures here, given what we'd seen before. But... erm...

PS C:\src\php\general> vendor\bin\phpunit .\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php --filter=testMyOtherMethodCallsDependencyMethodEnoughTimes                                   PHPUnit 8.5.0 by Sebastian Bergmann and contributors.

FF...                                                               5 / 5 (100%)

Time: 64 ms, Memory: 4.00 MB

There were 2 failures:

1) me\adamcameron\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest::testMyOtherMethodCallsDependencyMethodEnoughTimes with data set "it expects myOtherMethod to be called 3 times" (Mock_MyDependency_6a23ba62 Object (...), 3)
me\adamcameron\general\phpunit\expectsWithDataProviderIssue\MyDependency::someOtherMethod() was not expected to be called more than 3 times.

C:\src\php\general\src\phpunit\expectsWithDataProviderIssue\MyService.php:21
C:\src\php\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php:59

2) me\adamcameron\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest::testMyOtherMethodCallsDependencyMethodEnoughTimes with data set "it expects myOtherMethod to be called 4 times" (Mock_MyDependency_6a23ba62 Object (...), 4)
me\adamcameron\general\phpunit\expectsWithDataProviderIssue\MyDependency::someOtherMethod() was not expected to be called more than 4 times.

C:\src\php\general\src\phpunit\expectsWithDataProviderIssue\MyService.php:22
C:\src\php\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php:59

FAILURES!
Tests: 5, Assertions: 3, Failures: 2.
PS C:\src\php\general>                   


OK so this is a bit perplexing. When the expected call count is less than the actual call count we get it failing correcty. But when it's more... no failure. Now I'm flummoxed. For good measure I also have a test not creating the dependency in the data provider, but instead inline in the test:

/** @dataProvider provideTestCasesForMyOtherMethodTests */
public function testMyOtherMethodCallsDependencyMethodEnoughTimesWithDependencyCreatedInTest($_, $expectedCount) {
    $dependency = $this->createMock(MyDependency::class);
    $dependency
        ->expects($this->exactly($expectedCount))
        ->method('someOtherMethod')
        ->willReturn("MOCKED VALUE");

    $service = new MyService($dependency);

    $result = $service->myOtherMethod();

    $this->assertEquals("MOCKED VALUE", $result);
}


And this one behaves correctly:

PS C:\src\php\general> vendor\bin\phpunit .\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php --filter=testMyOtherMethodCallsDependencyMethodEnoughTimesWithDependencyCreatedInTest                                                   PHPUnit 8.5.0 by Sebastian Bergmann and contributors.

FF.FF                                                               5 / 5 (100%)

Time: 67 ms, Memory: 4.00 MB

There were 4 failures:

1) me\adamcameron\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest::testMyOtherMethodCallsDependencyMethodEnoughTimesWithDependencyCreatedInTest with data set "it expects myOtherMethod to be called 3 times" (Mock_MyDependency_2707ab91 Object (...), 3)
me\adamcameron\general\phpunit\expectsWithDataProviderIssue\MyDependency::someOtherMethod() was not expected to be called more than 3 times.

C:\src\php\general\src\phpunit\expectsWithDataProviderIssue\MyService.php:21
C:\src\php\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php:99

2) me\adamcameron\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest::testMyOtherMethodCallsDependencyMethodEnoughTimesWithDependencyCreatedInTest with data set "it expects myOtherMethod to be called 4 times" (Mock_MyDependency_2707ab91 Object (...), 4)
me\adamcameron\general\phpunit\expectsWithDataProviderIssue\MyDependency::someOtherMethod() was not expected to be called more than 4 times.

C:\src\php\general\src\phpunit\expectsWithDataProviderIssue\MyService.php:22
C:\src\php\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php:99

3) me\adamcameron\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest::testMyOtherMethodCallsDependencyMethodEnoughTimesWithDependencyCreatedInTest with data set "it expects myOtherMethod to be called 6 times" (Mock_MyDependency_2707ab91 Object (...), 6)
Expectation failed for method name is "someOtherMethod" when invoked 6 time(s).
Method was expected to be called 6 times, actually called 5 times.

4) me\adamcameron\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest::testMyOtherMethodCallsDependencyMethodEnoughTimesWithDependencyCreatedInTest with data set "it expects myOtherMethod to be called 7 times" (Mock_MyDependency_2707ab91 Object (...), 7)
Expectation failed for method name is "someOtherMethod" when invoked 7 time(s).
Method was expected to be called 7 times, actually called 5 times.

FAILURES!
Tests: 5, Assertions: 6, Failures: 4.
PS C:\src\php\general>                                                                                                                                                                                                                                               

I started diving back through the PHPUnit code, and found the point where my two test variations began to demonstrate different behaviour. I edited InvocationHandler::verify to spit out some telemetry:

public function verify(): void
{
    printf(PHP_EOL . "InvocationHandler::verify called with %d matchers" . PHP_EOL, count($this->matchers));

    foreach ($this->matchers as $matcher) {
        $matcher->verify();
    }



And here's what I get running the two tests:

PS C:\src\php\general> vendor\bin\phpunit .\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php --filter=testMyMethodViaDataProvider                                                                                                    PHPUnit 8.5.0 by Sebastian Bergmann and contributors.

.                                                                   1 / 1 (100%)
InvocationHandler::verify called with 0 matchers


Time: 60 ms, Memory: 4.00 MB

OK (1 test, 1 assertion)
PS C:\src\php\general> vendor\bin\phpunit .\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php --filter=testMyMethodViaInlineMock                                                                                                      PHPUnit 8.5.0 by Sebastian Bergmann and contributors.

F                                                                   1 / 1 (100%)
InvocationHandler::verify called with 1 matchers


Time: 62 ms, Memory: 4.00 MB

There was 1 failure:

1) me\adamcameron\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest::testMyMethodViaInlineMock
Expectation failed for method name is "someMethod" when invoked 10 time(s).
Method was expected to be called 10 times, actually called 1 times.

FAILURES!
Tests: 1, Assertions: 2, Failures: 1.
PS C:\src\php\general>                             

Well. So when we run our tests using the mock created in the test code... we get our matcher "loaded" (for lack of a better term); when we run our test using the mock created in the data provider: it's nowhere to be seen. Interesting.

Looking further up the call stack, we find that this verify method is called from TestCase::verifyMockObects:
private function verifyMockObjects(): void
{
    foreach ($this->mockObjects as $mockObject) {
        if ($mockObject->__phpunit_hasMatchers()) {
            $this->numAssertions++;
        }

        $mockObject->__phpunit_verify(
            $this->shouldInvocationMockerBeReset($mockObject)
        );
    }


(I dunno what's going on with the method name there, but I did verify the call stack... PHPUnit is doing some special magic there).

verifyMockObjects in turn is called from runBare in the same class, and that is run from TestResult::run.

The key thing here is verifyMockObjects being run in the context of the TestCase object, and it looks at $this->mockObjects. Where does $this->mockObjects come from? When we create the mock:

public function getMock(): MockObject
{
    $object = $this->generator->getMock(
        $this->type,
        !$this->emptyMethodsArray ? $this->methods : null,
        $this->constructorArgs,
        $this->mockClassName,
        $this->originalConstructor,
        $this->originalClone,
        $this->autoload,
        $this->cloneArguments,
        $this->callOriginalMethods,
        $this->proxyTarget,
        $this->allowMockingUnknownTypes,
        $this->returnValueGeneration
    );

    $this->testCase->registerMockObject($object);

    return $object;
}

// ...

public function registerMockObject(MockObject $mockObject): void
{
    $this->mockObjects[] = $mockObject;
}

(remember that our createMock call is just a wrapper for a getMockBuilder()...->getMock() call).

So what's the bottom line here? Well my initial wariness about creating the mock in the data provider method has legs... when a mock is created it's registered in the TestCase object it's created within... and the one that the data provider method is running in is not the same TestCase as the one the test is run in. So... none of the mock's constraints are checked when the test is run. That's mostly it.

But what's the story with how that last test still failed on the variants with only 3/5 and 4/5 calls made? That check is done when the mocked method is actually called. So given this:

function myMethod() {
    return $this->dependency->someMethod();
}

When the mocked someMethod is called, a check is made then as to whether it's been called too many times for the expectation. So that's one within the mock itself at the time it's executed, hence it still "works". It's just the check that the exact number of call times was hit that is done at test level, not mock level if that makes sense.

This all just goes to show that one should just use data provider methods for providing data values for test variations, not to treat them like some sort of looping mechanism that the test is just a callback for. That's not the intent, and when one tries to be tricky... one is quickly found out to not be as clever as one hopes... ;-)

That's an awful lot of carry on for what ended up to be an easy / simple explanation. I'm mostly writing this up in case anyone else gets the same issue, and also to show how I went about troubleshooting and investigating it. I'll be putting this in front of my team shortly...

Righto.

--
Adam