Showing posts with label PHPUnit. Show all posts
Showing posts with label PHPUnit. Show all posts

Saturday, 6 March 2021

Symfony & TDD: adding endpoints to provide data for front-end workshop / registration requirements

G'day:

That's probably a fairly enigmatic title if you have not read the preceding article: "Vue.js: using TDD to develop a data-entry form". It's pretty long-winded, but the gist of it is summarised in this copy and pasted tract:

Right so the overall requirement here is (this is copy and pasted from the previous article) to construct an event registration form (personal details, a selection of workshops to register for), save the details to the DB and echo back a success page. Simple stuff. Less so for me given I'm using tooling I'm still only learning (Vue.js, Symfony, Docker, Kahlan, Mocha, MariaDB).

There's been two articles around this work so far:

There's also a much longer series of articles about me getting the Docker environment running with containers for Nginx, Node.js (and Vue.js), MariaDB and PHP 8 running Symfony 5. It starts with "Creating a web site with Vue.js, Nginx, Symfony on PHP8 & MariaDB running in Docker containers - Part 1: Intro & Nginx" and runs for 12 articles.

In the previous article I did the UI for the workshop registration form…

… and the summary one gets after submitting one's registration:

Today we're creating the back-end endpoint to fetch the list of workshops in that multiple select, and also another endpoint to save the registration details that have been submitted (ran out of time for this bit). The database we'll be talking to is as follows:

(BTW, dbdiagram.io is a bloody handy website... I just did a mysqldump of my tables (no data), imported it their online charting tool, and... done. Cool).

Now… as for me and Symfony… I'm really only starting out with it. The entirety of my hands-on exposure to it is documented in "Part 6: Installing Symfony" and "Part 7: Using Symfony". And the "usage" was very superficial. I'm learning as I go here.

And as-always: I will be TDDing every step, using a tight cycle of identify a case (eg: "it needs to return a 200-OK status for GET requests on the endpoint /workshops"); create tests for that case; do the implementation code just for that case.


It needs to return a 200-OK status for GET requests on the /workshops endpoint

This is a pretty simple test:

namespace adamCameron\fullStackExercise\spec\functional\Controller;

use adamCameron\fullStackExercise\Kernel;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;

describe('Tests of WorkshopController', function () {

    beforeAll(function () {
        $this->request = Request::createFromGlobals();
        $this->kernel  = new Kernel('test', false);
    });

    describe('Tests of doGet', function () {
        it('needs to return a 200-OK status for GET requests', function () {

            $request = $this->request->create("/workshops/", 'GET');
            $response = $this->kernel->handle($request);

            expect($response->getStatusCode())->toBe(Response::HTTP_OK);
        });
    });
});

I make a request, I check the status code of the response. That's it. And obviously it fails:

root@58e3325d1a16:/usr/share/fullstackExercise# vendor/bin/kahlan --spec=spec/functional/Controller/workshopController.spec.php --lcov="var/tmp/lcov/coverage.info" --ff

[error] Uncaught PHP Exception Symfony\Component\HttpKernel\Exception\NotFoundHttpException: "No route found for "GET /workshops/"" at /usr/share/fullstackExercise/vendor/symfony/http-kernel/EventListener/RouterListener.php line 136

F                                                                   1 / 1 (100%)


Tests of WorkshopController
  Tests of doGet
    ✖ it needs to return a 200-OK status for GET requests
      expect->toBe() failed in `.spec/functional/Controller/workshopController.spec.php` line 22

      It expect actual to be identical to expected (===).

      actual:
        (integer) 404
      expected:
        (integer) 200

Perfect. Now let's add a route. And probably wire it up to a controller class and method I guess. Here's what I've got:

# backend/config/routes.yaml
workshops:
  path: /workshops/
  controller: adamCameron\fullStackExercise\Controller\WorkshopsController::doGet
// backend/src/Controller/WorkshopsController.php
namespace adamCameron\fullStackExercise\Controller;

use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\JsonResponse;

class WorkshopsController extends AbstractController
{
    public function doGet() : JsonResponse
    {
        return new JsonResponse(null);
    }
}

Initially this continued to fail with a 404, but I worked out that Symfony caches a bunch of stuff when it's not in debug mode, so it wasn't seeing the new route and/or the controller until I switched the Kernel obect initialisation to switch debug on:

// from workshopsController.spec.php, as above:

$this->kernel  = new Kernel('test', true);

And then the actual code ran. Which is nice:

Passed 1 of 1 PASS in 0.108 seconds (using 8MB)

(from now on I'll just let you know if the tests pass, rather than spit out the output).

Bye Kahlan, hi again PHPUnit

I've been away from this article for an entire day, a lot of which was down to trying to get Kahlan to play nicely with Symfony, and also getting Kahlan's own functionality to not obstruct me from forward progress. However I've hit a wall with some of the bugs I've found with it, specifically "Documented way of disabling patching doesn't work #378" and "Bug(?): Double::instance doesn't seem to cater to stubbing methods with return-types #377". These render Kahlan unusable for me.

The good news is I sniffed around PHPUnit a bit more, and discovered its testdox functionality which allows me to write my test cases in good BDD fashion, and have those show up in the test results. It'll either "rehydrate" a human-readable string from the test name (testTheMethodDoesTheThing becomes "test the method does the thing"), or one can specify an actual case via the @testdox annotation on methods and the classes themselves (I'll show you below, outside this box). This means PHPUnit will achieve what I need, so I'm back to using that.

OK, backing up slightly and switching over to the PHPUnit test (backend/tests/functional/Controller/WorkshopsControllerTest.php):

/**
 * @testdox it needs to return a 200-OK status for GET requests
 * @covers \adamCameron\fullStackExercise\Controller\WorkshopsController
 */
public function testDoGetReturns200()
{
    $this->client->request('GET', '/workshops/');

    $this->assertEquals(Response::HTTP_OK, $this->client->getResponse()->getStatusCode());
}
> vendor/bin/phpunit --testdox 'tests/functional/Controller/WorkshopsControllerTest.php' '--filter=testDoGetReturns200'
PHPUnit 9.5.2 by Sebastian Bergmann and contributors.

Tests of WorkshopController
it needs to return a 200-OK status for GET requests

Time: 00:00.093, Memory: 12.00 MB

OK (1 test, 1 assertion)

Generating code coverage report in HTML format ... done [00:00.682]
root@5f9133aa9de3:/usr/share/fullstackExercise#

Cool.


It returns a collection of workshop objects, as JSON

The next case is:

/**
 * @testdox it returns a collection of workshop objects, as JSON
 */
public function testDoGetReturnsJson()
{
}

This is a bit trickier, in that I actually need to write some application code now. And wire it into Symfony so it works. And also test it. Via Symfony's wiring. Eek.

Here's a sequence of thoughts:

  • We are getting a collection of Workshop objects from [somewhere] and returning them in JSON format.
  • IE: that would be a WorkshopCollection.
  • The values for the workshops are stored in the DB.
  • The WorkshopCollection will need a way of getting the data into itself. Calling some method like loadAll
  • That will need to be called by the controller, so the controller will need to receive a WorkshopCollection via Symfony's DI implementation.
  • A model class like WorkshopCollection should not be busying itself with the vagaries of storage. It should hand that off to a repository class (see "The Repository Pattern"), which will handle the fetching of DB data and translating it from a recorset to an array of Workshop objects.
  • As WorkshopsRepository will contain testable data-translation logic, it will need unit tests. However we don't want to have to hit the DB in our tests, so we will need to abstract the part of the code that gets the data into something we can mock away.
  • As we're using Doctrine/DBAL to connect to the database, and I'm a believer in "don't mock what you don't own", we will put a thin (-ish) wrapper around that as WorkshopsDAO. This is not completely "thin" because it will "know" the SQL statements to send to its connector to get the data, and will also "know" the DBAL statements to get the data out and pass back to WorkshopsRepository for modelling.

That seems like a chunk to digest, and I don't want you to think I have written any of this code, but this is the sequence of thoughts that leads me to arrive at the strategy for handling the next case. I think from the first bits of that bulleted list I can derive sort of how the test will need to work. The controller doesn't how this WorkshopCollection gets its data, but it needs to be able to tell it to do it. We'll mock that bit out for now, just so we can focus on the controller code. We will work our way back from the mock in another test. For now we have backend/tests/functional/Controller/WorkshopsControllerTest.php

/**
 * @testdox it returns a collection of workshop objects, as JSON
 * @covers \adamCameron\fullStackExercise\Controller\WorkshopsController
 */
public function testDoGetReturnsJson()
{
    $workshops = [
        new Workshop(1, 'Workshop 1'),
        new Workshop(2, 'Workshop 2')
    ];

    $this->client->request('GET', '/workshops/');

    $resultJson = $this->client->getResponse()->getContent();
    $result = json_decode($resultJson, false);

    $this->assertCount(count($workshops), $result);
    array_walk($result, function ($workshopValues, $i) use ($workshops) {
        $workshop = new Workshop($workshopValues->id, $workshopValues->name);
        $this->assertEquals($workshops[$i], $workshop);
    });
}

To make this pass we need just enough code for it to work:

class WorkshopsController extends AbstractController
{

    private WorkshopCollection $workshops;

    public function __construct(WorkshopCollection $workshops)
    {
        $this->workshops = $workshops;
    }

    public function doGet() : JsonResponse
    {
        $this->workshops->loadAll();

        return new JsonResponse($this->workshops);
    }
}
class WorkshopCollection implements \JsonSerializable
{
    /** @var Workshop[] */
    private $workshops;

    public function loadAll()
    {
        $this->workshops = [
            new Workshop(1, 'Workshop 1'),
            new Workshop(2, 'Workshop 2')
        ];
    }

    public function jsonSerialize()
    {
        return $this->workshops;
    }
}

And thanks to Symphony's dependency-injection service container's autowiring, all that just works, just like that. That's the test for that end point done.

Now there was all thant bumpf I mentioned about repositories and DAOs and connectors and stuff. As part of the refactoring part of this, we are going to push our implementation right back to the DAO. This allows us to complete the parts of the code in the WorkshopCollection, WorkshopsRepository and just mock-out the DAO for now.

class WorkshopCollection implements \JsonSerializable
{
    private WorkshopsRepository $repository;

    /** @var Workshop[] */
    private $workshops;

    public function setRepository(WorkshopsRepository $repository) : void
    {
        $this->repository = $repository;
    }

    public function loadAll()
    {
        $this->workshops = $this->repository->selectAll();
    }

    public function jsonSerialize()
    {
        return $this->workshops;
    }
}

My thinking here is:

  • It's going to need a WorkshopsRepository to get stuff from the DB.
  • It doesn't seem right to me to pass in a dependency to a model as a constructor arguments. The model should work without needing a DB connection; just the methods around storage interaction should require the repository. On the other hand the only thing the collection does now is to be able to load the stuff from the DB and serialise it, so I'm kinda coding for the future here, and I don't like that. But I'm sticking with it for reasons we'll come to below.
  • I also really hate model classes with getters and setters. This is usually a sign of bad OOP. But here I have a setter, to get the repository in there.

The reason (it's not a reason, it's an excuse) I'm not passing in the repo as a constructor argument and instead using a setter is because I wanted to check out how Symfony's service config dealt with the configuration of this. If yer classes all have type-checked constructor args, Symfony just does it all automatically with no code at all (just a config switch). However to handle using the setRepository method I needed a factory method to do so. The config for it is thus (in backend/config/services.yaml):

adamCameron\fullStackExercise\Factory\WorkshopCollectionFactory: ~
adamCameron\fullStackExercise\Model\WorkshopCollection:
    factory: ['@adamCameron\fullStackExercise\Factory\WorkshopCollectionFactory', 'getWorkshopCollection']

Simple! And the code for WorkshopCollectionFactory:

class WorkshopCollectionFactory
{
    private WorkshopsRepository $repository;

    public function __construct(WorkshopsRepository $repository)
    {
        $this->repository = $repository;
    }

    public function getWorkshopCollection() : WorkshopCollection
    {
        $collection = new WorkshopCollection();
        $collection->setRepository($this->repository);

        return $collection;
    }
}

Also very simple. But, yeah, it's an exercise in messing about, and there's no way I should have done this. I should have just used a constructor argument. Anyway, moving on.

The WorkshopsRepository is very simple too:

class WorkshopsRepository
{
    private WorkshopsDAO $dao;

    public function __construct(WorkshopsDAO $dao)
    {
        $this->dao = $dao;
    }

    /** @return Workshop[] */
    public function selectAll() : array
    {
        $records = $this->dao->selectAll();
        return array_map(
            function ($record) {
                return new Workshop($record['id'], $record['name']);
            },
            $records
        );
    }
}

I get some records from the DAO, and map them across to Workshop objects. Oh! Workshop:

class Workshop implements \JsonSerializable
{
    private int $id;
    private string $name;

    public function __construct(int $id, string $name)
    {
        $this->id = $id;
        $this->name = $name;
    }

    public function jsonSerialize()
    {
        return (object) [
            'id' => $this->id,
            'name' => $this->name
        ];
    }
}

And lastly I mock WorkshopsDAO. I can't implement any further down the stack of this process because the DAO is what uses the DBAL Connector object, and I don't own that, so if I actually started to use it, I'd be hitting the DB. Or hitting the ether and getting an error. Either way: no good for our test. So a mocked DAO:

class WorkshopsDAO
{
    public function selectAll() : array
    {
        return [
            ['id' => 1, 'name' => 'Workshop 1'],
            ['id' => 2, 'name' => 'Workshop 2']
        ];
    }
}

Having done all that refactoring, we check if our test is still good, and it is. I can verify this is not a trick of the light by changing some of that data in the DAO, and watch the test break (which it does). I can also now go back to the test and stick some more code-coverage annotations in:

/**
 * @testdox it returns a collection of workshop objects, as JSON
 * @covers \adamCameron\fullStackExercise\Controller\WorkshopsController
 * @covers \adamCameron\fullStackExercise\Factory\WorkshopCollectionFactory
 * @covers \adamCameron\fullStackExercise\Repository\WorkshopsRepository
 * @covers \adamCameron\fullStackExercise\Model\WorkshopCollection
 * @covers \adamCameron\fullStackExercise\Model\Workshop
 */

And see that all the code is indeed covered:


It returns the expected workshops from the database

But now we need to implement the real DAO. once we do that, our test will break because the DAO will suddenly start hitting the DB, and we'll be getting back whatever is in the DB, not our expected canned response. Plus we don't want this test to hit the DB anyhow. So first we're gonna mock-out the DAO using PHPUnit's mocks instead of our code-mock. To do this turned out to be a bit tricky, initially, given Symfony's DI container is looking after all the dependencies for us, but fortunately when in test mode, Symfony allows us to hack into that container. I've updated my test, thus:

public function testDoGetReturnsJson()
{
    $workshopDbValues = [
        ['id' => 1, 'name' => 'Workshop 1'],
        ['id' => 2, 'name' => 'Workshop 2']
    ];

    $this->mockWorkshopDaoInServiceContainer($workshopDbValues);

    // ... unchanged ...

    array_walk($result, function ($workshopValues, $i) use ($workshopDbValues) {
        $this->assertEquals($workshopDbValues[$i], $workshopValues);
    });
}

private function mockWorkshopDaoInServiceContainer($returnValue = []): void
{
    $mockedDao = $this->createMock(WorkshopsDAO::class);
    $mockedDao->method('selectAll')->willReturn($returnValue);

    $container = $this->client->getContainer();
    $workshopRepository = $container->get('test.WorkshopsRepository');

    $reflection = new \ReflectionClass($workshopRepository);
    $property = $reflection->getProperty('dao');
    $property->setAccessible(true);
    $property->setValue($workshopRepository, $mockedDao);
}

We are popping a mocked WorkshopsDAO into the WorkshopsRepository object in the container So when the repo calls it, it'll be calling the mock.

Oh! to be able to access that 'test.WorkshopsRepository' container key, we need to expose it via the services_test.xml container config:

services:
  test.WorkshopsRepository:
    alias: adamCameron\fullStackExercise\Repository\WorkshopsRepository
    public: true

And running that, the test works, and is ignoring the reallyreally DAO.

To test the final DAO implementation, we're gonna do an end-to-end test:

class WorkshopControllerTest extends WebTestCase
{
    private KernelBrowser $client;

    public static function setUpBeforeClass(): void
    {
        $dotenv = new Dotenv();
        $dotenv->load(dirname(__DIR__, 3) . "/.env.test");
    }

    protected function setUp(): void
    {
        $this->client = static::createClient(['debug' => false]);
    }

    /**
     * @testdox it returns the expected workshops from the database
     * @covers \adamCameron\fullStackExercise\Controller\WorkshopsController
     * @covers \adamCameron\fullStackExercise\Factory\WorkshopCollectionFactory
     * @covers \adamCameron\fullStackExercise\Model\WorkshopCollection
     * @covers \adamCameron\fullStackExercise\Repository\WorkshopsRepository
     * @covers \adamCameron\fullStackExercise\Model\Workshop
     */
    public function testDoGet()
    {
        $this->client->request('GET', '/workshops/');
        $response = $this->client->getResponse();
        $workshops = json_decode($response->getContent(), false);

        /** @var Connection */
        $connection = static::$container->get('database_connection');
        $expectedRecords = $connection->query("SELECT id, name FROM workshops ORDER BY id ASC")->fetchAll();

        $this->assertCount(count($expectedRecords), $workshops);
        array_walk($expectedRecords, function ($record, $i) use ($workshops) {
            $this->assertEquals($record['id'], $workshops[$i]->id);
            $this->assertSame($record['name'], $workshops[$i]->name);
        });
    }
}

The test is pretty familiar, except it's actually getting its expected data from the database, and making sure the whole process, end to end, is doing what we want. Currently when we run this it fails because we still have our mocked DAO in place (the real mock, not the… mocked mock. Um. You know what I mean: the actual DAO class that just returns hard-coded data). Now we put the proper DAO code in:

class WorkshopsDAO
{
    private Connection $connection;

    public function __construct(Connection $connection)
    {
        $this->connection = $connection;
    }

    public function selectAll() : array
    {
        $sql = "
            SELECT
                id, name
            FROM
                workshops
            ORDER BY
                id ASC
        ";
        $statement = $this->connection->executeQuery($sql);

        return $statement->fetchAllAssociative();
    }
}

And now if we run the tests:

> vendor/bin/phpunit --testdox
PHPUnit 9.5.2 by Sebastian Bergmann and contributors.

Tests of WorkshopController
it needs to return a 200-OK status for GET requests
it returns a collection of workshop objects, as JSON

Tests of baseline Symfony install
it displays the Symfony welcome screen
it returns a personalised greeting from the /greetings end point

PHP config tests
gdayWorld.php outputs G'day world!

Webserver config tests
It serves gdayWorld.html with expected content

End to end tests of WorkshopController
it returns the expected workshops from the database

Tests that code coverage analysis is operational
It reports code coverage of a simple method correctly

Time: 00:00.553, Memory: 22.00 MB

OK (8 tests, 26 assertions)

Generating code coverage report in HTML format ... done [00:00.551]
root@5f9133aa9de3:/usr/share/fullstackExercise#

Nice one!

And if we look at code coverage:

I was gonna try to cover the requirements for the process ot save the form fields in the article too, but it took ages to work out how some of the Symfony stuff worked, plus I sunk about a day into trying to get Kahlan to work, and this article is already super long anyhow. I now have the back-end processing sorted out to update the front-end form to actually use the values from the DB instead of test values. I might look at that tomorrow (see "TDDing the reading of data from a web service to populate elements of a Vue JS component" for that exercise). I need a rest from Symfony.


It needs to drink some beer

I'm rushing the outro of this article because I am supposed to be on a webcam with a beer in my hand in 33min, and I need to proofread this still…

Righto.

--
Adam

Friday, 8 January 2021

Part 3: PHPUnit

G'day:

Please note that I initially intended this to be a single article, but by the time I had finished the first two sections, it was way too long for a single read, so I've split it into the following sections, each as their own article:

  1. Intro / Nginx
  2. PHP
  3. PHPUnit (this article)
  4. Tweaks I made to my Bash environment in my Docker containers
  5. MariaDB
  6. Installing Symfony
  7. Using Symfony
  8. Testing a simple web page built with Vue.js using Mocha, Chai and Puppeteer
  9. I mess up how I configure my Docker containers
  10. An article about moving files and changing configuration
  11. Setting up a Vue.js project and integrating some existing code into it
  12. Unit testing Vue.js components

As indicated: this is the third article in the series, and follows on from Part 2: PHP. It's probably best to go have a breeze through the earlier articles first.

PHPUnit

I got slightly ahead of myself and added PHPUnit into composer.json when I was working through the PHP configuration part of this exercise, so it's already installed. Before I run it though, I need a phpunit.xml file, so I'll chuck one of those in:

<?xml version="1.0" encoding="UTF-8"?>
<phpunit
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/9.5/phpunit.xsd"
        colors="true"
        forceCoversAnnotation="true"
        cacheResult="false"
>
    <coverage>
        <include>
            <directory suffix=".php">src</directory>
        </include>
        <report>
            <html outputDirectory="public/test-coverage-report/" />
        </report>
    </coverage>
    <testsuites>
        <testsuite name="Functional tests">
            <directory>test/functional/</directory>
        </testsuite>
        <testsuite name="Unit tests">
            <directory>test/unit/</directory>
        </testsuite>
    </testsuites>
</phpunit>

There's no real surprises here. I've got two separate test suits: one for functional tests in which I'll test those test web-browseable files I created in the previous article; and one for unit tests. To test the code coverage config here I'll need some actual code to test and cover.

I had some drama getting PHPUnit working with code coverage, and that in itself is covered in a separate article, PHPUnit: get code coverage reporting working on PHP 8 / PHPUnit 9.5. The stuff I've written there is very focused on PHPUnit and not so much on the Docker side of things, or the testing in the context of this notional application I'm putting together, hence splitting it out into its own article, and also so I can focus on the Docker side of things here.

First things first, I need some tests! I decided to functional-test the two web browseable files: gdayWorld.html and gdayWorld.php. As a reminder their contents are (respectively):

<!doctype html>

<html lang="en">
<head>
    <meta charset="utf-8">

    <title>G'day world</title>
</head>

<body>
<h1>G'day world</h1>
<p>G'day world</p>
</body>
</html>

And:

<?php

$message = "G'day World";
echo $message;

So for the tests I'm just gonna make sure I can hit them, and the HTML one has the expected title, heading and content; and the PHP one just has the expected string. I could horse around with PHP's native curl implementation to make these work, but its programming interface written like something out of 1995, so I tend to avoid it where I can. I'm gonna use Guzzle instead. Also, and this is slightly OTT I know, but I like using Symfony's Response constants when checking for HTTP status codes to make the code more clear, so I'm adding in symfony/http-foundation. Lastly as I'll be using PHP's DOM API for testing the HTML, I'm placating a warning in PHPStorm that says "ooh but ext-dom might not be installed!" So I'm forcing that too. Oh and I like keeping my code tidy so I'm also slinging PHPMD and PHPCS in there too. My composer.json file becomes:

{
    "name": "adamcameron/full-stack-exercise",
    "description": "Full Stack Exercise",
    "license": "GPL-3.0-or-later",
    "require-dev": {
        "phpunit/phpunit": "^9.5",
        "guzzlehttp/guzzle": "^7",
        "symfony/http-foundation": "^5.2",
        "ext-dom": "*",
        "phpmd/phpmd": "^2.9",
        "squizlabs/php_codesniffer": "^3.5"
    },
    "autoload": {
        "psr-4": {
            "adamCameron\\fullStackExercise\\": "src/"
        }
    },
    "autoload-dev": {
        "adamCameron\\fullStackExercise\\test\\": "test/"
    }
}

Now I can write some tests. I did all this incrementally, but you know how to do that, so here is the "final" (see below for why I put that in quotes) version of WebServerTest:

namespace adamCameron\fullStackExercise\test\functional\_public; // "public" is reserved

use GuzzleHttp\Client;
use PHPUnit\Framework\TestCase;
use Symfony\Component\HttpFoundation\Response;

class WebServerTest extends TestCase
{
    /** @coversNothing */
    public function testGdayWorldHtmlReturnsExpectedContent()
    {
        $expectedContent = "G'day world";


        $client = new Client([
            'base_uri' => 'http://localhost/'
        ]);

        $response = $client->get('gdayWorld.html');

        $this->assertEquals(Response::HTTP_OK, $response->getStatusCode());

        $html = $response->getBody();
        $document = new \DOMDocument();
        $document->loadHTML($html);

        $xpathDocument = new \DOMXPath($document);

        $hasTitle = $xpathDocument->query('/html/head/title[text() = "' . $expectedContent . '"]');
        $this->assertCount(1, $hasTitle);

        $hasHeading = $xpathDocument->query('/html/body/h1[text() = "' . $expectedContent . '"]');
        $this->assertCount(1, $hasHeading);

        $hasContent = $xpathDocument->query('/html/body/p[text() = "' . $expectedContent . '"]');
        $this->assertCount(1, $hasContent);
    }
}

And I triumphantly run this:

/usr/share/fullstackExercise # vendor/bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

Warning:       No code coverage driver available

E                                                                   1 / 1 (100%)

Time: 00:00.438, Memory: 6.00 MB

There was 1 error:

1) adamCameron\fullStackExercise\test\functional\_public\WebServerTest::testGdayWorldHtmlReturnsExpectedContent
GuzzleHttp\Exception\ConnectException: cURL error 7: Failed to connect to localhost port 80: Connection refused (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost/gdayWorld.html

Doh! But… but… but… I quickly went and hit http://localhost/gdayWorld.html in my browser and it was fine. Then the penny dropped. I'm not running the tests from my host machine. I'm running them from with the PHP container. I've told the host machine about the Nginx container's web server; but I've not told the PHP container about it. Reminder as to what the docker-compose.yml is like at the moment:

version: '3'

services:
  nginx:
    build:
      context: ./nginx
    volumes:
      - ../public:/usr/share/fullstackExercise/public
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/sites/:/etc/nginx/sites-available
      - ./nginx/conf.d/:/etc/nginx/conf.d
      - ../log:/var/log
    depends_on:
      - php-fpm
    ports:
      - "80:80"
    stdin_open: true # docker run -i
    tty: true        # docker run -t

  php-fpm:
    build:
      context: ./php-fpm
    volumes:
      - ..:/usr/share/fullstackExercise
    stdin_open: true # docker run -i
    tty: true        # docker run -t

I read a whole bunch of stuff on networking in Docker. I didn't find the Docker docs very useful at the time, but now I re-read them knowing how I'm supposed to interpret them, they seem clear. Not sure if that's an indictment of me or the docs. Or both. I also looked at a whole bunch of Stack Overflow Q&A and the answers were conflicting and divergent. However after distilling what I could from all these sources, it's really pretty easy. Here's the updated version:

version: '3'

services:
  nginx:
    build:
      context: ./nginx
    volumes:
      - ../public:/usr/share/fullstackExercise/public
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/sites/:/etc/nginx/sites-available
      - ./nginx/conf.d/:/etc/nginx/conf.d
      - ../log:/var/log
    depends_on:
      - php-fpm
    ports:
      - "80:80"
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    networks:
      - backend

  php-fpm:
    build:
      context: ./php-fpm
    volumes:
      - ..:/usr/share/fullstackExercise
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    networks:
      - backend
        
networks:
  backend:
    driver: "bridge"

I just added the networks section, and then told the Nginx and PHP containers to be on that backend network. NB: backend has no significance as a word here, it's just a label, and one used in the docs I was reading. After rebuild, I could now see the Nginx server from the PHP container:

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker exec --interactive --tty docker_php-fpm_1 /bin/sh
/usr/share/fullstackExercise # curl http://nginx/gdayWorld.php
G'day World


/usr/share/fullstackExercise #

Note that I'm using the Nginx services container name there as the host name, ie:

services:
  nginx:

Not that it matters, but that seems a bit manky to me, so I wanted to specify a hostname here. That's just a matter of giving the Nginx container a hostname:

services:
  nginx:
    build:
      context: ./nginx
    hostname: webserver.backend
    volumes:
      # etc
adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker exec --interactive --tty docker_php-fpm_1 /bin/sh
/usr/share/fullstackExercise # curl http://webserver.backend/gdayWorld.php
G'day World


/usr/share/fullstackExercise #

Now my functional tests should work, provided I use that new hostname:

/usr/share/fullstackExercise # vendor/bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

Warning:       No code coverage driver available

..                                                                   1 / 1 (100%)

Time: 00:00.462, Memory: 6.00 MB

OK (2 test, 6 assertions)
/usr/share/fullstackExercise #

Cool!

Oh there was a separate test for gdayWorld.php too:

namespace adamCameron\fullStackExercise\test\functional\_public; // "public" is reserved

use GuzzleHttp\Client;
use PHPUnit\Framework\TestCase;
use Symfony\Component\HttpFoundation\Response;

class PhpTest extends TestCase
{
    /** @coversNothing */
    public function testGdayWorldPhpReturnsExpectedContent()
    {
        $client = new Client([
            'base_uri' => 'http://webserver.backend/'
        ]);

        $response = $client->get('gdayWorld.php');

        $this->assertEquals(Response::HTTP_OK, $response->getStatusCode());

        $content = $response->getBody()->getContents();

        $this->assertSame("G'day world", $content);
    }
}

I'm glad I wrote these tests, because this one identified a small bug I had introduced into gdayWorld.php. I'd fixed it by the time I catpured that output above, but the first run was less positive:

/usr/share/fullstackExercise # vendor/bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

Warning:       No code coverage driver available

F.                                                                  2 / 2 (100%)

Time: 00:00.509, Memory: 6.00 MB

There was 1 failure:

1) adamCameron\fullStackExercise\test\functional\_public\PhpTest::testGdayWorldPhpReturnsExpectedContent
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'G'day world'
+'G'day World'

/usr/share/fullstackExercise/test/functional/PhpTest.php:24

FAILURES!
Tests: 2, Assertions: 6, Failures: 1.
/usr/share/fullstackExercise #

Yay for testing! I did not contrive this situation as an example of "always test first!", but there it is. This is hugely trivial code, but I still messed it up, and simply eyeballing it did not spot the bug.

Speaking of being observant… you will no-doubt have noticed the warning about code coverage driver above. I still need to install XDebug to make this work. This is a matter of adding this into the Dockerfile:

FROM php:fpm-alpine
RUN apk --update --no-cache add git
RUN docker-php-ext-install pdo_mysql
RUN pecl install xdebug-3.0.1 && docker-php-ext-enable xdebug
COPY --from=composer /usr/bin/composer /usr/bin/composer
WORKDIR  /usr/share/fullstackExercise/
CMD composer install ; php-fpm
EXPOSE 9000

Or at least that was the theory:

adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$ docker-compose up --build --detach
Creating network "docker_backend" with driver "bridge"
Building php-fpm
Step 1/8 : FROM php:fpm-alpine
---> 6bd7d9173974
Step 2/8 : RUN apk --update --no-cache add git
---> Using cache
---> 098d91282e3e
Step 3/8 : RUN docker-php-ext-install pdo_mysql
---> Using cache
---> 6f74a2ec5bb1
Step 4/8 : RUN pecl install xdebug-3.0.1 && docker-php-ext-enable xdebug
---> Running in 12444e0a094c
downloading xdebug-3.0.1.tgz ...
Starting to download xdebug-3.0.1.tgz (214,467 bytes)
.............................................done: 214,467 bytes
87 source files, building
running: phpize
Configuring for:
PHP Api Version: 20200930
Zend Module Api No: 20200930
Zend Extension Api No: 420200930
Cannot find autoconf. Please check your autoconf installation and the
$PHP_AUTOCONF environment variable. Then, rerun this script.

ERROR: `phpize' failed
ERROR: Service 'php-fpm' failed to build : The command '/bin/sh -c pecl install xdebug-3.0.1 && docker-php-ext-enable xdebug' returned a non-zero code: 1
adam@DESKTOP-QV1A45U:/mnt/c/src/fullstackExercise/docker$

Sigh. I have encountered this sort of thing before with other containers. Alpine is really really pared down, and doesn't include a bunch of packages that tools might need to run. This is fine for a lot of situations, but it's also a pain in the arse for others. In this case I added in autoconf, but then I needed to install a C compiler too. And then after that I think it wanted something else. Sod that. I just stopped using Alpine and wend back to the Debian version of the container, in the Dockerfile:

FROM FROM php:8.0-fpm
RUN apt-get update --yes && apt-get install git --yes
ENV XDEBUG_MODE=coverage
NB that changed from:
FROM :fpm-alpine
RUN apk --update --no-cache add git

Note: the apk / apt-get change is just the difference in package manager between Alpine and Debian. I won't show you the installation of this because it's 70kB of telemetry, all of which culminates in:

Creating docker_php-fpm_1 ... done Creating docker_nginx_1 ... done

And now we can run our functional tests, and the code coverage report should create (even if it hasn't got anything in it yet, cos I'm not code-covering those functional tests):

root@2e5f56af2f54:/usr/share/fullstackExercise# vendor/bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

Warning: Incorrect filter configuration, code coverage will not be processed
..                                                                  2 / 2 (100%)

Time: 00:00.493, Memory: 6.00 MB

OK (2 tests, 6 assertions)
root@2e5f56af2f54:/usr/share/fullstackExercise#

Grrrr… what now?. I reviewed the docs and my phpunit.xml was legit-looking, and validated fine, so I was flumoxed. But then I came across this issue with PHPUnit: Misleading error message when no files present in coverage path. That sums it up. I have this in my phpunit.xml file:

<coverage>
    <include>
        <directory suffix=".php">src</directory>
    </include>
    <report>
        <html outputDirectory="public/test-coverage-report/" />
    </report>
</coverage>

But I don't actually have a src/ directory yet. Once I added that, and also added the public/ directory into that <include> block, I get a report. At the same time, I will add a stub PHP class and test thereof into the src/ directory as well, to better test the reporting:

root@2e5f56af2f54:/usr/share/fullstackExercise# vendor/bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

...                                                                 3 / 3 (100%)

Time: 00:02.050, Memory: 12.00 MB

OK (3 tests, 7 assertions)

Generating code coverage report in HTML format ... done [00:00.780]
root@2e5f56af2f54:/usr/share/fullstackExercise#

And the coverage report generates fine too:
above



below

The test for MyClass's needsTesting method is as simple as you might imagine:

namespace adamCameron\fullStackExercise\test\unit;

use adamCameron\fullStackExercise\MyClass;
use PHPUnit\Framework\TestCase;

/** @coversDefaultClass adamCameron\fullStackExercise\MyClass */
class MyClassTest extends TestCase
{
    private $myClass;

    protected function setUp(): void
    {
        $this->myClass = new MyClass();
    }

    /** @covers ::needsTesting */
    public function testNeedsTesting()
    {

        $needsTesting = $this->myClass->needsTesting();
        $this->assertTrue($needsTesting);
    }
}

OK, so I'm in a good place to be able to TDD some PHP code now. But before I do that, I want to get a MariaDB container added into the mix as well, and write an intergration test for its connectivity. Before I get to that though, as I've been doing the work on this project up until now - especially when troubleshooting the networking and the PHPUnit issue I had - I've tweaked some stuff with my Bash environment, and I wanted to rip that out into a separate wee article. So before we get onto MariaDB, I'll write that up: Tweaks I made to my Bash environment in my Docker containers.

Righto.

--
Adam

Wednesday, 9 December 2020

PHPUnit: get code coverage reporting working on PHP 8 / PHPUnit 9.5

 G'day:

I'm still beavering away getting a PHP dev environment working via Docker. As I mentioned in the previous article, I'm writing an article about how I'm doing that as I go, but I've had a few issues today that have made me pause that.

My current issue has been getting PHPUnit working properly. This has nothing to do with the Docker environment (as far as I can tell), it's just... me and PHPUnit not playing nicely together. The solution to my issue was not immediately apparent to me... it's taken me about three hours to sort this out. So I figured it might be worthwhile writing down.

I have a barebones "G'day World" website running in a coupla Docker containers (one for Nginx, one for PHP). That's all working fine, I can get "G'day world" being delivered to the browser via both HTML and PHP requests. Before I start writing any really-really code, I decided I better get PHPUnit working. TDD 'n' all that. I will admit I felt dirty yesterday when I wrote the gdayWorld.html and gdayWorld.php and ran them without tests; I just browsed to them and went "seems legit". Not the way I like to operate.

I sat down to write some tests. My gdayWorld.php is... basic:

<?php

$message = "G'day World";
echo $message;

I don't have any framework or anything installed as yet, so that's as good as it gets. That is in the web root. To test that I'd be needing to test that by curling it and checking the response, then I remembered Symfony has a WebTestCase for this sort of thing, then I decided I could not be arsed looking into that right now. All I need is a test to run to check PHPUnit is running. So... erm... here we go:

    /** @coversNothing */
    public function testNothing()
    {
        $this->assertTrue(false);
    }

This is enough to test PHPUnit. I'll write a better test once I know everything is working.

root@5fdd4abf5d20:/usr/share/fullstackExercise# vendor/bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

F                                 1 / 1 (100%)

Time: 00:00.166, Memory: 6.00 MB

There was 1 failure:

1) adamCameron\fullStackExercise\test\functional\public\WebServerTest::testNothing
Failed asserting that false is true.

/usr/share/fullstackExercise/test/functional/public/WebServerTest.php:12

FAILURES!
Tests: 1, Assertions: 1, Failures: 1.
root@5fdd4abf5d20:/usr/share/fullstackExercise#

Perfect. Now I can make it pass ;-)

$this->assertTrue(true);
root@5fdd4abf5d20:/usr/share/fullstackExercise# vendor/bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

.                                 1 / 1 (100%)

Time: 00:00.127, Memory: 6.00 MB

OK (1 test, 1 assertion)
root@5fdd4abf5d20:/usr/share/fullstackExercise#

So far, so good. My next step whilst I'm configuring stuff on PHPUnit, I decided to get the code coverage working too. I just added the code-coverage reporting bit to phpunit.xml and re-ran the tests:

root@eb28587ca66f:/usr/share/fullstackExercise# vendor/bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

Warning: No code coverage driver available

.                                 1 / 1 (100%)

Time: 00:00.210, Memory: 6.00 MB

OK (1 test, 1 assertion)
root@eb28587ca66f:/usr/share/fullstackExercise#

Doh! Forgot about that. I need to install Xdebug.

This was way more horsing around than it could have been, but this is down to the vagaries of the Docker set-up I was using. Once I decided to change direction it was a one-liner in me Dockerfile, and all good.

root@37d06fcf9209:/usr/share/fullstackExercise# php -v
PHP 8.0.0 (cli) (built: Dec 1 2020 03:33:03) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.0-dev, Copyright (c) Zend Technologies
with Xdebug v3.0.1, Copyright (c) 2002-2020, by Derick Rethans
root@37d06fcf9209:/usr/share/fullstackExercise#

Run the tests again...

root@37d06fcf9209:/usr/share/fullstackExercise# vendor/bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

Warning: XDEBUG_MODE=coverage or xdebug.mode=coverage has to be set

.                                 1 / 1 (100%)

Time: 00:00.220, Memory: 6.00 MB

OK (1 test, 1 assertion)
root@37d06fcf9209:/usr/share/fullstackExercise#

OK. Fine. Makes sense I s'pose. I added in the ini setting into my phpunit.xml:

<php>
  <ini name="xdebug.mode" value="coverage" />
</php>

No difference. I figured the ini setting is probably changed too late in the piece, as the xdebug.mode will need to be set before PHP starts running any code. And perhaps the environment variable approach would work such that the actual environment variable was set before hand. As it happens it ain't - all that happens is the value is added to the $_ENV array, which is hardly the same thing. From the docs:



However this step is still a relevant one cos we see a behavioural change here:

root@37d06fcf9209:/usr/share/fullstackExercise# vendor/bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.
Segmentation fault
root@37d06fcf9209:/usr/share/fullstackExercise#

Splat!

I'm gonna spare you a coupla hours of looking in to how to troubleshoot segmentation faults here. Nothing I read helped, and there's nothing useful relating to getting them with PHPUnit, other than a bunch of people getting these segmentation faults.

I started to wonder if it was an issue with Xdebug and PHP8, given PHP8 is so new and sometimes in the last I've noted that Xdebug has needed updating when there's a point release of PHP. But looking into this (at Xdebug: Supported Versions and Compatibility), the current version of Xdebug has been around for a while, and they say it's good to go with PHP8:

I was now thinking about versions of stuff, so I decided to rollback PHP to 7.4, and PHPUnit to 8.5. I realise I should have only done one at a time, but it did not occur to me when doing this that when troubleshooting things, only change one thing per test iteration.

root@41f51888adff:/usr/share/fullstackExercise# php -v
PHP 7.4.13 (cli) (built: Dec 1 2020 04:25:48) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
with Xdebug v3.0.1, Copyright (c) 2002-2020, by Derick Rethans
root@41f51888adff:/usr/share/fullstackExercise# vendor/bin/phpunit --version
PHPUnit 8.5.13 by Sebastian Bergmann and contributors.

root@41f51888adff:/usr/share/fullstackExercise# vendor/bin/phpunit
PHPUnit 8.5.13 by Sebastian Bergmann and contributors.

.Code coverage needs to be enabled in php.ini by setting 'xdebug.mode' to 'coverage'
root@41f51888adff:/usr/share/fullstackExercise#

This verifies we are back to PHP 7.4 and PHPUnit 8.5. And we don't have the segmentation fault any more, but... now PHP is ignoring both of the settings in phpunit.xml. I have this in place:

<php>
  <env name="XDEBUG_MODE" value="coverage"/>
  <ini name="xdebug.mode" value="coverage"/>
</php>

At this point I read up on the <ini> and <env> settings in phpunit.xml, and confirmed both of them are set too late for Xdebug to read them anyhow. Quite odd the way PHPUNit 9.5 reacts to the XDEBUG_MODE being set in that case. Anyway, I need to set them before PHP is run, so I set a really-really environment variable in my Dockerfile:

ENV XDEBUG_MODE=coverage

After rebuilding my containers:

root@6053a26dc3eb:/usr/share/fullstackExercise# echo $XDEBUG_MODE
coverage
root@6053a26dc3eb:/usr/share/fullstackExercise# vendor/bin/phpunit
PHPUnit 8.5.13 by Sebastian Bergmann and contributors.

..                                 2 / 2 (100%)

Time: 1.01 seconds, Memory: 6.00 MB

OK (2 tests, 2 assertions)

Generating code coverage report in HTML format ... done [596 ms]
root@6053a26dc3eb:/usr/share/fullstackExercise#

Hurrah! Let's have a look at the report:

This is correct: I'm not testing that code in /public yet.

I also quickly tried to use the xdebug.mode ini file setting. I dropped a /usr/local/etc/php/conf.d/phpunit-code-coverage-xdebug.ini with just xdebug.mode=coverage in it into my container's file system:

root@f3cd4a0715a5:/usr/share/fullstackExercise# cat /usr/local/etc/php/conf.d/phpunit-code-coverage-xdebug.ini
xdebug.mode=coverage
root@f3cd4a0715a5:/usr/share/fullstackExercise# php --ini
Configuration File (php.ini) Path: /usr/local/etc/php
Loaded Configuration File: (none)
Scan for additional .ini files in: /usr/local/etc/php/conf.d
Additional .ini files parsed: /usr/local/etc/php/conf.d/docker-php-ext-pdo_mysql.ini,
/usr/local/etc/php/conf.d/docker-php-ext-sodium.ini,
/usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini,
/usr/local/etc/php/conf.d/phpunit-code-coverage-xdebug.ini

root@f3cd4a0715a5:/usr/share/fullstackExercise# php -i | grep xdebug.mode
xdebug.mode => coverage => coverage
root@f3cd4a0715a5:/usr/share/fullstackExercise# vendor/bin/phpunit
PHPUnit 8.5.13 by Sebastian Bergmann and contributors.

..                                 2 / 2 (100%)

Time: 1.07 seconds, Memory: 6.00 MB

OK (2 tests, 2 assertions)

Generating code coverage report in HTML format ... done [869 ms]
root@f3cd4a0715a5:/usr/share/fullstackExercise#

Perfect.

My supposition here is that the issue all along was that I needed to set one of either the ini file option, or the environment variable option before PHPUnit runs. PHPUnit 8 correctly sez "OI!" if I don't; whereas PHPUnit 9 doesn't check correctly, and instead messes things up. Let's roll forward to PHP8 and PHPUnit 9.5 again and see:

root@f7d29f40ad62:/usr/share/fullstackExercise# php -v
PHP 8.0.0 (cli) (built: Dec 1 2020 03:33:03) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.0-dev, Copyright (c) Zend Technologies
with Xdebug v3.0.1, Copyright (c) 2002-2020, by Derick Rethans
root@f7d29f40ad62:/usr/share/fullstackExercise# vendor/bin/phpunit --version
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.

root@f7d29f40ad62:/usr/share/fullstackExercise# vendor/bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.
..                                 2 / 2 (100%)

Time: 00:01.535, Memory: 10.00 MB

OK (2 tests, 2 assertions)

Generating code coverage report in HTML format ... done [00:00.605]
root@f7d29f40ad62:/usr/share/fullstackExercise#

Hoo-frickin-ray!!

So that supposition was correct then.

I guess I could have saved myself a bunch of time here by just setting the .ini setting or the environment variable in the correct environment right from the outset. Then again had PHPUnit 9 not changed its warnings, I would have known that straight away: PHPUnit 8 was pretty clear on this. The problem and the solution here were pretty mundane, but I learned a lot about how to do things with Docker (which will all go into the other article I'm writing at the same time as this one), some stuff about Xdebug, some stuff about PHP and some stuff about PHPUnit. So it was a worthwhile afternoon of scratching my head. I'm also pleased I've sat here and done 8hrs of "work" for the second day running. Hopefully this will stick and I'll work my way out of my unemployed malaise before long. We'll see.

PHPUnit 9.5 on PHP 7.4

As I said earlier, I should not have changed two things at once when testing this, as it makes it harder to draw accurate conclusions. I omitted to say I did then go back and test PHPUnit 9.5 on PHP 7.4, so as to check whether the issue was due to PHPUnit 9.5, or whether it was some combination of that and PHP version. Here are the test results when running PHPUnit 9.5 on PHP 7.4 with XDEBUG_MODE set in phpunit.xml and not in the environment. xdebug.mode is not set anywhere:

root@81c63ae50626:/usr/share/fullstackExercise# php -v
PHP 7.4.13 (cli) (built: Dec 1 2020 04:25:48) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
with Xdebug v3.0.1, Copyright (c) 2002-2020, by Derick Rethans
root@81c63ae50626:/usr/share/fullstackExercise# vendor/bin/phpunit --version
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.
root@81c63ae50626:/usr/share/fullstackExercise# vendor/bin/phpunit
PHPUnit 9.5.0 by Sebastian Bergmann and contributors.
Segmentation fault
root@81c63ae50626:/usr/share/fullstackExercise#

To me this demonstrates the issue is with PHPUnit 9.5 and nothing else.

Anyway... now for a beer.

Righto.


--
Adam

Friday, 20 December 2019

PHPUnit: trying to be clever with data providers and failing

G'day:
Yesterday we had a seemingly strange situation with a test one of my colleagues had written. It was behaving as if PHPUnit's exactly matcher was not working. The developer concerned was in the process of concluding that that is exactly what the problem is: PHPUnit. I suggested we pause and back-up a bit: given thousands of people uses PHPUnit and the exactly matcher every day (including ourselves) and have not reported this issue, whereas the code we are running it with was freshly written and not used by anyone yet... almost certainly the problem lay with our code.

Before I get on with what the problem was, this bears repeating:

If you are doing something new and it's going wrong, the problem will almost certainly not be related to any established third-party tools you are using. It'll be down to how you're using them.

I find it's a really helpful mindset when troubleshooting problems to just own the issue from the outset, rather than looking around for something else to blame. And if you've looked and looked and can't see the problem with your own code, this probably just means you haven't spotted the problem yet. Absence of evidence is not evidence of absence.

Anyway, the issue at hand. Here's a pared-down repro of the situation. The code is meaningless as I've removed all business context from it, but we have this:

class MyService
{
    private $dependency;

    function __construct(MyDependency $dependency){
        $this->dependency = $dependency;
    }

    function myMethod() {
        return $this->dependency->someMethod();
    }
}


And MyDependency is this:

class MyDependency
{
    function someMethod(){
        return "ORIGINAL VALUE";
    }
}

myMethod is new, so the dev had written a test for this:

class MyServiceTest extends TestCase
{

    /** @dataProvider provideTestCasesForMyMethodTests */
    public function testMyMethodViaDataProvider($dependency)
    {
        $dependency
            ->expects($this->once())
            ->method('someMethod')
            ->willReturn("MOCKED VALUE");

        $service = new MyService($dependency);

        $result = $service->myMethod();

        $this->assertEquals("MOCKED VALUE", $result);
    }

    public function provideTestCasesForMyMethodTests(){
        return [
            'it should call someMethod once' => [
                'dependency' => $this->createMock(MyDependency::class)
            ]
        ];
    }
}

I hasten to add that it's completely daft to have the data provider in this example. However in the real-world example there were valid test variations, and given the method being tested was a factory method, part of the test was to check that the relevant method had been called on the relevant object being returned by the factory method. The data provider was legit.

This test passes:

PS C:\src\php\general> vendor\bin\phpunit .\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php
PHPUnit 8.5.0 by Sebastian Bergmann and contributors.

.                                                                   1 / 1 (100%)

Time: 77 ms, Memory: 4.00 MB

OK (1 test, 1 assertion)
PS C:\src\php\general>

However when we were trouble shooting something, we ended up with this assertion:

$dependency
    ->expects($this->exactly(10))
    ->method('someMethod')
    ->willReturn("MOCKED VALUE");


And the test still passed:

PS C:\src\php\general> vendor\bin\phpunit .\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php
PHPUnit 8.5.0 by Sebastian Bergmann and contributors.

.                                                                   1 / 1 (100%)

Time: 77 ms, Memory: 4.00 MB

OK (1 test, 1 assertion)
PS C:\src\php\general>

Whut the?

When we were looking at this I asked my colleague to simplify their tests a bit. The IRL version was testing this case via a data provider and test that was testing a bunch of things in a method that had some complexity issues, and I wanted to pare the situation back to just one simple test, testing this one thing. We ended up with this test:

public function testMyMethodViaInlineMock()
{
    $dependency = $this->createMock(MyDependency::class);
    $dependency
        ->expects($this->once())
        ->method('someMethod')
        ->willReturn("MOCKED VALUE");

    $service = new MyService($dependency);

    $result = $service->myMethod();

    $this->assertEquals("MOCKED VALUE", $result);
}


The difference here being we are not using a data provider any more, the mocking of the dependency is in the test. And this passes:

PS C:\src\php\general> vendor\bin\phpunit .\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php                 PHPUnit 8.5.0 by Sebastian Bergmann and contributors.

.                                                                   1 / 1 (100%)

Time: 61 ms, Memory: 4.00 MB

OK (1 test, 2 assertions)
PS C:\src\php\general>                                                                                                                              

Good, fine, but now we tested out the ->expects($this->exactly(10)) issue again:

PS C:\src\php\general> vendor\bin\phpunit .\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php                 PHPUnit 8.5.0 by Sebastian Bergmann and contributors.

F                                                                   1 / 1 (100%)

Time: 102 ms, Memory: 4.00 MB

There was 1 failure:

1) me\adamcameron\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest::testMyMethodViaInlineMock
Expectation failed for method name is "someMethod" when invoked 10 time(s).
Method was expected to be called 10 times, actually called 1 times.

FAILURES!
Tests: 1, Assertions: 2, Failures: 1.
PS C:\src\php\general>                                                                                                                                                    

This is more like it. But... erm... why?

I'd had "unexpected" behaviour previously when referring to $this inside a data provider method, because the data provider method is not called on the same instance of the test class that the tests are run with (why? Buggered if I know, but it is), and had my suspicions that this could be contributing to it. This is when we just shrugged and left the test as it was - working now - and I undertook to investigate some more later. The code I've been sharing here is the start of my investigation. I was at this point none the wiser though.

On a whim I tried another test variation, to see if there was any change in behaviour across the threshold of the count (so checking "off by one" situations). Here I have a new method that calls the same dependency method five times:
function myOtherMethod(){
     $this->dependency->someOtherMethod();
     $this->dependency->someOtherMethod();
     $this->dependency->someOtherMethod();
     $this->dependency->someOtherMethod();
    return  $this->dependency->someOtherMethod();
}
And a test for it that just checks what happens when we expect it to be called 3-7 times:

/** @dataProvider provideTestCasesForMyOtherMethodTests */
public function testMyOtherMethodCallsDependencyMethodEnoughTimes($dependency, $expectedCount) {
    $dependency
        ->expects($this->exactly($expectedCount))
        ->method('someOtherMethod')
        ->willReturn("MOCKED VALUE");

    $service = new MyService($dependency);

    $result = $service->myOtherMethod();

    $this->assertEquals("MOCKED VALUE", $result);
}

public function provideTestCasesForMyOtherMethodTests(){
    return [
        'it expects myOtherMethod to be called 3 times' => [
            'dependency' => $this->createMock(MyDependency::class),
            'expectedCount' => 3
        ],
        'it expects myOtherMethod to be called 4 times' => [
            'dependency' => $this->createMock(MyDependency::class),
            'expectedCount' => 4
        ],
        'it expects myOtherMethod to be called 5 times' => [
            'dependency' => $this->createMock(MyDependency::class),
            'expectedCount' => 5
        ],
        'it expects myOtherMethod to be called 6 times' => [
            'dependency' => $this->createMock(MyDependency::class),
            'expectedCount' => 6
        ],
        'it expects myOtherMethod to be called 7 times' => [
            'dependency' => $this->createMock(MyDependency::class),
            'expectedCount' => 7
        ]
    ];
}


All things being equal, I was expecting no failures here, given what we'd seen before. But... erm...

PS C:\src\php\general> vendor\bin\phpunit .\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php --filter=testMyOtherMethodCallsDependencyMethodEnoughTimes                                   PHPUnit 8.5.0 by Sebastian Bergmann and contributors.

FF...                                                               5 / 5 (100%)

Time: 64 ms, Memory: 4.00 MB

There were 2 failures:

1) me\adamcameron\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest::testMyOtherMethodCallsDependencyMethodEnoughTimes with data set "it expects myOtherMethod to be called 3 times" (Mock_MyDependency_6a23ba62 Object (...), 3)
me\adamcameron\general\phpunit\expectsWithDataProviderIssue\MyDependency::someOtherMethod() was not expected to be called more than 3 times.

C:\src\php\general\src\phpunit\expectsWithDataProviderIssue\MyService.php:21
C:\src\php\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php:59

2) me\adamcameron\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest::testMyOtherMethodCallsDependencyMethodEnoughTimes with data set "it expects myOtherMethod to be called 4 times" (Mock_MyDependency_6a23ba62 Object (...), 4)
me\adamcameron\general\phpunit\expectsWithDataProviderIssue\MyDependency::someOtherMethod() was not expected to be called more than 4 times.

C:\src\php\general\src\phpunit\expectsWithDataProviderIssue\MyService.php:22
C:\src\php\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php:59

FAILURES!
Tests: 5, Assertions: 3, Failures: 2.
PS C:\src\php\general>                   


OK so this is a bit perplexing. When the expected call count is less than the actual call count we get it failing correcty. But when it's more... no failure. Now I'm flummoxed. For good measure I also have a test not creating the dependency in the data provider, but instead inline in the test:

/** @dataProvider provideTestCasesForMyOtherMethodTests */
public function testMyOtherMethodCallsDependencyMethodEnoughTimesWithDependencyCreatedInTest($_, $expectedCount) {
    $dependency = $this->createMock(MyDependency::class);
    $dependency
        ->expects($this->exactly($expectedCount))
        ->method('someOtherMethod')
        ->willReturn("MOCKED VALUE");

    $service = new MyService($dependency);

    $result = $service->myOtherMethod();

    $this->assertEquals("MOCKED VALUE", $result);
}


And this one behaves correctly:

PS C:\src\php\general> vendor\bin\phpunit .\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php --filter=testMyOtherMethodCallsDependencyMethodEnoughTimesWithDependencyCreatedInTest                                                   PHPUnit 8.5.0 by Sebastian Bergmann and contributors.

FF.FF                                                               5 / 5 (100%)

Time: 67 ms, Memory: 4.00 MB

There were 4 failures:

1) me\adamcameron\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest::testMyOtherMethodCallsDependencyMethodEnoughTimesWithDependencyCreatedInTest with data set "it expects myOtherMethod to be called 3 times" (Mock_MyDependency_2707ab91 Object (...), 3)
me\adamcameron\general\phpunit\expectsWithDataProviderIssue\MyDependency::someOtherMethod() was not expected to be called more than 3 times.

C:\src\php\general\src\phpunit\expectsWithDataProviderIssue\MyService.php:21
C:\src\php\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php:99

2) me\adamcameron\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest::testMyOtherMethodCallsDependencyMethodEnoughTimesWithDependencyCreatedInTest with data set "it expects myOtherMethod to be called 4 times" (Mock_MyDependency_2707ab91 Object (...), 4)
me\adamcameron\general\phpunit\expectsWithDataProviderIssue\MyDependency::someOtherMethod() was not expected to be called more than 4 times.

C:\src\php\general\src\phpunit\expectsWithDataProviderIssue\MyService.php:22
C:\src\php\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php:99

3) me\adamcameron\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest::testMyOtherMethodCallsDependencyMethodEnoughTimesWithDependencyCreatedInTest with data set "it expects myOtherMethod to be called 6 times" (Mock_MyDependency_2707ab91 Object (...), 6)
Expectation failed for method name is "someOtherMethod" when invoked 6 time(s).
Method was expected to be called 6 times, actually called 5 times.

4) me\adamcameron\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest::testMyOtherMethodCallsDependencyMethodEnoughTimesWithDependencyCreatedInTest with data set "it expects myOtherMethod to be called 7 times" (Mock_MyDependency_2707ab91 Object (...), 7)
Expectation failed for method name is "someOtherMethod" when invoked 7 time(s).
Method was expected to be called 7 times, actually called 5 times.

FAILURES!
Tests: 5, Assertions: 6, Failures: 4.
PS C:\src\php\general>                                                                                                                                                                                                                                               

I started diving back through the PHPUnit code, and found the point where my two test variations began to demonstrate different behaviour. I edited InvocationHandler::verify to spit out some telemetry:

public function verify(): void
{
    printf(PHP_EOL . "InvocationHandler::verify called with %d matchers" . PHP_EOL, count($this->matchers));

    foreach ($this->matchers as $matcher) {
        $matcher->verify();
    }



And here's what I get running the two tests:

PS C:\src\php\general> vendor\bin\phpunit .\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php --filter=testMyMethodViaDataProvider                                                                                                    PHPUnit 8.5.0 by Sebastian Bergmann and contributors.

.                                                                   1 / 1 (100%)
InvocationHandler::verify called with 0 matchers


Time: 60 ms, Memory: 4.00 MB

OK (1 test, 1 assertion)
PS C:\src\php\general> vendor\bin\phpunit .\test\phpunit\expectsWithDataProviderIssue\MyServiceTest.php --filter=testMyMethodViaInlineMock                                                                                                      PHPUnit 8.5.0 by Sebastian Bergmann and contributors.

F                                                                   1 / 1 (100%)
InvocationHandler::verify called with 1 matchers


Time: 62 ms, Memory: 4.00 MB

There was 1 failure:

1) me\adamcameron\general\test\phpunit\expectsWithDataProviderIssue\MyServiceTest::testMyMethodViaInlineMock
Expectation failed for method name is "someMethod" when invoked 10 time(s).
Method was expected to be called 10 times, actually called 1 times.

FAILURES!
Tests: 1, Assertions: 2, Failures: 1.
PS C:\src\php\general>                             

Well. So when we run our tests using the mock created in the test code... we get our matcher "loaded" (for lack of a better term); when we run our test using the mock created in the data provider: it's nowhere to be seen. Interesting.

Looking further up the call stack, we find that this verify method is called from TestCase::verifyMockObects:
private function verifyMockObjects(): void
{
    foreach ($this->mockObjects as $mockObject) {
        if ($mockObject->__phpunit_hasMatchers()) {
            $this->numAssertions++;
        }

        $mockObject->__phpunit_verify(
            $this->shouldInvocationMockerBeReset($mockObject)
        );
    }


(I dunno what's going on with the method name there, but I did verify the call stack... PHPUnit is doing some special magic there).

verifyMockObjects in turn is called from runBare in the same class, and that is run from TestResult::run.

The key thing here is verifyMockObjects being run in the context of the TestCase object, and it looks at $this->mockObjects. Where does $this->mockObjects come from? When we create the mock:

public function getMock(): MockObject
{
    $object = $this->generator->getMock(
        $this->type,
        !$this->emptyMethodsArray ? $this->methods : null,
        $this->constructorArgs,
        $this->mockClassName,
        $this->originalConstructor,
        $this->originalClone,
        $this->autoload,
        $this->cloneArguments,
        $this->callOriginalMethods,
        $this->proxyTarget,
        $this->allowMockingUnknownTypes,
        $this->returnValueGeneration
    );

    $this->testCase->registerMockObject($object);

    return $object;
}

// ...

public function registerMockObject(MockObject $mockObject): void
{
    $this->mockObjects[] = $mockObject;
}

(remember that our createMock call is just a wrapper for a getMockBuilder()...->getMock() call).

So what's the bottom line here? Well my initial wariness about creating the mock in the data provider method has legs... when a mock is created it's registered in the TestCase object it's created within... and the one that the data provider method is running in is not the same TestCase as the one the test is run in. So... none of the mock's constraints are checked when the test is run. That's mostly it.

But what's the story with how that last test still failed on the variants with only 3/5 and 4/5 calls made? That check is done when the mocked method is actually called. So given this:

function myMethod() {
    return $this->dependency->someMethod();
}

When the mocked someMethod is called, a check is made then as to whether it's been called too many times for the expectation. So that's one within the mock itself at the time it's executed, hence it still "works". It's just the check that the exact number of call times was hit that is done at test level, not mock level if that makes sense.

This all just goes to show that one should just use data provider methods for providing data values for test variations, not to treat them like some sort of looping mechanism that the test is just a callback for. That's not the intent, and when one tries to be tricky... one is quickly found out to not be as clever as one hopes... ;-)

That's an awful lot of carry on for what ended up to be an easy / simple explanation. I'm mostly writing this up in case anyone else gets the same issue, and also to show how I went about troubleshooting and investigating it. I'll be putting this in front of my team shortly...

Righto.

--
Adam