Friday 27 October 2023

I'm a big meany again

G'day

Chortle:

I'm new to cf development and recently joined the cfml slack community. I noticed some curmudgeonly posts from this fella at first and didn't think anything of it... I'm now realizing that he is an a**h*** to people all the time. For example, there was a question posted today that currently has 14 replies. A few of them are from others trying to answer OP's question; 7 are this guy explaining how stupid of a question it is. What gives? I'm sure he's a knowledgeable and respected developer, but the pedantry is off the charts!

DJgrassyknoll on Reddit

Guilty as charged I suppose.

However the charges are pretty specious. I think the thread in question was the one with a question (paraphrase) "is there a function in CFML that can convert JSON to XML?". This is a largely non-sensical question, and without further elaboration, can only be furnished with non-sensical and unhelpful answers. I don't wanna do that. I wanna get the person's problem solved. I did point this out to the bod in question, and explained why, with both a real-world analogy, plus a code demonstration! Here's the code version:

json = serializeJson({key="value"})
xml = xmlParse("<nowItIsXml>#encodeForXml(json)#</nowItIsXml>")

writeDump([
    json = json,
    xml = xml
])

Completely answers the question as stated, but is a completely frickin useless answer. Other than to illustrate the shortcomings of the question.

I've been answering people's CFML questions for years, and the biggest blocker to actually solving their problem is coercing what their actual problem is out from their brain. Really a lot of people don't seem to get we can't see the code in front of them, or the Jira ticket (sic) they're trying to deal with, or any other thoughts on the matter they might have had beyond what they type in as their question. So… not to be put off… I will explain this, and try to get a decent question out of them. So we all can understand it and then get their issue sorted. I will also often post links to How To Ask Questions The Smart Way and/or The SSCCE. I'm not going to apologise for that. It's good advice and if more people took it on board, they would improve, and also their chances of getting good help when they need it would improve. How is that bad?

We're all busy people, so the quicker and easier it is for us to get the info we need in front of us to help: the more likely we are to help. As one of the participants in that thread said "[when] people ask shitty questions, I mostly will just ignore it". That's your call pal. But I won't ignore them. I'll still try to help.

Ah - fuck it - people can say what they like about me. I don't care (and hey it's given me an in to blow some of the dust off this thing). But try to have a well-formed train of thought before doing so. And also poss sink some time into helping people to understand what it's like before stirring the pot in yer little sewing circle.

But yeah cheers for the new strapline quote.

Righto.

--
Adam

PS: props for using "curmudgeonly" though. Good word.

Monday 1 May 2023

PHP / Symfony: working through "Symfony: The Fast Track", part 5: Twig stuff, and irritation

G'day:

I'm back on the case working through "Symfony: The Fast Track". This will be part five of this series, after the first four:

The page I'm starting on is Building the User Interface, which seems more interesting than the previous section. I also see mention of my old friend Twig in the first subsection of this, so it will be good to look at that some more and see how it's moved along since I last did PHP UI stuff about 7-8 years ago (god is it that long?).


Using Twig for the Templates

The first thing I need to do is to restore a twig file I deleted during the last exercise, thinking it was redundant. So: back comes templates/base.html.twig. I'll be referring back to this, so will dump it out here, even though it's boilerplate, and not my code:

<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8">
    <title>{% block title %}Welcome!{% endblock %}</title>
    <link
    	rel="icon"
        href="
            data:image/svg+xml,
            <svg
            	xmlns=%22http://www.w3.org/2000/svg%22
                viewBox=%220 0 128 128%22
            >
                <text y=%221.2em%22 font-size=%2296%22>⚫️</text>
            </svg>
        "
    >
    {# Run `composer require symfony/webpack-encore-bundle` to start using Symfony UX #}
    {% block stylesheets %}
        {{ encore_entry_link_tags('app') }}
    {% endblock %}

    {% block javascripts %}
        {{ encore_entry_script_tags('app') }}
    {% endblock %}
</head>
<body>
{% block body %}{% endblock %}
</body>
</html>

And one for templates/conference/index.html.twig:

{% extends 'base.html.twig' %}

{% block title %}Conference Guestbook{% endblock %}

{% block body %}
    <h2>Give your feedback!</h2>

    {% for conference in conferences %}
        <h4>{{ conference }}</h4>
    {% endfor %}
{% endblock %}

Note that it inherits from the base one, and overrides its title and body blocks. I'll not discuss all the syntax vagaries as they're well documented: Twig Documentation. It's easy enough to read and infer what's going on, anyhow.

The next section is entitled "Using Twig in a Controller" and I panicked as I thought they meant they were gonna put Twig code into a controller method (remember they've been a bit woolly about what goes in a controller already, so it would not have surprised me), but fortunately no. It's just describing how to use the twig file:

public function index(Environment $twig, ConferenceRepository $conferenceRepository): Response
{
    // return of inline HTML (!!) removed

    return new Response($twig->render(
        "conference/index.html.twig",
        ["conferences" => $conferenceRepository->findAll()]
    ));
}

The conferences there are then used in the loop in the twig.


Creating the Page for a Conference

OK so here's a cool thing Symfony does. I am creating the page for the route /conference/{id}, and if my controller method takes a Conference object, Symfony is clever enough to go "I see what yer doing there" and goes and gets the conference for that ID. EG, I've added this routing to get to my controller method:

config/routes/conference.yaml

# …

show:
    path: /{id}
    methods: [ GET ]
    controller: adamcameron\symfonythefasttrack\Controller\ConferenceController::show

And the controller method the exercise has me write is this (nothing noteworthy really):

public function show(Environment $twig, Conference $conference, CommentRepository $commentRepository): Response
{
    return new Response($twig->render(
        "conference/show.html.twig",
        [
            "conference" => $conference,
            "comments" => $commentRepository->findBy(
                ["conference" => $conference],
                ["createdAt" => "DESC"]
            )
        ]
    ));
}

The routing takes an ID; the controller receives the equivalent Conference object.

Note that whilst it's neat that Symfony does this, that controller they've had me write is borderline not great MVC in my books. I don't really think controllers should be aware of things like "repositories". Repositories are a storage consideration, and should be hidden away with the model. What should be happening here I think is that the controller receives its Conference, but the Conference should implement whatever is necessary to get its Comment collection from wherever it put it (and via whatever mechanism it did it with). This is not the controller's job. This is a good example of how using generic ORM crap leads to subpar code. I realise this is a tutorial about using Twig, but I think on the whole this book is encouraging some pretty poor design decisions because that better shows-off Symfony's bells and whistles.

Ahem. Moving along…

For reference here's the twig template for the view:

{% extends 'base.html.twig' %}

{% block title %}Conference Guestbook - {{ conference }}{% endblock %}

{% block body %}
    <h2>{{ conference }} Conference</h2>

    {% if comments|length > 0 %}
        {% for comment in comments %}
            {% if comment.photofilename %}
                <img src="{{ asset('uploads/photos/' ~ comment.photofilename) }}"/>
            {% endif %}

            <h4>{{ comment.author }}</h4>
            <small>
                {{ comment.createdAt|format_datetime('medium', 'short') }}
            </small>

            <p>{{ comment.text }}</p>
        {% endfor %}
    {% else %}
        <div>No comments have been posted yet for this conference.</div>
    {% endif %}
{% endblock %}

And the test shows the initial part of the template implementation presents the detail from the correct conference correctly:

/** @testdox /conference/{id} end point returns a 200 OK for a valid conference */
public function testShow()
{
    $kernel = new Kernel("test", true);
    $kernel->boot();

    $container = $kernel->getContainer();

    $connection = $container->get("database_connection");
    $testConference = $connection->executeQuery("
        SELECT id, city, year
        FROM conference LIMIT 1
    ")->fetchAssociative();
    if (!$testConference) {
        $this->markTestSkipped("No conferences in database, test aborted");
    }

    $client = static::createClient();
    $client->request('GET', '/conference/' . $testConference['id']);

    $this->assertResponseIsSuccessful();

    $this->assertSelectorTextContains(
        'title',
        sprintf(
            "Conference Guestbook - %s %s",
            $testConference['city'],
            $testConference['year']
        )
    );
}

I presumed it would 404 if the {id} requested wasn't in the DB:

/** @testdocs /conference/{id} end point returns a 404 if the conference is not found */
public function testShowNotFound()
{
    $client = static::createClient();
    $client->request('GET', '/conference/999999');

    $this->assertResponseStatusCodeSame(Response::HTTP_NOT_FOUND);
}

And this passes. NB: the 404 page is a general Symfony one, but that's fine. It's handling it correctly.

It's important to note that I have not written any code relating to Conferences or the storage thereof in this chapter of the exercise. I'm simply using the stuff Symfony generated for me during the previous exercise.

Oh one additional thing I did need to do was to install the twig/intl-extra dependency, because I'm using that format_datetime Twig filter. How did I know I needed to do this? Symfony told me (well, OK, I have also just scrolled ahead in the lesson, and it also told me there):

That's quite cool. I note it's claiming it's a syntax error though, which it is not. If it was a syntax error, the code wouldn't compile, let alone run to the point to which it works out I have a Twig filter missing, and which one it is and where to get it from. It's a MissingTwigExtensionError or something. But a runtime error of some sort, not a compile-time one. Still: semantics aside, it's helpful. It does seem a bit odd to me though that something in either Twig or Symfony clearly knows what all the extensions are and what external libs they're in. This seems a bit of an inappropriate overreach to me, and a bit of a break of the Open-Closed Principle. Still: productivity over dogma, eh?

Oh! So far I have no comments in the DB, so all that Twig bit about rendering them doesn't do anything. All I'm getting is this:

Time to test the comments bit, before I move on.

[… more time passes than I would like to admit…]

I went round the houses with this a bit. I had to install a bunch of Node.js stuff in my Node container because one of the Twig function's internal workings checked for [something] and if it didn't exist: threw a PHP exception. I didn't preserve the error, but this code (in show.html.twig) was the culprit:

<img src="{{ asset('uploads/photos/' ~ comment.photofilename) }}"/>

The asset function was complaining that public/build/entrypoints.json was missing. I googled about and found out I needed to have symfony/webpack-encore-bundle installed, the instructions for which (that previous link) seem to not be as complete as they could be for codebases not using Symfony Flex. Still: the residual issues I had were all fairly googleable, and easily fixed. Sorry to be a bit vague there, but I did this lot on Fri/Sat - it's now Sun - and I can't be arsed digging through my browser history to recreate the - fairly unrewarding - sequence of hurdles I bumbled over to get my PHP app to stop erroring because I didn't have some webpack shit configured properly.

BTW: this is another thing I don't think a server-side MVC framework should be investing its time in. There's plenty of client-side frameworks for handling client-side stuff. Server-side frameworks should just busy themselves with their own job. Seems to me like another case of "if your only tool is a hammer, every problem starts looking like a nail".

Anyway, my test stopped erroring, and after some small effort I got it to go green. I actually ended up with two tests here:

/** @testdox /conference/{id} also displays any comment associated with the conference */
public function testShowComments()
{
    $testConference = $this->getConferenceToTestWith();

    $client = static::createClient();
    $client->request("GET", "/conference/" . $testConference->getId());

    $this->assertResponseIsSuccessful();
    $this->assertCommentersAreCorrect($client, $testConference);
}

private function getConferenceToTestWith(): Conference
{
    $container = Container::getContainer();

    $repo = $container->get("testing.ConferenceRepository");
    $qb = $repo->createQueryBuilder("cf");

    $qb->select("cf")
        ->setMaxResults(1)
        ->leftJoin("cf.comments", "cm")
        ->groupBy("cf.id")
        ->having("count(cm.id) > 1");
    $testConference = $qb->getQuery()->getResult();

    if (!$testConference) {
        $this->markTestSkipped("No suitable conferences in database, test aborted");
    }

    return $testConference[0];
}

private function getExpectedCommenters(Conference $testConference): array
{
    $expectedCommenters = $testConference->getComments()->toArray();
    usort($expectedCommenters, function ($e1, $e2) {
        return $e2->getCreatedAt() <=> $e1->getCreatedAt();
    });
    $expectedCommenters = array_map(function ($comment) {
        return $comment->getAuthor();
    }, $expectedCommenters);
    return $expectedCommenters;
}

private function assertCommentersAreCorrect(KernelBrowser $client, Conference $testConference): void
{
    $commenters = $client
        ->getCrawler()
        ->filter("body > h4")
        ->each(function ($node) {
            return $node->text();
        });

    $expectedCommenters = $this->getExpectedCommenters($testConference);
    $this->assertEquals($expectedCommenters, $commenters);
}

This one goes right through from the controller to the DB and back: comparing the data from the DB (expected) matching the data returned in the mark-up (actual).

I don't wanna be hitting the DB in my tests as a rule, especially if it's only a mechanism to get data to then see what the logic does with it. If I'm testing the logic, I want the expediency of using canned data that best exercises the logic. I tried to write a unit test of this, but it proved to be more difficult than I had patience for. The problem being that all the "logic" is in the controller method:

public function show(
    Environment $twig,
    Conference $conference,
    CommentRepository $commentRepository
): Response {
    return new Response($twig->render(
        "conference/show.html.twig",
        [
            "conference" => $conference,
            "comments" => $commentRepository->findBy(
                ["conference" => $conference],
                ["createdAt" => "DESC"]
            )
        ]
    ));
}

So to unit test that, I need to be able to mock the collaborators. Mocking my ones - the Conference and the CommentsRepository - was easy. Dealing with the Twig Environment was beyond my patience. And I wasn't even trying to do anything fancy with it: all I wanted was to spy on it so I could test the values exiting my test unit. But Symfony is very very convoluted in how it configures things, and I could not work out a minimum Environment that would actually initialise without erroring. Please note that I was very aware of "don't mock what you don't own" during all of this, and beyond a point I had decided that this was the wrong approach as well as being a fool's errand.

In the end I wrote a more functional test. I stuck with the Environment that Symfony had configured in its DI container, and the checked the results of its rendering. This is not so bad: it's all just done in code and no hits to external services (the DB), but it resulted in more code than I'm really happy with. There's no need to pore over this: I'm not gonna discuss it further. I only included to show the amount of effort involved.

/** @testdox /conference/{id} displays correct conference and comments */
public function testShow()
{
    $conferenceTitle = "MOCKED_CONFERENCE";
    list($twigEnvironment, $conference, $commentRepository) = $this->getTestDependencies($conferenceTitle);
    $controller = new ConferenceController();

    $result = $controller->show($twigEnvironment, $conference, $commentRepository);

    $this->assertResponseIsCorrect($result, $conferenceTitle, $commentRepository);
}

private function getTestDependencies(string $conferenceTitle): array
{
    $twigEnvironment = self::getContainer()->get(TwigEnvironment::class);

    $conference = $this->getMockedConference($conferenceTitle);

    $commentRepository = $this->getMockedCommentRepository();
    return array($twigEnvironment, $conference, $commentRepository);
}

private function getMockedConference(string $conferenceTitle): Conference
{
    $conference = $this->getMockBuilder(Conference::class)
        ->disableOriginalConstructor()
        ->getMock();
    $conference
        ->expects($this->any())
        ->method("__toString")
        ->willReturn($conferenceTitle);
    return $conference;
}

private function getMockedCommentRepository(): CommentRepository
{
    $commentCount = 3;
    $commentsToTestFor = range(1, $commentCount);

    $testComments = array_map(function ($i) {
        return (new Comment())
            ->setAuthor("COMMENTER" . $i)
            ->setText("COMMENT" . $i)
            ->setCreatedAt(new \DateTimeImmutable("2020-01-0{$i}"));
    }, $commentsToTestFor);

    $commentRepository = $this->getMockBuilder(CommentRepository::class)
        ->disableOriginalConstructor()
        ->getMock();
    $commentRepository
        ->method("findBy")
        ->willReturn($testComments);
    return $commentRepository;
}

private function assertResponseIsCorrect(Response $result, string $conferenceTitle, mixed $commentRepository): void
{
    $this->assertEquals(Response::HTTP_OK, $result->getStatusCode());

    $xpathDocument = $this->getContextAsXpathDocument($result);
    $this->assertTitleIsCorrect($xpathDocument);
    $this->assertSubheadingIsCorrect($xpathDocument, $conferenceTitle);
    $this->assertCommentsAreCorrect($xpathDocument, $commentRepository);
}

private function getContextAsXpathDocument(Response $result): DOMXPath
{
    $content = $result->getContent();
    $document = new DOMDocument();
    $document->loadHTML($content, LIBXML_NOWARNING | LIBXML_NOERROR);
    $xpathDocument = new DOMXPath($document);
    return $xpathDocument;
}

private function assertTitleIsCorrect(DOMXPath $xpathDocument): void
{
    $title = $xpathDocument->query('/html/head/title[text()]');
    $this->assertCount(1, $title);
    $this->assertEquals("Conference Guestbook - MOCKED_CONFERENCE", $title->item(0)->textContent);
}

private function assertSubheadingIsCorrect(DOMXPath $xpathDocument, string $conferenceTitle): void
{
    $h2 = $xpathDocument->query('/html/body/h2[text()]');
    $this->assertCount(1, $h2);
    $this->assertEquals("$conferenceTitle Conference", $h2->item(0)->textContent);
}

private function assertCommentsAreCorrect(DOMXPath $xpathDocument, CommentRepository $commentRepository): void
{
    $testComments = $commentRepository->findBy([], []);
    $commentCount = count($testComments);

    $commentAuthors = $xpathDocument->query('/html/body/h4[text()]');
    $this->assertCount($commentCount, $commentAuthors);
    $commentTexts = $xpathDocument->query('/html/body/p[text()]');
    $this->assertCount($commentCount, $commentTexts);
    $commentDates = $xpathDocument->query('/html/body/small[text()]');
    $this->assertCount($commentCount, $commentDates);

    array_walk($testComments, function ($comment, $i) use ($commentAuthors, $commentTexts, $commentDates) {
        $this->assertEquals($comment->getAuthor(), $commentAuthors->item($i)->textContent);
        $this->assertEquals($comment->getText(), $commentTexts->item($i)->textContent);

        $commentCreatedAtFormatted = $comment->getCreatedAt()->format("M j, Y, h:i A");
        $this->assertEquals($commentCreatedAtFormatted, trim($commentDates->item($i)->textContent));
    });
}

There's a moral to all this. "Don't mock what you don't own" issues aside, testing code that is tightly-coupled to the framework - like a controller method pretty much is - is gonna have overhead. As such: keep as little code in yer controllers as possible: have yer business logic within yer model, because you "own" this, and there will be less over-engineers framework code getting in the way.


Linking Pages Together & Pagination & some refactoring

Next they've got some more Twig advice about using its methods to build predictable URLs insted of hard-coding them, eg:

<a href="/conference/{{ conference.id }}">View</a>
<a href="{{ path('conference', { id: conference.id }) }}">View</a>

Good advice. I'm not gonna dwell on it further than that, other than sticking this code into my Twig template (after I test for it, obvs). I'll spare you the code, but it'll be in source control if you decide you do want to look at it. I got slightly tripped-up there as I had changed my route's name to be conference_show, and that reference to conference is the route's name. Easily fixed though.

Next it shows how to paginate record sets in the UI, which is pleasingly straight forward. A method in the repository:


public const PAGINATOR_PER_PAGE = 2;

// …

public function getPaginator(Conference $conference, int $offset): Paginator
{
    $query = $this->createQueryBuilder('c')
        ->andWhere('c.conference = :conference')
        ->setParameter('conference', $conference)
        ->orderBy('c.createdAt', 'DESC')
        ->setMaxResults(self::PAGINATOR_PER_PAGE)
        ->setFirstResult($offset)
        ->getQuery();

    return new Paginator($query);
}

Pass a paginator instead of the articles from the controller:

public function show(
    Request $request,
    Environment $twig,
    Conference $conference,
    CommentRepository $commentRepository
): Response {
    $offset = max(0, $request->query->getInt('offset', 0));
    $paginator = $commentRepository->getPaginator($conference, $offset);

    return new Response($twig->render(
        "conference/show.html.twig",
        [
            "conference" => $conference,
            "comments" => $commentRepository->findBy(["conference" => $conference], ["createdAt" => 'DESC']),
            "comments" => $paginator,
            'previous' => $offset - CommentRepository::PAGINATOR_PER_PAGE,
            'next' => min(count($paginator), $offset + CommentRepository::PAGINATOR_PER_PAGE)
        ]
    ));
}

And then add the pagination stuff to the view:

<div>There are {{ comments|length }} comments.</div>
{% for comment in comments %}
    {% if comment.photofilename %}
        <img src="{{ asset('uploads/photos/' ~ comment.photofilename) }}"/>
    {% endif %}

    <h4>{{ comment.author }}</h4>
    <small>
        {{ comment.createdAt|format_datetime('medium', 'short') }}
    </small>

    <p>{{ comment.text }}</p>
{% endfor %}

{% if previous >= 0 %}
    <a href="{{ path('conference_show', { id: conference.id, offset: previous }) }}">Previous</a>
{% endif %}
{% if next < comments|length %}
    <a href="{{ path('conference_show', { id: conference.id, offset: next }) }}">Next</a>
{% endif %}

The only change I made to this was that the suggested name for the pagination method was getCommentPaginator, but given it was in the CommentRepository, I thought repeating "comment" there was tautological, so changed it to getPaginator.

Oh, the page now looks like this:

Cool.

One thing to note here is that in refactoring my tests of the ConferenceController show method to accommodate the pagination, I pushed the coupling of the test class over the top of what PHPMD was happy with (the threshold is the default 13 classes), so I refactored that into three classes: Test, Assertions and Dependencies.

tests
`-- php
    `-- Functional
        `-- Controller
            `-- ConferenceController
                |-- Index
                |   `-- Test.php
                `-- Show
                    |-- Assertions.php
                    |-- Dependencies.php
                    `-- Test.php

I think this is all caused by testing code in a controller; having to mess around with Requests and Environments and having to test for rendered mark-up instead of just values-in and value-out adds "unnecessary" overhead. I am realy irked (in a laughing way) by what I just read in the next section which is the refactoring bit…

You might have noticed that both methods in ConferenceController take a Twig environment as an argument. Instead of injecting it into each method, let's leverage the render() helper method provided by the parent class

Symfony the Fast Track › Building the User Interface › Refactoring the Controller

Oh FFS. Ha. OK. Fine. Tell me that now.

So I can simplify things a bit:

public function show(
    Request $request,
    Environment $twig,
    Conference $conference,
    CommentRepository $commentRepository
): Response {
    $offset = max(0, $request->query->getInt('offset', 0));
    $paginator = $commentRepository->getPaginator($conference, $offset);

    return new Response($twig->render('conference/index.html.twig', [
    return $this->render(
        "conference/show.html.twig",
        [
            "conference" => $conference,
            "comments" => $paginator,
            'previous' => $offset - CommentRepository::PAGINATOR_PER_PAGE,
            'next' => min(count($paginator), $offset + CommentRepository::PAGINATOR_PER_PAGE)
        ]
    );
    ]));
}

That's good, but I really don't know why they didn't lead with that. There was nothing that relied on the Twig Environment in those methods, so why pass it in? Just to give an extra paragraph at the end?

Oh I also had to tweak a couple of the tests that broke because of this: for that render method to be available, I needed to initialise the controller class with the DI container. This was easy though:

$controller = new ConferenceController();
$container = self::getContainer();
$controller->setContainer($container);

End of section / article

Man. I only got through one page of the docs with this article, and it also took me four bites to write, even though it didn't really cover much ground. I think the issue is that a) I lack self-discipline; b) I'm not finding this stuff particularly compelling. I also got deeply frustrated with the amount of dicking around I had to do to try to convince Symfony to let me at its dependencies when I was trying to test stuff, and also the waste of time that was getting Encore working, because - as I said earlier - client-side libs are nothing to do with a server-side MVC framework, and should not get in the way. The more I use Symfony, the more I miss Silex.

I'm not looking forward to the re-drafting & read-throughs ofthis before I press "send", as I suspect it's a bit of a half-hearted mess. But I've come this far, so I'm buggered if I'm not publishing it. Well done if you read this far.

Righto.

--
Adam

Monday 17 April 2023

CFML: Into the Box 2023

Howdy partner:

Well here's something interesting (well: to me, anyhow). I will be attending the Into the Box conference in Houston, TX in May (17th - 19th). This has come about due to a very kind gesture from Luis / Ortus Solutions. Thank-you so much for this!

I know nothing about Houston - I have been through the airport on the way to somewhere else a coupla times, and that's it - so am getting there a coupla days early to have a look around and make sure the local craft beer bars are patronised.

The schedule of Into the Box itself looks pretty good too, and cover a lot of bases other than just the CFML / Boxiverse side of things. I see "testing" and "TDD" mentioned a coupla times in the schedule, so will be in my element.

The best thing about conferences though is to get to put faces to names, and meet people properly. I might be a surly anti-social type, but I really value the friendships I have built from meeting people at industry conferences, and having an agendaless chat with them over a beer or a watercooler, and got to know them better. I do hope to catch up with a bunch of people from the community who I know only as words appearing in a chat app / forum / etc. And hopefully re-meet some familiar faces too.

And the proof is in the pudding on this "conference socialising" thing: after saying farewell to Houston on the Sunday after the conference, I'm heading over to North Carolina for a few days to hang out with Dan & Jennette Skaggs (and family). I first met D & J @ CF.Obective() (or maybe Dev.Objective()?) many years ago, and we've become really good friends since then. And having had them come visit me on my side of the pond a coupla times; it's my turn to come see their part of the world. I can't wait.

So come and say "G'day" if you see me around the conference, and let's talk about... whatevs. And maybe let's go for a beer :-D

Y'all come back now, hear? (*)

--
Adam


(*) yeah, all right, enough of that.

Tuesday 11 April 2023

Getting Windows Terminal to open my Ubuntu Bash session in the right directory

G'day:

This is a follow on from the previous two articles:

All that is working well, but I had a wee problem with switching my Windows Terminal config from starting Git Bash shells to do the equivalent with directly running a Bash shell on my WSL Ubuntu filesystem.

For Git Bash, I have profiles set up like this:

{
    "commandline": "C:\\apps\\Git\\git-bash.exe",
    "guid": "{be9a184b-1c89-4ac5-88da-3ef93cd5ec98}",
    "hidden": false,
    "icon": "C:\\apps\\Git\\mingw64\\share\\git\\git-for-windows.ico",
    "name": "Git Bash (SymfonyFastTrack)",
    "startingDirectory": "\\\\wsl$\\Ubuntu\\home\\adam\\src\SymfonyFastTrack",
    "tabTitle": "Git Bash (SymfonyFastTrack)"
}

And when I open a terminal with that profile I get:

adam@DESKTOP MINGW64 //wsl$/ubuntu/home/adam/src/SymfonyFastTrack (main)
$

I figured it would be easy with an Ubuntu one, I'd copy the default one:

{
    "guid": "{2c4de342-38b7-51cf-b940-2309a097f518}",
    "hidden": false,
    "name": "Ubuntu",
    "source": "Windows.Terminal.Wsl"
}

Change its GUID, give it a different name and a startingDirectory value and done. But no: one cannot have more than one profile for Windows.Terminal.Wsl it would seem: It didn't show up (it also didn't error, which Windows Terminal is pretty good at when it doesn't like something).

So now I have two issues: getting a second Ubuntu Bash shell profile working at all, and then: point it to the correct starting directory.

The first thing I found out is that in the current version of Windows Terminal, the startingDirectory is not honoured for Ubuntu anyhow. So that's a non-starter. I did a lot of googling, and that turned up nothing. Then I turned to a more powerful search engine: I Mingo-ed it. He didn't quite nail the solution, but he got me onto the right track: asking ChatGPT. I dunno why I didn't start there. After a bit of back and forth, ChatGPT and I came up with this:

{
    "commandline": "wsl.exe -d Ubuntu /bin/bash --rcfile <(echo \"source ~/.bashrc; cd ~/src/SymfonyFastTrack\")",
    "guid": "{3933fa46-657f-4db9-ad6a-2bee51554bc5}",
    "icon": "C:\\Users\\camer\\AppData\\Local\\wt\\bash.png",
    "name": "Bash (SymfonyFastTrack)",
    "tabTitle": "Bash (SymfonyFastTrack)"
}

Explanation:

  • wsl.exe -d Ubuntu is the long form of what I tried before with source:Windows.Terminal.Wsl; name: Ubuntu.
  • /bin/bash says to run Bash.
  • --rcfile says "using this RC file (eg: instead of .bashrc.
  • <(echo [etc]): instead of using an actual file, take it from stdout.
  • source ~/.bashrc: first my actual .bashrc.
  • cd ~/src/SymfonyFastTrack\: but then switch to this directory.

And when I use this profile, I get what I want:

adam@DESKTOP //wsl$/ubuntu/home/adam/src/SymfonyFastTrack (main) $

Cool. All done.

This was another article which is largely me fumbling around being a n00b, but we all start that way with things I guess, so maybe this will short-circuit all the goolging, mingoing and ChatGPTing I had to do to arrive at the solution.

And - seriously Mingo - cheers for helping with this.

Righto.

--
Adam

Sunday 9 April 2023

Changing my WSL Bash prompt to include my current Git branch

G'day:

Note: there's no real original research / thinking in this. it's just the result of me googling stuff, and arriving at a result. I'm writing it mostly so I have a record of it.

Yesterday I shifted my source code directories from my Windows file system (eg: C:\src\myapp) to using the WSL file system (eg: ~/src/myapp). See Changing where I home my source code dramatically speeds up my Windows / WSL2 / Docker environment.

Up until now I'd been doing my dev from C:, my Docker stuff in WSL, and my Git carry-on from Git Bash. Given my files are in the WSL file system now, and I have Git installed in there too, I figured I might as well ditch Git Bash and use Git directly via Bash.

The only thing I needed to do to shift to this work pattern is to update my shell prompt to reflect what branch I'm currently in, if it's a version-controlled directory.

EG, in Bash I see this:

adam@DESKTOP MINGW64 //wsl$/ubuntu/home/adam/src/SymfonyFastTrack (main)
$

But in Bash I just see this:

adam@DESKTOP //wsl$/ubuntu/home/adam/src/SymfonyFastTrack $

I don't need the MINGW64 in there (I don't even know what it means, nor have I ever had to care. Also: no need to pipe up and tell me), but I do need the branch to be shown the current branch I'm on if I'm in a source-controlled directory, like how Git Bash does it.

I googled about how to change the Bash prompt, and found a few helpful notes. Links to those @ the bottom of this article. The end result is setting my prompt thus:

PS1='\[\e[32m\]\u@\h\[\e[39m\]:\[\e[94m\]\w \[\e[36m\]$(__git_ps1 "(%s)")\[\e[39m\]\$ '

This renders as:

adam@DESKTOP //wsl$/ubuntu/home/adam/src/SymfonyFastTrack (main) $

Explanation:

  • I got almost all my guidance for this from How to show current git branch with colors in Bash prompt.
  • This goes in my ~/.bashrc file. I slung it in at the bottom.
  • PS1 is the variable containing Bash's primary prompt formatting. There's also PS0 -> PS4, but we don't need to worry about those.
  • \[\e[32m\] is an escape sequence that sets the colour for all characters thenceforth. The colour codes I'm using are:
    32
    Green - for the user / computer name
    39
    Default
    94
    Light blue - for the path
    36
    Cyan - for the branch
  • \u@\h is the Bash escape sequence for the current user and the short host name, separated by a literal @.
  • \w is the escape sequence for the current path.
  • $(__git_ps1 "(%s)") is the Git bit. __git_ps1 is a function that returns the branch, basically. How it does this is explained in those docs.

And that's it. As I said, none of this info is my own work, it's all from the references listed below. Cheers to the bods behind that lot.

Righto.

--
Adam


References:

Saturday 8 April 2023

Changing where I home my source code dramatically speeds up my Windows / WSL2 / Docker environment

G'day:

This is more an admission of "not initially thinking things through" on my part, but the outcome has helped me a lot, so in case there are others out there who don't think things through, maybe this will be helpful to them as well.

Or people can just point and laugh at me for being so thick.

Either way, perhaps someone will get something out of this.

My dev environment is Windows (nono, that's not the "not thinking things through" bit, just behave please). All my applications run in Docker containers, and the way I get the code into the container during dev is via a volume from my file system. For example this snippet from one of my docker-compose.yml files:

version: "3"
services:

    # ...

    php:
        build:
            context: php
            dockerfile: Dockerfile

        env_file:
            - envVars.public
            - envVars.private

        stdin_open: true
        tty: true

        volumes:
            - ..:/var/www

I'm just using a volume there to mount my app directory as /var/www in the container.

So the source code for the app is in - say - C:\src\myApp.

When I'm building and starting my containers, I drop into a shell in WSL, navigate to /mnt/c/src/myApp/docker, and do the docker compose up from there.

On Windows 10 and with older versions of WSL2 and Docker, this worked reasonably well. The app was a bit slow, but only as much as a shrug seemed to be a reasonable reaction to it. It's only dev.

When I migrated to Windows 11 things slowed down a chunk more, and it's been getting progressively worse. I've been working on a Symfony app recently, and clearing its cache is taking about 3-4min. Clearly this is ballocks cos it's PHP and nothing is measured in minutes with PHP.

Also my rig was comparatively slower than the other bods in my team. For me the unit tests in our CFML project have gone from taking - about a year ago - 5min to run (already not great) to about 10min now. Obviously a lot of this is that the tests we inherited were not great (almost all hit the DB), and we've also been adding a lot more tests in that intervening year. Recently though I found out that for other teams members it was slow, but they were only meaning like 3min was slow. Oh I wish they only took me 3min to run.

Clearly something is wrong on this machine. It's 4yrs old, but it was reasonably high spec when I bought it, and its drive is an SSD. So: no excuses there. And it's not like I'm Bitcoin mining; I'm just doing file system operations.

Whatever it is: I need to fix it.

I concluded it was something to do with misconfiguration of Docker or WSL making file operations from my host machine being dog slow when run from the container. I googled around a bit and it seems a lot of other people have had similar problems; but various settings, registry hacks, and even disabling Windows Defender (not a viable solution long-term, but something to try) were not helping.

Then someone mentioned "when the files are in the native part of the WSL file system, not the /mnt/c partition, then the overhead of the WSL->Windows file system processing doesn't occur". Their solution was to develop the code locally, then automatically deploy it via SSH into the container.

At the same time, I read that whilst there is the /mnt/c mount inside WSL, there is also the reverse: \\wsl.localhost points to the WSL file system, specifically for me \\wsl.localhost\Ubuntu is the filesystem for the Ubuntu distro I am using.

Putting two and two together to see how close to four I could get it, I did this:

  • Got rid of my code from C: drive.
  • Instead: I checked-out my code within WSL into ~/src/myApp.
  • Ran all my docker stuff from there, in ~/src/myApp/docker.
  • In VSCode and IntelliJ, homed my projects in \\wsl.localhost\Ubuntu\home\adam\src\myApp.

When I run those tests that before took >10min to run, now they take around 50sec. That is more than an order of magnitude faster.

In my Symfony project the cache-clear now takes a few seconds. And the tests there run in a second or so too.

I realise I am perhaps inheriting some slowness in reverse by accessing \\wsl.localhost\Ubuntu from Windows, but I am only dealing with occasional file edits and such like. Speed there is not a problem. Not one I could perceive anyhow.

I wish I had sat down to sort this out a few months back now. I had aimlessly googled in the past for 10min or so trying to find an easy silver bullet, but never found it and each time I looked I saw the same stuff. Today I rolled up my sleeves and said "right, I'm fixing this", and after about an extra 45min of googling and trying stuff (and then backing-out each thing that didn't work again), I landed on the solution.

Righto.

--
Adam

Friday 7 April 2023

PHP / Symfony: working through "Symfony: The Fast Track", part 4: not really about Symfony, this one

G'day:

Once again I'm gonna continue working through "Symfony: The Fast Track". This will be part four of this series, after the first three:

The page I'm starting on is Setting up an Admin Backend, which sounds very situation-specific to me (so: not very portable knowledge to acquire), but I guess I'll learn "Symfony's way of ~". And there might be some useful stuff in there. Let's see.


Setting up an Admin Backend

Installing more dependencies

First they describe some Symfony concepts: Symfony Components: low-level stuff like routing, HTTP etc; and Symfony Bundles: higher-level stuff like wrappers for third-party libs. Fair enough.

Hrm. More Symfony opinions now. There's a "feature" of Symfony "aliases". Their wording:

Aliases are shortcuts for popular Composer packages. Want an ORM for your application? Require orm. Want to develop an API? Require api. These aliases are automatically resolved to one or more regular Composer packages. They are opinionated choices made by the Symfony core team.
Symfony: The Fast Track › Setting up an Admin Backend › Installing more Dependencies

Righto then.

This is a bit pathetic:

Another neat feature is that you can always omit the symfony vendor. Require cache instead of symfony/cache.
ibid

Mate. Yer bragging about omitting seven keystrokes. Well: other than the fact I've got to type in symfony composer rather than just composer. How was this even a good use of anyone's dev time, let alone brag about it being not only a feature, but a "neat" one? Sigh.

They're getting me to do this: symfony composer req "admin:^4", which installs easycorp/easyadmin-bundle, and adds a reference to it in config/bundles.php,and adds config/packages/uid.yaml. The latter looks intriguing, I guess I'll find about about it later.


Configuring EasyAdmin

Next up I'm configuring this EasyAdmin thing:

root:/var/www# symfony console make:admin:dashboard

 Which class name do you prefer for your Dashboard controller? [DashboardController]:
 >

 In which directory of your project do you want to generate "DashboardController"? [src/Controller/Admin/]:
 >

[OK] Your dashboard class has been successfully generated.
Next steps: * Configure your Dashboard at "src/Controller/Admin/DashboardController.php" * Run "make:admin:crud" to generate CRUD controllers and link them from the Dashboard. root:/var/www#

This has added a controller with some boilerplate (I'll spare you, but I'll link through to it in source control once it's there). There's a slight bug in the generation, in that it has not picked up my app's PSR-4 namespace, so it's written out this:

namespace App\Controller\Admin;
Which I will change to be:
namespace adamcameron\symfonythefasttrack\Controller\Admin;

I kinda think this should have looked at the maker.yaml config file that Symfony itself suggested I create to set the root_namespace (see It looks like your app may be using a namespace other than "App" in the second article of this series). Ah well: never mind: it's an easy fix.

It's also put an annotation in to handle the routing which I am not gonna run with, I'll stick it in the routing config where it belongs.

And because I'm changing something here, I'm gonna make a test for it. I should really have done this before running the wizard I guess. I didn't think about it. I also didn't know what the wizard was gonna do at the time, however I could have read ahead. Anyways, this has configured a /admin/ end point, so I will make sure it returns a 200:

namespace adamcameron\symfonythefasttrack\tests\Acceptance\Controller;

use Symfony\Bundle\FrameworkBundle\Test\WebTestCase;
use Symfony\Component\HttpFoundation\Response;

/** @testdox Tests the endpoints in the DashboardController */
class DashboardControllerTest extends WebTestCase
{
    /** @testdox The index endpoint returns a 200 response */
    public function testIndex()
    {
        $client = static::createClient();
        $client->request('GET', '/admin/');

        $this->assertEquals(Response::HTTP_OK, $client->getResponse()->getStatusCode());
    }
}

(This changes in the next step to be a 302, but I adjusted the test accordingly behind the scenes).

I've added a new test suite too, for acceptance tests (hitting the "front" of pages, and verifying they do whatever they are supposed to. For now, 200ing will be fine).

This test still passes after I reconfigure the routing. And I also get a page rendering in the browser:

Next I'm building a CRUD UI for the entities I've created. This is done via symfony console make:admin:crud.I'll spare you the detail unless there's something interesting.

[…]

Nope, nothing interesting except it asked me what namespace to use this time, so I was able to give it the correct one. It defaulted to Symfony's default App one though.

The process has created two skeleton controller classes. They don't have any routing in them, which is "interesting". The next step is wiring them into that initial dashbaord page I created, so let's see - just in the browser - what happens when I've done that.

public function configureMenuItems(): iterable
{
    yield MenuItem::linkToDashboard('Dashboard', 'fa fa-home');
    yield MenuItem::linktoRoute('Back to the website', 'fas fa-home', 'homepage');
    yield MenuItem::linkToCrud('Conferences', 'fas fa-map-marker-alt', Conference::class);
    yield MenuItem::linkToCrud('Comments', 'fas fa-comments', Comment::class);
}

A controller should only be busying itself with marrying-up which model should provide the data for the response, and which view to use to render the data. Here it is defining the data too. I mean this is a third-party package it's demonstrating here, so this approach is not strictly Symfony's doing; but Symfony is choosing to use this package, so I don't think it's great that official Symfony guidance should be encouraging poor practice like this.

It's also got me to reconfigure the index route handler method to redirect to the CRUD UI for conferences:

public function index(): Response
{
    $routeBuilder = $this->container->get(AdminUrlGenerator::class);
    $url = $routeBuilder->setController(ConferenceCrudController::class)->generateUrl();

    return $this->redirect($url);
}

When I reloaded the index page, I got a 500 error: the conference table was missing. Ah. I have rebuilt the containers since the last article, and I blew away the volume the DB data was in. I need to rerun the migration to get those tables back.


"Doctrine: know your limits!"(*)

Oh for goodness sake.

I just re-ran the migration to recreate the Conference and Comment tables, and I noticed... Symfony blew away the rest of the tables in the database (OK, granted, there was only one other table). WTH, Symfony? Then I looked at the migration file:

public function up(Schema $schema): void
{
    // this up() migration is auto-generated, please modify it to your needs
   // [snip for brevity]
    $this->addSql('DROP TRIGGER IF EXISTS notify_trigger ON messenger_messages;');
    $this->addSql('CREATE TRIGGER notify_trigger AFTER INSERT OR UPDATE [etc]');
    $this->addSql('ALTER TABLE comment ADD CONSTRAINT FK_9474526C604B8382 FOREIGN KEY [etc]');
    $this->addSql('DROP TABLE test');
}

You what, son?. Who the hell told you to do that? (/me hastily checks the make:migration process to confirm I didn't just go "yeah yeah yeah, delete the other tables. Cos like of course I want you to do that". No, I had not).

I googled about the place, and found this: Doctrine › Migrations › Generating Migrations › Ignoring Custom Tables:

If you have custom tables which are not managed by Doctrine you will need to tell Doctrine to ignore these tables. Otherwise, everytime you run the diff command, Doctrine will try to drop those tables. You can configure Doctrine with a schema filter.

What? Just: no. Doctrine, I have said "map these entities". Your job is to - let's see if you follow my thinking here - map these entities. Nothing else. Leave the rest of the frickin DB alone. It's not your business. Jesus.

At least they go on to say:

If you use the DoctrineBundle with Symfony you can set the schema_filter option in your configuration.

And over on the Symfony side of things: Symfony › Bundles › DoctrineMigrationsBundle › Manual Tables:

If you follow a specific scheme you can configure doctrine/dbal to ignore those tables. Let's say all custom tables will be prefixed by t_. In this case you just have have to add the following configuration option to your doctrine configuration:

doctrine:
    dbal:
        schema_filter: ~^(?!t_)~

Note that if you have multiple connections configured then the schema_filter configuration will need to be placed per-connection.

OK, two things.

  • What if you're not a lunatic from the 1990s and don't put hungarian notation on the beginning of things, so there isn't a prefix to match all the rest of the tables in your database?
  • Why the hell are you tightly coupling this to connection config? If I don't want Doctrine to delete my shit, then that's not gonna be - by default - connection specific. It's gonna be a blanket "don't delete my shit, you weirdo!?" across the board. I can see how maybe it could be overridden on a connection-by-connection basis (maybe?), but this should be a top-level Doctrine config thing (and "don't do it" being the default behaviour).

Ugh. But OK, I've added this to my connection:

doctrine:
    dbal:
        default_connection: default
        connections:
            default:
                wrapper_class: Doctrine\DBAL\Connections\PrimaryReadReplicaConnection
                dbname: '%env(resolve:POSTGRES_PRIMARY_DB)%'
                host: '%env(resolve:POSTGRES_PRIMARY_HOST)%'
                port: '%env(resolve:POSTGRES_PRIMARY_PORT)%'
                user: '%env(resolve:POSTGRES_PRIMARY_USER)%'
                password: '%env(resolve:POSTGRES_PRIMARY_PASSWORD)%'
                driver: pdo_pgsql
                server_version: 15
                charset: utf8
                schema_filter: ~^(?!test)~
                replicas:
                    replica1:
                        dbname: '%env(resolve:POSTGRES_REPLICA_DB)%'
                        host: '%env(resolve:POSTGRES_REPLICA_HOST)%'
                        port: '%env(resolve:POSTGRES_REPLICA_PORT)%'
                        user: '%env(resolve:POSTGRES_REPLICA_USER)%'
                        password: '%env(resolve:POSTGRES_REPLICA_PASSWORD)%'
                        charset: utf8

I guess I'm lucky I only have the one table to exclude, or that "pattern" could get quite weighty, quite quickly.

As an aside: I found out that when running symfony console make:migration, it'll try to run all the files in the migrations directory. As each has its own time-stamp-unique file name, I probably only want to keep the most recent one in source control? Or at least only one in that directory at any given time, anyhow. There might be a way of telling it to only run one migration. I should look into that.

Anyway: I have all my tables back in the DB now, so the page renders an empty Conference CRUD page:

And it all works fine once I allow the /admin/ route to process POST requests instead of just GETs (this is on me: I didn't realise I'd be needing to allow POSTs when I set up the routing).

I had less luck initially adding in comments, as the steps are slightly out of order in the book. I got onto this fix via googling "symfony the fast track comment UI broken", and landed on the GitHub issue to get it fixed: [Book] Step 9 Issue: Cannot create new Comment. So: in case you are following along here: after the step of testing/trying out the "Add conference" functionality, don't continue to add a comment; skip ahead and do the "Customizing EasyAdmin" bit first. It's just the next step.

This step adds code to configure how the form fields should work:

class CommentCrudController extends AbstractCrudController
{
    public static function getEntityFqcn(): string
    {
        return Comment::class;
    }

    public function configureCrud(Crud $crud): Crud
    {
        return $crud
            ->setEntityLabelInSingular('Conference Comment')
            ->setEntityLabelInPlural('Conference Comments')
            ->setSearchFields(['author', 'text', 'email'])
            ->setDefaultSort(['createdAt' => 'DESC'])
        ;
    }

    public function configureFilters(Filters $filters): Filters
    {
            return $filters
                    ->add(EntityFilter::new('conference'))
            ;
    }

    public function configureFields(string $pageName): iterable
    {
        yield AssociationField::new('conference');
        yield TextField::new('author');
        yield EmailField::new('email');
        yield TextareaField::new('text')
                ->hideOnIndex()
            ;
        yield TextField::new('photoFilename')
                ->onlyOnIndex()
            ;

        $createdAt = DateTimeField::new('createdAt')->setFormTypeOptions([
                'html5' => true,
                'years' => range(date('Y'), ((int)date('Y')) + 5),
                'widget' => 'single_text',
            ]);
        if (Crud::PAGE_EDIT === $pageName) {
            yield $createdAt->setFormTypeOption('disabled', true);
        } else {
            yield $createdAt;
        }
    }
}

(Again: directly in the controller. Bleah).


That's it for that page. It didn't really have much to do with Symfony though, did it? It was all about this third-party admin app that I have no interest in whatsoever. This is no slight on EasyAdmin, it looks slick. But this is a Symfony book, not an EasyAdmin book.

I'm a bit annoyed at my experiences working through that page, so I'm leaving off for now.

There not much code that I would be willing to put my name next to in this effort, but I'll link to it anyhow: 1.8.

The next part is here: PHP / Symfony: working through "Symfony: The Fast Track", part 5: Twig stuff, and irritation.

Righto.

--
Adam

(*) [cough]