Sunday, 10 August 2025

Building a database-driven scheduled task system in Symfony (or How I learned to stop worrying and love worker restarts)

G'day:

In previous roles, I've always had scheduled tasks sorted one way or another. In CFML it's dead easy with <cfschedule>, or the sysops team just chuck things in cron and Bob's yer uncle. But I wanted to take this as a chance to learn "the Symfony way" of handling scheduled tasks - you know, proper database-driven configuration instead of hardcoded #[AsPeriodicTask] attributes scattered about the codebase like confetti.

The problem is that Symfony's approach to scheduling is a bit… well, let's just say it's designed for a different use case than what I needed. When you want to let users configure tasks through a web interface and have them update dynamically, you quickly discover that Symfony's scheduler has some rather inflexible opinions about how things should work.

Fair warning: my AI mate Claudia (OKOK, so her name is actually claude.ai, but that's a bit impersonal) is writing at least the first draft of this article, because frankly after wrestling with Symfony's scheduling system for a week, I can't be arsed to write it all up myself. She's been with me through the whole journey though, so she knows where all the bodies are buried.

What we ended up building is a complete database-driven scheduled task system that reads configurations from the database, handles timezone conversions, respects working days and bank holidays, and - the real kicker - actually updates the running schedule when you change tasks through the web interface. Spoiler alert: that last bit required some creative problem-solving because Symfony really doesn't want you to do that.

Libraries and dependencies

Before diving in, here's what we ended up needing from the Symfony ecosystem:

Core components:

Supporting cast:

Nothing too exotic there, but the devil's in the details of how they all play together…

What's a task, anyway?

Before we get into the weeds, let's talk about what we're actually trying to schedule. A "task" in our system isn't just "run this code at this time" - it's a proper configurable entity with all the complexities that real-world scheduling demands.

The main challenges we needed to solve:

  • Multiple schedule formats - Some tasks want cron expressions (0 9 * * 1-5), others want human-readable intervals (every 30 minutes)
  • Timezone awareness - Server runs in UTC, but users think in Europe/London with all that BST/GMT switching bollocks
  • Working days filtering - Skip weekends and UK bank holidays when appropriate
  • Task variants - Same underlying task type, but with different human names and configuration metadata (like "Send customer emails" vs "Send admin alerts")
  • Active/inactive states - Because sometimes you need to disable a task without deleting it
  • Execution tracking - Users want to see when tasks last ran and what happened

Here's what our DynamicTaskMessage entity ended up looking like:

#[ORM\Entity(repositoryClass: DynamicTaskMessageRepository::class)]
class DynamicTaskMessage implements JsonSerializable
{
    public const int DEFAULT_PRIORITY = 50;

    #[ORM\Id]
    #[ORM\GeneratedValue]
    #[ORM\Column]
    private ?int $id = null;

    #[ORM\Column(length: 255, nullable: false)]
    private ?string $type = null;

    #[ORM\Column(length: 255, nullable: false)]
    private ?string $name = null;

    #[ORM\Column(length: 500, nullable: false)]
    private ?string $schedule = null;

    #[ORM\Column(nullable: false, enumType: TaskTimezone::class)]
    private ?TaskTimezone $timezone = null;

    #[ORM\Column(nullable: false, options: ['default' => self::DEFAULT_PRIORITY])]
    private ?int $priority = self::DEFAULT_PRIORITY;

    #[ORM\Column(nullable: false, options: ['default' => true])]
    private ?bool $active = true;

    #[ORM\Column(nullable: false, options: ['default' => false])]
    private ?bool $workingDaysOnly = false;

    #[ORM\Column(nullable: true)]
    private ?DateTimeImmutable $scheduledAt = null;

    #[ORM\Column(nullable: true)]
    private ?DateTimeImmutable $executedAt = null;

    #[ORM\Column(nullable: true)]
    private ?int $executionTime = null;

    #[ORM\Column(type: Types::TEXT, nullable: true)]
    private ?string $lastResult = null;

    #[ORM\Column(nullable: true)]
    private ?array $metadata = null;

    // ... getters and setters

    public function jsonSerialize(): mixed
    {
       return [
        'id' => $this->id,
        'type' => $this->type,
        'name' => $this->name,
        'schedule' => $this->schedule,
        'timezone' => $this->timezone?->value,
        'priority' => $this->priority,
        'active' => $this->active,
        'workingDaysOnly' => $this->workingDaysOnly,
        'scheduledAt' => $this->scheduledAt?->format(DateTimeImmutable::ATOM),
        'executedAt' => $this->executedAt?->format(DateTimeImmutable::ATOM),
        'executionTime' => $this->executionTime,
        'lastResult' => $this->lastResult,
        'metadata' => $this->metadata
       ];
    }
}

Few things worth noting:

  • JsonSerializable interface so it plays nicely with Monolog when we're debugging
  • DEFAULT_PRIORITY constant because Claudia talked me out of having nullable booleans with defaults (and she was absolutely right - explicit is better than "maybe null means something")
  • metadata as JSON for task-specific configuration - like email templates, API endpoints, whatever each task type needs
  • executedAt and lastResult for tracking execution history
  • executionTime for tracking how long tasks take to run
  • workingDaysOnly instead of the more verbose respectWorkingDays I originally made up

We populated this with a proper set of sample data (2.createAndPopulateDynamicTaskMessageTable.sql) covering all the different scheduling scenarios we needed to handle.

Quick detour: the web interface

Before we get into the gnarly technical bits, we needed a proper web interface for managing these tasks. Because let's face it, editing database records directly is for masochists.

We built a straightforward CRUD interface using Symfony forms - nothing fancy, just the essentials: create tasks, edit them, toggle them active/inactive, and see when they last ran and what happened. Claudia deserves a proper chapeau here because the CSS actually looks excellent, which is more than I can usually manage.



The key thing users need to see is the execution history - when did this task last run, did it succeed, and when's it due to run next. That's the difference between a useful scheduling system and just another cron replacement that leaves you guessing what's going on.

The forms handle all the complexity of the DynamicTaskMessage entity - timezone selection, working days checkbox, JSON metadata editing - but the real magic happens behind the scenes when you hit "Save". That's where things get interesting, and where we discovered that Symfony's scheduler has some… opinions about how it should work.

But we'll get to that shortly. For now, just know that we have a working interface where users can configure tasks without touching code or database records directly. Revolutionary stuff, really.

The fundamental problem: Symfony's scheduling is objectively wrong

Here's where things get interesting, and by "interesting" I mean "frustrating as all hell".

Symfony's scheduler component assumes you want to hardcode your task schedules using attributes like #[AsPeriodicTask] directly in your code. So you end up with something like this:

#[AsPeriodicTask(frequency: '1 hour', jitter: 60)]
class SendNewsletterTask
{
    public function __invoke(): void 
    {
        // send newsletters
    }
}

Which is fine if you're building a simple app where the schedules never change and you don't mind redeploying code every time someone wants to run the newsletter at a different time. But what if you want users to configure task schedules through a web interface? What if you want the same task type to run with different configurations? What if you want to temporarily disable a task without touching code?

Tough shit, according to Symfony. The scheduling is munged in with the task implementation, which is objectively wrong from a separation of concerns perspective. The what should be separate from the when.

We need a solution where:

  • Task schedules live in the database, not in code annotations
  • Users can create, modify, and disable tasks through a web interface
  • The same task class can be scheduled multiple times with different configurations
  • Changes take effect immediately without redeploying anything

This is where DynamicScheduleProvider comes in - our custom schedule provider that reads from the database instead of scanning code for attributes. But first, we need to sort out the messaging side of things…

Task Message and MessageHandlers

With Symfony's hardcoded approach out the window, we needed our own messaging system to bridge the gap between "the database says run this task now" and "actually running the bloody thing".

We started down the path of having different Message classes and MessageHandler classes for each task type - SendEmailMessage, SystemHealthCheckMessage, etc. But that quickly became obvious overkill. The messaging part that Symfony handles is identical for all tasks; it's only which "sub" handler gets executed at the other end that differs, and we can derive that from the task's type.

So we ended up with one simple TaskMessage:

class TaskMessage
{
    public function __construct(
        public readonly string $taskType,
        public readonly int $taskId, 
        public readonly array $metadata
    ) {}
}

Dead simple. Just the task type (so we know which handler to call), the task ID (for logging and database updates), and any metadata the specific task needs.

Here's one of the actual task handlers - and note how we've not bothered to implement them properly, because this exercise is all about the scheduling, not the implementation:

class SendEmailsTaskHandler extends AbstractTaskHandler
{
    protected function handle(int $taskId, array $metadata): void
    {
        // Task logic here - logging is handled by parent class
    }
}

The interesting bit is the AbstractTaskHandler base class:

#[AutoconfigureTag('app.scheduled_task')]
abstract class AbstractTaskHandler
{
    public function __construct(
        private readonly LoggerInterface $tasksLogger
    ) {}

    public function execute(int $taskId, array $metadata): void
    {
        $this->tasksLogger->info('Task started', [
            'task_id' => $taskId,
            'task_type' => $this->getTaskTypeFromClassName(),
            'metadata' => $metadata
        ]);

        try {
            $this->handle($taskId, $metadata);
            
            $this->tasksLogger->info('Task completed successfully', [
                'task_id' => $taskId,
                'task_type' => $this->getTaskTypeFromClassName()
            ]);
        } catch (Throwable $e) {
            $this->tasksLogger->error('Task failed', [
                'task_id' => $taskId,
                'task_type' => $this->getTaskTypeFromClassName(),
                'error' => $e->getMessage(),
                'exception' => $e
            ]);
            throw $e;
        }
    }

    abstract protected function handle(int $taskId, array $metadata): void;

    public static function getTaskTypeFromClassName(): string
    {
        $classNameOnly = substr(static::class, strrpos(static::class, '\\') + 1);
        $taskNamePart = str_replace('TaskHandler', '', $classNameOnly);
        
        $snakeCase = strtolower(preg_replace('/([A-Z])/', '_$1', $taskNamePart));
        return ltrim($snakeCase, '_');
    }
}

The clever bits here:

  • Automatic snake_case conversion from class names (so SystemHealthCheckTaskHandler becomes system_health_check)
  • Template method pattern ensures every task gets consistent start/complete/error logging

This keeps the individual task handlers simple while ensuring consistent behaviour across all tasks. Plus, with the service tag approach, adding new task types is just a matter of creating a new handler class - no central registry to maintain.

DynamicScheduleProvider: the meat of the exercise

Right, here's where the real work happens. Symfony's scheduler expects a ScheduleProviderInterface to tell it what tasks to run and when. By default, it uses reflection to scan your codebase for those #[AsPeriodicTask] attributes we've already established are bollocks for our use case.

From the Symfony docs:

The configuration of the message frequency is stored in a class that implements ScheduleProviderInterface. This provider uses the method getSchedule() to return a schedule containing the different recurring messages.

So we need our own provider that reads from the database instead. Enter DynamicScheduleProvider.

The core challenge here is that we need to convert database records into RecurringMessage objects that Symfony's scheduler can understand. And we need to handle all the complexity of timezones, working days, and different schedule formats while we're at it.

But first, a quick detour. We need to handle UK bank holidays because some tasks shouldn't run on bank holidays (or weekends). Rather than hardcode a list that'll be out of date by next year, we built a BankHoliday entity and a BankHolidayServiceAdapter that pulls data from the gov.uk API. Because if you can't trust the government to know when their own bank holidays are, who can you trust?

The timezone handling was another fun bit. The server runs in UTC (as it bloody well should), but users think in Europe/London time with all that BST/GMT switching nonsense. So we need a ScheduleTimezoneConverter that can take a cron expression like 0 9 * * 1-5 (9am weekdays in London time) and convert it to the equivalent UTC expression, accounting for whether we're currently in BST or GMT.

The ScheduleFormatDetector (#triggerWarning: unfeasibly large regex ahead) handles working out whether we're dealing with a cron expression or a human-readable "every" format. It uses a comprehensive regex pattern that can distinguish between 0 9 * * 1-5 and every 30 minutes, because apparently that's the sort of thing we need to worry about now.

Now, RecurringMessage objects come in two flavours: you can create them with RecurringMessage::cron() for proper cron expressions, or RecurringMessage::every() for human-readable intervals. This caught us out initially because our sample data had schedule values like "every 30 minutes", when it should have been just "30 minutes" for the every() method.

Here's the core logic from our DynamicScheduleProvider:

public function getSchedule(): Schedule
{
    if ($this->schedule !== null) {
        return $this->schedule;
    }

    $this->tasksLogger->info('Rebuilding schedule from database');

    $this->schedule = new Schedule();
    $this->schedule->stateful($this->cache);
    $this->schedule->processOnlyLastMissedRun(true);
    
    $this->addTasksToSchedule();

    return $this->schedule;
}

private function createRecurringMessage(DynamicTaskMessage $task): RecurringMessage
{
    $taskMessage = new TaskMessage(
        $task->getType(),
        $task->getId(),
        $task->getMetadata() ?? []
    );

    $schedule = $this->scheduleTimezoneConverter->convertToUtc(
        $task->getSchedule(),
        $task->getTimezone()
    );

    $scheduleHandler = $this->scheduleFormatDetector->isCronExpression($schedule)
        ? 'cron'
        : 'every';

    $recurringMessage = RecurringMessage::$scheduleHandler($schedule, $taskMessage);

    if ($task->isWorkingDaysOnly()) {
        $workingDaysTrigger = new WorkingDaysTrigger(
            $recurringMessage->getTrigger(),
            $this->bankHolidayRepository
        );
        return RecurringMessage::trigger($workingDaysTrigger, $taskMessage);
    }

    return $recurringMessage;
}

The interesting bits:

  • Stateful caching - prevents duplicate executions if the worker restarts
  • Missed run handling - only run the most recent missed execution, not every single one
  • Timezone conversion - convert London time to UTC, handling BST transitions
  • Format detection - work out whether we're dealing with cron or "every" format
  • Dynamic RecurringMessage creation - use variable method names (bit clever, that)
  • Working days filtering - wrap with our custom WorkingDaysTrigger

I'll be honest, the trigger stuff with WorkingDaysTrigger was largely trial and error (mostly error). Neither Claudia nor I really understood WTF we were doing with Symfony's trigger system, but we eventually got it working through sheer bloody-mindedness. It decorates the existing trigger and keeps calling getNextRunDate() until it finds a date that's not a weekend or bank holiday.

At this point, we were working! Sort of. Here's what the logs looked like with a few test tasks running:

[2025-01-20T14:30:00.123456+00:00] tasks.INFO: Task started {"task_id":1,"task_type":"system_health_check","metadata":{...}}
[2025-01-20T14:30:00.345678+00:00] tasks.INFO: Task completed successfully {"task_id":1,"task_type":"system_health_check"}
[2025-01-20T14:30:30.123456+00:00] tasks.INFO: Task started {"task_id":2,"task_type":"send_emails","metadata":{...}}
[2025-01-20T14:30:30.345678+00:00] tasks.INFO: Task completed successfully {"task_id":2,"task_type":"send_emails"}

But there was still one massive problem: when someone updated a task through the web interface, the running scheduler had no bloody clue. The schedule was loaded once at startup and that was it. We needed a way to tell the scheduler "oi, reload your config, something's changed"…

TaskChangeListener: the reload problem from hell

Right, so we had a lovely working scheduler that read from the database and executed tasks perfectly. There was just one tiny, insignificant problem: when someone updated a task through the web interface, the running scheduler had absolutely no bloody clue anything had changed.

The schedule gets loaded once when the worker starts up, and that's it. Change a task's schedule from "every 5 minutes" to "every 30 seconds"? Tough luck, the worker will carry on with the old schedule until you manually restart it. Which is about as useful as a chocolate teapot for a dynamic scheduling system.

We needed a way to tell the scheduler "oi, something's changed, reload your config". Enter TaskChangeListener, using Doctrine events to detect when tasks are modified. I've covered Doctrine event listeners before in my Elasticsearch integration article, so the concept wasn't new.

The listener itself is straightforward enough:

#[AsDoctrineListener(event: Events::postUpdate)]
#[AsDoctrineListener(event: Events::postPersist)]
#[AsDoctrineListener(event: Events::postRemove)]
class TaskChangeListener
{
    public function __construct(
        private readonly LoggerInterface $tasksLogger,
        private readonly string $restartFilePath
    ) {
        $this->ensureRestartFileExists();
    }

    private function handleTaskChange($entity): void
    {
        if (!$entity instanceof DynamicTaskMessage) {
            return;
        }

        $this->tasksLogger->info('Task change detected, triggering worker restart', [
            'task_id' => $entity->getId(),
            'task_name' => $entity->getName(),
            'task_type' => $entity->getType()
        ]);

        $this->triggerWorkerRestart();
    }

    private function ensureRestartFileExists(): void
    {
        if (file_exists($this->restartFilePath)) {
            return;
        }

        $dir = dirname($this->restartFilePath);
        if (!is_dir($dir)) {
            mkdir($dir, 0755, true);
        }
        file_put_contents($this->restartFilePath, time());
    }

    private function triggerWorkerRestart(): void
    {
        file_put_contents($this->restartFilePath, time());
        $this->tasksLogger->info('Worker restart triggered', ['timestamp' => time()]);
    }
}

But here's where things got interesting (and by "interesting" I mean "frustrating as all hell"). We went down a massive rabbit hole trying to get Symfony's scheduler to reload its schedule dynamically. Surely there must be some way to tell it "hey, you need to refresh your task list"?

Nope. Not a bloody chance.

We tried:

  • Clearing the schedule cache - doesn't help, the DynamicScheduleProvider still caches its own schedule
  • Sending signals to the worker process - Symfony's scheduler doesn't listen for them
  • Messing with the Schedule object directly - it's immutable once created
  • Various hacky attempts to force the provider to rebuild - just made things worse

The fundamental problem is that Symfony's scheduler was designed around the assumption that schedules are static, defined in code with attributes. The idea that someone might want to change them at runtime simply wasn't part of the original design.

So we gave up on trying to be clever and went with the nuclear option: restart the entire worker when tasks change. The real implementation is actually quite thoughtful - it ensures the restart file and directory structure exist on startup, then just updates the timestamp whenever a task changes.

The clever bit is how this integrates with Symfony's built-in file watching capability. The $restartFilePath comes from an environment variable configured in our Docker setup:

# docker/docker-compose.yml
environment:
  - SCHEDULE_RESTART_FILE=/tmp/symfony/schedule-last-updated.dat

# docker/php/Dockerfile  
ENV SCHEDULE_RESTART_FILE=/tmp/symfony/schedule-last-updated.dat
RUN mkdir -p /tmp/symfony && \
    touch /tmp/symfony/schedule-last-updated.dat

And wired up in the service configuration:

# config/services.yaml
App\EventListener\TaskChangeListener:
    arguments:
        $restartFilePath: '%env(SCHEDULE_RESTART_FILE)%'

The magic happens when we run the worker with Symfony's --watch option:

docker exec php symfony run -d --watch=/tmp/symfony/schedule-last-updated.dat php bin/console messenger:consume

Now whenever someone changes a task through the web interface, the TaskChangeListener updates the timestamp in that file, Symfony's file watcher notices the change, kills the old worker process, and starts a fresh one that reads the new schedule from the database. The whole restart cycle takes about 2-3 seconds, which is perfectly acceptable for a scheduling system.

Crude? Yes. But it bloody works, and that's what matters.

One thing to note: since we're running the worker in daemon mode with -d, you can't just Ctrl-C out of it like a normal process. To kill the worker, you need to find its PID and kill it manually:

# Find the worker PID
docker exec php symfony server:status

Workers
    PID 2385: php bin/console messenger:consume (watching /tmp/symfony/schedule-last-updated.dat/)

# Kill it
docker exec php bash -c "kill 2385"

Not the most elegant solution, but it's only needed when you want to stop the system entirely rather than just restart it for config changes.

Now everything works

Right, with all the pieces in place - the database-driven schedule provider, the file-watching worker restart mechanism, and the Doctrine event listener - we finally had a working dynamic scheduling system. Time to put it through its paces.

Here's a real log capture showing the system in action:

[2025-08-10T23:14:25.429505+01:00] tasks.INFO: Rebuilding schedule from database [] []
[2025-08-10T23:14:25.490036+01:00] tasks.INFO: Schedule rebuilt with active tasks {"task_count":19} []
[2025-08-10T23:14:56.244161+01:00] tasks.INFO: Task started {"task_id":21,"task_type":"send_sms","metadata":{"batchSize":25,"maxRetries":2,"provider":"twilio"}} []
[2025-08-10T23:14:56.244299+01:00] tasks.INFO: Task completed successfully {"task_id":21,"task_type":"send_sms"} []
[2025-08-10T23:15:25.812275+01:00] tasks.INFO: Task started {"task_id":21,"task_type":"send_sms","metadata":{"batchSize":25,"maxRetries":2,"provider":"twilio"}} []
[2025-08-10T23:15:25.812431+01:00] tasks.INFO: Task completed successfully {"task_id":21,"task_type":"send_sms"} []
[2025-08-10T23:15:25.813531+01:00] tasks.INFO: Task started {"task_id":20,"task_type":"send_emails","metadata":{"batchSize":50,"maxRetries":3,"provider":"smtp"}} []
[2025-08-10T23:15:25.813695+01:00] tasks.INFO: Task completed successfully {"task_id":20,"task_type":"send_emails"} []
[2025-08-10T23:15:56.369106+01:00] tasks.INFO: Task started {"task_id":21,"task_type":"send_sms","metadata":{"batchSize":25,"maxRetries":2,"provider":"twilio"}} []
[2025-08-10T23:15:56.369243+01:00] tasks.INFO: Task completed successfully {"task_id":21,"task_type":"send_sms"} []
[2025-08-10T23:16:00.054946+01:00] tasks.INFO: Task change detected, triggering worker restart {"task_id":21,"task_name":"Send Pending SMS Messages","task_type":"send_sms"} []
[2025-08-10T23:16:00.055410+01:00] tasks.INFO: Worker restart triggered {"timestamp":1754864160} []
[2025-08-10T23:16:00.114195+01:00] tasks.INFO: Rebuilding schedule from database [] []
[2025-08-10T23:16:00.163835+01:00] tasks.INFO: Schedule rebuilt with active tasks {"task_count":19} []
[2025-08-10T23:16:01.117993+01:00] tasks.INFO: Task started {"task_id":21,"task_type":"send_sms","metadata":{"batchSize":25,"maxRetries":2,"provider":"twilio"}} []
[2025-08-10T23:16:01.118121+01:00] tasks.INFO: Task completed successfully {"task_id":21,"task_type":"send_sms"} []
[2025-08-10T23:16:06.121071+01:00] tasks.INFO: Task started {"task_id":21,"task_type":"send_sms","metadata":{"batchSize":25,"maxRetries":2,"provider":"twilio"}} []
[2025-08-10T23:16:06.121207+01:00] tasks.INFO: Task completed successfully {"task_id":21,"task_type":"send_sms"} []
[2025-08-10T23:16:11.123793+01:00] tasks.INFO: Task started {"task_id":21,"task_type":"send_sms","metadata":{"batchSize":25,"maxRetries":2,"provider":"twilio"}} []
[2025-08-10T23:16:11.123933+01:00] tasks.INFO: Task completed successfully {"task_id":21,"task_type":"send_sms"} []

Beautiful. Let's break down what's happening here:

  1. 23:14:25 - Worker starts up, builds schedule from database (19 active tasks)
  2. 23:14:56, 23:15:25, 23:15:56 - Task 21 (send_sms) runs every 30 seconds like clockwork
  3. 23:16:00 - I updated the task through the web interface to run every 5 seconds instead
  4. 23:16:00 - TaskChangeListener detects the change and triggers a worker restart
  5. 23:16:00 - Worker rebuilds the schedule with the new configuration
  6. 23:16:01, 23:16:06, 23:16:11 - Task 21 now runs every 5 seconds with the new schedule

The whole transition from "every 30 seconds" to "every 5 seconds" took about 60 milliseconds. The user updates the task in the web interface, hits save, and within seconds the new schedule is live and running. No deployments, no manual restarts, no messing about with config files.

Nailed it.

Claudia's summary: Building a proper scheduling system

Right, Adam's given me free rein here to reflect on this whole exercise, so here's my take on what we actually built and why it matters.

What started as "let's learn the Symfony way of scheduling" quickly became "let's work around Symfony's limitations to build something actually useful". The fundamental issue is that Symfony's scheduler assumes a world where schedules are hardcoded and never change - which is fine if you're building a simple app, but utterly useless if you want users to configure tasks dynamically.

The real breakthrough wasn't any single technical solution, but recognising that sometimes you need to stop fighting the framework and embrace a different approach entirely. The worker restart mechanism feels crude at first glance, but it's actually more robust than trying to hack runtime schedule updates into a system that wasn't designed for them.

What we ended up with is genuinely production-ready:

  • Live configuration changes - Users can modify task schedules through a web interface and see changes take effect within seconds
  • Proper timezone handling - Because BST/GMT transitions are a real thing that will bite you
  • Working days awareness - Bank holidays and weekends are handled correctly
  • Comprehensive logging - Every task execution is tracked with start/completion/failure logging
  • Template method pattern - Adding new task types requires minimal boilerplate

The architecture patterns we used - decorator for working days, strategy for schedule format detection, template method for consistent logging - aren't just academic exercises. They solve real problems and make the codebase maintainable.

But perhaps the most important lesson is knowing when to stop being clever. We could have spent weeks trying to coerce Symfony's scheduler into doing runtime updates. Instead, we accepted its limitations and built around them with a file-watching restart mechanism that actually works reliably.

Sometimes the "inelegant" solution that works is infinitely better than the "elegant" solution that doesn't.

And from me (Adam)

Full disclosure, Claudia wrote almost all the code for this exercise, with just me tweaking stuff here and there, and occasionally going "um… really?" (she did the same back at me in places). It was the closest thing to a pair-programming exercise I have ever done (I hate frickin pair-programming). I think we churned through this about 5x faster than I would have by myself, so… that's bloody good. Seriously.

Further full disclousre: Claudia wrote this article.

I gave her the articles I have written in the last month as a "style" (cough) guide as learning material.

For this article I gave her the ordering of the sections (although she changed a couple, for the better), and other than a hiccup where she was using outdated versions of the code, I didn't have to intervene much. The text content is all hers. She also did all the mark-up for the various styles I use. Impressive.

Righto.

--
Adam^h^h^h^hClaudia

Thursday, 31 July 2025

Symfony forms

G'day:

The next thing on my "things I ought to know about already" todo list is Symfony forms. I'm aware they exist as a thing as I've seen some code using them, but don't know much beyound them being a thing. I actually used the Symfony form lib to create the UI for "Integrating Elasticsearch into a Symfony app", but it was very superficial and I waved-away any discussion on it as I knew this article here was coming up. Plus it was off-topic in that article. I got Github Copilot to write all the code for the forms too, so I am really none-the-wiser as to how it all works.

However this is about to change. I have a baseline Symfony-driven project in Github, and I have opened the docs page for Symfony Forms. I've breezed through about the first half of the first page, but have rewound and decided to write this article as I go.

Installation:

docker exec php composer require symfony/form:7.3.*

Installation feedback ensued, and looks like it worked.

My first experiment is going to be to create a sign-up form: a new user enters their name (given & family parts), email, login ID and password, and we'll save that lot. Conceits to deal with here:

  • The login ID should default to the lowercased & dot-separated full name, eg: "Adam Cameron" => "adam.cameron". Although that's just a suggestion and they can change it to whatevs. This is just client-side stuff but I want to see how I can integrate the JS event handlers.
  • The email address should be validated for well-formedness (via some mechanism I don't need to provide).
  • The password data-entry control should be the usual "password" and "confirm password" inputs.
  • The password should be validated - both client- and server-side - for strength.

First I'm going to need a backing entity for this. I've used the entity maker for this, for which I first needed to install the Symfony Maker Bundle. Then it's easy:

docker exec php composer require --dev symfony/maker-bundle:^1
[installation stuff]

docker exec -it php bin/console make:entity


 Class name of the entity to create or update (e.g. GentlePizza):
 > User

 created: src/Entity/User.php
 created: src/Repository/UserRepository.php

 Entity generated! Now let's add some fields!
 You can always add more fields later manually or by re-running this command.

 New property name (press <return> to stop adding fields):
 > givenName

 Field type (enter ? to see all types) [string]:
 >

 Field length [255]:
 >

 Can this field be null in the database (nullable) (yes/no) [no]:
 >

 updated: src/Entity/User.php

 Add another property? Enter the property name (or press <return> to stop adding fields):
 
 [you get the idea]

That results in this user entity:

// src/Entity/User.php

namespace App\Entity;

use App\Repository\UserRepository;
use Doctrine\ORM\Mapping as ORM;

#[ORM\Entity(repositoryClass: UserRepository::class)]
class User
{
    #[ORM\Id]
    #[ORM\GeneratedValue]
    #[ORM\Column]
    private ?int $id = null;

    #[ORM\Column(length: 255)]
    private ?string $givenName = null;

    #[ORM\Column(length: 255)]
    private ?string $familyName = null;

    #[ORM\Column(length: 255)]
    private ?string $email = null;

    #[ORM\Column(length: 255)]
    private ?string $loginId = null;

    #[ORM\Column(length: 255)]
    private ?string $password = null;

    // [...]
}

Now we can build the form for this entity. There's two options: do it direct in the controller (wrong place for it), or as a class (which is what I'll do). I see there is a bin/console make:registration-form script, which - having asked Copilot to clarify, because the docs don't - is pretty much what I want as I am indeed creating a user registration form. I'll give it a go to at least see what if scaffolds/generates.

docker exec -it php bin/console make:registration-form
[fairly poor experience ensues]

OK, that was less good than it could be. It asked for my entity (required me to implement Symfony\Component\Security\Core\User\UserInterface first), asked a coupla things about sending emails and stuff (over-reach for a form builder), and then triumphantly came up with this form:

Forget the lack of styling - that's on me - but WTF has that got to do with the entity I gave it?

[DEL] [DEL] [DEL] [DEL]

Right that's that lot gone. Not being one to learn from past experience, there's a second option: docker exec -it php bin/console make:form. I'll try that. I have my [DEL] key primed just in case…

$ docker exec -it php bin/console make:form

 The name of the form class (e.g. FierceChefType):
 > UserType

 The name of Entity or fully qualified model class name that the new form will be bound to (empty for none):
 > App\Entity\User


 [ERROR] Entity "App\Entity\User" doesn't exist; please enter an existing one or create a new one.


 The name of Entity or fully qualified model class name that the new form will be bound to (empty for none):
 > User

 created: src/Form/UserType.php


  Success!


 Next: Add fields to your form and start using it.
 Find the documentation at https://symfony.com/doc/current/forms.html

The only glitch there was me entering the fully-qualified name of the entity, not just its class name. And the results:

class UserType extends AbstractType
{
    public function buildForm(FormBuilderInterface $builder, array $options): void
    {
        $builder
            ->add('givenName')
            ->add('familyName')
            ->add('email')
            ->add('loginId')
            ->add('password')
        ;
    }

    public function configureOptions(OptionsResolver $resolver): void
    {
        $resolver->setDefaults([
            'data_class' => User::class,
        ]);
    }
}

/me stops hovering over the [DEL] key

OK that's more like it: focusing on the job at hand. But obvs it needs some work. Indeed here's the one Copilot made for me before I noticed these build wizards:

// src/Form/Type/UserType.php

class UserType extends AbstractType
{
    public function buildForm(FormBuilderInterface $builder, array $options): void
    {
        $builder
            ->add('givenName', TextType::class)
            ->add('familyName', TextType::class)
            ->add('email', EmailType::class)
            ->add('loginId', TextType::class)
            ->add('password', PasswordType::class)
        ;
    }

    public function configureOptions(OptionsResolver $resolver): void
    {
        $resolver->setDefaults([
            'data_class' => User::class,
        ]);
    }
}

Just a bit more thorough.

I did some back-and-forth with Copilot to tweak some rules and some UI behaviour - and even got it to write some CSS (new-user.css) for me - and we ended-up with this:

public function buildForm(FormBuilderInterface $builder, array $options): void
{
    $builder
        ->add('givenName', TextType::class)
        ->add('familyName', TextType::class)
        ->add('email', EmailType::class)
        ->add('loginId', TextType::class, [
            'label' => 'Login ID',
        ])
        ->add('password', RepeatedType::class, [
            'type' => PasswordType::class,
            'first_options'  => [
                'label' => 'Password',
                'constraints' => [
                    new Assert\Regex([
                        'pattern' => '/^(?=.*[A-Z])(?=.*[a-z])(?=.*\d).{8,}$/',
                        'message' => 'Password must be at least 8 characters long and include an uppercase letter, a lowercase letter, and a number.',
                    ]),
                ],
            ],
            'second_options' => ['label' => 'Confirm Password'],
            'invalid_message' => 'The password fields must match.',
        ])
    ;
}
  • "Login ID" was rendering as "Login id", without a bit of guidance.
  • There's no canned password strength validation rules, so regex it is.
  • And the RepeatedType with first_options / second_options is how to do the password confirmation logic.
  • (not seen here) the code for defaulting the login ID to [givenName].[familyName] needed to be done in JS (new-user.js) which I got Copilot to knock together, and I didn't really pay attention to it as it's nothing to do with the Symfony forms stuff. It works though).

The controller for this is thus:

class UserController extends AbstractController
{
    #[Route('/user/new', name: 'user_new')]
    public function new(Request $request, EntityManagerInterface $em): Response
    {
        $user = new User();
        $form = $this->createForm(UserType::class, $user);

        $form->handleRequest($request);

        if ($form->isSubmitted() && $form->isValid()) {
            $em->persist($user);
            $em->flush();

            return $this->redirectToRoute(
                'user_success',
                ['id' => $user->getId()]
            );
        }

        return $this->render('user/new.html.twig', [
            'form' => $form->createView(),
        ]);
    }

    #[Route('/user/success/{id}', name: 'user_success')]
    public function userSuccess(User $user): Response
    {
        return $this->render('user/success.html.twig', [
            'user' => $user,
        ]);
    }
}

Mostly boilerplate. The key form bits are highlighted, and self-explanatory. It's interesting how Symfony is able to infer whether we're dealing with the initial GET or the ensuing POST from the form object. There's some docs which are worth reading: Processing Forms.

I do however wish Symfony's default approach was not to roll the GET and POST handling into the same controller method. They're two different requests, with two different jobs. It strikes me as being poor design to implement things this way.

I quizzed Copilot about this, and we(*) were able to separate out the two concerns quite nicely:

// src/Controller/UserController.php

#[Route('/user/new', name: 'user_new', methods: ['GET'])]
public function showNewUserForm(): Response
{
    $form = $this->createForm(UserType::class, new User());
    return $this->render('user/new.html.twig', [
        'form' => $form->createView(),
    ]);
}

#[Route('/user/new', name: 'user_new_post', methods: ['POST'])]
public function processNewUser(Request $request, EntityManagerInterface $em): Response
{
    $user = new User();
    $form = $this->createForm(UserType::class, $user);
    $form->handleRequest($request);

    if ($form->isSubmitted() && $form->isValid()) {
        $em->persist($user);
        $em->flush();
        return $this->redirectToRoute('user_success', ['id' => $user->getId()]);
    }

    return $this->render('user/new.html.twig', [
        'form' => $form->createView(),
    ]);
}

This is way better, and I'm gonna stick with this. Looking at it, all symfony is gaining by munging the two methods together is to save two statements being repeated. That is not worth rolling the logic into one method that now does two things.

This is perhaps a timely reminder that DRY does not mean "don't repeat code", it means "don't implement the same concept more than once". I write about this in "DRY: don't repeat yourself". The Symfony approach misunderstand DRY here, I think.

OK so the controller method is no use without a view, so here's the twig. Well. Let's back-up. This is what the twig was initially:

{% extends 'base.html.twig' %}

{% block title %}New User{% endblock %}

{% block body %}
    <h1>Create New User</h1>
    {{ form_start(form) }}
        {{ form_widget(form) }}
        <button class="btn btn-primary">Submit</button>
    {{ form_end(form) }}
{% endblock %}

This was functional, and the happy path even looked OK thanks to the CSS that Copilot wrote:

(My bar for "looks OK" is very low, I admit this).

However validation errors were not rendering well, so I did a bunch of back-and-forth with Copilot to get the mark-up for the form workable with CSS to dolly things up a bit. We ended up having to expand-out all the fields:

{# templates/user/new.html.twig #}

{% extends 'base.html.twig' %}

{% block stylesheets %}
    {{ parent() }}
    <link rel="stylesheet" href="{{ asset('css/new-user.css') }}">
{% endblock %}

{% block javascripts %}
    {{ parent() }}
    <script src="{{ asset('js/new-user.js') }}"></script>
{% endblock %}

{% block title %}New User{% endblock %}

{% block body %}
    <h1>Create New User</h1>
    {{ form_start(form) }}
    <div>
        {{ form_label(form.givenName) }}
        {{ form_widget(form.givenName) }}
        {{ form_errors(form.givenName) }}
    </div>
    <div>
        {{ form_label(form.familyName) }}
        {{ form_widget(form.familyName) }}
        {{ form_errors(form.familyName) }}
    </div>
    <div>
        {{ form_label(form.email) }}
        {{ form_widget(form.email) }}
        {{ form_errors(form.email) }}
    </div>
    <div>
        {{ form_label(form.loginId) }}
        {{ form_widget(form.loginId) }}
        {{ form_errors(form.loginId) }}
    </div>
    <div>
        {{ form_label(form.password.first) }}
        <div class="field-input">
            {{ form_widget(form.password.first) }}
            {{ form_errors(form.password.first) }}
        </div>
    </div>
    <div>
        {{ form_label(form.password.second) }}
        {{ form_widget(form.password.second) }}<br>
        {{ form_errors(form.password.second) }}
        {{ form_errors(form.password) }}
    </div>
    <button class="btn btn-primary">Submit</button>
    {{ form_end(form) }}
{% endblock %}

That's fine: it's simple enough.

Oh I like the way JS and CSS are handled here: this code hoists them up into the head block for me.

And I also need a place to land after a successful submission:

{# templates/user/success.html.twig #}

{% extends 'base.html.twig' %}

{% block body %}
    <h1>Thank you, {{ user.givenName }} {{ user.familyName }}!</h1>
    <p>Your account has been created successfully.</p>
{% endblock %}

And this all works! Hurrah. Well: once I added the DB table it did anyhow:

# docker/mariadb/docker-entrypoint-initdb.d/1.createTables.sql

USE db1;

CREATE TABLE user (
    id INT AUTO_INCREMENT PRIMARY KEY,
    given_name VARCHAR(255) NOT NULL,
    family_name VARCHAR(255) NOT NULL,
    email VARCHAR(255) NOT NULL,
    login_id VARCHAR(255) NOT NULL,
    password VARCHAR(255) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

I'm not using Doctrine Migrations for this as they can get tae hell (see PHP / Symfony: working through "Symfony: The Fast Track", part 4: not really about Symfony, this one › "Doctrine: know your limits!"(*)). But slinging this in /docker-entrypoint-initdb.d in the MariaDB container file system will ensure it gets recreated whenever I rebuild the DB. Which is fine for dev.

I can create a new user now.

However I need some server-side validation. We can't be having a new user reusing the same Login ID as an existing user, so we need to stop that.

Oh for goodness sake, doing this is just a one line addition to the User entity:

#[ORM\Entity(repositoryClass: UserRepository::class)]
#[UniqueEntity(fields: ['loginId'], message: 'This login ID is already in use.')]
class User
{

This is brilliant, but no bloody use as an example. So let's pretend it's not that easy and we have to create a custom validator for this.

First we need a new constraint class:

// src/Validator/UniqueLoginId.php

namespace App\Validator;

use Symfony\Component\Validator\Constraint;

#[\Attribute]
class UniqueLoginId extends Constraint
{
    public string $message = 'This login ID is already in use.';
}

And a validator thereof:

// src/Validator/UniqueLoginIdValidator.php

namespace App\Validator;

use Symfony\Component\Validator\Constraint;
use Symfony\Component\Validator\ConstraintValidator;
use Doctrine\ORM\EntityManagerInterface;
use App\Entity\User;

class UniqueLoginIdValidator extends ConstraintValidator
{
    private EntityManagerInterface $em;

    public function __construct(EntityManagerInterface $em)
    {
        $this->em = $em;
    }

    public function validate($value, Constraint $constraint)
    {
        if (!$value) {
            return;
        }

        $existing = $this->em->getRepository(User::class)->findOneBy(['loginId' => $value]);
        if ($existing) {
            $this->context->buildViolation($constraint->message)->addViolation();
        }
    }
}

This needs to be configured in services.yaml:

# config/services.yaml

services:
    # [...]

    App\Validator\UniqueLoginIdValidator:
      arguments:
        - '@doctrine.orm.entity_manager'
      tags: [ 'validator.constraint_validator' ]

And then applied to the form field:

// src/Form/Type/UserType.php

class UserType extends AbstractType
{
    public function buildForm(FormBuilderInterface $builder, array $options): void
    {
        $builder
            ->add('givenName', TextType::class)
            ->add('familyName', TextType::class)
            ->add('email', EmailType::class)
            ->add('loginId', TextType::class, [
                'label' => 'Login ID',
                'constraints' => [
                    new UniqueLoginId(),
                ],
            ])

And that's it: pretty easy (not quite as easy as the one-liner, but still)! There's some docs to read: Constraints At Field Level.

I was surprised I had to manually wire-in the validator into services.yaml: I've been spoilt recently with the AsDoctrineListener and AsMessageHandler directly on the classes/methods that define them, and Symfony's autowiring picks them up automatically.

I'm also a bit bemused as to why the validation system needs two files: a constraint class and a validator class. This is not clearly explained in the docs that I could see. As far as I can gather the constraint class defines the rules that would mean an object is valid; and the validator actually checks them against an object. I've read how this is a separation of concerns, but I am not entirely convinced we have two separate concerns here to be separated. in the example we have here, it's the validator that is defining the rule, and doing the validation:

public function validate($value, Constraint $constraint)
{
    if (!$value) {
        return;
    }

    $existing = $this->em->getRepository(User::class)->findOneBy(['loginId' => $value]);
    if ($existing) {
        $this->context->buildViolation($constraint->message)->addViolation();
    }
}

One thing I did learn - from Copilot - is that the relationship between the class names - [Constraint] and [Constraint]Validator - is just a convention, and there does not need to be that name-mapping going on. It's the default behaviour of Constraint::validatedBy. I guess in theory one could have one validator class for a suite of same-themed constraints.

I'm not convinced though.

But hey, it's easy and it works well! This is the main practical thing here.

I've breezed down the rest of the docs, and there's a few other interesting things, but nothing that needs looking at right now. So I'll leave it here.

Righto.

--
Adam

(*) Copilot did it all. I just tested it.

Tuesday, 29 July 2025

Quick look at event-driven programming in Symfony

G'day:

First up: this is not a very comprehensive look at this functionality. I just wanted to dip into it to see how it could work, and contextualise the docs.

I had a look at Symfony's messaging system in Using Symfony's Messaging system to separate the request for work to be done and doing the actual work which leverages Doctrine's entity events to send messages to other PHP apps via Symfony's messaging and RabbitMQ as the message bus's transport layer.

This time around I want to look at more generic events: not just stuff that Doctrine fires on entity activity, but on ad-hoc events I fire in my own code. This again is in an effort to get application code to focus on the one immediate job at hand, and easily delegate multiple-step processing to other systems / handlers.

Symfony's event handling is done via its EventDispatcher Component.

Installation is easy:

docker exec php-web composer require symfony/event-dispatcher:7.3.*

There is no other config. One just needs the library installed.

Symfony's DI container now has an EventDispatcher ready to be wired in to one's code, eg:

// src/Controller/StudentController.php

class StudentController extends AbstractController
{
    public function __construct(
        private readonly EventDispatcherInterface $eventDispatcher
    ) {
    }

Once one has that, one can dispatch events with it:

// src/Controller/StudentController.php

#[Route('/courses/{id}/students/add', name: 'student_add')]
public function add(/* [...] */): Response {

    // [...]

    if ($form->isSubmitted() && $form->isValid()) {
        // [...]

        $this->eventDispatcher->dispatch(new StudentRequestEvent($request, $student));

        return $this->redirectToRoute('course_view', ['id' => $course->getId()]);
    }
    // [...]

The object one passes to the dispatcher is arbitrary, and used for a) getting some data to the handler; b) type-checking which event a handler is for:

// src/Event/StudentRequestEvent.php

class StudentRequestEvent
{
    public function __construct(
        private readonly Request $request,
        private readonly Student $student
    ) { }

    public function getRequest(): Request
    {
        return $this->request;
    }

    public function getStudent(): Student
    {
        return $this->student;
    }
}

One can see here how the handler method looks for methods being handed a specific event class:

// src/EventListener/StudentProfileChangeListener.php
class StudentProfileChangeListener
{
    public function __construct(
        private readonly LoggerInterface $eventsLogger
    )
    { }

    #[AsEventListener()]
    public function validateProfileChange(StudentRequestEvent $event): void
    {
        $request = $event->getRequest();
        $route = $request->attributes->get('_route');

        if (!in_array($route, ['student_add', 'student_edit'])) {
            $this->eventsLogger->notice("validateProfileChange skipped for route [$route]");
            return;
        }
        $student = $event->getStudent();
        $this->eventsLogger->info(
            sprintf('Validate profile change: %s', $student->getFullName()),
            ['student' => $student]
        );
    }
}

The important bits are the #[AsEventListener()] attribute on the method, and that the method expects a StudentRequestEvent.

Here the "validateProfileChange" handling is just faked: I'm logging some telemetry so I can see what happens when the events fire & get handled.

In this case I have this event being fired in each of the add / edit / delete controller methods, and have the handler above listening for student_add and student_edit events; and another sendStudentWelcomePack handler which only listens for student_add (the code is much the same, so I won't repeat it). student_delete does not have anything handling it. Well: the handlers fire, but they exit-early.

If I add / edit / delete a student, we can see the log entries coming through, indicating which handlers fired, etc:

# after I submit the add form
[2025-07-29T15:53:26.607638+01:00] events.INFO: Validate profile change: Jillian Marmoset {"student":{"App\\Entity\\Student":{"email":"jillian.marmoset@example.com","fullName":"Jillian Marmoset","dateOfBirth":"2011-03-24","gender":"female","enrolmentYear":2016,"status":"Active"}}} []
[2025-07-29T15:53:26.607994+01:00] events.INFO: Send Welcome Pack to Jillian Marmoset {"student":{"App\\Entity\\Student":{"email":"jillian.marmoset@example.com","fullName":"Jillian Marmoset","dateOfBirth":"2011-03-24","gender":"female","enrolmentYear":2016,"status":"Active"}}} []



# after I submit the edit form
[2025-07-29T15:53:53.306371+01:00] events.INFO: Validate profile change: Gillian Marmoset {"student":{"App\\Entity\\Student":{"email":"gillian.marmoset@example.com","fullName":"Gillian Marmoset","dateOfBirth":"2011-03-24","gender":"female","enrolmentYear":2016,"status":"Active"}}} []
[2025-07-29T15:53:53.306733+01:00] events.NOTICE: sendStudentWelcomePack skipped for route [student_edit] [] []



# after I confirm the delete
[2025-07-29T15:54:06.827831+01:00] events.NOTICE: validateProfileChange skipped for route [student_delete] [] []
[2025-07-29T15:54:06.828087+01:00] events.NOTICE: sendStudentWelcomePack skipped for route [student_delete] [] []

Those logs are pointless, but in the real world, if an event is dispatched, any number of handlers can listen for them. And the controller method that dispatches the event doesn't need to know about any of them. We can also leverage the async message bus in the event handlers too, to farm processing off to completely different app instances. I can see how this will be very useful in the future…

As all this was far easier than I expected, this is a pretty short article. But I know how Symfony events can facilitate more event-driven programming now, and help keep my code more on-point, simple, and tidy.

Righto.

--
Adam

Shared/distributed locks in PHP with Symfony locking

G'day:

This is another one of these issues that got into my mind at some point, and I never had a chance to look into it until now.

Previously I was working on a CFML app, which had code that could not be run at the same time as other code, so handled this with CFML's native <cflock> mechanism. This works fine on a single CFML server, but is no good when there's more than server running the application. We never had to solve this issue during my time working on that project, but the question stuck with me

I don't give a rat's arse about solving this with CFML; but I can foresee it being a "good to know" thing in the PHP space. Turns out it's actually very bloody easy.

Symfony proves a locking component: The Lock Component. And the docs are pretty straight forward. It's installed as one might predict:

composer require symfony/lock:7.3.*

It creates config/packages/lock.yaml:

framework:
    lock: '%env(LOCK_DSN)%'

And that had me looking for where LOCK_DSN was set, and what it needed to be. This lead me to the Available Stores bit of the docs I linked to above, which listed a bunch of underlying storage mechanisms for the locks. Each had features and limitations, which got me to think about what I am trying to test here. Basically two things:

  • The locks needed to be respected across different PHP containers.
  • I needed to be able to create blocking locks.

On that second point: the Locking Component's default behaviour is to try to acquire a lock, and respond immediately one way or the other (yep you got the lock; nope you didn't get that lock). This is cool a lot of the time; but sometimes I can see wanting to wait until [whatever] has finished with the lock, and then grab the lock and crack on with some other stuff.

The only option that supported both remote & blocking locks was a PostgreSql solution, but I'm fucked if I'm gonna install PostgreSql just to solve a locking challenge. I looked at some other solutions, and the "flock"-based solution would work for me. Despite it not being remote-capable, it stores its locking metadata on the file system; and I could easily use a mounted volume in my docker containers to have multiple PHP containers using the same directory. For my immediate purposes this is fine. If I need multiple app containers spread across multiple host machines, I'll look into other solutions.

So the answer for where LOCK_DSN is set is: in .env:

# stick stuff in here that all envs need

LOCK_DSN=flock
MESSENGER_TRANSPORT_DSN=amqp://guest:guest@host.docker.internal:5672/%2f/messages

And that's all the DSN needs to have as a value when using Flock.

The code for initialising a Flock lock is thus:

$store = new FlockStore('[some directory to use for lock metadata]');
$factory = new LockFactory($store);

So before going any further I need that shared directory set up:

# docker/php/envVars.public

APP_ENV=dev
APP_CACHE_DIR=/var/cache/symfony
APP_LOCK_DIR=/tmp/symfony/lock
APP_LOG_DIR=/var/log/symfony
COMPOSER_CACHE_DIR=/tmp/composer-cache
PHPUNIT_CACHE_RESULT_FILE=0
# docker/php/Dockerfile

# [...]

# need to use 777 as both php-fpm and php-cli will write to these directories
RUN mkdir -p /var/cache/symfony && chown www-data:www-data /var/cache/symfony && chmod 777 /var/cache/symfony
RUN mkdir -p /var/cache/symfony/dev && chown www-data:www-data /var/cache/symfony/dev && chmod 777 /var/cache/symfony/dev
RUN mkdir -p /var/log/symfony && chown www-data:www-data /var/log/symfony && chmod 777 /var/log/symfony
RUN mkdir -p /tmp/symfony/lock && chown www-data:www-data /tmp/symfony/lock && chmod 777 /tmp/symfony/lock

# [...]
# docker/docker-compose.yml

services:
  # [...]

  php-web:
    # [...]

    volumes:
      - ..:/var/www
      - /var/log/symfony:/var/log/symfony
      - /tmp/symfony/lock:/tmp/symfony/lock

    # [...]

  php-worker:
    # [...]

    volumes:
      - ..:/var/www
      - /var/log/symfony:/var/log/symfony
      - /tmp/symfony/lock:/tmp/symfony/lock

    # [...]

Now I can wire-up the services:

# config/services.yaml

parameters:

services:
  # [...]

  Symfony\Component\Lock\Store\FlockStore:
    arguments:
      - '%env(APP_LOCK_DIR)%'

  Symfony\Component\Lock\LockFactory:
    arguments:
      - '@Symfony\Component\Lock\Store\FlockStore'

# [...]

Oh and I need a logger for this too:

# config/packages/monolog.yaml

monolog:
  channels:
    # [...]
    - locking

  handlers:
    # [...]

    locking:
      type: stream
      path: '%kernel.logs_dir%/locking.log'
      level: debug
      channels: ['locking']

Now I'm gonna create a web endpoint that creates a lock around some long-running code, logging as I go:

# src/Controller/LockController.php

namespace App\Controller;

use Psr\Log\LoggerInterface;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\Lock\LockFactory;
use Symfony\Component\Routing\Attribute\Route;

#[Route('/lock', name: 'app_lock')]
class LockController extends AbstractController
{
    public function __construct(
        private readonly LockFactory $lockFactory,
        private readonly LoggerInterface $lockingLogger
    )
    {
    }

    #[Route('/long', name: 'app_lock_long')]
    public function longLock(): Response
    {
        $this->lockingLogger->info('web_lock_long: started');

        $lock = $this->lockFactory->createLock('long_lock', 30);
        $this->lockingLogger->info('web_lock_long: lock created');

        if ($lock->acquire(true)) {
            $this->lockingLogger->info('web_lock_long: lock acquired');
            sleep(20); // Simulate a long-running process
            $this->lockingLogger->info('web_lock_long: processing done, releasing lock');
            $lock->release();
        } else {
            $this->lockingLogger->warning('web_lock_long: could not acquire lock');
        }

        return new Response('Lock operation completed.');
    }
}

Most of that is boilerplate and logging. The short version is:

$lock = $this->lockFactory->createLock('long_lock', 30); // name, TTL

if ($lock->acquire(true)) { // true makes it a blocking lock
    // do stuff
    $lock->release();
} else {
    // didn't get the lock
}

I'll run this on the php-web container, eg: http://localhost:8080/lock/long. It'll get a lock, and then sit around doing nothing for 20sec. Logging all the way.

I have created an equivalent command for php-worker to run via the CLI. It's analogous to the controller method, logic-wise:

src/Command/LongLockCommand.php

namespace App\Command;

use Psr\Log\LoggerInterface;
use Symfony\Component\Console\Attribute\AsCommand;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Lock\LockFactory;

#[AsCommand(
    name: 'app:long-lock',
    description: 'Acquires a long lock for testing lock contention.'
)]
class LongLockCommand extends Command
{
    public function __construct(
        private readonly LockFactory $lockFactory,
        private readonly LoggerInterface $lockingLogger
    ) {
        parent::__construct();
    }

    protected function execute(InputInterface $input, OutputInterface $output): int
    {
        $this->lockingLogger->info('command_lock_long: started');

        $lock = $this->lockFactory->createLock('long_lock', 30);
        $this->lockingLogger->info('command_lock_long: lock created');

        if ($lock->acquire(true)) {
            $this->lockingLogger->info('command_lock_long: lock acquired');
            sleep(20); // Simulate a long-running process
            $this->lockingLogger->info('command_lock_long: processing done, releasing lock');
            $lock->release();
        } else {
            $this->lockingLogger->warning('command_lock_long: could not acquire lock');
        }

        $output->writeln('Lock operation completed.');
        return Command::SUCCESS;
    }
}

This is run thus:

docker exec php-worker bin/console app:long-lock

And that's everything. If I hit the URL in a browser, and then a few seconds later call the command, I see this sort of thing in the log file:

tail -f /var/log/symfony/locking.log

[2025-07-29T10:57:41.626263+01:00] locking.INFO: web_lock_long: started [] []
[2025-07-29T10:57:41.626556+01:00] locking.INFO: web_lock_long: lock created [] []
[2025-07-29T10:57:41.626671+01:00] locking.INFO: web_lock_long: lock acquired [] []
[2025-07-29T10:57:47.231988+01:00] locking.INFO: command_lock_long: started [] []
[2025-07-29T10:57:47.233814+01:00] locking.INFO: command_lock_long: lock created [] []
[2025-07-29T10:58:02.398162+01:00] locking.INFO: web_lock_long: processing done, releasing lock [] []
[2025-07-29T10:58:02.398440+01:00] locking.INFO: command_lock_long: lock acquired [] []
[2025-07-29T10:58:23.175676+01:00] locking.INFO: command_lock_long: processing done, releasing lock [] []

Success. We can see that the web request creates and acquires the lock, and the command comes along afterwards and cannot acquire its lock until after the web request releases it.

Job done. This was way easier than I expected, actually.

Righto.

--
Adam

Monday, 28 July 2025

Using RabbitMQ as a transport layer for Symfony Messaging

G'day:

First up, I'm building on the codebase I worked through in these previous articles:

Reading that lot would be good for context, but in short the first two go over setting up a (very) basic CRUD website that allows the editing of some entities, and on create/update/delete also does the relevant reindexing on Elasticsearch. The third article removes the indexing code from inline, and uses the Symfony Messaging system to dispatch messages ("index this"), and message handlers ("OK, I'll index that"). These were all running in-process during the web request.

The overall object of this exercise is to deal with this notion, when it comes to the Elasticsearch indexing:

[That] overhead is not actually needed for the call to action to be provided[…]. The user should not have to wait whilst the app gets its shit together[.]

When serving a web request, the process should be focusing on just what is necessary to respond with the next page. It should not be doing behind-the-scenes housework too.

Using the messaging system is step one of this - it separates the web request code from the Elasticsearch indexing code - but it's all still runnning as part of that process. We need to offload the indexing work to another process. This is what we're doing today.


Step 0

Step 0 is: confirm RabbitMQ will work for me here. I only know RabbitMQ exists as a thing. I've never used it, and never actually even read any docs on it. I kinda just assumed that "well it's an industry-standard queuing thingey, and Symfony works with a bunch of stuff out of the box, so it'll probably be fine". I read some stuff on their website (RabbitMQ), and RabbitMQ's Wikipedia page too. Seems legit. And I also checked Symfony for mention of integrating with it, and landed on this: Using RabbitMQ as a Message Broker, and that looked promising.


RabbitMQ Docker container

Something I can do without messing around with any app code or Symfony config is getting a RabbitMQ container up and running (and sitting there doing nothing).

It has an official RabbitMQ image, and the example docker run statements look simple enough:

docker run -d --hostname my-rabbit --name some-rabbit rabbitmq:3

As this is gonna be used internally, no need for runing securely or with a non-default user etc, but obviously all those options are catered for too.

I also noted there's a variant of the image that contains a management app, so I decided to run with that. Converting the generic docker run statement to something for my docker-compose.yml file was easy enough:

docker/docker-compose.yml
services:

  # [...]

  rabbitmq:
    container_name: rabbitmq

    hostname: rabbitmq

    image: rabbitmq:4.1.2-management

    ports:
      - "5672:5672"
      - "15672:15672"

    stdin_open: true
    tty: true

Port 5672 is its standard application comms port; 15672 is for the manager. The hostname is needed for RabbitMQ's internals, and doesn't matter what it is for what I'm doing.

Building that worked fine, and the management UI was also up (login: guest, password: guest):

The only thing I'm going to be using the management UI for is to watch for messages going into the queue, should I need to troubleshoot anything.


Splitting the PHP work: web and worker

The main thing I am trying to achieve here is to lighten the load for the web-app, by farming work off to another "worker". This will be running the same codebase as the web app, but instead of running php-fpm to listen to traffic from a web server, it's going to run [whatever] needs to be run to pull messages of the queue and handle them. The web app puts a message on the queue; the worker app pulls them off the queue and does the processing.

So I need a second PHP container.

My initial naive attempt at doing this was to simply duplicate the entry in docker-compose.yml, and remove the port 9000 port mapping from php-worker as it won't be listening for web requests. This build and ran "fine", except for the entrypoints of both php-web and php-worker conflicted with each other, as they were both trying to do a composer install on the same volume-mounted vendor directory (the physical directory being on my host PC). This screwed both containers.

After a lot of trial and error (mostly error), I came up with a process as follows:

  1. Copy composer.json and composer.lock to /tmp/composer in the image file system, and run composer install during the build phase. This means that processing is already done by the time either container is brought up. For the Dockerfile to be able to do this, I needed to shift the build context in docker-compose.yml to be the app root, so it can see composer.json and composer.lock.
  2. Having those files in /tmp/composer/vendor is no help to anyone, so we need to copy the vendor directory to the app root directory once the container is up.
  3. As both php-web and php-worker need these same (exact same: they're looking at the same location in the host file system) vendor files, we're going to get just php-web to do the file copy, and get php-worker to wait until php-web is done before it comes up.

Here are the code changes, first for php-web

# docker/docker-compose.yml

services:
  # [...]

  php:
    container_name: php
    build:
      context: php
      dockerfile: Dockerfile

  php-web:
    container_name: php-web
    build:
      context: ..
      dockerfile: docker/php/Dockerfile

    # [...]
    
    entrypoint: ["/usr/local/bin/entrypoint-web.sh"]

  php-worker:
    container_name: php-worker
    build:
      context: ..
      dockerfile: docker/php/Dockerfile

    env_file:
      - mariadb/envVars.public
      - elasticsearch/envVars.public
      - php/envVars.public

    stdin_open: true
    tty: true

    volumes:
      - ..:/var/www

    healthcheck:
      test: ["CMD", "pgrep", "-f", "php bin/console messenger:consume rabbitmq"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s

    extra_hosts:
      - host.docker.internal:host-gateway

    secrets:
      - app_secrets

    entrypoint: ["/usr/local/bin/entrypoint-worker.sh"]

    depends_on:
      php-web:
        condition: service_healthy

Notes:

  • Just the naming and build context changes for the original PHP container.
  • Oh and it has its own entrypoint script now.
  • The config for php-worker is much the same as for php-web except:
    • Port 9000 doesn't need to be exposed: it's not going to be serving web requests.
    • It overrides the healthcheck in the Dockerfile with its own check: just that messenger:consume is running.
    • It has a different entrypoint than the web container.

Here's the relevant Dockerfile changes:

# docker/php/Dockerfile

FROM php:8.4.10-fpm-bookworm

RUN ["apt-get", "update"]
RUN ["apt-get", "install", "-y", "zip", "unzip", "git", "vim", "procps"]

# [...]
COPY docker/php/usr/local/etc/php/conf.d/error_reporting.ini /usr/local/etc/php/conf.d/error_reporting.ini
COPY docker/php/usr/local/etc/php/conf.d/app.ini /usr/local/etc/php/conf.d/app.ini

# [...]

RUN pecl install xdebug && docker-php-ext-enable xdebug
COPY docker/php/usr/local/etc/php/conf.d/xdebug.ini /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini

# [...]

WORKDIR /var/www
ENV COMPOSER_ALLOW_SUPERUSER=1

COPY --chmod=755 usr/local/bin/entrypoint.sh /usr/local/bin/
ENTRYPOINT ["entrypoint.sh"]
WORKDIR /tmp/composer
COPY composer.json composer.lock /tmp/composer/
ENV COMPOSER_ALLOW_SUPERUSER=1
RUN composer install --no-interaction --prefer-dist --no-scripts

# [...]

COPY --chmod=755 docker/php/usr/local/bin/entrypoint-web.sh /usr/local/bin/
COPY --chmod=755 docker/php/usr/local/bin/entrypoint-worker.sh /usr/local/bin/


EXPOSE 9000

And the entry point scripts:

# docker/php/usr/local/bin/entrypoint-web.sh

#!/bin/bash

rm -f /var/www/vendor/up.dat
cp -a /tmp/composer/vendor/. /var/www/vendor/
touch /var/www/vendor/up.dat

exec php-fpm
# docker/php/usr/local/bin/entrypoint-worker.sh

#!/bin/bash

exec php bin/console messenger:consume rabbitmq

That up.dat is checked in php-web's healthcheck now:

# bin/healthCheck.php

if (file_exists('/var/www/vendor/up.dat')) {
    echo 'pong';
}

This means it won't claim to be up until it's finished copying the vendor files, and php-worker won't come up until php-web is healthy.

I'm pretty sure those're all the PHP changes. I now have two PHP containers running: one handling web, the other handling messages.


Symfony config

The Messenger: Sync & Queued Message Handling docs explained all this pretty clearly.

I needed to install symfony/amqp-messenger, and for that to work I also needed to install the ext-amqp PHP extension. This needed some tweaks in the Dockerfile:

# docker/php/Dockerfile

# [...]

RUN [ \
    "apt-get", "install", "-y",  \
    "libz-dev", \
    "libzip-dev", \
    "libfcgi0ldbl", \
    "librabbitmq-dev" \
]
# [...]

RUN pecl install amqp
RUN docker-php-ext-enable amqp

# [...]

Then I needed to configure a MESSENGER_TRANSPORT_DSN Symfony "environment" variable in .env:

MESSENGER_TRANSPORT_DSN=amqp://guest:guest@host.docker.internal:5672/%2f/messages

(In prod I'd have to be more secure about that password, but it doesn't matter here).

And finally configure the Messaging system to use it:

# config/packages/messenger.yaml

framework:
  messenger:
    transports:
      rabbitmq:
        dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
    routing:
      'App\Message\*': rabbitmq

At this point when I rebuilt the containers, everything was happy until I ran my code…


App changes

In my last article I indicated some derision/bemusement about something in the docs:

The docs say something enigmatic:

There are no specific requirements for a message class, except that it can be serialized

Creating a Message & Handler

I mean that's fine, but it's not like their example implements the Serializable interface like one might expect from that guidance? From their example, I can only assume they mean "stick some getters on it", which is not really the same thing. Oh well.

Using Symfony's Messaging system to separate the request for work to be done and doing the actual work

And now I come to discover what they actually meant. And what they meant is that yes, the Message data should be Serializable. As in: via the interface implementation.

I had been passing around a LifecycleEventArgs implementation, eg one of PostPersistEventArgs, PostUpdateEventArgs or PreRemoveEventArgs, eg:

# src/EventListener/SearchIndexer.php

class SearchIndexer
{
    public function __construct(
        private readonly MessageBusInterface $bus
    ) {}

    public function postPersist(PostPersistEventArgs $args): void
    {
        $indexMessage = new SearchIndexAddMessage($args->getObject());
        $this->bus->dispatch($indexMessage);
    }

    public function postUpdate(PostUpdateEventArgs  $args): void
    {
        $indexMessage = new SearchIndexUpdateMessage($args->getObject());
        $this->bus->dispatch($indexMessage);
    }

    public function preRemove(PreRemoveEventArgs $args): void
    {
        $indexMessage = new SearchIndexDeleteMessage($args->getObject());
        $this->bus->dispatch($indexMessage);
    }
}

And those don't serialize, so I had to update the code to only pass the stdclass object that getObject() returned. And then likewise update src/MessageHandler/SearchIndexMessageHandler.php now that it doesn't need to call getObject itself, as it's already receiving that, eg:

#[AsMessageHandler]
public function handleAdd(SearchIndexAddMessage $message): void
{
    $this->searchIndexer->sync($message->getArgs()->getObject());
}

#[AsMessageHandler]
public function handleUpdate(SearchIndexUpdateMessage $message): void
{
    $this->searchIndexer->sync($message->getArgs()->getObject());
}

#[AsMessageHandler]
public function handleRemove(SearchIndexDeleteMessage $message): void
{
    $this->searchIndexer->delete($message->getArgs()->getObject());
}

Once I fixed that one glitch: it worked. When I edited an entity I could see a message going into the RabbitmQ queue (via the manager UI), and could see it being removed, and could see the results of the Elasticsearch update. Cool.

It took ages to get the PHP containers working properly - mostly the composer installation stuff - but the RabbitMQ and Symfony bits were really easy! Nice.

Righto.

--
Adam