Executive summary
There has been on and off discussion I’ve seen about the benefits of adding a Queue system to TYPO3. I agree with them in general, so I decided to do some research on what would be involved and what the options are.
I surveyed 13 PHP libraries sourced through the PHP grapevine. I then made a specific attempt to make use of two of them: Symfony Messenger and Enqueue. My throw-away noodling-about code can be found here:
https://github.com/Crell/queue-test
Based on that research, I see three options for how to proceed.
Enqueue
Enqueue is a dedicated and robust Queue library. It is built as a Queue from the ground up. It has widespread use, although it is in maintenance-only mode at this point. (Arguably it is feature complete, so that is not a major drawback.) It exposes its queue-ness front-and-center, for better or worse. On the upside it is the most robust queue-centric option available, and has fairly good documentation (although there are still a few gaps). On the downside, some of its design decisions are quite odd, and undesirable. In particular:
- Although it includes a Doctrine DBAL driver for a backend, its mechanism for hooking it up is rather clunky, undocumented, and frankly bizarre. There would be some contortions to connect it to our existing Doctrine connection, although those would be hidden away from extension developers.
- Its messages are bare strings, not classed objects. Presumably that is helpful if you want cross-language support, but as that is not relevant for TYPO3 that is all down-side, and a major downside at that.
Enqueue has the most complete and robust set of support for different queue backends, and many other queue systems bridge in whole or in part to Enqueue’s available backends (called “Transports”) to avoid duplication.
Symfony Messenger
Symfony Messenger is first and foremost a low-end Command Bus. I say low-end here because it lacks most of the built-in functionality and robustness of tools like Broadway, Prooph, Ecotone. Its closest competition in the PHP space would be Tactician from the PHP League. However, it has plugins available that allow command messages to be deferred via a queue backend (which it also calls Transports). However, I find its design for doing so quite clunky and suboptimal.
Messenger is widely used within Symfony applications, and while it can be used on its own I do not have any data on how often it is used outside of Symfony.
Messenger’s documentation for use within the Symfony framework is fairly good. Its documentation for using outside of Symfony is decidedly poor. I was able to figure out how to run it thanks to help from Symfony’s documentation lead, and once you understand how it works it’s fairly tractable, just not at all self-evident.
The queue portion of Messenger includes a Doctrine transport as well as a number of others, although not as many as Enqueue does. It also has Enqueue bridges available separately to use Enqueue’s transports in some cases.
- Messenger’s setup is, in my view, over-abstracted. However, most of that would be/can be insulated from extension developers.
- Its worker script is not as robust as Enqueue’s. We would potentially need to do more work here to build a CLI tool for background running, although the level of effort is non-zero in either case.
- On the plus side, Messenger uses objects throughout its design, so Messages are classed value objects. That is a much nicer developer experience.
Using Messenger would have the additional result of being available as a command bus, which is a useful architectural style on its own. Unlike the more robust libraries, it doesn’t ask you to redesign your whole application around it. Making a command bus available may help with on-going efforts to better separate out code within the system.
Effectively, whether we would use Messenger as a command bus and getting queue support as a nice bonus or use it as a queue system and get a command bus along the way for free is a matter of perspective, but we would get both, even if not best-of-breed of either one.
Rolling our own
Writing our own queue system is always an option. It is not, I would argue, a good option in this case, as the problem space has a lot more gotchas than are readily apparent. At bare minimum, if we took this approach we should use the Transports from Enqueue rather than trying to write our own drivers for RabbitMQ, SQS, etc. This would also take the longest.
On the upside, we could of course purpose-build the exact API developer experience we want. On the downside, we’d have to argue about what API and developer experience we want, which would no doubt take additional time.
Recommendation
The state of PHP’s queue support is sadly rather poor at this point. Efforts to form a PHP-FIG working group to improve it were unsuccessful, mainly due to lack of interest from the major players. The unfortunate reality is that there are no great options at the moment.
Given that, I believe the least-bad option would be to integrate the Symfony Messenger component into TYPO3. We already use a number of Symfony components, which reduces the additional dependency weight. It also gives us the benefit of both a command bus and a queue backend, even if not the best version of either.
The out-of-the-box configuration should use Doctrine DBAL as a transport, although a way to switch it to some other service should be well-documented.
Additionally, cron should, by default, try to run queued tasks so that administrators do not need to take any extra steps to have the queue “just work” for low-end use cases. However, a simple toggle should disable that behavior. A provided alternative persistent worker script CLI command should be the recommended alternative to use if possible.
Background
For the purposes of this discussion, a Queue system is a library or subsystem that would allow either extensions or core code to send instructions to a separate process that will take a potentially-asynchronous action with no return value (that is, no Promise object). The use cases are for actions that need to be taken but their result is not necessary for handling the current request, or cases where some process has an extremely large number of similar subtasks that can be handled asynchronously from the current request. The canonical example is emailing a large number of subscribers when a particular piece of data is updated, but there are numerous others.
In the typical case, messages are enqueued by the system to some queue storage server. A separate, persistent process then listens for items in the queue (the mechanism may vary) and processes them in FIFO order. Importantly, running multiple of these “workers” in parallel is both possible and mundane to allow for easy horizontal scaling. However, more low-end implementations can simulate that process with a relatively frequent cron task. (Some frameworks recommend running cron every minute as a form of pseudo-queue, but that is a generally bad idea. If tasks need that frequent an update, using a proper persistent queue runner is greatly preferred.)
Many dedicated queue servers exist, both those that can be installed locally (RabbitMQ, ZeroMQ, etc.) and 3rd party services (Amazon SQS, IronMQ, etc.). It’s also possible to use many general purpose datastores as a queue store through polling mechanisms (Any SQL database, MongoDB, Redis, etc.), and some have specific queue-supporting features. In short, there are ample options we would need to support and abstract over.
In particular, since not everyone will want or need to install a dedicated server for queue tasks we must have a default of storing queued items in the main SQL database and allowing tasks to be processed from a cron job if no other worker is running. That is, from an extension author point of view it “just works” out of the box, and as needed can be cleanly separated to a persistent worker command and/or a dedicated server without modifying extension code.
Command and Event busses
Command Buses, Event buses, and CQRS Command/Request buses are a separate but often related topic to queues. They already provide a clean separation between the requesting side and the processing side of a request, and communicate with a defined message object of some sort. Inserting a delay into that connection via a queue server is an obvious add-on, and many such libraries do exactly that. As a result, many of the more robust queue options in PHP today are actually add-ons to Command buses, some of them more hacky than others.
Command bus libraries, however, have their own complexity independent of the queue system. They may have considerable additional machinery on the assumption that they’ll be used for CQRS, EventSourcing, or other such architectural designs. If those designs are of benefit on their own, that’s great, and the ability to defer message handling in some cases is icing on the cake. If not, then the extra machinery is mostly dead weight to make debugging harder.
I considered a few Command bus libraries for this analysis, but the more robust ones I determined to be out of scope at this time. Adopting a high-end CQRS or EventSourcing library would have its benefits, and queue support would be one among many, but that is a far larger discussion with much deeper implications, including a host of backward compatibility questions.
Raw research notes
The following libraries are listed in roughly the order in which I looked into them. This list was primarily collected through the tried-and-true research method of “asking for suggestions on Twitter.”
Lines marked with a @ are informational, or neutral.
Lines marked with (+) are positive
Lines marked with (-) are negative
Queue-Interop
@ GitHub - queue-interop/queue-interop: Promoting the interoperability of message queue objects.
@ MIT license
@ Highly engineered version of what a PSR would be if it had been done in FIG
@ Based on the Java equivalent (JMS)
(+) 76 Packagist dependents, nearly 12 million installs (+)
(-) Most of the dependents major seem to all be parts of Enqueue, from the same author
(-) Many of the dependencies are forks of Enqueue. Hm.
(-) No stable release. Latest tag is 1.0 alpha 2, but author says to use 0.8.1?
(+) 0.8.1 branch lists PHP 8 support.
(-) Author says it’s unmtaintained: Still maintained? · Issue #39 · queue-interop/queue-interop · GitHub
Enqueue
@ https://php-enqueue.github.io/
@ MIT license
(+) 7 million Packagist installs (Meaning over half of queue-interop’s installs are Enqueue)
@ Based on the Queue-Interop project; same author, it’s the reference implementation
@ Has a Giter chat room
(+) Seems robust
(-) Only supports string message bodies. (Does not auto-serialize more complex values.)
(+) Supports lots of different backends, including Doctrine DBAL
(-) Some of the backends are reporting failed tests, according to GitHub
(+) Has had recent updates
(+) PHP 7.3+, including PHP 8.0+
(+) Existing bridges for Laravel, Symfony, Magento2, and others
(-) Author says it is in maintenance mode only
PHP-Message-Queue
@ GitHub - bozerkins/php-message-queue: message stack on php using local files
@ MIT license
(-) Clearly abandoned; no stable releases, no code changes in 4 years.
Symfony Messenger component
@ The Messenger Component (Symfony Docs)
@ MIT license
@ 19.1 million installs on Packagist (what percentage of those are Symfony apps vs not is unclear)
(+) Well-maintained by a known entity
(-) Documentation mostly assumes you’re using Symfony, not just the component
(+) Supports many transports, including doctrine DBAL with Postgres additions, Redis, SQS, in-mem, etc.
(+) Extremely flexible. Which could be a downside because that means a lot of stuff to configure
(-) Wiring it all up for non-Symfony use is likely to be a not-small task because of all the abstraction that assumes you have FrameworkBundle’s magic. Documentation for using outside of Symfony is almost non-existent.
@ Uses an “envelope” around each message to carry metadata from one middleware to another, if necessary.
@ The 5.4 version has 5 Symfony dependencies for production that aren’t always needed (eg, doctrine-messenger, amqp-messenger, deprecation-contracts, etc.). 11 more for dev. The 6.0 version fixes this and is down to just psr/log, which is good.
(-) Currently has unnecessary soft-dependencies on the EventDispatcher component rather than PSR-14. I reported it and it’s been fixed in 6.1, and there are workarounds possible for earlier versions.
Swarrot
@ GitHub - swarrot/swarrot: A lib to consume message from any Broker
@* MIT license
(+) PHP 8-friendly. Appears to be maintained.
(-) Only handles the consumer side, not the producer side. We need both.
Castor
@ GitHub - castor-labs/queue: A simple queue abstraction for your PHP projects
@ MIT license
(+) Small and easy to setup
(-) Very little usage. Probably just the author. 
(-) Only supports strings as messages.
(-) Very basic
Tactician
@ https://tactician.thephpleague.com/
@ MIT license
(+) Managed the PHP League, a known entity with high standards.
(+) 5.8 million Packagist installs
(+) PHP 7.4 and up
@ Mainly a message bus, with a queue middleware plugin.
(-) The queue plugin is… 6 years old and has no stable releases. Latest is 0.6.0 from 6 years old.
@ Has a 3rd party bridge to link to Enqueue
@ Lots of configuration options, for good or ill.
(-) Ross Tuck (Tactician maintainer) recommends against using queue plugin, and message buses generally for queue interfaces. (cf: https://twitter.com/rosstuck/status/1511083883160293377)
Bernard
@ GitHub - bernardphp/bernard: Bernard is a multi-backend PHP library for creating background jobs for later processing.
@ MIT license
@ Dedicated queue library
@ 1.2 million downloads on Packagist
(+) Latest commit is less than a month ago, handling PHP 8.1 support.
(-) Latest actual release is 0.13.0, 3 years ago.
(+) Can route messages to different queues based on many factors, including class name.
(+) Drivers available for many backends, including Doctrine DBAL, Redis, AppEngine, SQS, etc.
@ Has Symfony CLI command integration for making quick CLI runners.
@ Has its own serializer and envelope system.
@ Has existing bridge libraries for Symfony, Laravel, and Silex(!)
PHP-FPM-Queue
@ GitHub - makasim/php-fpm-queue: Use php-fpm as a simple built-in async queue
@ MIT license
@ 106 installs on Packagist
@ More of a hack to use PHP-FPM as a pseudo-queue in memory
(-) Chews up the FPM worker pool
@ Uses the Queue-Interop interfaces
(-) No stable release; latest tag is 0.1.2 from 2018
(-) More of a “because we can” demo than a real system.
Laravel Illuminate Queue
@ GitHub - illuminate/queue: [READ ONLY] Subtree split of the Illuminate Queue component (see laravel/framework)
@ MIT License
@ 13.9 million installs (unclear if that includes Laravel full framework installs or not)
(+) Well-maintained by a known entity
(+) Requires PHP 8.0.2+
(-) No standalone docs, only in the main Laravel framework docs.
(-) Includes drivers for SQS, Redis, Beanstalkd, but NOT Doctrine DBAL. (There is a synchronous driver for testing.) There is a “database” driver, but that’s for Laravel’s DB API. We’d have to write one for Doctrine.
(-) Has 8 dependencies on the rest of Laravel, plus 2 3rd-party (Symfony Process and Ramesey/UUID). That includes its own Container, among other things. It’s really not intended for stand-alone use.
(-) Uses arrays for the payload, not an object.
(-) Seems to use lots of magic traits.
Yii2 Queue
@ GitHub - yiisoft/yii2-queue: Yii2 Queue Extension. Supports DB, Redis, RabbitMQ, Beanstalk and Gearman
@ BDS-3 license
@ 3.9 million downloads on Packagist
(+) Recent activity suggests it’s still maintained.
(+) Uses objects for messages, not arrays.
(-) Master branch build is failing, though.
(-) Requires superclosure. What?
(-) PHP version 5.5 and up. That’s… old.
(-) PHPUnit 4.4. What?
PHP Resque
@ GitHub - resque/php-resque: An implementation of Resque in PHP.
@ MIT license
@ 415,000 Packagist downloads
(-) Listed as unmaintained as of 2020
(-) Redis only
(-) PHP 5.3+
Ecotone
@ GitHub - ecotoneframework/ecotone: Ecotone shifts the focus to the business code. In order to make it happen, enables Message-Driven Architecture with DDD, CQRS, Event Sourcing in PHP.
@ 48,9000 Packagist installs
@ MIT license
(+) Requires PHP 8.0+
(+) Only two notable dependencies, ramsey/uuid and friendsofphp/proxy-manager-lts.
(+) Very extensive documentation with tutorials
(-) Mainly an EventSourcing/CQRS tool, not a queue system specifically
@ Driven/configured through PHP Attributes
@ Its Doctrine/queue functionality appears to make use of Enqueue? Or at least Enqueue’s connection handling.
Excluded EventSourcing tools
These tools provide EventSourcing/CQRS/CommandBus functionality for PHP, which can often have a queue-based component to them. I did not evaluate them at this time, as a CQRS bus is a much larger topic worthy of its own consideration. In short, “out of scope.”
Prooph: https://getprooph.org/
Symfony experiments
The Symfony docs are characteristically awful when it comes to using Messenger outside of Symfony itself. Or for how the system is put together. In fact, the graphics for the architecture are actively misleading.
How it actually works is that a message is wrapped into an Envelope along with “Stamps” (markers). Then it’s passed through a series of middleware steps, which are entirely arbitrary and can terminate the chain at any time by just returning, rather than calling the next step in the middleware chain. Middleware steps that want to avoid being run multiple times must include a stamp on the envelope after they’re done to avoid running processing a message multiple times. (More on that in a moment.)
One of the middlewares is HandleMessageMiddleware, which is usually last. Its job is to unwrap the message and delegate it to a handler (callable). The handler is actually derived using a HandlersLocator, which is pluggable. Common versions of it include a k/v map on the class name and magic derivation (eg, Foo is handled by FooHandler, etc.). Although it’s usually last, it doesn’t have to be and will dutifully call other middleware if there are any.
Another middleware is the SendMessageMiddleware, which will “send” the message to a queue backend (or, really, anything that implements a SenderInterface) and then, importantly, terminate the pipeline. It also has a Locator to map to the right Sender, which is swappable. The most common one maps a Message class name to a container service ID. Each queue backend (Redis, Doctrine, RabbitMQ, etc.) has its own Sender implementation.
A Receiver is a class that pulls messages out of a queue and puts them back into the bus from the start. Usually, a Sender and Receiver are implemented together in a single class, called a Transport. A Transport is just a Sender + Receiver.
The SendMessageMiddleware adds a stamp to the Envelope before it sends it to the Sender that says “I already did this.” That means when the message is passed through the bus a second time, SendMessageMiddlware will ignore it and let it continue on to the next middleware, which is usually (but doesn’t have to be) HandleMessageMiddleware.
To reiterate, any middleware that are listed before SendMessageMiddleware must account for the fact that any run-immediately message will go through the pipeline once, but any deferred-to-a-queue message will run through the first half of the pipeline twice (anything before SendMessageMiddleware). It’s up to the middleware what “account for” means, but usually it means adding its own custom “I’ve seen this” stamp.
Configuring how to map messages to handlers and messages to transports/senders/queues is mostly the job of the various Locators, which can be configured via injected DI properties or whatever else. This is where the TYPO3 glue will go, most likely.
Once values are in the queue, something needs to pull them back out to re-insert into the pipeline. That is the Worker class, which takes a list of Receivers to check (again, usually a combined Transport object) and the bus to pull from, plus some other details. Its only job is to pull stuff out of the corresponding queue and toss it into the bus again for a second round of processing. It’s designed to run in an infinite loop as a CLI command.
In particular, the Doctrine Transport can self-initialize the tables it needs, including having multiple queues in one table or separate tables. However, the mechanism to do so is entirely undocumented and non-obvious. It also involves a number of layers of indirection to wrap a connection into a Messenger connection wrapper, then into a Transport, which is then registered as a container service, and then message classes may be mapped to it. It is ugly, but once setup is entirely insulated from most end-users.
Overall, Messenger is not a queue system. It’s an over-abstracted message bus with a queue hack bolted to the side. It works, but it’s definitely not purpose built as a queue. (A purpose-built queue system, I would argue, would be built the other way around; assume all messages go to a queue and architect around that, with one of the queues just happening to “execute immediately.”)
In its defense, Messenger does offer a number of queue-related features via stamps and extra middlewares, if properly configured. For instance, when sending a message you can include a stamp to delay running it for some period of time, or have a separate queue for messages that have failed to get re-tried or analyzed as “these failed, why?” There are also various other middleware included that may or may not be useful.
The Worker, Sender, and Receiver classes currently optionally depend on the Symfony EventDispatcher rather than PSR-14. There’s no architectural reason they have to, and it was most likely an accidental oversight. I reported it and it has already been fixed in 6.1, but likely won’t be backported to 5.4/6.0. There is a workaround available, however. (cf: https://github.com/symfony/symfony/issues/45963 and https://github.com/symfony/symfony/pull/45967) The 5.4 version also has a number of additional dependencies that became optional in 6.0, mainly all of the supported transports.
Thanks to Symfony’s Ryan Weaver for his help in explaining how all of this works.
Enqueue experiments
The queue-interop project was a non-FIG attempt by the author to, essentially, port Java’s JSR 914 (its queue spec) to PHP. Enqueue is the reference implementation. Queue-interop never really caught on, and the maintainer now considers it abandoned. However, some of its transports (queue backends) have bridges to other queue systems, including Symfony Messenger. At this point, we should view queue-interop as a historical artifact that happens to have a lot of transports built for it, but that’s about it.
Enqueue’s documentation is more complete than Symfony’s, though still not ideal. It’s also built as a queue, specifically. That means many queue-related features – such as pub/sub for multiple receivers from one message, delays, TTL, etc. – are first-class citizens in Enqueue’s API whereas they’re sideways supported in Messenger via stamps. (Whether or not TYPO3 would use them often enough for that to matter is debatable.)
What is missing in the documentation is knowledge of some less-used parts of Doctrine. You can have Enqueue’s Doctrine transport make its own Doctrine connection, or use Doctrine’s ManagerRegistry interface… which Enqueue provides no documentation for using, and neither does Doctrine, and in fact Doctrine has no full implementation of that interface at all, just an abstract class. From what I’ve been able to determine, ManagerRegistry is an ancient alternative to having a DI Container that has never been jettisoned; why Enqueue is using that, rather than something that leverages PSR-11 I do not know, but it will make integrating with TYPO3/Doctrine DBAL more challenging.
Enqueue also has a first-class API for processing a single message and then continuing, in addition to a separate persistent runner. (Symfony may be able to do this, but it’s either buried or undocumented or both.) It also has a built-in “run only for a certain amount of time” flag, whereas Symfony needs a Symfony-specific event that gets tossed into the event dispatcher as a way to sneak a message into it. Enqueue definitely has the stronger runner here.
On the downside, Enqueue is much less maintained than Symfony Messenger. According to the author it’s in maintenance mode only, and while he’ll accept PRs from others he isn’t actively developing it. (That may or may not be necessary; it is a stable library at this point, so it’s unclear what new features would even be added.)
Enqueue also has no built-in “just run it now anyway” mechanism, and it’s unclear how to write one. That is arguably by design, since it’s a queue, not a message bus, but it is a factor to consider.
The other major downside is that its message body payload only supports strings. It cannot do its own serialization, so you have to serialize messages to strings first, and then manually decode them from strings in the consumer. This is a major usability flaw, in my view, and makes the otherwise reasonable (if somewhat convoluted in places) architecture much less reasonable. The serialization/deserialization really should be insulated from the user.
The likely reason for that is to support producers and consumers in different languages, so PHP’s serialized format would not work. However, there are ample ways to make that more portable if necessary, which Symfony Messenger does. This is also not a benefit that is relevant for TYPO3 so we get no value from this trade-off.
(Without an explicit license, it’s illegal to use.)