I’ve chatted a bit more with Benni Mack and others, and based on the previous two posts (1, 2) and their feedback I want to now offer a proposal for the next several steps in overhauling configuration.
The final picture
The endgame for this Epic is as follows. There is a directory tree (in composer or legacy mode, no difference) like this, in every TYPO3 site:
config/
EnvironmentOverrides.php
features.yaml
default/
Development/
mail.yaml
Testing/
mail.yaml
Production/
mail.yaml
somestring/
mail.yaml
mail.yaml
sites/
default/
mail.yaml
sitea/
config.yaml
overrides/
Development/
mail.yaml
mail.yaml
siteb/
overrides/
mail.yaml
- All configuration objects are defined canonically by a typed PHP class. That PHP class corresponds 1:1 with a YAML file. (The exact naming of the YAML files is still TBD.)
- Every config class is expected to have a default for every property, so that it can be loaded without a YAML file existing. It also MUST be re-loadable via
__set_state(). (Traits provided to make that easy.) - YAML is chosen as the config format on-disk not because it’s good (it isn’t), but because it’s popular. XML would be more self-documenting, but arguing for XML is an uphill battle I don’t want to burn karma on.
(Also, as of this writing Serde supports YAML but not XML yet, since it’s more annoying to parse.) features.yaml- This config object (there is a class for it) is technically outside of “configuration”. It is read into a corresponding object and that object is then exposed to the system via DI. It contains all feature flags, with readonly properties. It replaces the feature flag portion ofTYPO3_CONF_VARS.
EnvironmentOverrides.php is part of the environment system. It consists of:
- The Symfony DotEnv component.
- The override file.
- A class that such data is read into, which may or may not be the same as the existing
Environmentclass (TBD). - A new
$connectionsarray. (More on that in a moment.)
The Environment
The existing TYPO3_CONTEXT variable is used as an environment mode differentiator. Many systems already have such a flag. We can use it as is. It is used mainly for config directory resolution.
There are additional env vars to define common per-environment things, such as a default database connection. (The exact list is TBD.)
The DotEnv library allows for those to be set via a .env file in development environments. ddev’s existing support for setting env vars is unaffected.
During startup, the environment variables are read into local variables along with a new variable named $connections, and then EnvironmentOverrides.php is invoked. That file may then mutate whatever variables it wants. This is primarily for host-specific glue code on cloud-based hosts (to map the host’s env vars into TYPO3’s env vars). Developers may also enhance the $connections array as desired. This file does not have meaningful access to any true-globals.
Connections
The $connections array includes all connections to external services: SQL database, Redis, Solr, Elasticsearch, etc. It’s structure is approximately thus:
$connections['db']['default'] = [ ... ];
$connections['db']['legacy'] = [ ... ];
$connections['solr']['main'] = [ ... ];
$connections['elasticsearch']['main'] = [ ... ];
$connections['cache']['redis'] = [ ... ];
A single connection type for each main type of connection is defined by the system, and populated automatically by environment variables. Additional options may be added in EnvironmentOverrides.php. So, for example, the typical site will only need to set these env vars:
DB_TYPE=mysql
DB_HOST=localhost
DB_USER=me
DB_PASS=secret
DB_PORT=3608
And that will automatically map to
$connections['db']['default'] = [
'type' => 'mysql',
'host' => 'localhost',
'user' => 'me',
'pass' => 'secret',
'port' => 3608,
];
There are similar env vars pre-defined for Solr, Redis, and whatever else we decide to predefine.
After that array is populated, it is deserialized into a readonly object (with nested objects, most likely), which is then available through the DI system. Any service that needs to can get that Connections object injected and read whatever it needs.
Any extension that wants to configure additional connections… is only allowed to do so by having the site admin define them with an appropriate key in that array. Defining new connections through the UI is explicitly not supported.
(It may be possible to make all parsing of that information lazy the first time the Connections service is requested. TBD.)
Sites v2
Extending the current sites definition logic, each site is defined by a config/sites/$key/config.yaml file. A site does not exist unless it is defined there first. That is, its presence in the page tree is dependent on that file existing. It’s structure is essentially the same as now, but is parsed into a readonly object to expose to the system. That is how one reads the site. (A “current site” DI service can handle the resolution logic and return the appropriate object.)
Configuration
All other configuration objects vary based on only two axes: TYPO3_CONTEXT and the current site. When a configuration object is loaded, a file is looked up according to the following order. The first file found is the whole configuration object.
config/$current_site/overrides/$current_context/mail.yaml
config/$current_site/overrides/mail.yaml
config/default/$current_context/mail.yaml
config/default/mail.yaml
The first file found is what gets used. It is deserialized into the corresponding object, which is available as a readonly service through DI. (Note: The order of the two middle lines–that is, what happens if there is a site-specific and context-specific file but not one for both–is important but easy to hard code either direction. I don’t much care which it is. That’s a separate discussion we can have at a later time, but should be trivial to change in code once we make up our minds.)
The loaded object is also cached out to disk using var_export() to typo3temp/config/$site/$context/ (or similar). That way, subsequent reads can just include that file if it exists and skip the file resolution. Because that process does not require a database lookup, it can be done lazily (for development) or all at once during a deployment process to pre-warm the cache. (The cache warmer is a low-priority, later feature.)
Wither TypoScript
I have concluded that, from an architectural and workflow perspective, we should treat TypoScript as content, not configuration. While one could debate what it is semantically, from a workflow perspective it is… not configuration, but content, because it is data that varies potentially per-page. It is therefore explicitly excluded from this discussion for now.
We may (and should) revisit TypoScript later to put more structure into it and rethink how it should be able to vary, but for now it should be viewed as content, not configuration, and thus out of scope of this discussion.
Because this system is more self-evident and easy to work with than ext_conf_template.txt, though, it may incentivize some extension developers to shift behavior out of TypoScript and into the formal configuration system.
One thing we should do, however, is expose configuration values to TypoScript, so that TS authors can read (but not manipulate) those values as appropriate. What that looks like syntactically, I don’t know right now.
Editing?
In v11, extension configuration defined via ext_conf_template.txt has a basic UI auto-generated for it, and is GUI editable. TYPO3_CONF_VARS proper does not, although some bits of it may be manipulated through the Install Tool.
In this revised model, by default, none of those configuration files are editable through the GUI. In practice I think this is a very small regression, but it is technically a regression. What we get in exchange, however, is a much more explicitly defined, self-documenting, and hand-editable configuration system. More importantly, it has native support for site-based and env mode-based overrides, which are far and away the most common things by which you need to vary configuration.
The configuration, $connections, and environment systems also make TYPO3 vastly more compatible with cloud-based hosts that offer readonly file systems. That’s the win.
A possible (and I stress this is possible; I am not promising it) extension, however, would be to allow the system to detect if it is running in an environment where the config directory is writable. If it is, then all configuration objects can have edit forms auto-generated for them (give or take an opt-in/opt-out flag and some attributes for form customization; I’m pretty sure the form engine configuration should not be UI-exposed, at all). Those forms would write back to the YAML files on disk directly. (Excluding features.yaml and EnvironmentOverrides.php, of course.) Updating the file clears the cached var_export() for that object, allowing it to be regenerated.
If the directory is read only, then all of those forms automatically become read only as well. That is, they become a way to review what the configuration is, but not to edit it.
That would allow local development environments to manipulate the configuration via the GUI, and automatically produce files that are git commit-able. When deploying to production, however, all of those files become locked and readonly and the only way to update them is via a new git push, which is the correct way to update production.
The tricky part is how the forms would interact with the context/site override logic. I think that can be handled by putting some toggles onto the forms themselves. The specific UI/UX of that I am not sure at the moment, but that’s something that would define if this addition is possible. (My prediction: It won’t happen by 12.0, but there’s a better than 50% chance of it happening by v12 LTS, assuming we can make it work at all. Again, I am not promising.)
Transition
This setup replaces TYPO3_CONF_VARS, LocalConfiguration.php, AdditionalConfiguration.php, and FactoryConfiguration.php. It also replaces the ext_conf_template.txt files. Naturally that means we need some transition phase for backward compatibility.
My intent here is that those systems all remain essentially unchanged in v12, but get deprecation warnings if you access them directly. (Assuming we can find a place to put such a warning; if not, we just document it.) Then, appropriate sections of the TYPO3_CONF_VARS array become a final fallback for the config objects if not defined elsewhere. For example, if the mail system is not defined anywhere in the following files:
config/$current_site/overrides/$current_context/mail.yaml
config/$current_site/overrides/mail.yaml
config/default/$current_context/mail.yaml
config/default/mail.yaml
Then it is hard-coded somewhere to look at TYPO3_CONF_VARS['MAIL']. That array is then used to populate the config object and cached. Config objects do not need to correspond directly to a top-level TYPO3_CONF_VARS array; in fact they very much should not. However, most will likely correspond to some segment of that array in order to make the transition easier. (For instance, the Locking class will correspond to locking.yaml, and if one is not found then it will look at TYPO3_CONF_VARS['SYS'][locking']['strategies'].
In v13, we just remove that last fallback step and TYPO3_CONF_VARS will be gone.
Implementation steps
Getting here is a multi-step process, of course. I can and will kick it off but it will require help from others, especially to convert core to use the new API rather than reading from TYPO3_CONF_VARS directly. The general 12-step plan is:
- Clean up the
Environmentclass into an injectable object, cf Feature #94995: Expose environment object through DI - TYPO3 Core - TYPO3 Forge - Add Symfony/DotEnv as a dependency and wire it in.
- Add Crell/Serde and its ancillary tools as dependencies. (Doing this as its own step will make merges easier, as it’s less work to keep
composer.jsonin sync.) - Add the new
features.yamlfile. - Introduce the
EnvironmentOverrides.phpfile and$connectionsarray, and associated object. - Convert the DB system to read from the
Connectionsservice value object. - Update the installer to write a
.envfile instead ofLocalConfiguration.php. (Or maybe in addition to at the moment, until the transition is further along.) - Update how the sites files are read to use Serde and injected objects. This includes appropriately updating the “site figuring-out” logic.
- Pick one or two easy-ish config objects (mail, logging, and gfx are good candidates) and build out the config reading and fallback system using those as trial balloons. This will be the most complex task, I imagine.
- Build out config objects for the rest of configuration. This is a very crowd-sourceable task.
- Convert all uses of
TYPO3_CONF_VARSin core to use the new config system. This is a very crowd-sourceable task. - Build auto-forms for configuration objects.
As for who does them:
- Steps 1, 3, 4, 5, and 9 I intend to tackle directly myself.
- Steps 2, 6, and 8 I could do, but so could most contributors so if someone else wants to pitch in, that’s a place to do it.
- Step 7 is best done by someone with more knowledge of the installer system.
- Steps 10 and 11 can and should be done by as many people as possible as a way to get developers practice with the new APIs.
- Step 12 is going to require someone with very dedicated form API knowledge, working closely with me to keep it as elegant as possible. (Volunteers welcome.)
Conclusion
So, this is my proposed roadmap for configuration in TYPO3 v12 and beyond. If anyone would like to chime in with support, approval, pointing out something stupid that I missed, or rotten tomatoes, now is the time.
