Future of TYPO3 command line

Archive Discuss Topics
1

Should we have a different cli binary for executing maintenance related tasks?

Background

The main challenge we face with the current state of the typo3 binary is, that it tries to fully bootstrap TYPO3 in the same way a backend request does. In order to make the binary functional in a non installed state, we must change the booting sequence in a way that low level commands can be executed.

While trying to solve this challenge, the suggestion was made to introduce a second binary, which can execute low level tasks. Commands executed with this binary, can then use a stripped down bootstrap that makes them available without a full install. Similar to the install tool web UI. The idea behind this proposal is to have different binaries for different use cases.

Alternative 1: One tool for each use case

Currently we have only one cli binary, which can execute a variety of commands for different use cases.
The question now is, whether it makes sense to have separate binaries for different use cases.

  1. Maintenance binary, things you would use (and are safe) in your deployment scripts – typo3ctl cache:flush, typo3ctl upgrade:all
  2. Business-Logic binary – typo3 import:foobar, typo3 scheduler:run

Very much like the install tool, which is the goto point for a maintenance UI, while the regular backend is for business logic.

Patch is here: https://review.typo3.org/#/c/58300/

Pros

  • A clear distinction between different concerns
  • These independent tools can be developed independently tailored to the different needs
  • Technically easier to implement, as we can define that the typo3ctl binary always starts with a stripped down bootstrap

Cons

  • Two different entry and goto points for users
  • No easy way to list all available tasks
  • Maintenance commands are not always low level commands. upgrade:run needs a full bootstrap in the end and might still fail in a not set up state of TYPO3
  • Convenience commands like server:run to start a simple PHP webserver (does not exist for TYPO3 yet) are clearly not maintenance commands, but still need a stripped down bootstrap to work
  • Some tasks might fall in different categories. Like in which category do the following commands fit? extension:activate, extension:deactivate, backend:lock, backend:unlock, language:update, referenceindex:update
  • It is not clear where to put commands use cases that fit into both like language:update or referenceindex:update

Alternative 2: One tool for all use cases

Instead of having two binaries, we keep having one and change the code in a way, that the single binary can execute low level tasks and regular tasks.

Patch is here: https://review.typo3.org/#/c/58298/

Pros

  • One goto point for users
  • Different types of commands can still be grouped in different namespaces like maintenance: to make the intent more clear
  • Similar to the integration of the install tool in the backend, we have one unified “UI” to accomplish multiple tasks

Cons

  • The technical implementation is more challenging. When executing a single binary, we must delay the decision whether to boot up to full extent at a late point in the execution. But convenience commands which are clearly not meant for maintenance are sometimes also useful to have a stripped down bootstrap (see above server:run), so we need to accept this challenge anyway.

Further considerations

To be able to cleanly dispatch low level commands AND high level commands within a single binary, we must know which command needs a full boot and which not before we initiate a TYPO3 boot. This means, we need to know which commands are available at all and where to get this information from.
Since commands are defined in active extensions, we currently need to boot TYPO3 up to a point where we can ask the PackageManager which commands are active. For that we need to boot up to at least the point where we can ask the PackageManager. This means with the current code base we need to decide wether we want to boot cached or uncached before we know anything. The consequence is, that in my approach (https://review.typo3.org/#/c/58298/), with the current code base we have, all commands will always be run completely uncached.

This chicken / egg problem can however be overcome at a later point, by using the same strategy as for class loading:

In composer mode we write a compiled file with all (via composer installed) extensions to a know location and in the binary, we can just read that file and the information which commands of which type are available and run a fully cached or uncached bootstrap for each command.

In non composer mode we can also go first with the approach we have for class loading. Write the command list of all required core extension in one place and add an additional command list for all active extensions in a second place when installing TYPO3 and activating/deactivating extensions.

In the long run it is planned anyway to get rid of the active/inactive state anyway and always load all extensions that are present in a (to be defined) location. Then finding commands will be trivial in all cases, as we only need to know the location where extensions reside and load the command definitions from there.

Long story short:

  • We need a more flexible booting mechanism for commands
  • To achieve that is a solvable technical problem, even if keep one binary
  • Would it still be benefitial for users to have two separate binaries for different use cases?

Organizational

Topic Initiator: Helmut Hummel

2

I Think Soulution 2) " One tool for all use cases" is the Best Way to aproach this.

because Especially Beginners will not understand which tool offers which functionality. and option 1) will miss the main point of the discussion of removing multiple CLI entry Points from TYPO3.

in my opinion i Rather have one tool which shows me all options. and Group them into usefull Categories

3

General

The motivation for this change/decision is to fix the upgrade:run command which was introduced in 9.4. It is a lowlevel command and requires a failsafe boot, like the install tool does. The respective issue can be found at https://forge.typo3.org/issues/86248
Therefore we need some kind of decision prior the release of 9LTS.

Alternative 1) One tool for each use case – Additional Pros

  • Clear guidance to the integrator, that there is a designated maintenance tool (typo3ctl), intended for deployment scripting
  • Quick command lookup due to shorter command lists
  • bin/typo3ctl may be used next to helhum/typo3-console. While helhum/typo3-console replaces bin/typo3 a separate bin/typo3ctl utility allows faster adaption of the core solution. The developer is still able to use helhum/typo3-console for features not yet available in core, but can use the new bin/typo3ctl tool for maintenance tasks – allows easier preparation for the future.

Alternative 1) One tool for each use case – Cons

This is actually a “Pro”, as the user has to read less commands in tool help in total. They can find the command for their usecase quicker.

In needs to bootstrap in the end, not in the beginning. Thats an important fact and actually a Pro. A late boot allows to properly catch errors in upgrade:run (like the install tool does as well)

I think this example is better suited to a task runner (use compoer scripts or makefiles). Not a Con, rather a Pro, when we instruct people to use proper task runners.

If something really belongs to both categories, then it may be provided in both tools. Though I’m not saying we should do this. We should rather aim for implementing it in one tool. (means maintenance for you examples)

Alternative 2) One tool for all use cases – Pros

I see this as a Con, the user doesn’t know what environment a command requires and in which situations it may be used.

From an end-user standpoint it’s quite similar, but it would only be similar if we would spawn a subprocess.
Also, despite the UI in the Backend, there is still a separate Install Tool available, which is the goto point when you do an upgrade/maintenance. That seperate tool is missing with the one-tool-solution.

Alternative 2) One tool for all use cases – Additional Cons

  • No clear distinction, which commands might be used in which scenario (especially deployment)
  • Requires uncached Bootstrap
  • It prohibits further improvements to the Bootstrap code, as it limits to uncached execution. We will have hard times integrating a compiled container like symfony/dependency-injection which would be required to run uncached (that means compilation for every CLI run, that’ll be performance hell)

Further considerations

That means the maintenace tool is composer. (which generates the command map)
Which (maintenance?) command repairs that command map in classic mode?

Both solutions should be solvable, somehow, or we wouldn’t need to discuss here :wink:
But “solvable” doesn’t mean that the code is clean or maintainable.

To summarize (my thoughts):
bin/typo3 can be made quite solid in future (for composer installations), but will be a nightmare during development having to use bin/typo3 dump:commandmap for every new command
bin/typo3ctl solves this now and additionally provides clear separation of usecases.

4

Let me start with what I think does not make sense to discuss in the first place and that are technical details.

Of course you can read “solvable” as “hacky”. Be rest assured, that I have no intent to suggest unclean and unmaintainable code. When I wrote “solvable” I meant a clean and simple technical solution.
Besides that, what you propose is having to maintain two separate code paths to execute commands. Every change to the bootstrap will likely need adaptions in both places. I propose to have one code path to execute commands, which is flexible enough to execute true low level commands (work in all conditions) as well as high level commands (only work in a healthy and set up TYPO3). This one code base could even serve the base for two binaries, which brings to the point which I want to discuss.

What I think makes more sense to discuss first, is what we think is the best user experience for interacting with TYPO3 from the command line.

From my point of view, if there is a single chance, that anything can be seen as part of one tool as well as part of the other, then the whole concept of a strict separation makes no sense any more. It in fact outlines the inconsistency of the separation.

Yes, we already have commands with different concerns, but they still serve a single purpose and that is:
Remote controlling / interacting with a TYPO3 instance via command line

The message for users could be simple as that:
“If you want to interact with a TYPO3 instance from command line use bin/typo3. Some commands fail in the current condition, but we marked (or have hidden them, depending on what we prefer) them for you in the list of commands”

I find this much easier to understand than:

“If you want to interact with a TYPO3 instance from command line it depends on what you want to accomplish and on the state the instance is in. You may find some commands of some type here (bin/typo3), if TYPO3 is in a good condition and set up. In any case, you can also find some other commands here: bin/typo3ctl. Some commands however you can find in both places, the only difference is, that when you use bin/typo3 you need to be sure TYPO3 is set up and in a good condition, before you can execute them. So for commands these commands we recommend to use bin/typo3ctl. However there are commands available with bin/typo3ctl that may still fail under certain conditions.”

These two quotes contradict each other. When the benefit of a separate binary should be, that the “user does not need to know what environment a command requires”, but commands like upgrade:run still need a full boot, thus a completely set up TYPO3 in a healthy condition (no broken extensions), then a separate binary does not hold its promises. From a users perspective, it does not matter whether the command failed on “early boot” or on “late boot with better error handling”. The command failed, despite the fact it has been called through the “you don’t need to know about the state” (typo3ctl) binary.

These two things alone:

  1. It is unclear which command belongs exactly to which binary/category
  2. Commands added to the typo3ctl binary might very well fail in certain conditions

Almost completely void any benefit from the separation.

Almost, because there is indeed one concern and that is dealing with a long list of commands with all different concerns.
I agree that we need to tackle that.
One solution could be better organize commands in namespaces (already available symfony/command feature). The other could be to add some kind of grouping mechanism, which is independent from command names. So commands would be ordered by group and only inside the group ordered by name. We could show only commands of one group with an option.
All of that would be achievable with little effort and we would get both: good discovery for commands of certain types AND a single goto point for command line interaction with TYPO3. Having such grouping in place, one could even create command aliases e.g. bin/typo3 --group deploy as t3deploy or bin/typo3 --group maintenance as typo3ctl

There are two usability concerns at hand. One is limited discovery for beginners (which was the entry command name for this typo3 task I want to run again?) and one for experts (holy crap, where is the deploy command I need in this endless list of commands).

Both can in my opinion be elegantly solved with the suggestion above.

5

Now some random comments on statements

Sure. Instead of saying “if you want to run a simple web server for the project just type typo3 server:run”, we could document “read the PHP documentation how to accomplish that, then read the Makefile documentation how to package that into a neat task for make, oh and make sure you have make installed on your Windows machine. (of course we can provide a Makefile already, but this Makfile might still need adaption for different environments and the make command needs to be available anyway)”

There are always multiple ways to accomplish things and I’m fine with different people having different preferences. If someone wants to use make, go ahead, use it. If we want to have a simple readme in the base distribution, we can add a server:run command to the dev dependencies and users who can run composer install will be able to run bin/typo3 server:run.

6

This is one suggestion for an intermediate solution, which would be obsolete with what I outlined above:

In fact, it is already trivial, because any approach we discussed needs \TYPO3\CMS\Core\Core\SystemEnvironmentBuilder::run() to be executed before anything else happens.
And the only true dependency to state the (Failsafe)PackageManager has, is the path to typo3conf/PackageStates.php

If we’d instanciate the PackageManager right after we set up the environment (yes the PackageManager itself would be uncached, but this is mostly irrelevant for the command line), then the complete other bootstrap can be failsafe or cached. With the PackageManager at hand we could read the commands from all extensions in runtime, no command map needed. (I personally would nevertheless prefer the command map, as I prefer to defer things to a compile time and not do it in runtime, but both ways have obvious advantages and disatvantages).

Main point here still is: It can be solved in a clean straightforward and maintainable and cachable way, so we can concentrate on deciding whether users should look for (different) commands in different places or in one.

7

I forgot to mention, that I actually understand the drive to separate concerns. Assumed that there is no doubt, which command satisfies which concern, the separation into two commands could make sense.

Nevertheless I would sacrifice separation for an easier initial discoverablity (like outlined above)

Unfortunately I have no numbers at hand, but that TYPO3 Console replaces the TYPO3 binary isn’t my invention, but resulted from user feedback. People were confused by the fact, that two binaries existed, which weren’t identical, but similar. Due to the fact, that TYPO3 Console is a full replacement for the TYPO3 binary, it was decided to let TYPO3 Console to replace it. The situation @bfr describes above would be similarly confusing, as commands of the same concern (management) would be available through different dispatching scripts. No matter how the decision here is taken, I would avoid that at all cost. I’d either replace the typo3ctl binary as well, or make TYPO3 Console commands available for this dispatcher as well.

I have no proof for it, but from my previous experience I would assume, that many of our users will be confused by the separation, not matter how clean it is.

To summarize: While I can see a theoretical benefit in the separation, I would prefer the most practical solution for the majority of our users, which in my opinion is to keep a single command line tool approach with the message: “If you want to interact with or remote control your TYPO3 instance, use bin/typo3”

8

I’m in favor of a unified binary, because it’s more convenient. I only have to memorize one command instead of two, which also bears less risk of frustration. As a user I simply don’t want having to think about different bootstrapping statuses. Long list of commands can be solved through hierarchy / subcommands.

9

Even for experienced user it will be completely unclear which of the two tools must be used for a certain action. A single entry point would be the way to go for me. Even if certain commands need a technically different environment this can somehow be solved.
Ideally this unified binary would support additional features from other packages such as the typo3 console. This way we could even move certain commands from those packages to the core once they are matured and if that makes sense.

10

I have one problem with how the question is posed. One vs. two tools in itself is like saying “do you like peace or war” - when peace might mean simply killing everyone alive that may not be the best decision.

From what I understand we either have

  • one tool, where I as a user need to remember / know which commands are safe to use in which context
  • two tools, where I as a user need to remember which tool to use for which command

if that’s the question, then: I don’t care because the cognitive load for the user is the same in both cases. And a lot of the arguments here don’t make any difference. The argument with one tool and having a possible script which emulates the two tool solution can be completely turned around (with two tools we could also have a script providing one entry point). In any case the user needs to remember / know something before using it, so really, do whatever is cleaner to program (or whatever whoever does the work prefers).

[I’d only prefer the one tool solution if we can have one tool that always works in all cases.]

11

desertum faciunt et pacem appellant

Apart from jokes, I agree with Susanne’s points about one/two tools and I tend to prefer the solution that will be judged as the better to be mantained or developed.
From my POV as long as the documentation is clear about what should be used when, there is little difference in learning one or two separate commands

12

This topic was automatically closed after 14 days. New replies are no longer allowed.