LSP implementation for Fluid

by Manfred Egger

What is your idea about?
Code-/Autocompletion/Intellisense for Fluid tags is only possible in IDEs which support XSD based autocompletion for HTML (and not only XML) and only for tags not for shorthand syntax. With a language server we could have autocompletion in all IDEs which support LSP. And not only in HTML but also other template formats and shorthand syntax.

What do you want to achieve by the end of Q1 2024?
Basic implementation of code completion for tags and shorthand syntax, “Goto definition” for tags (should open the PHP class of the tag). Maybe diagnostics based on the TYPO3Fluid parser.

What is the potential impact of your idea for the overall goal?
Developers can have code completion in Fluid templates.

Which budget do you need for your idea?
5.000 Euro

3 Likes

Hi Manfred,

I like that idea very much because it would lower the barrier to Fluid and would make template maintenance much easier. I have three questions about your approach:

  1. Would the language server be written in PHP?

  2. Are there any requirements to the Fluid parser implementation to be able to accomplish this? Or would the basic feature set be independent of it?

  3. How much additional effort is required to get from a language server “binary” to an IDE extension nowadays? And is this part of your initiative?

Thanks!
Simon

1 Like

For PHPStorm and intellij in general, this is already possible: https://plugins.jetbrains.com/plugin/11151-typo3-fluid--enterprise

Maybe it make sense to combine efforts?

1 Like

Hi Simon!

I would like to clarify a few things in advance:

The approach I have in mind - and which I already made some testing for - is, to have an TYPO3 extension which works as LSP. So this wouldn’t be a standalone LSP “binary” but “just” a TYPO3 console command. Otherwhise the LSP implementation would require to have PHP parsing, TypoScript parsing, etc. for different TYPO3 versions and always keep up and maintain different version’s behaviour. And without access to the database we still don’t know, which TypoScripts are loaded (and in which order) so specific things (like resolving template paths) are much more complicated or even not possible.

If this is an extension, it makes many things much easier IMHO and the coding can focus on the LSP features. And I guess it would make things easier to extend some features of the language server. For example an extension could provide additional completion for specific ViewHelper arguments to provide possible values. Let’s use <f:translate /> as an example for this: the LSP could provide possible values for the argument “extensionName” by listing all installed packages in upper camel case or for “key” by listing all possible keys and their values of an extension’s locallang file.
But I guess for ViewHelpers of the core some of this completion will be part of the LSP implementation :wink:

To answer your questions:

1.: yes, Im planning to write this in PHP ;-). I did some research of already available PHP based implementations and my tests are based on the language server implementation of phpactor (phpactor/language-server)

2.: As the extension uses the features/packages of the TYPO3 installation, the parser is “typo3fluid/fluid” which provides some necessary features, but not all. For example to have working auto completion, we need to detect if there is a fluid tag at the current cursor position. And if so, provide the possible namespaces, functions or arguments. This is not possible with “typo3fluid/fluid” as it would require more features which are not necessary in a “normal” fluid context. And I guess, the maintainers wouldn’t want to add this just for a feature like LSP. Which I can very much understand :wink:
Other things like at least parts of diagnostics or the AST of syntactically valid files can be provided by the core parsers.

3.: It depends on the IDE you use. Neovim for example has an generic LSP interface so you (more or less) just provide the path to the binary (or in this case to the TYPO3 console command). For VS Code you need an extension. There are some generic VS Code LSP extensions to use a LSP with not much possibility for individual settings. And there are a vast majority of language specific VS Code extensions if you have specific configuration needs. I don’t have any experience with other IDEs so there might be additional client implementation needed depending on the IDE you use.
I will try to provide usage examples for neovim as it is my main IDE atm. And if there is enough time write a client extension for VS Code (which I used before neovim, so I have at least some needed knowledge). But I’m confident that the generic LSP VS Code extensions are working OK for testing purposes.
Of course any contribution is welcome :grin:

3 Likes

Hi Felix!

I already came accross Stefan Galinski’s extensions for IntelliJ while researching for my LSP implementation. As far as I understand it has standalone implementations of Fluid parsing (and TypoScript parsing in another extension). But it is closed source and written in Java (or maybe I got something wrong) so at the moment it doesn’t seem to fit well in the planned implementation of the LSP. Allthough any experience which supports the implementation would help :smile:

Can you briefly outline how you plan to deal with the following:

  • Third-party ViewHelper scopes (some of which require a full TYPO3 application including database)?
  • Merged ViewHelper namespaces consisting of multiple PHP namespaces?
  • Template pre-processors which may change how things are parsed, might (invisibly) change the template source that Fluid’s parser would actually operate on?
  • Expressions (custom and built-in) which change how a variable-accessor-like inline expression is handled?

Also, although this probably isn’t part of your idea right now: what are the perspectives of enabling a click-to-navigate capability for rendering of partials and sections (one which respects combined template paths with lookup priority)?

Hi Claus!

Let me start with the questions I don’t understand:
“Third-party ViewHelper scopes” and “Expressions”: Sorry, I don’t know what you mean. Could you explain this in more detail?

And now my answers your other questions:

Merged ViewHelper namespaces: If I get this correct, you mean something like the “f” namespace, which consists of ViewHelpers of TYPO3Fluid\Fluid\ViewHelpers and TYPO3\CMS\Fluid\ViewHelpers, right? As it is planned for the LSP to be an extension which has to be installed in your TYPO3 installation, imho it should have access to all globally configured namespaces (for example registered via $GLOBALS['TYPO3_CONF_VARS']['SYS']['fluid']['namespaces']). If there is a ViewHelper with the same name, the order of registration within the global namespace matters.
Long story short: ViewHelper reolving should work exactly the same in the LSP as in TYPO3’s fluid “context”. Any ViewHelper of any installed extension can be provided in auto completion.

Template pre-processors: I don’t think it’s possible to provide auto completion for this automatically, as the preprocessors don’t have any mechanism to get the part of the code which makes changes. So we can not predict, which code triggers a specific preprocessor. Maybe in the future the LSP extension could provide some kind of interface to let TYPO3 extensions provide auto completion for their preprocessors, but it is not part of my plans for the first version atm.
For the parsing of templates in the LSP implementation I guess any preprocessors shouldn’t be applied. For example with {parsing off} in the fluid template the auto completion would not work, because there is no parsing. But maybe this could be part of the LSP settings but this has to be well thought out.

Click-to-navigate: It’s on my ToDo list. I think the extension can “read” the view configurations of a certain TYPO3 installation (as the LSP extension is part of the installation) so it should be possible to map the template/partial/layout paths to their controllers/extensions at least to some extent. Navigating to partials or layouts should work, if it’s clear which file is meant via the TypoScript/view config. If there are multiple possibilities, I think a list of possible targets can be provided. And if it’s not possible to resolve the target, well then there will be no navigation :wink:
But as this feature has some complexity, I don’t know, if it will make it into the first version. And if so, idk what edge cases are not taken into account.

2 Likes

By Third-party ViewHelper scopes I just mean namespaces that aren’t part of TYPO3 - for example, extensions like VHS. So with other words, would the tool also provide autocompletion for ViewHelpers like those from VHS, when VHS is installed. It sounds like the answer is yes. This also relates to my question about template pre-processors because it is actually a template pre-processor that is responsible for extracting a namespace definition from xmlns - there are three built-in processors (https://github.com/TYPO3/Fluid/tree/main/src/Core/Parser/TemplateProcessor), but in addition to those, TYPO3 allows you to add your own which might do different things. The thing about those template pre-processors is that you don’t actually know what they do (which effect they have) until you execute them - and they can change how the template is parsed and executed. By far the most important one would be NamespaceDetectionTemplateProcessor which extracts and registers namespaces defined with xmlns - without this, the information about which PHP namespace belongs to a certain Fluid namespace might be lost.

Very important detail: the global array that contains Fluid namespaces is not the only way a Fluid namespace can be registered so it isn’t sufficient to respect those namespaces! It actually needs to be done on a per-template basis and ideally has to happen by triggering the namespace detection pre-processor and let it add detected namespaces to the ViewHelperResolver via RenderingContext.

And the Expressions are a similar feature. There are some built-in expressions: https://github.com/TYPO3/Fluid/tree/main/src/Core/Parser/SyntaxTree/Expression (Math, Ternary, Casting) which you trigger e.g. with {variable + 1} - those are similar to the template pre-processors in that you don’t actually know what they do, until you’ve triggered them - and any TYPO3 extension can register additional ones. You can see in this factory the two additional global configuration arrays that are used to hold such third-party pre-processors and expression types in https://github.com/TYPO3/typo3/blob/main/typo3/sysext/fluid/Classes/Core/Rendering/RenderingContextFactory.php#L58.

It would be a challenge to support those and provide for example auto-completion and syntax validation for those things that are provided by them, and to take into account how they might change how a template is parsed and rendered (for example, those two that add namespaces might change which PHP class a given ViewHelper points to). And I was wondering if those things are on your radar because without them, an LSP tool for Fluid would be quite limited in usefulness.

Merged ViewHelper namespaces: yes, that’s exactly what I meant. If the LSP extension is truly within a real TYPO3 context that shouldn’t be a problem. I would then suggest using the ViewHelperResolver to convert a namespace prefix and ViewHelper name to the FQCN of the PHP class that handles it (assuming you also did something to trigger the pre-processors which register the Fluid namespaces so the right FQCN is resolved).

Template pre-processors: You’re absolutely right that it’s probably not going to be possible to provide auto-completion for those - although it would make sense imho to hardwire auto-completion for the built-in ones at least, to assist with writing namespace registrations and to disable further validation and auto-completion of the passthrough pre-processor, {parsing off} is used (since this makes the template valid and un-handled even if you use broken ViewHelper syntax in it).

Click-to-navigate: Good to know it’s on your list, because that would be a very useful feature. I’d even say that a first iteration which just shows every potentially matching partial would be useful. I was just wondering if that was something you were also considering - and now I see that you did :slight_smile:

As a final note, I think the answer to Simon’s second question in the start may actually turn out to be “yes” if the LSP tool is to cover the things I’ve mentioned here. Some parts may need to become public which currently aren’t - other parts may need to be formalised a bit more (expression types to document their supported and matched expression formats - template pre-processors to document which strings they would match and replace).

Thanks for clarification.

To summarize regarding ViewHelpers: the plan is to support auto completion for any ViewHelper of any TYPO3 extension as long as it’s installed in the same TYPO3 installation :wink:

Expressions: I think supporting this would be challenging and I guess this will not make it into the first version. And as you stated some kind of “information transfer” is needed to get this work. As I have already mentioned before, maybe hooks or events could be established so authors of TYPO3 extensions can provide additional completions.