Yes, I know, to some, the mere mention of drastic changes to something they’re used to or have well mastered is to them, like an act of war. They first wipe their face and head for fresh air. This topic is not for the faint at heart. It’s not for every end-user or 'used-to’s developers or even every ‘experienced’ TYPO3 developers per se. This is for TYPO3 architects, the software architects among you. Those who sit down and plan an infrastructure. It’s for people who like to see things cleaned up or done in a proper, clear, consistent, conceptualized and planned fashion. Forgive me if am not so eloquent. Storytelling is not exactly my strongest suit. I can put you to sleep, quickly with boredom. Am just bringing to the table good experience and code analysis. So, thank you.
This is not a complaint. It’s a note on the inefficiencies and inadequacies of the TCA in its current form, its power, prowess and the limitations it places on any developer. I won’t get into specifics but just give a general ground.
This will benefit all those who work with tables, forms, content etc.
Experience
I was using TYPO3 when version 4 came out and have since used every version and have stuck to TYPO3 on a daily basis since then to this day with the current version 10.4.5. There are several things that are great about TYPO3 to this day that keep getting better with every other release. These include the localization features (though its changing for use in classes), the caching system, imaging, the yaml settings, the DBAL (or so I call it, the connection to the database features), the menu system and typoscript (for design). For others, the more things change, the more they remain the same, such as the extension manager and the MVC. Others require heavy-duty changes. These include - the normalization of the backend and the TCA. Others keep getting worse like lack of proper documentation (what TYPO3 calls documentations are actually mostly white-papers), the database limitations, long development times given that no two websites are exactly the same, among other things. For some of these I shall open separate topics for them (if you don’t mind) so you can weigh-in and suggest improvements and strategies that are simple and cheap. I will too.
TCA
The TCA desperately needs an extreme makeover. A complete overhaul. As demands for complexity and sophistication increases with every passing day, the TCA has been left behind to such a degree that innovation in websites using TYPO3 is slowly becoming difficult. Some follow TCA like a cult, I don’t understand why, it not capable of very much and would like you to sacrifice more and more on its alter- ‘Oh give me more. I want more… feed me moooooore…’. The rest of us say, it’s too big and evil it should be broken up, like Amazon. Am talking from an end-use-development outsider perspective - not from an insider/establishment type. Using the TCA right now feels like a cookie-cutter type of software where you either fit in or you quit. It’s too restrictive, too cumbersome and a lot more difficult and time-consuming - for an example, it should allow for custom queries that override TYPO3 data process generated 'queries. It should allow for user-designed joins for foreign tables or fields, etc. - it should do that for you. Heck it doesn’t allow for such. It simply says - show me your license or step out of your car with hands up. In other words, it demands perfection rather than compliance, conformity rather than freedom. I’ve read about people pouting and quitting because it more and more takes away more ‘freedom’ from you that is necessary for a developer to really showcase his talents and compete readily with the rest of the world - all because of the TCA. These things TYPO3 can change so easily. TYPO3 is actually best suited for all these with some clear-cut changes to the TCA, which don’t take much.
Short-comings
Point 1
TCA has no proper way of defining tables, their fields and the relationships between them. Yes, I know I know. It pretends to. But it doesn’t know how to. Currently, it simply requires pre-definitions (that is, simply mapping - which is not even that good) - and hence you get a pre-determined outcome - which you mostly thumb-down. Looking at many extensions (a variety of them), although they lay out their tables based on the rules (which need changing also - in favor of a standard), on the inside, they later resort to defining their own rules using CRUD in their repository and other methods, disregarding and without benefiting from the already set rules. And this begs the question, who then are these rules for? And the sheer number of them is alarming. Nearly every extension out there that uses the database has some mechanism that overrides some aspect of the way the TCA works - which is ok but which is also my point - showing that the TCA’s mechanisms are not satisfactory. One can say, it’s not supposed to satisfy everyone. But if the developer were instead allowed to set the rules for their own processing, this would greatly reduce development time - after all, the developer will be processing using his own rules via the TCA. And in the absence of own rules, a best case scenario should kick-in.
Point 2
Currently, TCA is inefficient. Over the years a lot has been added to the TCA. And rather than redesign it; codes, snippets, patches and hacks have been made to it, many with hind-sight but no fore-sight or concern about their impact to usabilities or robustness. This you soon find out when you have a new table layout. Data processing using TCA is also inefficient. Retrieval of child records is also inefficient - easy and better to use UNIONs and such. When doing a multi-table query, in the repository, all children of a foreign field are returned, then stored. Without using hacks, it should be possible to return only the children you want and not all as is the case currently. Imagine having thousands of children. If you detach them, they get deleted. Storage of children is another issue all together.
Point 3
TCA has no clear separation of concerns. TCA does too many things, all lumped up together, all synchronized as one - bad bad software design. Database table definitions, their fields, relationships plus their conditions and accompanying models should have a separate definition, away from backend forms and its interpretations (for those who chose to have backend forms for either of their tables), away from data retrievals and away from data storage and storages. This would also require re-designing the form engine so one can design it as they please and have its rules in a completely separate file. By separating these concerns, the total processing time would greatly reduce (since only what is needed is included), developers can be given the ability to manipulate the data between stages, and add or remove capabilities as they see fit, among many other benefits like having the ability to develop database administration tools and such.
Point 4
The TCA has no way of declaring which of your fields in a table is the primary field and which one is the parent field it is all assumed to be uid and pid. If TYPO3 can do these, I along with many others can consider this a BIG win! Additionally, it has no way of knowing whether the parent field exists in the same table or not and how to connect to it. This by itself opens up a whole can of worms for developers who want to use TYPO3 to create unique experiences for their users and want to manage saving of data normally through the repositories. Lighten the load on the developer.
Point 5
Some of fields in the TCA configuration only work under certain conditions and might be relevant for other uses as well (based on type). Table configuration should exist completely separately all by itself together with its fields and associations.
— You can stop reading here —
Proposition
By now, I think you realize, I don’t like talking much about short-comings. I just leave them open-ended. I’ve looked at TCA related questions on the internet and there are people with a lot more to say than I do. They’re many. Can’t list all their concerns here. These are just to give you a general look and feel of the current state of the TCA and the need to conceptualize it - modularize it - modernize it. Talking about the solutions is far more exciting since the possibilities are endless. My belief is that a redesign of the whole concept of the TCA will revolutionize TYPO3 and give it an edge to adequately deal with challenges of the future, which are already knowcking at the door.
Beauty in the wilderness
There’s a location called Persistence under Extbase in the Configuration folder that defines Models and their tables. This, in my opinion, would be a perfect location to move all table configurations from the TCA files - each one of them. Perhaps a start? I don’t know. Allow it to handle table all definitions and all matters of storage and retrievals. Every configuration moved there should be subject to storage. Retrieval configurations can appear separately in the same folder or elsewhere. Table relationships and field relationships can also appear in this location. Each field can be an array of its configuration with enable/disable option, and its control feature declarations including sql for its retrievals, formatting and validation classes for its storage. Tables themselves can carry own SQL so that whenever they are references, the SQL executes in place of the generated one. These can also be provided for example in array form for the FROM, WHERE, GROUPBY,… clause for each table. Conditions for data storage can also be introduced for uses to supply their classes and if the condition return TRUE then data is stored otherwise data is not stored or retrieved.
There’s no one example that can adequately sum-up the deficiencies inherent in the TCA. But I think the following example can illustrate most of them. Imagine having just two table for your extension: Parent table and Child table. The parent table only hold the index record and relations between index records and such fields for each index as created, modified, deleted, etc. It doesn’t contain any actual data. The associated data records are stored in the Child table as title, first name, last name,… as individual records with a field to show which record is which. Unfortunately, the TCA in its present state cannot cope with this setup as it would treat the Child records as sub-ordinate to the Parent record and would treat the foreign field in the child table as the uid in the parent record when it’s not. Hard-core users would argue this, every which way - even I can, but they point being that having a table configuration defined by the DEVELOPER (not TYPO3’s internally generated scripts), separate from backend form interpretations and separate from table structure and data would greatly enhance the way we use TYPO3 - a great software due to its principles (not necessarily its processes). Backend table forms can benefit greatly from these kind of setup as well queries in the frontend.
Related areas
While we are here, I might as well mention a few related things.
SQL
Table structures are highly restricted even for miniscule tasks such as having a MYSQL generated date or hash column. Common keywords like ON or bool are not permitted also. It would be nice to have a switch in configuration extension file LocalConfiguration.php under connections or in table definition or other means by which a developer can be allowed to use these keywords, if he so chooses. Additionally, I work in tourism and so spatial keywords in MYSQL would do as great benefit if allowed, however they keep being flagged for changing whenever we analyze the database. and can’t be allowed to create them. The following examples are perfect illustrations of what’s possible with TYPO3 but won’t run under current TYPO3 database setup conditions:
modified DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
total_item_price DECIMAL(10,2) AS (quantity * product_price),
Repositories
Repositories have no way of dealing with kids (I mean children). So one has to iterate. Imagine thousands of children. I had to port a Laravel collections into typo3 (a small script) which I use for those tasks. (The collections are developed by a German company and available for free in GitHub). I don’t know if any thoughts have been given in the capabilities to work with child object of a parent field in result sets produced by createQuery function in the repository. These can easily be configured in a robust TCA.
In the Repositories, children are a joy to work with - with the know-how but are a pain when it comes to putting them to sleep (persisting them). You add the children to the parent and persist the parent. Unfortunately, the persistent manager persists ALL children - again! - a disaster. And if you exempt certain children from storage, they get deleted - dear Lord! So you can’t win with children. Isn’t that how it’s supposed to be… everywhere? Am just kidding here. But you get it, there ought to be a way to exclude a child from updating without deleted them and getting the children you want, not all of them.
What do you think? Have I summed it up well? I think these changes are long overdue.