Migrating from Pootle to Crowdin

Archive Discuss Topics
1

The multilingual backend has always been a great benefit of TYPO3. Since 2011 the software pootle has been used on translation.typo3.org to provide translators an interface to translate labels of TYPO3 core and extensions.

About Crowdin

Crowdin is a cloud-based solution to localize content. It offers a free plan for open source projects and is used by multiple other well known projects like Joomla! CMS, Neos, PrestaShop or Magento. The showcases can be found at Projects search - Crowdin.

The team of Crowdin is very supportive and is highly interested in providing a good solution for the TYPO3 community.

The proof of concept can be found at Crowdin

Benefits

The following benefits have been identified. The UI (User Interface) and UX (User Experience) follows current web applications you already know. You will find well known and used features like login via Google/GitHub and it is easy to use.

Support During Translations

Crowdin supports translators during the work by providing state of the art features:

  • Translate the same text used in different versions and parts of your product only once
  • Machine Translation - is a great assistance for human-translators. Work goes faster with translators post-editing translations suggested by machine.
  • A glossary can be used to make sure specific words are properly translated (e.g. Template in german, TypoScript or SEO)

Inline Translation handling

While translating text into a different language it is sometimes very handy to know how and where this string is actually used in the backend.
By using the inline translation handling of crowdin it is actually possible to translate TYPO3 in the backend.

In-Context Localization
In-Context Localization1280×1024 51.3 KB

Providing Screenshots to Translators

Along with the in-context feature it is also possible to use screenshots and tag the source strings to them. This way translators get additional context for some strings that aren’t really easy to find in the ui or alike. See Screenshots | Crowdin Documentation for more information about that.
The following screenshot is shown when the string “Edit and Advanced functions” is getting localized.

Crowdin Screenshot example
Crowdin Screenshot example1948×1174 71.3 KB

Less work on our shoulders

The current translation server has been mainly maintained by Xavier Perseguers (thanks a lot for your work) and the server team. Switching to a SaaS solution enables us to concentrate more on the actual work of translation.

Risks

The following risks are currently known:

  • Extensions are not supported anymore for phase 1. Currently those are beeing migrated to semi-automatically to Crowdin
  • Instead of using self-hosted software a cloud-based solution is used. As the translations are exported daily there would be nothing lost if the service won’t be some time not available anymore.
  • Currently XML export format is not implemented
  • Not tested enough

My plans are to support TYPO3 9 & 10 and keep the current translation packages on the server for all older installations.

Next steps

Near future (Q4 2019)

I have dug into the API of Crowdin and TYPO3 and got a working solution already which covers the following parts:

  • Add the source strings (the text in English) from Git to Crowdin
  • Download translations from the current translation server and upload those translation at Crowdin
  • Download translations from Crowdin and create zip files which can later be consumed by TYPO3 sites.

Of course those APIs need to be stabilized.

To ease the switch to a new infrastructure I propose a feature switch in TYPO3 Install Tool which would allow users to test the new packages on sites.

Far future (Q1/2 2020)

For the start it is not possible anymore to fetch also extension’s xlff files from crowdin. However the plans are to create projects for extensions. The advantages would be huge as this would allow extension authors and the translators to reuse the translation memory of TYPO3 CMS itself.
If an extension author is eager to use crowdin already, it is of course possible to create a custom project. EXT:news and tt_address uses this already, see TYPO3 Extension news dashboard in Crowdin.

Decisions which are not part of this migration

The following topics need to be solved as well but are not but of this migration:

  • Future of CSH (Context Sensitive Help)
  • Unify tons of xlff files and reducing duplicates
  • Deprecate the usage of XML files
  • Support of plurals
  • How to attract more people contributing localizations

Time plan

  • 09/2019 Decide about using Crowdin or not
  • 10/2019 Stabilize API & Make required changes to TYPO3 CMS
  • 11/2019 Testing
  • 01/2020 Public start of localization
  • 04/2020 Using the new API

Organizational

Topic Initiator: Georg Ringer
Topic Mentor: Georg Ringer

2

As I have already tested translating ext:news in Dutch on Crowdin, I can only say it works quite nice to translate. As long as we stay in control of the translated files (and I know we do), this is a great step forward to make it easier for people to translate TYPO3.

So a +1 for me

3

I worked with Crowdin a couple of years ago. I really like the UI for translating. Translation suggestions are really good too. It’s really fun to translate – and fast.

Monster feature

The inline translation handling within the TYPO3 backend is THE monster feature.

Forger Leaderboard

With the upcoming crowdin UI, gamification will be also introduced. Based on this upcoming data AND the motivated Crowdin team, we can probably also take this data and can use it for the new upcoming forger feature.

4

Hello,
I’m really glad to read about this feature request. I’m also aware of the past work and decisions from the Serverteam and Xavier Perseguers.
In the past years I talked a lot about this proposal with Preben Rather Sørensen, who is keeping danish translations up to date and has currently the most submissions on https://translation.typo3.org.
In 2018 he already started a decision to ask for improving translation handling for TYPO3.
https://decisions.typo3.org/t/a-future-for-a-translated-typo3/344
Since then, we gave up trying to ask or involve someone to improve the current handling of translations for TYPO3 core and extensions, cause of no response.
Unfortunately, setting up an initiative was not an option for us, cause of missing consist with other people.

I would appreciate to have an Initiative which will help to switch to such modern translationtool.

Currently it is not smart to keep translations maintained.

5

I wasn’t aware of this posting, at least now there could be a solution :wink:

6

I think I will need help on how to use it, but the inline translation seems really useful! +1 for me

7

Hi there, thanks Georg for opening this discussion we started few days ago here.

As you know I’m basically the only one to take care of the TYPO3 translation server for many years.

For the history, the switch from ll-XML (what Georg refers as “XML files”), that is locallang.xml, is something that was the official l10n format until TYPO3 4.5. Prior to that we even had a PHP-based format.

Starting from TYPO3 4.6 the official format became XLIFF which is also XML based but just something that is officially known as a l10n file format.

Our translation server is still supporting ll-XML format but since pre-TYPO3 4.6 usage is now totally obsolete, this is something we definitely don’t need to worry about anymore!

Key features we currently have with our (Pootle-based) translation server:

  • Central place for every extension, everyone may ask to get his/her extension there, I only put some “restrictions” because Pootle was never meant at handling that many subprojects and translation files and each and every extension added made a migration to a new version harder and harder. But apart from those gentle restrictions, I responded to every request to add a project there
  • Connection with TYPO3 for users. Although, yes, Pootle chose to drop support of LDAP and thus prevented us from having a real SSO like we had back then. This is something that is actually not “really” needed anymore if one may connect with his/her GitHub account or alike, even with dedicated account.
  • Project lead developer was granted full “admin” on his/her extension. This is something that I added over the years but this allowed the developer to accept suggestions in any language for his/her own project and choose who would be able to do so as well. This is something I consider important when moving to another system
  • Support for multiple versions of TYPO3, with labels being added, removed, … and files being moved around. This is something that has some known limitations, but should be supported with another solution. It should even be enhanded in regards to what we know is “needed” nowadays. I would even suggest to think about enabling this feature for any extension.

I won’t discuss here why we did not upgrade to the latest version of Pootle yet, long story, nothing easy and nobody to blame for. What’s important is that Pootle did not evolve “much” during the years and had to face even a big step backward (I can very happilly explain more on that but this is not the purpose of that post). In the end, even latest version of Pootle would not be “much better” than what we have currently, period.

What I’d like to see with a better alternative (no order):

  • Shared translation memory. Translating something in Core or extension X should be usable to translate something in another file/extension. This is only possible in Pootle within the same XLIFF file of a given project. So even similar labels (think of locallang.xlf and locallang_db.xlf) cannot benefit from that in Pootle
  • Integration of Deepl translation engine (possibly dreaming) or at least external engine such as Google Translate for helping/suggesting
  • Inline editing in Backend (Georg has a PoC for that it seems, hopefully it works/could work for any extension, not only TCA and Core)
  • Inline editing in Frontend (extend Fluid’s TranslateViewHelper so that one can do the same in plugin’s output, if enabled)
  • Easy export/import if one wants to work with external translation agencies
  • Full import of current Pootle install to the alternative, I happily help here of course
  • Support for GitHub (and possibly alike) projects, allowing to prepare the translation of upcoming release. Something that is not currently possible because I programmed Pootle update to base on TER releases
  • Integration with extensions.typo3.org and alike, it should be straightforward to contribute to translation of any extension
  • Think of possibly encompassing translation of documentation (don’t even know if this is still supporting by current docs.typo3.org but it was and this is something important since TYPO3 should not be English only if we want to grab more people). I can help with knowlegde about do’s and dont’s by personal experience on that topic.

Not directly related to the change but what I’d like to see in future:

  • Unified l10n for Backend/Frontend/JS in TYPO3
  • Support for plurals in our API (a feature available in XLIFF but that we never implemented, unlike Neos)

All in all, I agree that something “new” is needed. I’m not sure this is the way to go, but I’m naturally not against it. I’d just want this post to be taken seriously as finding a technical solution is always the easy and appealing part, but supporting it for many years and ensuring it is actually viable without sacrificing available and highly useful features is quite more challenging :slight_smile:

8

Thanks Xavier for your public post! I know how lonely it can be sometimes and really appreciate your insights. I want to give answer to some points

This is perfectly possible

Possible, see https://support.crowdin.com/configuring-machine-translation-engines/

This works for any translation in the backend and also for extensions.

Not tested that but that should work already as well by just adding some additional JS

Works. I currently also use an API to fetch translations from translation server and upload that to Crowdin. You can also drop a zip file with the xlf files at Crowdin

I will certainly need your help to finish my API!

Works out of the box already. Crowdin pushs back translations as pull requests

This will be solved. Crowdin is currently finishing their beta version which would be a perfect fit with lots of nice features.

Haven’t thought about that yet

I also got another question:

  • Yes, it is all based on xliff, so no change there!
9

I’m fan of this suggestion!

For some years, I’ve been involved in translation work. Not only TYPO3 but also other OS (Magento, Piwik/Matomo, Atlassian-stuff, etc), and translation work for customers in general. My general approach is and has always been, that every good piece of software should ‘speak’ Danish, if used by Danes.

I’ve tired a good handfull of translation tools and to me, the Pootle we use on translation.typo3.org has a lot of missing features.
I only know Crowdin through my Magento contributions but it has a lot of the same functionality and features as Transifex - in general both modern and mature tools.
As mentioned above by Mr. Jokumer, I did an overlooked attempt to raise awareness on the lack of features in our Pootle some month ago, and from my post, I would like to emphasise a few missing feature:

  • A list of files that I take care of. On the Pootle I take care of core and 10-12 other extensions. The number of ‘projects’ are around 350 on Pootle - most of them obsolete or unmaintained btw.
  • Notification, when labels are added, when suggestions are made and new projects/extensions added.

I don’t know if Crowdin will support it in some ways …

And to raise another issue, that we need to focus on too: No matter what tool we choose for helping us making great translation, it’ll be important how we communicate and share the daily work between us. How do we decide, review and proof? How do we organise the translation work? I’ve seen a lot of machine translation - and human can do machine translation too - and we need a way to qualify the work we do.

I’ll (still) be happy to help and support

Go on Georg!

10

@xperseguers:
Support for GitHub (and possibly alike) projects, allowing to prepare the translation of upcoming release

@just2b:
Works out of the box already. Crowdin pushs back translations as pull requests

Just to make it clear, general use case (I know you tricked it with EXT:news, at least to test, and that’s fine but should not become the way to go) should not be to store localization into the extension itself, it should be kept as it is now, separate from the extension.

Current update workflow for Pootle (https://translation.typo3.org) is to regularly check for new releases in TER for each and every registered extension and if so download the latest zip and use that as the new “source” (= English) version to be translated. Existing translations are retained of course.

What I suggest, if possible, would be an option so that the master branch of the corresponding extension be already available for translation prior to the actual release. We could make it an option with some special file in the repo or alike. Purpose is to possibly have a translated extension on day 1 of the release. This is not needed for TYPO3 Core since we have intermediate releases and in fact we even connect directly to the Core git repo. But for extensions, this could be an interesting option.

Not a must have, but definitely a nice to have with some enhanced system.

11

To get everybody on same level. There are 2 ways to ship localizations for extensions

  • Use the translation server
  • Ship the localizations with the extension

The 1st is the correct one, the 2nd the more used one, e.g. powermail is using that and of course all extensions which are not public.

Crowdin supports 2nd out of the box. For final integration extension will connect Github/Gitlab/… with Crowdin and can localize any branch and the localizations will be distributed as used from translation server.

12

I am against this switch, but not for technical reasons. The product itself looks good, but:

  1. Crowdin is, as far as I can tell, completely for-profit but offer opensource / educational licenses. This model looks like the one used by Slack to gain traction. The problem is, it puts us in bed with a privately owned company which now benefits from us saying “Want to translate? Get a Crowdin account” just like when we said “Want support? Get a Slack account”. The old saying, that when the product is “free” that almost certainly means that you’re the product. I believe this holds true here.
  2. Crowdin terms of service retain rights for Crowdin to use the data for “research” or “educational” purposes. I don’t like my data being used for research or education without consent and I would not be able to withdraw such automatically-given consent. If I don’t consent, I cannot translate.
  3. Crowdin applies its own content licensing terms on top of whichever terms are set by the client (TYPO3). I’m not even sure this is legally compatible with GPL (which does not allow changing the license).
  4. Crowdin retains ownership over the comment track associated with the translation process, much like Slack retains ownership over the discussion history. Should we ever want to stop using Crowdin, I can only assume that we would not be able to take our history with us, although we would be able to take the most recent state of the translation files with us.

For the above reasons I believe it is an unwise idea to get involved with a company whose strategy appears to be the strategy of Slack HQ. Which I am against for the very same reasons.

It’s all about who owns what and who’s allowed to do what with the data. In that context I do not feel safe choosing a privately run for-profit company.

These are just my two cents.

PS: I would appreciate disclosure about how Crowdin reached us (did they reach out? Did you? Why and how?)

13

We are currently checking all those questions and will get here with feedback

14

I made some test translations with crowdin in the past days and wanted to share my 2 cents.
The first noticeable difference compared to TYPO3’s translation server is it’s speed. Crowdin is a lot faster.
And remembering my first steps with pootle, figuring things out in crowdin was a piece of cake. I found it really intuitive to use.
And if this inline translation in the backend really works (that was not a joke, right?), it would be a dream come true.
So, from a UX point of view, Crowding would be a welcomed change.

I understand that the technical aspects of integrating and automating stuff with TYPO3 can be worked out. On my part, as an occasional translator and extension developer, I don’t mind to change my workflow or compromise in other areas.

The concerns raised by Claus are not to be dismissed though. In the end, it comes to the point of what compromises TYPO3 and it’s contributors are willing to make.

15

Crowdin gives me mixed feelings. It’s yet another external service we want to rely upon. For some services it would be inconvenient if they shut down / get acquired by others / change their plans, but it would not stop people from using all features of TYPO3.
Translations are somewhere in the middle of the spectrum. We would still offer the downloadable packages but making translations would not be possible for a while if Crowdin would stop.

The fact that it doesn’t have an SSO with the t3o LDAP is another inconvenience. Pootle has the same problem but at least the accounts are managed by ourselves and we don’t need to accept the terms of an external service. If you don’t agree with the terms of crowdin you simply wouldn’t be able to do translations for TYPO3.

Support for translations of extensions is a must have for me. If extension authors would have to include the translations in the extension then the community needs to rely on the releases of that extension for updates of the translations. The beauty of the translations in TYPO3 is that they are separated from the code releases. As soon as translations are available in automatically created packages they can be used by all installations in the world.
It’s hard enough to get translators so we need all translations (core + extensions) in one place.

To make doing bulk translations easy the interface needs some features:

  • It must be easy to find the translation suggestions to approve/disapprove/change the suggestions
  • There must be keyboard shortcuts for all these actions
  • It would be very convenient if one could select a branch/version to which the labels belong (translating labels for TYPO3 6.x has a lower priority than labels for v10)
  • TYPO3 allows to re-use labels in new major versions. If the source text of a label has changed the interface must show that and these changes must be easy to find for translators
  • the tree with language files is nice but really not very useful. For translations it’s rather irrelevant in which file the label is located

In context editing is nice. There will be plenty of labels that are not easy to spot and which only appear in certain conditions (error messages, flash messages, etc.). Also labels that are used in e.g. JavaScript, title attributes will be hard to translate with in context editing. In context translating will probably produce more suggestions (also for corrections), but for people who do translations in large numbers it will be less important.

Xavier’s list is something I agree with.
Wherever someone writes “integration with ” this must also include other public repository services (such as, but not limited to GitLab, BitBucket, BoilingCloud, repositoryhosting).

Although Pootle is far from ideal I don’t see Crowdin as a perfect replacement at the moment. It certainly needs more features that we really need. The beta status, the privacy concerns that were raised and the dependency upon a third party cloud service need to be thoroughly looked at. For that dependency it would be nice/essential to have a backup plan. If we won’t be able to use Crowdin for whatever reason it’s not an option to go without any translation possibilities for a long period of time.

16

Hey,

now feedback to all statements:

Of coourse Crowdin is a not a non-profit company but IMO there is a no chance to have a product like crowdin being completely open-source. It is already hard to support all my extensions properly…
Yes I know the quote about being the product but you can’t compare Slack to Crowdin as in Slack all information is closed source but in crowdin the main benefit are the translations which we will extract on a daily basis. Some official feedback from Crowdin

We indeed will benefit from having Typo3 among clients. It increases brand awareness and market penetration.
I assure we won’t ask Typo3 or any community member to pay for Crowdin as long as it is open project. Though, if any member from Typo3 community will decide to use Crowdin for another, commercial project this new project will follow our pricing policy
Concerning data ownership, Crowdin strictly follows GDPR regulation and more. Below is the short list of things Crowdin returns to customer per request along with the GDPR’s “right to be forgotten”

  • Account Information
  • Single Sign On connections information
  • Orders History information
  • Crowdin Notifications information
  • Conversations information
  • Configured Workflows information
  • Glossaries
  • Translation Memories
  • Project Managers information (what is visible to project owner)
  • Project Activity Feed
  • Tasks information
  • Project file and meta information
  • Files
  • Translations
  • API Logs
  • Strings meta information
  • Translators information (what is visible to project owner)
  • Screenshots & meta information
  • Configured Integrations information
  • Discussions threads
  • Comments information

and more

That is has been a misinterpretation, again a quote from Crowdin:

the section 12.2 explains how our users shall treat the content owned by Crowdin, that are any marketing texts, brochures, white papers and the like. I agree we should make it more clear from definitions what is Client Data and what is the Content owned by us.

Yes if Crowdin would stop we would need to switch to a new tool. Same if a really better tool comes up.
However the thing is: Currently we have the situation that really much is broken on our translation server and we are far away from having a CMS which is fully translated in various languages. So currently it is nearly impossible to provide translations on our current infrastructure too!

True but on the roadmap. To increase the user base it is IMO important to have a SSO not only with TYPO3 but with all other tech systems like GitHub. An example: The CEO of Crowdin would have liked to add feedback here but currently it is impossible to fully register at typo3.org because the users need to be enabled manually…

Support for extensions will be provided. Currently you need to ask Xavier to setup the extension on Pootle. In future this will be done by API

Currently this is not possible, so for extensions you need to release first and then you see the new sources on pootle. With Crowdin it is possible to connect it via CVS or API and translate first and release afterwards!

Just picking this, currently this is not possible.

I haven’t yet looked up that but IMO that should also still work within JS. The funny thing is: this is just an additional feature we can use. This is of course also not possible with pootle and you also don’t need to use it with Crowdin.

I don’t get it, please collaborate: Which features does Pootle provide which are not available for Crowdin?

We can certainly think about pushing the xlf files also to a git repository which would be a nice idea. However I think that the current situation is a no-go as we have a kind of broken system which is hardly maintained and we have no contributors there.

Regarding the question who approached whom: I tested Crowdin, liked it, asked the support tons of questions and just started using it for my extensions. This is how I came up with it and started playing more and more with it.

17

I understand you’re mistrusting companies like slack and so do I but I’d like to ask what’s more relevant here: Having an easy way to attract people to translate labels or to have a detailed history of why something has been translated at all, by whom and when? In the end, every single translation is a step forward to possibly gain traction in non german (and non english) speaking countries.

To me, this is the usual hen-egg issue. Do we first need to encourage people to use TYPO3 and let them do the localization or do we make the translation process as easy as possible, have a specific set of translations for every language (don’t know how atm) and attract users that way?

Let’s have a look at chinese, (brazilian) portuguese, hindi and other languages of emerging countries/markets. We have very few localizations for those languages and – except India – I don’t think those countries are known for their love for english.

To sum this up: I am not against pootle and I’d love to have a great open source solution but if there is a company providing a better tool for now that could really help the project (especially by machine learning and what not), I think it’s worth doing the switch.

18

I’d like to question this ever repeated argument in favor of SSO via typo3.org. There are 3 or 4 services in this world you can sign on/log in with your typo3.org account while there are thousands of sites you can sign on/log in with your github, google, facebook or whatever account. So, please help me here. Why is it relevant at all to have a SSO via typo3.org?

To me the much bigger inconvenience is the inability to use my github account to sign on/log in at typo3.org and all the services we have. Take a look from the outside and imagine what people out there are used to that are getting in touch with the TYPO3 ecosystem for the first time. It’s like to take a trip down memory lane. And a few of the services we use are just cumbersome.

Please don’t get me wrong. I am used to our environment and once you are, it’s fine. SSOing into gerrit is very helpful. But if you don’t have a typo3.org account, what’s the reason to create one in the first place? To be able to SSO into 3 services? I bet, we would have more users on gerrit if we allowed for a SSO via github. If you visit https://review.typo3.org/login/ for the first time, you don’t even know what kind of account you need. We can’t make it harder to get people aboard…

At Crowdin, I can sign in via github. What a relief! TYPO3 desperately needs contributors – in all areas – and the first impression counts. We need to spoil them instead of making them create a typo3.org account and get used to our services with little or no help.

So, to me, not being forced to have a typo3.org account is actually a +1 for crowdin.

19

Legal stuff. We need to make sure that users have agreed to our terms of service: https://my.typo3.org/privacy-policy
Not sure if that is needed in the case of translating labels, though…

However, I agree that LDAP should not be the only possibility to login with a typo3.org account. I’m thinking about having oAuth as an alternative for external services… Would this work with Crowdin?

20

I don’t want to open a huge discussion about a totally different topic but I doubt that this is the reason we need people to have an account. At least not for our external services like the translation server and gerrit. Like you mentioned yourself, you are not sure about the translation server. And reading the privacy policy, I see lots of stuff that is needless to state with GDPR as I have the rights given by the law, not by a consent. However, I am not an expert in this area but I seriously doubt that an account is needed to agree to the privacy policy that imho doesn’t affect external services at all. And if so, I’d like to know what my personal data is used for on those external services. If nobody knows, someone should file a request for personal information according to GDRP, so we can talk about facts (in another thread :wink:).