TYPO3 AI Localization with Crowdin Integration

edwardlenssen1 · November 12, 2024, 7:41am

What is your idea about?
Our project, TYPO3 AI Localization with Crowdin, addresses the challenge of time-consuming and labor-intensive localization within TYPO3. Currently, manual translation of XLIFF files for multiple languages is a tedious task for contributors.

The AI-driven extension streamlines the process by automating translation tasks with high accuracy and seamless integration into TYPO3’s backend, enabling contributors to publish directly to Crowdin with minimal effort. This solution allows TYPO3’s localization team to focus on refining translation quality rather than handling repetitive tasks.

What do you want to achieve by the end of Q1 2025?

Milestones of the Project:

Stable Release of the TYPO3 AI Extension: A fully developed, tested, and stable TYPO3 AI Localization Extension available directly within TYPO3, supporting TYPO3 core extensions, TER extensions, and custom extensions.
Seamless Localization Workflow in TYPO3: An accessible, three-step localization process within the TYPO3 backend:

• Access localization options via Admin Tools > Extension Manager and select the ““AI Localization”” function.
• Choose the AI model, target language, and the number of XLIFF files, and initiate the localization process with one click.
• Proofread AI-generated translations, make adjustments as needed, and publish directly to Crowdin from TYPO3.
Detailed Documentation and Training: Comprehensive guides, tutorials, and documentation to help TYPO3 community members utilize this extension efficiently, making it easy for both experienced and new contributors to join the localization process.
Community Feedback for Iterative Improvements: Gathering input from the TYPO3 localization team to ensure the extension aligns with their needs and expectations before a stable release, allowing for iterative refinement.

Scope of Work (SOW):

• Initiate Extension: Launch and configure the extension in TYPO3.
• Support for Multiple AI Models: Integrate models such as ChatGPT, DeepL, Gemini, and more for versatile translation options.
• Extension Settings: Comprehensive customization options for users.
• Add Feature at Admin Tools > Extensions: Centralized access to AI Localization features.
• Feature 1: Start AI Localization: One-click translation automation.
• Feature 2: Manual Localization: Manual adjustments for translation flexibility.
• Feature 3: Log Localization: Track changes and review translation history.
• Feature 4: Export XLF Files: Easily export translated files.
• Feature 5: Import XLF Files: Import XLIFF files directly into TYPO3.
• Feature 6: Explore at Crowdin: Direct access to Crowdin projects.
• Feature 7: Undo Localization (optional): Optional rollback feature to revert translations.
• Feature 8: Synchronise to Crodwin: Send whole extension’s XLIFF to Crowdin
• Integration of Crowdin Project ID: Link specific Crowdin projects for targeted translations for TER & custom extensions.
• Crowdin Glossary Feature: Maintain terminology consistency using Crowdin’s Glossary feature.
• Code Review & QA Testing: Rigorous testing and quality assurance.
• Core Contribution with Hindi Language: Real-world example by contributing Hindi translations to TYPO3 core.
• Feedback & Support: Engage the TYPO3 localization team and community for input and improvements.

Technical Approach:
The extension is built using a robust AI framework that integrates seamlessly with TYPO3’s localization infrastructure and Crowdin. By automating the end-to-end translation process, it minimizes manual touchpoints while ensuring that translations align contextually with TYPO3’s language standards.

Beta Version Ready for Testing:
The TYPO3 AI Localization Extension is currently in beta, fully functional and ready for hands-on testing. We invite the TYPO3 localization team to explore the extension and provide valuable feedback. This review phase is essential, as it allows us to fine-tune the extension’s capabilities and ensure it meets the high standards and specific needs of TYPO3’s multilingual community before releasing a stable version. Your insights will help shape a tool that truly supports TYPO3’s expanding global reach.

What is the potential impact of your idea for the overall goal?
This TYPO3 AI Extension has the potential to greatly enhance TYPO3’s localization capabilities and contribute to TYPO3’s global mission:

• Increased Localization Efficiency: AI automation reduces the manual workload, enabling contributors to work faster and concentrate on improving the quality, consistency, and contextual relevance of translations.

• Scalable Localization Process: By minimizing repetitive tasks, the extension allows TYPO3 to support a broader range of languages, making it more accessible to users worldwide. This scalability encourages TYPO3’s global adoption by communities across different linguistic backgrounds.

• Enhanced Community Involvement: This simple and intuitive workflow within TYPO3 encourages more community members to engage in localization efforts, further broadening TYPO3’s reach and accessibility.

• Long-Term Global Accessibility for TYPO3: With this TYPO3 AI Extension, the community can more easily support new languages, bringing TYPO3 to diverse audiences and enhancing TYPO3’s position as an inclusive and globally accessible platform.

Conclusion
By supporting this project, TYPO3 can harness the power of AI to transform localization, benefiting both contributors and end-users around the world. We invite the TYPO3 community to vote for this project, empowering us to bring the TYPO3 platform to even more languages, cultures, and communities.

Thank you for considering our proposal.

Which budget do you need for your idea?

5.000 Euro

Please note: After the start of the voting we can not change the idea description nor the idea outcome. If this idea is selected by the members, it must be archived as described.

just2b · November 18, 2024, 4:15pm

I am not really sure how this AI integration can look like. From my opinion the way to go would be to implement the so called In-Context Localization and not to implement anther AI tool chain.

The main reasons are:

Only by using the tools of crowdin can use the translation memory of all translations over all projects including extensions and core
There are a lot of features of crowdin which could be used more easily to help translators like providing screenshots

In the beginning of crowdin & TYPO3 I started an extension which is available at GitHub - georgringer/crowdin: Crowdin integration in TYPO3. Finishing the project was not that important for the TYPO3 Association and sadly I don’t have any private time left to do it but IMHO this would be the way to go.

If we really want to integrate AI tools for improving the translation status, we can add machine translations on crowdin (see Machine Translation Tools | Crowdin Marketplace) already and all translators would benefit from that.

nitsan · November 19, 2024, 6:33am

Hey @just2b

We deeply appreciate your thoughtful feedback and your willingness to share insights on this topic. Your experience is invaluable, and it helps us refine and align our approach with the community’s needs.

Addressing the Pain Point: Manual XLF Label Editing

Our main goal with this initiative is to solve a common challenge—manual, repetitive editing of XLF files, which we’ve found to be a pain point for many translators. Tools like In-Context Localization (backend one-bye-one label editing) and Machine Translation Tools (manually editing each XLF file) have similar limitations in terms of speed and efficiency. Our custom AI solution aims to eliminate this bottleneck and provide a faster alternative.

Exploring Crowdin Features for a Complete Solution

We completely agree with your points about translation memory and screenshots in Crowdin. These are powerful features, and we’re committed to investigating how they can be integrated into our solution to enhance its functionality further.

A Simplified, Three-Step Translation Process

What makes our approach unique is its simplicity:

Choose the Extension: Select the XLF files you want to translate.
Bulk Translate: Use AI-powered bulk translations to quickly process the labels.
Proofread and Synchronize: Fine-tune translations and sync them with Crowdin in a single click.

This streamlined process is designed to make translations faster, more efficient, and more accessible.

Invitation to Join the Localization Discussion

We’ll be discussing this initiative with the Localization Team during the TYPO3 Community Sprint on November 21. Your presence and expertise would be incredibly valuable to this discussion, and we’d love to incorporate your feedback into the project.

Upcoming Prototypes and Interactive Demos

We understand that without a prototype, it might be difficult to fully visualize the feature. Following the sprint, we plan to share detailed feedback from the Localization Team, along with interactive demos and screencasts, so the community can better evaluate the tool in action.

Together Towards an Innovative Solution

This initiative aims to innovate and make translation easier for TYPO3 contributors, and we’re committed to working closely with the community to achieve this goal.

Thank you once again for your input and for engaging with this proposal. We’re excited to collaborate further and improve the translation experience for everyone!

Cheers,
Sanjay

just2b · November 19, 2024, 6:51am

Hey,

Thanks for your answer!
I see some problems with translating technical stuff without context. As German I have quite a few challanges translating strings even when I know the backend by heart. Throwing more AI into that just makes the number of translations better but not the quality.

IMHO translators don’t have problem with xlf as this is only the format and there are lots of tools out there, especially for professionals.

I agree that a core module could be written which can do such things but there are already extensions for that use case.

To really improve the quality we need in context localisation combined with the translation memory

nitsan · November 19, 2024, 12:55pm

Hey Georg!

Thank you for sharing your valuable insights. We truly appreciate your focus on quality, especially when translating technical content where precision is essential.

1. AI Quality vs. Quantity

We fully acknowledge that translation quality is a critical concern, especially when dealing with technical content. However, AI models such as GPT4-o have made significant strides in understanding context and improving translation quality. Unlike traditional machine translation tools, these advanced AI models can generate more nuanced, context-aware translations.

You can explore how modern language models are reshaping localization:

• Large language models vs machine translation

• Better language models and their implications

2. Improving Translation Quality with Context

We understand that ensuring translation quality is vital, especially in technical documentation. That’s why our Prompt Engineering team is actively working on refining context-driven prompts to guide AI in delivering the best possible translations. By creating more accurate and relevant prompts, we aim to significantly improve translation quality.

3. Integrating Crowdin’s Glossary for Consistency

We plan to integrate Crowdin’s Glossary feature into our solution. This will allow us to ensure consistent terminology across translations, making sure the technical language remains accurate throughout all projects. This is a key aspect of improving translation quality and maintaining consistency in large-scale localization projects.

4. Addressing Gaps in Existing Extensions

Regarding existing extensions, we acknowledge that there are some useful extensions out there, but they often fall short in two critical areas: support for TYPO3 core extensions and seamless one-click synchronization with Crowdin. Our initiative is designed to bridge these gaps, offering an end-to-end solution that not only supports the entire TYPO3 ecosystem but also simplifies the workflow for both translators and developers. This will improve overall efficiency and ensure a smoother experience for everyone involved.

We truly value the feedback from the community and professionals like you. We are committed to ongoing improvement and would love to continue working together to refine this approach.

Cheers,
Sanjay

pkraume · November 21, 2024, 3:26pm

A proof of concept extension has been presented to the TYPO3 Localization Team today by Sanjay Chauhan from Nitsan. The extension is working and the usability for the backend users is easy. The translation results can be edited/proofread and then be stored locally and optionally transfered to Crowdin, the official TYPO3 translation handling tool.
The tool will not only work with TYPO3 Core but any extension available in Crowdin or outside.

We have not yet been able to assess the quality of the translations. We assume that the quality will vary greatly from language to language, especially if the AI tools used in the language have few references to TYPO3.

The release process with proofreading in the Crowdin web interface is deliberately not covered by the extension. This still requires native speakers. They are responsible for the quality of the translations.

The extension supports TYPO3 versions 11 to 13, but it is unclear if and how it will be adapted to future TYPO3 versions.

Advantages:

TYPO3 can be quickly and easily translated into different languages.

Disadvantages:

Using AI tools costs money. These costs must be covered by the users of the extension.
Crowdin could be flooded with many translation suggestions. It is not yet clear how Crowdin will deal with many different accounts submitting identical or very similar translations.
Proofreaders could be tempted to accept translations without checking them, which could damage TYPO3’s reputation.

If the TYPO3 Association approves a budget for the project, the following specifications should be made

The extension must be published in the TER
The source code must be publicly available on GitHub or elsewhere.
There must be no paid premium version of the extension
Documentation of the extension must be published on docs.typo3.org

nitsan · November 21, 2024, 4:17pm

Hey Peter and Localization Team,

Thank you for your detailed feedback after our TYPO3 AI Localization with Crowdin Integration demo. It was a pleasure to present this tool to the TYPO3 Localization Team, and we appreciate the opportunity to address some important considerations.

1. AI Translation Quality

We understand your concerns about the quality of AI-generated translations. We are working on improving the accuracy, especially for TYPO3-specific terminology, through prompt engineering. Additionally, we will integrate Crowdin’s Glossary to further ensure consistency and quality across all translations.

While AI can improve efficiency, we agree that human proofreading remains essential to ensure translation quality. The goal is not to replace this process but to accelerate it, allowing translators to focus on refining and adjusting translations rather than starting from scratch.

2. AI Tool Costs

In terms of costs, we’ve found that AI tools such as OpenAI’s GPT models are quite affordable. We’ve processed thousands of translations without exceeding a $25 charge, which we believe is a reasonable investment for faster translation. As we continue to scale, we will gather more data and share clearer figures with the community, but we’re confident that the cost won’t be prohibitive. One more thing, the TYPO3 association could spare a small budget and can set up a platform where translators can use the platform and contribute to the TYPO3 localization who are really concerned with the AI model’s cost.

3. Proofreading Process

The proofreading process will remain as it is with Crowdin today. Moderators and administrators will continue to play a key role in reviewing and approving translations. Our extension does not aim to change this workflow but instead speeds up the process of creating initial translations, giving translators more time to focus on quality improvements.

4. Crowdin Contribution and Moderation

We acknowledge that, as more people use this tool, we may see a greater volume of translations submitted to Crowdin. This is an opportunity for the TYPO3 localization community to grow, and we expect that more moderators and administrators will be needed to manage the process. The goal is to make it easier for people to contribute and expand TYPO3’s reach to more languages while maintaining high standards.

5. Support for Future TYPO3 LTS Versions

Rest assured, we are committed to supporting future TYPO3 LTS versions. We will ensure that this extension is updated for every new LTS version of TYPO3 as it is released. Additionally, we will continue to maintain the extension by providing bug fixes and security updates for all minor TYPO3 versions. This approach guarantees long-term compatibility with the TYPO3 ecosystem.

5. Commitment to TYPO3 Open Source and Future Updates

We are fully committed to making this extension available to the TYPO3 community, regardless of whether we secure funding. Our intention is to keep this project open-source and free for the entire community (at TER, Github, Docs etc). If we receive the budget, we’ll be able to make further improvements and ensure the tool is updated for future TYPO3 versions.

Thank you again for your support and valuable feedback. We’re excited about this project and look forward to continuing to work with the TYPO3 community to improve localization contribution for everyone.

Furthermore, As we discussed, Let’s keep stay connected on Emails & Slack

Cheers,
Sanjay