Augmented Quality Assessment (AQuA 💦)

Translation checking has a significant impact on the quality of Bible translation efforts, but also constrains their pace. The intent of the Augmented Quality Assessment (or AQuA) project is to develop capacity and increase the pace while still enabling quality throughout the Bible translation community. Harnessing the latest artificial intelligence (AI) techniques, AQuA seeks to automatically and objectively assess multiple facets of translation quality and to validate the usability of those assessments with translation experts. You can watch this video for more context.

At the beginning of the AQuA project, technologists and researchers in SIL observed that the artificial intelligence (AI) community at large had a number of new technologies for estimating the qualities of text. These technologies could generally be categorized into sets of technologies that can estimate:

• Similarity between two passages
• Readability, or the ability needed by the reader of a passage
• Naturalness of the grammar of a passage
• Comprehensibility based on automated answering of comprehension questions
• Adequacy, or equivalency of conveyed information between two passages
• Dynamicity, or the extent to which a passage is dynamic versus literal

At the same time, consultants and translators are already steeped in a process that ensures translation quality in three major categories:

• Accuracy - Does it communicate the original message as closely as possible? A translation is inaccurate if the meaning of the translation is different in any way from the original message (omission, addition, change).
• Clarity - Does the translation communicate the meaning in a way that the receptor audience will readily understand? Do people for whom the translation is being made understand what it means?
• Naturalness - Does the translation flow fluently? Is it the kind of language that the people of the intended audience speak every day? Does it sound good when read aloud?

Notice, there is a partial correspondence between the groupings of known AI technologies (for quality estimation) and the qualities that consultants and translators care about. The overlap is not one-to-one, but it does beg the question: could some of the technologies from the AI world act as probes into the big three qualities that consultants and translators care about (accuracy, clarity, and naturalness)? If so, maybe we could use these technologies to create:

• Early warning systems for translators, which would alert translators of relevant quality issues early on (or at any appropriate time) during the drafting process,
• Power tools for consultants, which would help the consultant diagnose the accuracy, clarity, and naturalness of a translation using a variety of granular patterns, flags, or characteristics generated by the AI technologies, and
• Equalizers for administrators and strategists, which would help compare (on an equal footing) products resulting from new or proposed translation methods (such as usage of machine generated first drafts and church based translation methods)

The answer to that question might be: No, AI technologies cant usefully provide any quality estimation information to translators, consultants, and administrators. The answer to that question might also be: Yes, the majority of AI technologies can be applied with great success in Bible translation quality assessment.

However, we expect that the answer to the question is not a binary Yes or No. We expect that AI technologies will NOT give us all the quality information we desire, but the technologies will give us some partial quality information that is useful.

That brings us to the next big question: how can we determine which AI technologies provide useful quality-related information in the context of Bible Translation, and how does that information map to things we care about (accuracy, clarity, naturalness)?

The AQuA project is currently has technologists with knowledge of existing AI and computational approaches to estimating quality, paired with consultants that are experts in the particular qualities that a Bible translation should possess (and how to assess those qualities in the real world). These two groups are working together to determine which, if any, of the AI approaches to quality estimation might help probe accuracy, naturalness, and clarity.

Generally, they are trying to accomplish the following phases of work for each of the categories of AI approaches (comprehensibility, similarity, naturalness, etc.):

• Phase #1 (Gathering Domain Knowledge) - The technologists on this project simply do not have the domain knowledge of the expert consultants. They understand the capabilities of the AI technology, but they can NOT (by themselves) map these capabilities to useful ways of probing accuracy, naturalness, and clarity for a draft translation. In Phase #1, technologists will carry out focus group style discussions with the consultants to determine how the technical capabilities that they understand map to the qualities of a Bible translation (even if that is discussed in the context of existing English translations at first). The goal of this phase is to clearly articulate (in writing) how a category of AI approaches (e.g., for measuring comprehensibility or similarity) maps to the qualities that are assessed in the real world by consultants.
• Phase #2 (Developing Examples) - Technologists are used to solving problems that are defined via structured data (i.e., give me this input and tell me what the expected output should look like). Once an understanding is developed in Phase #1 about how AI capabilities might map to useful outputs (which correspond to aspects of accuracy, naturalness and clarity), the technologists will, in Phase #2, develop a data set that includes a number of input/output examples (e.g., a number of verses or verse pairs, each annotated with a numerical score for naturalness or similarity). These examples are relevant to specific aspects of accuracy, naturalness, or clarity as guided by the domain knowledge gathered in Phase #1. This set of examples can then be used in the lab by the technologists to test and score methods on the input/output examples, thus determining how well the methods can actually produce the desired output.
• Phase #3 (Developing Assessment Methods) - With the example input/output data in hand, technologists start their work of testing and scoring potential methods to see which, if any, of the AI methods can consistently produce the desired output. It is very possible that one or multiple methods might be useful in producing output that probes accuracy, naturalness, and/or clarity, or it could be that no methods are useful in producing good output.
• Phase #4 (Reviewing and Validating Assessment Methods) - Even if the methods developed by technologists perform as expected on the example input/output data, that does not automatically mean that they will perform well in the real world. In Phase #4, technologists will rely on consultants and translators to look at example output from the assessment methods and determine efficacy.
• Phase #5 (Tool Development and Field Trials) - In the end, we will not know the true utility of these methods until we create prototypes (based on the developed assessment methods) and test those prototypes with real translation teams and new data. This is the focus of Phase #5.

Please contact dan_whitenack@sil.org for more information about the AQuA project or to get involved. Thanks for reading this long intro post!