Standards Development

Scope

This document discusses the concept of task in the case of AI applied to natural language, and proposes a landscaping of the AI tasks related to the analysis or generation of natural language, as well as other language-related functionalities that are associated to those AI systems. It identifies existing and competing terminologies, co-existing variants of the same tasks and functionalities, and how specific tasks can be affected by language diversity in terms of their role or their challenges. This includes all languages, dialects and variants spoken in Europe, whether official or not. The relations among tasks or functionalities, and their interactions within pipelines, are discussed and illustrated. In addition, the document provides references to existing standards and published guidelines associated to those tasks and functionalities, highlighting their differences in case of competing standards.

Purpose

AI systems that process (analyse or generate) natural language are associated with a wide diversity of tasks and other transversal functionalities, some of which have been referred to under various names, and sometimes with the same name for different tasks. The purpose of this project is to identify those tasks and functionalities, together with their existing terminology, while pinpointing current ambiguities. This analysis also includes the identification of existing standards already associated with those tasks. Compared to existing standards, and in particular ISO/IEC 22989, this project aims at getting broader in terms of task diversity (including for instance coverage of preprocessing components), while achieving higher technical precision (removing possible ambiguities on the nature of inputs and outputs).

In the context of the upcoming AI Act, there will be a need for clarifying how generic AI requirements apply technically in the case of natural language processing systems, and possibly a need for refinement.Such standardization gaps will be the object of future projects, while this project is meant to provide the necessary language to enable that future work. Europe also has the specificity to be highly multilingual, whereas at present most of the work in the natural language processing field is dedicated to English, often without consideration of how different languages would present technically different challenges. In that spirit, the project also includes a systematic analysis of how language diversity, and in particular the European linguistic diversity, can impact the definition of the tasks or their technical specificities. This work appears important in light of the existing European efforts for Europewide linguistic understanding between countries.