Standards Development

Scope

This document describes the framework of knowledge enhancement for pretrained machine learning models, and also provides guidance on methods of knowledge enhancement.

Purpose

Pre-trained machine learning models which are trained on text or multimodal corpus via self-supervised learning method, have yielded promising performance on various tasks in conversation, content generation, content processing, machine translation, Natural Language Processing, etc. Pretrained machine learning models with many parameters can effectively possess rich knowledge learned from training text. They can benefit downstream tasks at the fine-tuning stage However they still have some limitations, such as poor reasoning ability due to the lack of external knowledge.

 Knowledge enhancement for pretrained machine learning models can further improve the performances of models, espcially on industry specific tasks, based on the existing knowledge base from enterprises in that industry. For example, the knowledge base can provide industry or domain knowledge for the industrial application of the general or basic pre-trained model, to make up for the lack of professional knowledge in the corpus of the pre-trained model, e.g., finance, medical care, transportation, manufacturing, etc. The knowledge base can be used to design and incorporate specified constraints into the pre-trained model, to control the content generation appropriately, and improve its adaptability in industry application scenarios. In addition, knowledge enhancement provide feasible ways to protect private knowledge of the enterprises and integrate the existing knowledge base and the pretrained machine learning models.

Knowledge enhancement for pretrained machine learning models includes knowledge enhancement on pretraining of models, knowledge enhancement on supervison fine-tuning of models, knowledge enhancement on alignment fine-tuning of models, knowledge enhancement on reasoning of models, knowledge enhancement on output of models.Although several service platforms offer proprietary solutions based on the knowledge enhancement on pretrained models, no standard has been developed to ensure interoperability of such systems. This proposal aims to fulfill that role, by the framework it aims to develop.

Presently, ISO/IEC 5392:2024 Information technology — Artificial intelligence — Reference architecture of knowledge engineering has been published, which defines a reference architecture of Knowledge Engineering (KE) in Artificial Intelligence (AI) and is mainly focused on the construction process of knowledge bases . ISO/IEC AWI TS 25258 Information technology — Artificial intelligence — Hybrid AI inference framework for AI systems provides effective guidance for an AI inference framework for AI systems with multiple tasks to generate output in the deployment process. It involves some content on knowledge inference. By comparison with ISO/IEC 5392:2024 and ISO/IEC AWI TS 25258, this proposal is focused on the application of knowledge bases after construction and knowledge inference methods to enhance the proformace of pretrained models.

In addtion, ISO/IEC TS 4213:2022 Information technology — Artificial intelligence — Assessment of machine learning classification performance, ISO/IEC 23053:2022 Framework for Artificial Intelligence (AI) Systems Using Machine Learning (ML) and ISO/IEC AWI TS 42112 Information technology — Artificial intelligence — Guidance on machine learning model training efficiency optimisation are related to machine learning models in ISO/IEC JTC 1 SC 42. Resource Description Framework (RDF), Resource Description Framework Schema (RDFs), RDFs-PLUS, Ontology Web Language (OWL), Protocol and RDF Query Language (SPARQL) and ontology-related theories/standards provide solid foundation of tools and theories in the aspects of knowledge representation and knowledge modeling. Some related standards have developed by some organizations, such as World Wide Web Consortium (W3C), ISO/IEC JTC 1 SC 32, ISO/TC 211, ISO/IEC JTC 1/SC 29. However, the framework of knowledge enhancement for pretrained machine learning models and the related activties has not been discussed in these committees. This creats a number of difficulties for pretrained machine learning model developers, KE developers, integrators, users and other stakeholders.

The proposed document aims to define the framework of knowledge enhancement for pretrained machine learning models. It will provide a basic picture and main activities of the different enhancement types, including knowledge enhancement for pretraining of models, knowledge enhancement on supervison fine-tuning of models, knowledge enhancement on alignment fine-tuning of models, knowledge enhancement on reasoning of models, knowledge enhancement on output of models, etc. To facilitate collaboration amongst stakeholders of pretrained machine learning models, the framework and concerns of knowledge enhancement for pretrained machine learning models can be comprehensively described and categorized. Expected use of the document is to guide the construction of machine learning systems based on knowledge enhancement.

Besides the framework, there are futher standardization demands related to knowledge enhancement for pretrained machine learning models to be addressed. This series document plan to include the following parts:

Part 1: Framework

Part 2: Quality measurement of knowledge enhancement

Part 3: Management and governance of knowledge enhancement

Part 4: Requirements of Retrieval-augmented generation (RAG)