We use cookies to give you the best experience and to help improve our website
Find out what cookies we use and how to disable them
Standards Development
This document defines a basic framework with principles, characteristics and approaches for the realization and enhancement for automated artificial intelligence (AI) systems’ controllability.
The following areas are covered:
State observability and state transition
Control transfer process and cost
Reaction to uncertainty during control transfer
Verification and validation approaches
This document is applicable to all types of organizations (e.g. commercial enterprises, government agencies, not-for-profit organizations) developing and using AI systems during their whole life cycle.
1. Artificial intelligence (AI) techniques have been applied in applications in domains and markets, such as health-care, education, clean energy, sustainable living, etc. Despite being used to enable systems to perform automated predictions, recommendations or decisions, the use of AI has raised a wide range of concerns. Some characteristics (e.g. lack of full explanation) of AI (e.g. deep neural network based learning and inference) can introduce uncertainty to systems’ behaviour. This could bring risks to end-users with unpredictable hazards. For this reason, controllability of AI systems becomes very important. This document is primarily intended as a guidance for AI system design and use, in terms of controllability realization and enhancement.
2. To embrace the gains from AI in a sustainable and responsible way, controllability characteristics and principles of AI system can be identified. By investigating the needs of controllability in domainspecific contexts, the understanding of AI systems’ controllability can be strengthened. Controllability is an important fundamental characteristic supporting AI systems’ safety for end-users.
3. Automated systems (ref. ISO/IEC DIS 22989, Table 1 in 5.12) can be realized by using AI. The degree of external control or controllability is an important characteristic of automated systems. Heteronomous systems range over a spectrum from no external control to direct control. The amount of external control or controllability can be used to guide or manipulate systems at various levels of automation to behave as intended and within functional safety limits. Such requirements can be satisfied considering controllability features or taking specific preventive actions within each step of the AI system life cycle as defined in ISO/IEC DIS 22989, clause 6.
4. Bad consequences are possible if an AI system is permitted to take wrong decisions or actions without any external intervention, control or oversight. To realize controllability, key points of system state observation and state transition should be identified. Intervention calls for “transfer of control” between an AI system and a human or another external agent. The exact points where transfer of control has to be enabled should be considered during the design and implementation of an AI system.
5. The transfer of control for an intervention should be easily executable, within reasonable time, space, energy and complexity limits, while at the same time minimizing the interruption to both sides(i. e. the AI system and the external agent). Stakeholders consider the cost of control transfer of automated AI systems, which determines effectiveness of controllability realization in AI systems. Moreover, uncertainty during control transfer can exist on both sides.
Thus it is important to carefully design the control transfer processes, so that uncertainty and other undesired consequences are minimized or mitigated.
6. Given the design and implementation of control transfer, the effectiveness needs to be tested. This calls for principles and approaches for validation and verification of AI systems’ controllability.
You are now following this standard. Weekly digest emails will be sent to update you on the following activities:
You can manage your follow preferences from your Account. Please check your mailbox junk folder if you don't receive the weekly email.
You have successfully unsubscribed from weekly updates for this standard.
Comment on proposal
Required form fields are indicated by an asterisk (*) character.