1.1. Assumed User

This document is intended for users who have some experience using commonly used tools in AI development, such as Python, Jupyter notebooks, Docker, Git, and Airflow. Although this AI development environment is designed so that users do not need to directly operate Docker, Git, or Airflow, there may be occasions when it is necessary to check these background tools in case of any issues.

1.2. What is AIT?

AIT(AI system test package) is the package of a test program to be installed and executed on Qunomon testbed. It measures the concrete quality metrics of target ML components under test such as datasets and/or machine learning model.

For example, when developing a classification model and evaluating its quality, classification accuracy is often used as one of the quality metrics. In this case, the classification accuracy is calculated using test data and classification results, and the quality is measured based on whether the classification accuracy meets a certain standard. Automating this quality evaluation process can enable it to be used for ML quality assessment using Qunomon.

AIT can be developed by capsuling evaluation programs (written in python) and meta-data (which describes the charactersitics of that AIT) as an Docker image. This enables users to search, download, utilize that AIT without knowing the details of specification or implementation of an AIT. Since all AITs are running on the Docker container, Users can also avoid the conflict of the execution environment.

Qunomon is an open-sourced platform (testbed) for quality assessment of AI systems. Qunomon is equipped with several pre-installed AITs and also provides access to AITs that have been made publicly available on GitHub. Users (such as those evaluating AI systems with Qunomon, or those involved in researching and developing assessment metrics) can develop their own AITs based on these resources.

We have prepared the AIT development environment which contains some necessary templates and test execution environment to develop and deploy your own (new) AIT in the Qunomon. This guide provides developers of AI evaluation programs with information on how to develop AITs using the AIT development environment, as well as technical specifications for AITs.

1.2.1. What is AIT development environment?

We provide AIT development environment as initially included component of Qunomon platform. Users (such as QA engineer of AI/ML systems or developer/researcher working on novel quality assessment techniques or methods) can develop their own AIT using this environment.

This environment consist of: (1) Docker images for AIT developers which can launch Jupyter Notebook in the isolated containers on your local machine, and (2) AIT template which includes all required components of the AIT from the beginning.

1.2.1.1. Docker images for AIT developers

We include ready-to-launch Jupyter lab (for experimentation and development) in the Qunomon repository. These environment are isolated as containers using docker and provided through your browser so that developers don’t have to worry about breaking existing environment and dependencies.

1.2.1.2. AIT Template

AIT Template is a template directory that contains all the basic components of AIT. By using this, developers can concentrate on implementing algorithmic part of the AIT.

Following components are contained in the template so that developers can create a new AIT with only partial modification.

  • Dockerfile

    • Information for setting up an execution environment to isolate AI evaluation programs on Docker containers.

  • Pipeline Definition

    • AITs are executed via Qunomon’s pipeline execution engine (Airflow) after deployment.

  • AIT Manifest (Meta data of an AIT)

    • Each AIT must have AIT Manifest to coordinate with Qunomon’s other components smoothly so that guarantee portability and reusability of itself.

  • Basic design elements of an AIT

    • Pre-defined directory structure suitable for experimentation and implementation

    • License information

    • Third party notices

  • Docker images for AIT developers.

    • Scripts and the execution environment to capsulize an AIT as Docker image.

1.2.1.3. Prerequisites

These are the prerequisites for working in the AIT development environment.

  • The programming language used is Python3.9.

  • Machine learning framework is not installed.

  • In the AIT development environment, Jupyter Notebook is used on port 9888. Please ensure that this port is not used by other applications while the AIT development environment is running.

  • AIT can access external networks. Please implement functions with security risks in mind.

  • We recommend that the amount of memory used by AIT is approximately 8GB or less, although this may be due to containerization. Please keep in mind that memory usage depends on the execution environment.

  • AIT can use GPU. However, depending on GPU usage, shared memory is used, so there is a concern that more memory will be used.