1.1. What is AIT?

AIT(AI system test package) is the package of a test program to be installed and executed on Qunomon testbed. It measures the concrete quality metrics of target ML components under test such as datasets and/or machine learning model.

AIT can be developed by capsuling evaluation programs (written in python) and meta-data (which describes the charactersitics of that AIT) as an Docker image. This enables users to search, download, utilize that AIT without knowing the details of specification or implementation of an AIT. Since all AITs are running on the Docker container, Users can also avoid the conflict of the execution environment.

Qunomon is an open-sourced platform (testbed) for quality assessment of AI systems. Qunomon has some pre-installed AITs initially so that users can develop their own AIT based on them and install third-party AITs on their local Qunomon instance.

We have prepared the AIT development environment which contains some necessary templates and test execution environment to develop and deploy your own (new) AIT in the Qunomon. In this guide, we provide some information about development process and technical details of the AIT required to develop your own one.

1.1.1. What is AIT development environment?

We provide AIT development environment as initially included component of Qunomon platform. Users (such as QA engineer of AI/ML sysmtes or developer/researcher working on novel quality assessment techniques or methods) can develop their own AIT using this environment.

This environment consist of: (1) Docker images for AIT developers which can launch Jupyter Notebook in the isolated containers on your local machine, and (2) AIT template which includes all required components of the AIT from the beginning.

1.1.1.1. Docker images for AIT developers

We include ready-to-launch Jupyter lab (for experimentation and development) in the Qunomon repository. These environment are isolated as containers using docker and provided through your browser so that developers don’t have to worry about breaking existing environment and dependencies.

1.1.1.2. AIT Template

AIT Template is a template directory that contains all the basic components of AIT. By using this, developers can concentrate on implementing algorithmic part of the AIT.

Following components are contained in the template so that developers can create a new AIT with only partial modification.

  • Dockerfile and an execution pipeline definition of an AIT.

    • Each AIT is isolated in individual containers to avoid conflict with other AITs or user’s environments using Docker.

    • AITs are executed via Qunomon’s pipeline execution engine (Airflow) after deployment.

  • AIT Manifest (Meta data of an AIT)

    • Each AIT must have AIT Manifest to coordinate with Qunomon’s other components smoothly so that guarantee portability and reusability of itself.

  • Basic design elements of an AIT

    • Pre-defined directory structure suitable for experimentation and implementation, license information and third party notices.

  • Docker images for AIT developers.

    • Scripts and the execution environment to capsulize an AIT as Docker image.

1.1.1.3. Prerequisites for work

These are the prerequisites for working in the AIT development environment.

  • The programming language used is Python3.9.

  • TensorFlow version 2.9.1 shipped with Jupyter Notebook.

  • Jupyter Notebook uses port 9888. Please do not use this port with other applications while Jupyter Notebook is running.