1.2. List of preinstalled AITs

Qunomon has some pre-installed ready-to-run AITs. Quality indices that is measurable by AIT is not limited to this list, but they will be a good reference when developing new ones.

They are stored in https://github.com/aistairc/{name_of_AIT}. For detailed information such as procedure to download or the directory structure, see the section 2. Prerequisites and installation.

  • eval_dataset_image_brisque

    • Read a set of image files (pickle) and calculate BRISQUE score (based on basic statistics and natural scene statistics (NSS)) based on a publicly available SVM model.

    • Based on the standard deviation of the BRISQUE score, the naturalness of a group of image files (loss of naturalness due to noise) is evaluated as image quality.

  • eval_dataset_image_diversity_vae

    • Calculate the feature values of the training data for evaluation with the VAE model that has been trained with the features of the image data for training.

  • eval_dataset_table_feature_importance_anova

    • Compute p-values and b-values of the specified attributes for the given table data using analysis of variance (ANOVA).

  • eval_model_image_classify_acc_adversarial_example

    • Generate adversarial sample images from input images and calculate accuracy information (Accuracy, Precision, Recall, F value, AUC) of the input model (image classification model trained with input images).

    • From these accuracy information, the accuracy and stability of the machine learning model can be evaluated.

  • eval_dataset_image_brightness_kld

    • Input two image data groups and calculate KLD (KL divergence, KL information content) for their brightness distribution.

    • The closer KLD is to 0, the more the two image groups can be evaluated as reproducing the same brightness features.

  • eval_dataset_table_coverage_pairwise

    • Calculate how much coverage a dataset has for a particular pairwise combination.

    • The higher the coverage, the higher the coverage of the combination of datasets, which can be evaluated as the accuracy and stability of the machine learning model.

  • eval_dataset_fairness

    • This AIT evaluates the fairness of a dataset.

    • From the attributes of the dataset, we specify the protected attributes (attributes for which fairness is required e.g.. Race, gender, religion, etc.) and find the mean of the objective variable for each attribute value. The closer the difference between the means is to 0, the higher the fairness is judged to be.

  • eval_model_numberclassification_confusionmatrix

    • This AIT evaluates the predictive performance of a model trained on the mnist dataset

    • The test creates a confusion matrix and ROC curve, and calculates accuracy, precision, racall, f-measure, and AUC.

  • eval_model_peformance_pytorch

    • Given a dataset and a pytorch classification model, the model evaluates the accuracy of the model’s inference from the inference results of the dataset.

    • This test calculates the ACCURACY, AP (average precision), and BALANCED ACCURACY of the inference on the test data.

  • eval_model_regression_coefficient_determination

    • This test computes the adjusted coefficient of determination adjusted for degrees of freedom for a model of multiple regression analysis constructed with several explanatory variables and one objective variable.

  • eval_model_regression_rmse_and_mae

    • This test calculates the RMSE (Root Mean Square Error) and MAE (Mean Absolute Error) of a model of multiple regression analysis constructed with multiple explanatory variables and one objective variable.

  • alyz_dataset_table_feature_contribution_shap

    • Internally generate a classification model using XGBboost and calculate SHAP values for the model.

    • SHAP values can be graphed to visualize the contribution of feature variables and to infer which feature variables in the table data are useful and how useful they are for classification.

  • alyz_dataset_table_counts_comb_all_attr

    • Calculate the counts of attribute values and the percentage of occurrences by the combination of attributes in the table data.

    • Based on this summary information, the distribution trend of the frequency of occurrence of the table data can be inferred.

  • alyz_dataset_table_counts_comb_two_attr

    • Specify a combination of unnecessary (improbable) attribute values in the table data, and calculate how much of that unnecessary data is contained in the table data, as well as the counts and percentage of occurrence of the attribute values.

  • alyz_dataset_table_counts_comb_attr_bdd100k

    • This AIT calculates the total number of occurrences of high-risk cases, and from the total number of occurrences, the importance of responding to high-risk cases can be determined.