Journal Article

A scenario-based functional testing approach to improving DNN performance

Abstract

This paper proposes a scenario-based functional testing approach for enhancing the performance of machine learning (ML) applica- tions. The proposed method is an iterative process that starts with testing the ML model on various scenarios to identify areas of weakness. It follows by a further testing on the suspected weak scenarios and statistically evaluate the model’s performance on the scenarios to confirm the diagnosis. Once the diagnosis of weak scenarios is confirmed by test results, the treatment of the model is performed by retraining the model using a transfer learning technique with the original model as the base and applying a set of training data specifically targeting the treated scenarios plus a subset of training data selected at random from the original train dataset to prevent the so-call catastrophic forgetting effect. Finally, after the treatment, the model is assessed and evaluated again by testing on the treated scenarios as well as other scenarios to check if the treatment is effective and no side-effect caused. The paper reports a case study with a real ML deep neural network (DNN) model, which is the perception system of an autonomous racing car. It is demonstrated that the method is effective in the sense that DNN model’s performance can be improved. It provides an efficient method of enhancing ML model’s performance with much less human and compute resource than retrain from scratch.

Authors

Zhu Hong
Tam Tran, Thi Minh
Benjumea, Aduen
Bradley, Andrew

Oxford Brookes departments

School of Engineering, Computing and Mathematics

Dates

Year of publication: 2023
Date of RADAR deposit: 2023-07-03

---

“© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.”

---

Related resources

This RADAR resource is the Accepted Manuscript of A Scenario-Based Functional Testing Approach to Improving DNN Performance
This RADAR resource is the Accepted Manuscript of [arXiv preprint] A Scenario-Based Functional Testing Approach to Improving DNN Performance