Empirical Assessment and System Level Integration of Neural Network Supervisors

You are cordially invited to attend the PhD Dissertation Defence of Michael Weiss on Tuesday 28 March 2023 at 09:30 in room D1.13 (USI East campus).

Abstract:
We consider the task of deep neural network supervision, i.e., the monitoring of the model in production to identify inputs likely to cause a wrong prediction. This allows a deep learning-based system to avoid system failure by employing predefined healing strategies. The contributions are threefold: In the first part, we empirically compare common and generally applicable supervisors such as uncertainty quantification and surprise adequacy techniques in three different studies. Overall, we found non-dominance between different supervision techniques, i.e., no technique always performs best. Correspondingly, we formulate actionable usage guidelines to help practitioners when implementing a supervisor in practice. We then continue the thesis by leveraging supervisors in two system-level applications: We show how in a simulated self-driving car environment a black-box supervisor outperforms the state-of-the-art tool DeepRoad in predicting misbehaviors. Second, we show how a client-server architecture can significantly reduce the number of costly requests to a server-hosted large neural network by employing a small, imperfect supervised model on the client device. Lastly, we discuss a collection of tools and datasets we created to facilitate the implementation and assessment of supervisors for practitioners and for future research.

Dissertation Committee:

• Prof. Paolo Tonella, Università della Svizzera italiana, Switzerland (Research Advisor)
• Prof. Andrea Stocco, Technical University Munich, Germany (Research co-Advisor)
• Prof. Gabriele Bavota, Università della Svizzera italiana, Switzerland (Internal Member)
• Prof. Luca Maria Gambardella, Università della Svizzera italiana, Switzerland (Internal Member)
• Prof. Shaukat Ali, Simula Research Lab, Norway (External Member)
• Prof. Robert Feldt, Chalmers University of Technology, Sweden (External Member)