Human action analysis models in artificial intelligence based proctoring systems and dataset for them

Abstract

This paper describes the approach for building a specialized model for human action analysis in AI-based proctoring systems and proposes a prototype of dataset which contains data specific to the application area. Boosted development of machine learning technologies, the availability of devices and the access to the Internet are skyrocketing the development of the field of distance learning. And in parallel with distance learning systems the AI-based proctoring systems, that provide the functional analysis of student work by imitating the teacher's assessment, are developing as well. However, despite the development of image processing and machine learning technology, the functionality of modern proctoring systems is still at a primitive level. Within the image processing functionality, they focus entirely on tracking students' faces and do not track postures and actions. At the same time, assessment of physical activity is necessary not only as part of the learning process, but also to keep students healthy according to regulatory requirements, as they spend the entire duration of learning process in front of computers or other devices during the distance learning. In existing implementations, this process falls entirely on the shoulders of teachers or even the students themselves, who work through the lesson materials or tests on their own. Teachers, at the same time, have to either establish contact through video communication systems and social media (TikTok, Instagram) and/or analyse videos of students doing certain physical activities in order to organise physical activities evaluation. The lack of such functionality in AI-based proctoring systems slows down the learning process and potentially harms students' health in the long run. This paper presents additional functionality requirements for AI-based proctoring systems including human action analysis functionality to assess physical activity and to monitor hygiene rules for working with computers during the educational process. For this purpose, a foundation model called InternVideo was used for processing and analysis of student's actions. Based on it, the approach for building a specialized model for student action analysis was proposed. It includes two modes of student activity evaluation during the distance learning process: static and dynamic. The static mode (aka working phase) analyses and evaluates the student's behavior during the learning and examination process, where physical activity is not the main component of learning. The dynamic mode (aka physical education mode) analyses and assesses the student who purposefully performs physical activity (physical education lesson, exercises for children during the lesson, etc.). A prototype dataset designed specifically for this application area has also been proposed.