Multistage Audio-Visual Automotive Cab Monitoring

Described is a task for an automobile interior having at least one subject that creates a video input, an audio input, and a context descriptor input. The video input relates to the at least one subject and is processed by a face detection module and a facial point registration module to produce a first output. The first output is further processed by at least one of: a facial point tracking module, a head orientation tracking module, a body tracking module, a social gaze tracking module, and an action unit intensity tracking module. The audio input relating to the at least one subject is processed by a valence and arousal affect states tracking module to produce a second output and to produce a valence and arousal scores output. A temporal behavior primitives buffer produce a temporal behavior output. Based on the foregoing, a mental state prediction module predicts the mental state of at least one subject in the automobile interior.