Advice needed for building InterviewAI: a real-time AI interview feedback project

Hi everyone,

I’m working on a project called InterviewAI.

The idea is to build an AI-powered interview practice system that analyzes a user during a mock interview and gives feedback about their performance. The goal is not only to process what the user says, but also to analyze visual and behavioral signals from the camera.

In general, the system should be able to:

• Use the webcam in real time

• Detect and track the user’s face/body

• Analyze facial expressions or emotions

• Detect signs such as posture, stability, eye contact, and self-touch gestures

• Generate useful feedback for improving interview performance

• Eventually combine computer vision results with an AI feedback system

I’m still trying to understand the best technical direction for this project, especially for the computer vision and machine learning parts.

My main questions are:

1. What would be the recommended architecture for this kind of real-time AI interview analysis system?

2. Which Hugging Face models or libraries could help with facial emotion recognition, body/pose analysis, or behavioral analysis?

3. Should I train my own model, fine-tune an existing model, or use pretrained models directly?

4. What are the best practices for real-time webcam inference?

5. How should I evaluate if the system is reliable and not giving misleading feedback?

6. Are there datasets, models, or pipelines you recommend for this type of project?

7. What common mistakes should I avoid when building a project like this?

I’m looking for guidance on the correct workflow, possible model choices, and how to structure the project properly.

Any advice, examples, or recommended resources would be really appreciated.

Thank you!