Clever 3D Digital Human Assistants

3D Virtual Human Assistants: Science Fact, Not Science Fiction

Recent advances in parallel processing have boosted computing power, enabling complex graphics rendering and fast convergence of neural network-based algorithms. Complex graphical User Interfaces in the form of 3D Virtual Humans (3DVH) with high-fidelity may be developed faster and connected with user feedback for interactive behavior (1).

Types Of 3D Virtual Human Assistants

3DVH can be categorized on many dimensions based on their interactivity or lack of it, based on their emotional expressiveness and level of autonomy as follows:

Static 3DVH

These are virtual characters that do not move or interact with users. They are often used in video games or animated films.

Animated 3DVH

These are virtual characters that move and have some level of interactivity, but they do not respond to user input. They are often used in Virtual Reality experiences or in customer service applications.

Interactive 3DVH

These are virtual characters that can interact with users in real time using natural language processing and machine learning. They can respond to user input and carry out complex tasks. They are often used in customer service, training, and other applications where human-like interactions are required.

Emotionally Expressive 3DVH

These are virtual characters that can display emotions and can adjust their behavior accordingly. They can be used in virtual therapy, customer service, and other applications where emotional intelligence is a key factor.

Autonomous 3DVH

These are virtual characters that can make decisions, take actions, and adapt to the environment without human intervention. They are capable of learning and evolving over time. Such software systems are still in the Research and Development stage (R&D).

3DVH Portraying Human Emotion

One of the key components of 3D virtual human assistants’ development is the way in which they convey emotion and recognize it upon interaction with the user. Computer vision techniques may be used for human emotion recognition from facial expressions; however, emotion can also be detected through a combination of biosensor signal analysis (2). Automatic emotion recognition methods can be divided into the following categories:

• Facial expression analysis
  The study of the movements and configurations of the muscles in the face to determine the emotions expressed.
• Speech analysis
  The process of analyzing a person's voice tone, pitch, and rhythm to determine the emotions they are expressing.
• Body language analysis
  The process of analyzing a person's posture, movements, and gestures in order to determine the emotions they are expressing.
• Physiological measures
  This involves determining a person's emotional state using physiological measures such as heart rate, skin conductance, and brain activity.
• Machine learning and AI techniques
  This involves training algorithms to recognize emotions in real time, using large datasets of emotional expressions.
• Hybrid methods
  Some emotion recognition systems combine multiple methods, such as facial expression analysis and speech analysis, to provide a more accurate result.

The method(s) used will be determined by the specific application as well as the desired level of accuracy and complexity. Multiple methods may be used in conjunction in some cases to provide a more accurate and reliable assessment of a person's emotions. Combining high-fidelity computer graphics (CG) and machine learning techniques is the next step in 3DVH interface development. A recent study found that participants experienced lower levels of negative emotional states after sharing personal emotional events and interacting with 3DVH tasked with delivering emotional or cognitive support (3). Providing emotional and cognitive support to a patient while adjusting to real-time changes in their emotional state and expressions can yield better treatment outcomes.

eLearning-Friendly Technologies Employed By Emotion-Driven Intelligent 3DVH

1. Natural Language Processing

Natural language processing (NLP) is used by virtual humans to understand and respond to spoken and written language, enabling more human-like interactions.

2. Computer Vision

Virtual humans use computer vision to analyze and respond to nonverbal cues such as gestures and facial expressions, which improves interaction realism.

3. Machine Learning

On large datasets, virtual humans can be trained to make more informed decisions and provide better responses.

4. Interactive Animation Techniques

3D virtual humans are made more engaging for users by using interactive animation techniques to create lifelike movements and expressions.

5. Multimodal Interaction

3D virtual humans can interact with users more naturally by using multiple input modalities such as speech, gestures, and touch.

Conclusion

While emotional intelligence is still a challenge, these advancements in technology have enabled 3D virtual humans to become more intelligent, realistic, and human-like, making them useful to an even wider range of applications in the near future.

References:

(1) Unreal’s Metahuman Creator software, last accessed on 02.07.2023.

(2) Adyapady, R. R., and B. Annappa. 2023. "A comprehensive review of facial expression recognition techniques." Multimedia Systems 29: 73–103.

(3) Lisanne S. Pauw, Disa A. Sauter, Gerben A. van Kleef, Gale M. Lucas, Jonathan Gratch, and Agneta H. Fischer, 2022. "The avatar will see you now: Support from a virtual human provides socio-emotional benefits." Computers in Human Behavior 136.