Français
Advancing animation with M-body.ai dataset

Advancing Animation with M-body.ai’s Dataset

We are advancing animation with M-body.ai to break down barriers in the field. As mentioned in our first article, animation is inherently complex, and current datasets lack the detail and quality needed for production environments. As a result, models trained on these datasets often fail to deliver production-quality results. M-body.ai seeks to change this by developing datasets that meet production standards. By creating high-quality, synchronized multi-modal data, we aim to enable models to generate animation that is much closer to production-level quality.

M-body.ai is focused on establishing production standards and developing intuitive tools to make animation more accessible and efficient. A major innovation is the creation of an open-source, commercially viable, multi-modal conversational dataset featuring detailed, synchronized data. Our analysis of existing datasets and Generative AI research has revealed key opportunities for innovation in areas such as:

  • Character Architecture
  • Motion Capture
  • Error Mitigation

M-body.ai Dataset

Advancements in these areas will enable the creation of cutting-edge animation data for research and production. Our goal is for animators, researchers, and enthusiasts to use these tools and datasets to elevate animation and enhance its quality. This article highlights how M-body.ai is innovating current datasets and how this research will lead to better outcomes for the animation industry.

Production Quality System

M-body.ai retains data within a detailed skeleton hierarchy to improve machine-learning analysis of natural motion in conversations. We prioritize consistency and precision in character architecture to capture detailed human motion. Our Production Quality System is a comprehensive solution for high-quality animation data. Characters move and interact in lifelike ways, maintaining the individuality of each actor’s performance. This results in diverse and authentic animations in the dataset. Our streamlined process minimizes time and resources while maximizing quality. Our Production Quality System is an innovative solution for animation creation for Generative AI datasets. We’ve developed a methodology and workflow that includes custom tools and automated workflows to process our dataset.

Character Architecture

M-body.ai’s character architecture demonstrates a deep understanding of character functionality within an animation pipeline. It is designed to use advanced character capture and animation workflows to elevate the realism of human motions found in datasets. Emphasizing error reduction with an Inverse Kinematic retargeting system, M-body.ai is mitigating common issues that are inherent in raw data. This pipeline preserves lifelike, expressive character data, pioneering innovation in the animation industry.

Realistic Proportions

M-body.ai focuses on maintaining accurate actor proportions to enhance the precision of human movement data. By preserving each actor’s unique proportions, we ensure their distinct actions are retained. Unlike other datasets that adjust movements to fit a standard, neutral skeleton, our system keeps the individuality and accuracy of every actor. Featuring a diverse range of actors in terms of ethnicity, age, and gender, we capture unique, detailed motions. This preservation of realistic movements allows characters in the dataset to exhibit authentic behaviors, enriching their interactions and lifelike qualities.

Geometry Intersection

Animations that don’t account for accurate body proportions often result in characters intersecting unnaturally with their environments and themselves. Our solution retains original body proportions, allowing for realistic interactions between characters and their surroundings, and minimizing geometry intersections.

Skeleton Hierarchy and Geometry Topology

A skeleton hierarchy that retains essential bones is crucial for accurately capturing natural joint rotations. By combining this hierarchy with a standardized polygon geometry topology, we define the extremities of unique human forms. This approach is vital for capturing, depicting, and visualizing individual character movements. The result is a framework that allows M-body.ai to display unique character actions with precise accuracy and intricate detail.

Detailed Joint Distribution

M-body.ai uses a highly detailed joint distribution to capture all essential aspects of human movement. This approach ensures that the subtle intricacies of joint and limb movements are accurately represented, providing valuable and insightful information in the datasets. This precision allows complex animations to maintain natural motion and realism within the dataset.

Motion Capture

Multi-Modal Motion Capture

Our data capture pipeline includes precise calibration, multiple recording sessions, and synchronization of all data streams. It integrates high-resolution motion capture of full-body movements, facial expressions, and hand gestures with high-fidelity audio for accurate lip-sync and natural flow.

Body Capture

A primary focus is on capturing body animations with accurate joint rotations, essential for non-verbal communication. By precisely capturing these movements, the system ensures characters effectively convey emotions and intentions through synchronized body language and facial expressions.

Facial Capture

Advanced facial animation techniques capture lip sync and nuanced expressions of prosodic actions that include intent, micro-expressions and eye gaze. This ensures that realistic facial movements convey authentic emotions with synchronized multi-modal actions. This results in a dataset that naturally represents characters’ facial movements, aligning with their actions and achieving accurate lip sync

Hand Capture

Hand capture technology tracks individual finger and hand movements. Hand gestures are crucial for character expression and communication. This detailed capture enables realistic interaction with the environment and other characters. It adds depth and subtlety to non-verbal storytelling, making animations more expressive and convincing.

Voice Capture

We integrate audio capture to synchronize with body, facial, and hand movements, ensuring accurate lip-sync and natural conversational flow. This data creates a rich, multi-modal dataset that enhances the realism and expressiveness of animated characters.

Error Mitigation

Automation is crucial for enhancing the system’s functionality. Existing datasets struggle with inefficiencies from raw data that don’t account for errors like noise, impossible joint rotations, foot sliding, and incorrect weight distribution. The M-body.ai system uses automation to refine raw data, reducing errors with advanced inverse kinematics (IK) retargeting. This ensures natural joint rotations and mitigates motion capture issues. Automated batch processing minimizes human error, ensuring consistency and efficiency in animation sequences. We use advanced filtering to remove noise, machine learning algorithms for error correction, and manual reviews by expert animators to fine-tune the data. The production process corrects motions to accurately match performances. Procedural animation techniques and automated pipelines ensure efficient processing. Our approach guarantees realistic and efficient animations.

Retargeting System

The retargeting system is designed for flexibility, adapting motion data from actor’s performances to digital characters. We are using standardized skeletal structures and consistent topologies, but varying skeletal proportions. This allows for a wide range of diverse character designs that can retain similar information for the data set. We also include post-process control, giving animators the ability to modify and refine character animations. This flexibility allows for adjustments and fine-tuning of actions, ensuring each character performs natural actions.

Posture Correction

Our system adjusts character postures, addressing issues that arise from retargeting animations to skeletons with different proportions. By understanding the unique biomechanics of different character models, our solution can analyze realistic postures even when skeletons vary significantly.

Angle Rotation Correction

We correct inaccuracies from motion capture, such as incorrect marker matching, synchronization issues, and rotation angle impossibilities. Our system’s algorithms can identify these errors of approximation and automatically adjust them to ensure the resulting animations are of greater accuracy.

Root Shifting 

Inaccurate positioning problems, often occur from uncalibrated or misaligned positioning. Our solution adjusts for world root positions to maintain correct alignment, ensuring that characters remain in their intended planes regardless of environmental complexities.

Foot Sliding

Foot sliding and unwanted micro-movements are common issues found in raw data that disrupt realism in animations. Our system detects foot movement inconsistencies and corrects them to produce stable, grounded animations. It also rectifies geometry intersection issues with the ground, improving foot contact and roll precision.

Noise

Motion capture data is prone to noise due to hidden markers and joint axis flipping, which can distort the final animations. Our system applies advanced filtering techniques to eliminate noise, creating smooth and realistic character movements.

Final Thoughts – Reimagining Animation

M-body.ai reconsiders the process of dataset creation by implementing a production-level system that simplifies technical barriers, enhances workflow integration, and ensures high-quality outcomes. The platform tackles traditional challenges such as complexity and quality control, offering a streamlined, accurate, and creative approach to building datasets. With advanced joint distribution, a procedural pipeline, and adaptable retargeting systems, M-body.ai provides a robust framework that reduces errors, increases realism, and encourages creative flexibility. By blending cutting-edge technology with core principles of dataset creation, M-body.ai empowers experts and newcomers to produce reliable, data-driven character movements.

As part of our commitment to advancing animation technology, we are excited to announce the upcoming initial release of the M-body.ai dataset. This open-source, commercially usable, multi-modal conversational dataset features detailed, synchronized multi-modal data, showcasing the quality and intricacies of processed animations. Accompanied by comprehensive documentation, it supports production-quality animation, enhances machine learning quality, and empowers animation enthusiasts to apply high-quality animations to their creative projects. By making this dataset available, we aim to lower the barriers to high-quality animation and foster a collaborative environment where the community can explore, innovate, and push the boundaries of what is possible in animation. This groundbreaking approach heralds a new era of creativity and innovation in animation, opening vast possibilities for future storytelling.

About the author

Stephan Kozak, Screen Industries Research and Training (SIRT), Sheridan College

Stephan Kozak, Principal Investigator and Lead of CG, Animation, and VFX Research at Sheridan College’s SIRT, has over two decades of experience in character animation and advanced pipelines. He has led groundbreaking projects, including the world’s first AI virtual human with real-world citizenship and enhancements to Microsoft’s Azure phoneme system, earning him international recognition and two patents. At SIRT, Stephan guides a team focused on integrating virtual humans, avatars, and environments with cutting-edge pipelines, AI, and machine learning, transforming healthcare and entertainment sectors. His expertise and leadership in the M-body project are pivotal in advancing the capabilities of the character architecture and animation pipeline, facilitating cutting-edge research and collaboration across leading institutions.