Text-to-animation solution

T2M-GPT integration for human motion animation

Industry:

Entertainment

Services:

AI solution development

Custom web development

Software development

Solution design

Project overview

Abto Software has entered the cooperation to design and deliver an unmatched text-to-animation solution, empowering indie game developers to generate humanoid animations by providing detailed descriptions.

The implemented T2M-GPT model converts natural language instructions into in-depth motion sequences, enabling immersive and interactive storytelling experiences.

Main goals

Our client, a game assets provider, provides solutions changing indie game development.

With the rise of large language model applications, we discovered the opportunity for utilizing such algorithms to generate humanoid animations and replace motion capture that requires specific hardware and assistance. Enthusiastic about this thought, we suggested an MVP to assess the concept’s maturity level.

Our team has covered the review and evaluation of available text-to-animation models, T2M-GPT integration, and custom ML training to improve the output of the T2M-GPT model.

The project’s key objectives:

Attract more potential customers

Offer users the opportunity to harness large language model capabilities to generate humanoid animations

Demonstrate evident technological leadership

Successfully commercialize LLMs application, thereby ensuring market differentiation and competitiveness within this specific domain

How the solution works

With this text-to-animation solution, the user can provide textual descriptions to generate dynamic animations. Just conversing, walking, running, fighting, shooting, swinging axes – no matter the instructions, the solution will reproduce the behavior.

And this is how it works:

First, the ML service is triggered to generate an animation by following user prompts
Second, the T2M-GPT model is enabled to process the prompt and generate the animation in point cloud format
The animation is converted into the following formats:
– BVH, Bio Vision Hierarchy, a format for skeletal animation data
– FBX (Filmbox), a proprietary file format for 3D animation interchange
The latter is rigged with the optimal rig for the associated mesh
A custom ML model is applied to identify any errors and corruption
Post-processing algorithms are applied to resolve found inconsistencies

The animations are integrated into our existing pipeline that allows setting parameters of the said animations. That means, the user can make the movements either faster or slower, adjust exaggeration and smoothness, and play around with playback speed.

And here we go – an animation to use in indie game projects.

Our contribution

We covered:

Business analysis
Software development
ML training and integration
3D visualization environment development
Thorough testing (manual testing, unit testing)
Technical support and maintenance

Main challenges

During the project’s scope, our engineers faced challenges associated with:

Output conversion

The open-source text-to-animation model was chosen and integrated to showcase the model’s key capabilities. For that single purpose, creating animations in point cloud format was acceptable, but talking about launching the product – the functionality was insufficient.

To resolve this problem, we performed output conversion into widely-used BVH and FBX formats and created the appropriate skeletal framework.

Output categorization

The quality of the produced animation was ranging quite drastically in dependence of the provided description. Some animations were broken to the point where they couldn’t be used for anything.

To handle this challenge, we trained our custom ML model to perform output categorization, so that we could apply appropriate post-processing algorithms right away.

Tools and technologies

Tech stack:

React.js
Three.js
Python
JavaScript
Django

ML stack:

Python
PyTorch
NumPy
Matplotlib
Pandas
Scikit-learn

Timeline:

November 2023 – March 2024

Team:

1 project manager / business analyst
2 software developers
2 ML engineers
1 QA engineer

Value delivered to business

After announcing the to-be-launched text-to-animation solution, the users started showing profound interest – one month after release, the number of customers buying subscriptions has more than doubled.

By bringing the solution to market, our client can leverage:

New customers, higher demand, and increased revenue streams

As the text-to-animation solution will continue attracting more potential customers and gain greater traction, the company will experience further opportunities for scalability and growth, thus expanding revenue streams.

Industry leadership

With this text-to-animation solution, the company can leverage the positioning as forerunners in popularizing large language model applications for human motion animation.