Text-to-animation solution

Abto Software has entered the cooperation to design and deliver an unmatched text-to-animation solution, empowering indie game developers to generate humanoid animations by providing detailed descriptions.

The implemented T2M-GPT model converts natural language instructions into in-depth motion sequences, enabling immersive and interactive storytelling experiences.

Our client, a game assets provider, provides solutions changing indie game development.

With the rise of large language model applications, we discovered the opportunity for utilizing such algorithms to generate humanoid animations and replace motion capture that requires specific hardware and assistance. Enthusiastic about this thought, we suggested an MVP to assess the concept’s maturity level.

Our team has covered the review and evaluation of available text-to-animation models, T2M-GPT integration, and custom ML training to improve the output of the T2M-GPT model.

The project’s key objectives:

• Attract more potential customers

Offer users the opportunity to harness large language model capabilities to generate humanoid animations

• Demonstrate evident technological leadership

Successfully commercialize LLMs application, thereby ensuring market differentiation and competitiveness within this specific domain

With this text-to-animation solution, the user can provide textual descriptions to generate dynamic animations. Just conversing, walking, running, fighting, shooting, swinging axes – no matter the instructions, the solution will reproduce the behavior.

And this is how it works:

• First, the ML service is triggered to generate an animation by following user prompts
• Second, the T2M-GPT model is enabled to process the prompt and generate the animation in point cloud format
• The animation is converted into the following formats:
  – BVH, Bio Vision Hierarchy, a format for skeletal animation data
  – FBX (Filmbox), a proprietary file format for 3D animation interchange
• The latter is rigged with the optimal rig for the associated mesh
• A custom ML model is applied to identify any errors and corruption 
• Post-processing algorithms are applied to resolve found inconsistencies

The animations are integrated into our existing pipeline that allows setting parameters of the said animations. That means, the user can make the movements either faster or slower, adjust exaggeration and smoothness, and play around with playback speed.

And here we go – an animation to use in indie game projects.

During the project’s scope, our engineers faced challenges associated with:

• Output conversion

The open-source text-to-animation model was chosen and integrated to showcase the model’s key capabilities. For that single purpose, creating animations in point cloud format was acceptable, but talking about launching the product – the functionality was insufficient. 

To resolve this problem, we performed output conversion into widely-used BVH and FBX formats and created the appropriate skeletal framework.

• Output categorization

The quality of the produced animation was ranging quite drastically in dependence of the provided description. Some animations were broken to the point where they couldn’t be used for anything. 

To handle this challenge, we trained our custom ML model to perform output categorization, so that we could apply appropriate post-processing algorithms right away. 

After announcing the to-be-launched text-to-animation solution, the users started showing profound interest – one month after release, the number of customers buying subscriptions has more than doubled.

By bringing the solution to market, our client can leverage:

• New customers, higher demand, and increased revenue streams

As the text-to-animation solution will continue attracting more potential customers and gain greater traction, the company will experience further opportunities for scalability and growth, thus expanding revenue streams.

• Industry leadership

With this text-to-animation solution, the company can leverage the positioning as forerunners in popularizing large language model applications for human motion animation.