AI supported jump recognition and analysis

Our client was looking for partners with required technical expertise to deliver an enterprise school platform. The project was centered around integrating jump recognition and analysis by leveraging artificial intelligence.

Abto Software has formerly successfully delivered education software for the Ireland-based organization. Having entered into cooperation this time, our engineers have implemented artificial intelligence to deliver sensorless human activity recognition to extend an enterprise-level school platform to improve physical health of children at schools.

Our team was aiming to adapt complex algorithms that could:

• Accurately recognize physical activity in real-time
• Quickly evaluate & measure performed movements
• Automatically compare the movements with the reference pattern

Our engineers were working on using live streams from both:

• Web cameras from the user’s PC or laptop
• Front cameras from the user’s smartphone or tablet

The platform was created to accelerate physical health state assessment as well as motivation across schools. Our concept is designed for accurate jump recognition and analysis, but can be extended to assist in monitoring other exercises – stretching, squatting, and more.

The concept is simple:

1. The student is shown a easy-to-follow reference pattern of the physical exercise

2. The student then performs the exercise (at this project stage, it is jumps up)

3. The system quickly compares the execution with the reference template and evaluates
the score

4. The student or the responsible teacher can review the performance for further:

• Physical health state assessment
• Future training plan adjustment

Abto Software has entered the cooperation to implement artificial intelligence, in particular pose estimation. While working on the described concept, we covered:

• Business consulting
• Solution design and research
• PoC development
• Integration planning

To ensure quality control, we suggest to implement in-built guides to help the users:

• Position themselves in a proper distance to the used camera
• Detect obstacles that might block accurate landmark detection

In the first prototype, the designed CV model was working with ready-made, smartphone captured video files. After the solution’s integration, the trained CV algorithms are working with live video streams derived from different devices – PC’s, laptops, smartphones, tablets.

The delivered proof-of-concept consists of a linear pipeline (in Python) as a command line interface application. For integration, we suggest to conduct landmark and jump detection (in JavaScript) on the client’s side, ensuring smooth real-time performance.

The project’s main challenge was accurate AI based pose estimation, to handle which we:

• Created a reference pattern to compare and evaluate the performance of the executed exercises: those can have different dynamics, that’s why we worked on minimizing its impact
• Taught the CV model:
  • To recognize and assess both the start and end points of the executed jump
  • To localize the movements in time
• Stabilized the movement assessment
• Developed techniques to compare human proportions considering height and weight to standardize the parameters

Our team has covered business consulting, solution design and research, PoC development, and smooth integration planning to implement jump recognition and analysis for a school platform to improve public health.

The solution might benefit:

• National authorities that aim to improve public health
• Educational institutions that aim to improve student wellbeing and motivation
• Physical therapists, personal trainers, sports teams, and private fitness centers by helping to monitor the progress of clients and athletes, offer personalized workout plans, adjust techniques, and enhance overall performance
• Individual users working on improving their health