WEB DESK, June 20(ABC): “We used two kinds of data to build NeuroMechFly,” says Professor Pavan Ramdya at the School of Life Sciences at Ecole Polytechnique Fédérale de Lausanne (EPFL). “First, we took a real fly and performed a CT scan to build a morphologically realistic biomechanical model. The second source of data was the real limb movements of the fly, obtained using pose estimation software that we’ve developed in the last couple of years that allow us to precisely track the movements of the animal.”
Rosophila is the most commonly used insect in the life sciences and a long-term focus of Ramdya’s own research, who has been working on digitally tracking and modeling this animal for years. In 2019, his group published DeepFly3D, a deep-learning based motion-capture software that uses multiple camera views to quantify the movements of Drosophila in three-dimensional space.
Continuing with deep-learning, in 2021 Ramdya’s team published LiftPose3D, a method for reconstructing 3D animal poses from 2D images taken from a single camera. These kinds of breakthroughs have provided the exploding fields of neuroscience and animal-inspired robotics with tools whose usefulness cannot be overstated.
In many ways, NeuroMechFly represents a culmination of all those efforts. Constrained by morphological and kinematic data from these previous studies, the model features independent computational parts that simulate different parts of the insect’s body. This includes a biomechanical exoskeleton with articulating body parts, such as head, legs, wings, abdominal segments, proboscis, antennae, halteres (organs that help the fly measure its own orientation while flying), and neural network “controllers” with a motor output.
Why build a digital twin of Drosophila?
“How do we know when we’ve understood a system?” says Ramdya. “One way is to be able to recreate it. We might try to build a robotic fly, but it’s much faster and easier to build a simulated animal. So one of the major motivations behind this work is to start building a model that integrates what we know about the fly’s nervous system and biomechanics to test if it is enough to explain its behavior.”
“When we do experiments, we are often motivated by hypotheses,” he adds. “Until now, we’ve relied upon intuition and logic to formulate hypotheses and predictions. But as neuroscience becomes increasingly complicated, we rely more on models that can bring together many intertwined components, play them out, and predict what might happen if you made a tweak here or there.”
The testbed
NeuroMechFly offers a highly valuable testbed for studies that advance biomechanics and biorobotics, but only in so far as it accurately represents the real animal in a digital environment. Verifying this was one of the researchers’ main concerns. “We performed validation experiments which demonstrate that we can closely replicate the behaviors of the real animal,” says Ramdya.