Canada-based space technology firm MDA builds robotic arms used on the International Space Station. MDA realized it could use immersive mixed reality experiences to help engineers visualize designs, accelerate diagnostic issues, and train astronauts and ground controllers. Using Microsoft Azure Spatial Anchors, MDA provides remote training to multiple people in different locations. MDA will continue to develop mixed reality applications as it builds a new robotic arm for the Lunar Gateway space station orbiting the Moon.

" With HoloLens 2, we optimize designs and build what's required to meet the mission requirements. It's going to be a transformative tool in our business. "

Most terrestrials know the International Space Station (ISS) from pictures, where it appears dwarfed against the backdrop of Earth, so it's hard to get an appreciation for its size. Even MDA Vice President of Robotics and Space Operations Tim Kopra, who served aboard as astronaut and Commander, was impressed when he saw the full-scale ISS in an immersive mixed reality experience using Microsoft HoloLens 2.

“When I viewed the space station with HoloLens 2 from my home, the space station appeared so large that one of the wings was three houses over from me,” Kopra says. The mixed reality experience felt eerily familiar. “Standing on top of the space station, looking down gave me flashbacks of doing space walks. I got the exact same feeling as I did outside of the actual space station in a space suit and looking at the vast blackness of space,” he says.

As Vice President of Robotics and Space Operations at Canadian space technology pioneer MDA, Kopra now helps identify how HoloLens 2 can augment experiences to support space missions, the design of new space robots, and immersive astronaut training.

Based in Brampton, Ontario, MDA specializes in geointelligence, satellite systems, and robotics and space operations. The company is best known for developing and operating Canadarm and Canadarm2, robotic arms that have supported space shuttle missions, deployed satellites, serviced the Hubble Space Telescope, and played a major part in assembling the ISS. MDA also developed Dextre, a two-armed “robotic handyman” that has the strength and dexterity to perform tasks otherwise done in spacewalks. Canada's pride in MDA's innovations shows on the back of its five-dollar bill, which features an astronaut working with Dextre attached to the Canadarm2.

MDA has found many ways in which HoloLens 2 can help engineers, ground controllers, astronauts, and others involved in space missions.

From static models to lifelike experiences

Using the open-sourced Mixed Reality Toolkit (MRTK), MDA has started using HoloLens 2 to boost its research and design capabilities. “MRTK makes it very easy to put together mixed reality functionality,” says Bart Verzijlenberg, Senior Technical Staff Member in MDA’s software department.

Engineers are getting a deeper understanding of the complex mechanisms of the space robots they design. They love seeing their designs in front of them and having the ability to walk around and pull it apart. “When I hit a button and the entire model pops up, jaws drop,” says Verzijlenberg. “You can see all the core components individually and better understand how an object works.”

In the company's zero-gravity Dynamic Robotic Emulation and Mixed Reality (DREAMR) Lab, engineers can test how the hardware will behave in orbit. They can even take the cover off the robots and look inside during operation. “If two interfaces are joining together, you can put your head right at the interface, which would not be safe to do with the real robot,” says Verzijlenberg. “You can see where things might interfere with each other and get stuck.”

Mixed reality visualizations are helpful not just in the design phase, but also after space robots have been deployed in space. “If we have technical issues, we can look at the telemetry that we receive from the International Space Station and diagnose problems,” says Kopra. “In a 3D-immersive mixed reality environment, we can slice through hardware to identify exactly where some of these issues have occurred. That's something that you can't do with only telemetry or even a drawing.”

Before HoloLens, the team would look at many numbers in various ways to figure out what happened. “With HoloLens, we can simply press play and visualize all the movements of the arm, like when it picks up a payload,” Verzijlenberg says. “It brings the story to life.”

Anchoring the experience

When multiple people participate in an immersive experience, holograms need to show up in the same place for each user. To provide that consistency, MDA uses Microsoft Azure Spatial Anchors, a kind of 3D bookmark. “We use Spatial Anchors to pre-position all elements in the right positions in the room so that every user is able to observe the holograms from the right place,” says Verzijlenberg.

This works even when people view the experience from different locations. “Each room gets its own anchor. If someone in Montreal, for example, puts an anchor in her physical space and I do the same thing in my home, I will see her relative to the anchor. We both experience the same set of objects and visual,” Verzijlenberg explains.

Spatial Anchors eliminates the difficult task of creating and restoring persistent, accurate digital anchors. “I can simply walk around with HoloLens, say ‘record,’ and create an anchor that automatically loads and gets shared with others,” says Verzijlenberg.

MDA anticipates deepening this experience with Microsoft Mesh to connect and collaborate with greater depth and dimension.

Training astronauts using HoloLens 2

MDA uses mixed reality to train engineers, ground controllers, and astronauts. “HoloLens 2 cements learning in a way that is completely different than looking at a slide deck or a mock-up,” says Kopra.

Astronauts train on how to operate the Canadarm2 through a two-week intensive course. “We want to make the experience for the astronauts as realistic as possible,” says Kevin Nasimok, Operations Team Lead at MDA.

That training can now be done remotely. “We track the hardware in the lab to reflect in the virtual world where it's positioned. Then, a remote user can see from any angle how those virtual objects are interacting,” says Verzijlenberg. “And it's a much more personal connection than via a screen. You see somebody standing in front of you, see their hands moving, visual feedback you don't get on a webcam.”

Lunar Gateway

MDA is now developing Canadarm3, an AI-enabled robotic system for Lunar Gateway, which will serve as the home base for missions to the moon’s surface from 2024 onwards. “With HoloLens 2, we optimize designs and build what's required to meet the mission requirements. It's going to be a transformative tool in our business,” says Kopra.

One of the challenges ground controllers will face with Lunar Gateway is its distance from Earth. “We have a very limited amount of time to receive information, and you need to get an answer back. Using Microsoft HoloLens, you can quickly understand what happened and plan next steps,” says Verzijlenberg.

Exploring further capabilities

MDA has found HoloLens 2 to be a game-changer. “You feel the energy when you talk to the team,” says Kopra. “We're building robots for a space station that goes around the moon, how cool is that? And HoloLens is helping us do it.”

Find out more about MDA on Facebook, Twitter, Instagram, and LinkedIn.