How VR and AR Will Help in Remote Expert Assistance

The speed and quality of maintenance and repair are critical in the modern, technology-packed built environment. Consequently, these were considered in an experimental project that tested how remote expert assistance using VR and AR technologies could help improve the productivity of field service.

I’m in a hall overlooking white mountain tops. It’s snowing. In front of me stands an avatar that explains to me what we can do together in this virtual space. He jumps away but I can still hear his voice from behind me. He fetches a chair and hands it to me. I grab it and inspect it. The next moment, a video starts playing on the wall. Later, my host shows me how to draw in three dimensions, how to make sticky notes, how to share a PC desktop, and how to use other collaboration tools.

This experience took place at FAKE Production, a Helsinki-based digital image, animation, and VR/AR studio. With VR glasses and hand-held controllers, I had tried out Glue, their universal collaboration platform. This is a soon-to-be-released service that you can use with VR/AR gear and on mobile and desktop devices.

Glue is also one of the solutions tested in an experimental project called Expert assistance using VR and AR glasses. In this project, Sovelto, a Finnish educational company, wanted to explore the possibilities of using VR and AR solutions for field service. Over ten organizations took part in the project, which received funding from KIRA-digi, the national built environment digitalization program.

Maintenance and Repair Need to Improve

In Finland alone, maintenance and repair work costs society about 24 billion euros annually. Sovelto’s project was focused on field service, which often involve critical operations that take place at a customer’s premises.

According to the report, typically the first field service visit fails in one out of four cases, and on average, it takes 1.5 visits for the service worker to fix the problem. This inefficiency stems mostly from poor preparation, not having the right spare parts, and a lack of expertise.

Delayed maintenance and repair can cause substantial financial losses for the customer. Furthermore, incorrect fixes may lead to damage and can even make machines hazardous to use. Also, as technical systems become ever more complicated and globally distributed, it’s becoming harder to have a suitably skilled person quickly available wherever they are needed.

Some companies have started to address this problem through mobile and VR/AR technologies. For instance, in 2016, thyssenkrupp announced the use of Microsoft HoloLens in their worldwide elevator service operations. Using this system, their service technicians can visualize and identify problems ahead of a job and can also have remote access to technical and expert information on site.

Helping the Field Workers Virtually

One way to improve the productivity of field workers is to offer them assistance remotely. Matti Pouhakka, the Sales Director of FAKE Production, explains: “I see two specific situations where remote assistance can come to the rescue. First, there’s the maintenance worker who understands how to fix something, but needs an expert to tell them what to fix. And then there’s a person—a technical cleaner, for example—who sees a problem and could likely fix it themselves quickly if they could be guided by a professional on what to do.”

As the Glue example demonstrated, VR is a powerful tool for simulation, training, and collaboration. In a maintenance scenario, an expert could show and train a field worker or even several workers in a VR environment. A worker could then practice their skills with a digital counterpart of the actual machine. The sessions could even be recorded for further use.

Also, digital twins of whole buildings and technical installations open up new uses for VR. Real-time IoT data combined with accurate models could help planning changes and identify problems and allow them to be fixed remotely. Pouhakka emphasizes that digital twins are assets that grow in value over the years. Clients and designers should understand this and see the creation of comprehensive digital models as an investment rather than a cost.

Augmented or Mixed Reality Applications

Kim Nyberg, Technology Director at Trimble, gave presentations, showed demos, and carried out tests with prototypes and existing Trimble mixed reality products at Sovelto’s workshops. Kim is an expert in BIM and mixed reality solutions.

“Remote expert assistance is already feasible with the commercial solutions that we currently have,” says Nyberg. “For example, I showcased Trimble Connect HoloLens, which lets you collaborate with several people simultaneously and see them as avatars. The attendees can study the same virtual model of a machine or a building with HoloLens glasses and discuss it. You can either be physically in the same room or can attend remotely.”

Nyberg also demonstrated a mobile AR app, Trimble SiteVision. With the app, a person inspecting a machine or a structure can place a digital 3D model into its real environment. The user can then take a snapshot of the combination, add notes, and upload the image to Trimble Connect. From there, a field worker can carry out the necessary work.

Simplicity and Usability Are the Keys to Success

Sovelto’s project report and my interviews with Pouhakka and Nyberg point to a future where remote expert assistance is as easy as calling on a phone today. Existing solutions already make this possible, and at a reasonable price. However, for remote expert assistance to really take off, both hardware and software developers must provide solutions that work flawlessly in demanding field service situations.

Nyberg believes that in the next one or two years we’ll see lightweight MR glasses that will be feasible even for consumer use. He also envisions systems that you can control with eye movements and thoughts rather than needing to use the hands or voice.

Simplicity and usability are and will be the critical success factors. “The best test is to be able to do something useful with a system when you use it for the first 30 to 60 seconds,” Nyberg stated, summing up.

The title image is courtesy of Trimble.