How can the Microsoft HoloLens assist maintenance engineers in their daily jobs? Imagine an unexpected maintenance where the engineer on-site does not possess the knowledge on a specific issue. This results in a loss in time & money and bad customer experience because they need the assistance of a more experienced colleague to solve the issue. Isn’t it about time to utilize Mixed Reality for these kinds of cases? In this blog, Daan will showcase you how the Microsoft HoloLens can be used as a maintenance assistant in your day to day operations. He will explain how we used Unity3D and the HoloToolKit to create the application and which challenges came up during the development. Let’s introduce the Microsoft HoloLens Maintenance Assistant.
Want to know more? Check out our previous blog where we talked about the use of Microsoft HoloLens in the logistic sector.
Proof of Concept
The project was a proof of concept to showcase how the Microsoft HoloLens can assist workers with their daily jobs. Instead of having to fall back to a system maintenance guidebook, they are guided with the help of holograms, sound or even remotely by a colleague, all by using the Microsoft HoloLens.
The Experience
The interaction with the application is quick and easy. The user can use both gestures and speech to interact with the application. In return, the user receives an answer from the system through both sound and text. This means that the application can be used hands-free. Ideal for when you’re working with numerous tools. By now you’re probably curious about how all of this works! So here is a sneak preview video.
By loading the video, you agree to YouTube’s privacy policy.
Learn more
A Look Behind Scenes
You might wonder how does all of this work exactly. Let’s have a deeper look at the technical aspects of this project. The application itself was built using Unity and the HoloToolKit.
The Unity Editor
Unity is generally known as a game engine; however, it has been growing rapidly as an environment to create Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR) applications. The Unity editor enables an easy workflow between visuals and functionality. This includes functionality to test AR applications before deploying them on the Microsoft HoloLens. Unity offers an intuitive interface that allows for quick design of visuals, but also a great integration between Unity objects and your own scripts.
The HoloToolKit
The HoloToolkit (now succeeded by the Mixed Reality Toolkit) gives a solid set of important features that speed up the development of a Mixed Reality application. The toolkit deals with a lot of the deeper functionalities and exposes them to us. By combining our own scripts with the HoloToolkit it is possible to quickly create powerful experiences. Interested in trying out Mixed Reality Development yourself? Even without a Microsoft HoloLens, you can start building your first Mixed Reality Application. Check out this link with tons of information to get started.
Challenges
Development for the Microsoft HoloLens is a challenging environment. Especially in terms of design, there is limited information available. This means a lot of trial and error to understand what works and what does not. Some of the more interesting challenges for us were the alignment of virtual objects with the real world, the user interface and how the user is directed in an environment.
Challenge 1: the alignment of virtual information with the real-world object
Theoretically, it would be ideal to be able to recognize each of the specific components of a device. Unfortunately, this is not possible just yet. This project uses a virtual representation of the real-world object and requires the virtual object to be aligned with the real-world object through the user at the start of the application. The virtual object is anchored to its real-world position using spatial anchors. Once an object is mapped it is easy to add instructional steps for the specific object. Specific parts of the object can be matched with multiple instructional steps.
Challenge 2: less is more
During the development, multiple approaches to the display of information were tested. In our case, we found that “less is more”. While initially, the idea was to create a full 3D model with animated parts and overlay that on its real-world counterpart it quickly proved to be very distracting while you were working. Instead, as shown in the image below, only the instructional steps with a small arrow are shown to explain the steps to the user.
With this “less is more” approach, it became obvious that we also had to be smart with the rest of our interfaces. The goal was to create an interface that distracted the user as little as possible whilst working on their task. This led to a solution where the menu can appear and disappear in an instant. The menu is multi-functional as it behaves as a standard menu but also gives a reminder on which voice commands are possible.
Challenge 3: directing the user
The field of view of the Microsoft HoloLens is limited compared to human vision; when you wear it, only a part of your vision is covered with digital information. This makes it difficult to spot information or points of interest that you don’t see directly. Fortunately, this problem was tackled using the Spatial Sound feature of the Microsoft HoloLens. The points of interest in the application have a virtual sound source attached to them. When the user needs to pay attention to a specific point in the room the user will hear a sound. Of course, this sound disappears when the user is focusing on the point, preventing them from hearing the sound constantly.
Use Case
What is the professional use case for an application like this? Right now, we imagine that in the future using Mixed Reality in such a setting could help with training new personnel to use complicated machinery. This could potentially lower training costs significantly. With the current Microsoft HoloLens, there is still a noticeable movement of the overlay, about ten centimetres, when walking around the real-world object. This affects how precise information can be displayed and could cause difficulty with more detailed machinery.
Microsoft will be releasing the Microsoft HoloLens 2 soon. The demonstrations given by Microsoft show a lot of improvement in user experience and ergonomics compared to its predecessor. Hopefully, we will also see an increase in compute power and accuracy of the sensors.
This blog is written by Daan Gunnink, Software Engineer at Mediaan.