Bosch Connected Devices and Solutions
User Experience

Interplay of Augmented Reality and IoT

Experience the Bosch XDK in combination with Microsoft HoloLens

guy uses hololens

Extending the real world with virtual elements: Simply put, this is how Mixed- or Augmented Reality (AR) can be described. What is behind this technology? What potential does itoffer? This article will tell you more.

Popular Augmented Reality examples

Today AR is already being used in several areas. However, it’s not necessarily obvious at first glance. For example, in the gaming sector: be it the well-known AR app "Pokémon Go", the current hyped "The Walking Dead: our World" or "Harry Potter: Wizards Unite", which is still to be released in 2018. Moreover, AR-technology is also being used in the broadcasting of sporting events. Let us remember the 2018 World Cup in Russia: statistics and moving the line indicating the offside position were used to analyze the football matches.

pokémon Go Augmented Reality
Augmented Reality App: Pokémon Go

Augmented Reality in Industry

At the same time, Augmented Reality is slowly gaining ground in industry. In some areas, industrial workers use AR glasses as an auxiliary tool or as a Training Simulator to be prepared for specific situations. This is a good opportunity to train new workers for different production processes. Furthermore, AR glasses already extend reality for plant maintenance staff; device information on common and special models as well as maintenance instructions are displayed in the user's field of vision and facilitate repair work.

A popular example for AR glasses are the Microsoft HoloLens. They are so-called mixed reality glasses that display 3-dimensional projections into the natural environment. By using the HoloLens, reality is enriched with additional information such as 3D objects, controls and text through graphical representations.

What is the HoloLens Hackathon?

During a two-day HoloLens hackathon at the knowledge factory in Balingen-Weilstetten, the focus was on augmented and mixed reality. The technology and programming event focused on the use of Microsoft HoloLens technology and the associated future opportunities for industry. The goal of the Hackathon was to create innovative business applications for the HoloLens in a time span of 24 hours. Our team consisted of students Marius Binal, Jan Dietrich, Lisa Lämmle and Steffen Hecht and was led by our BCDS staff members Tharani Tharan Muthukumaran and Dr. Frederik Wegelin as the team manager.

Among our competitors at the Hackathon were many interesting ideas. One team, for example, linked the HoloLens with Google Earth, while another team ensured that object recognition using QR code is possible.

HoloLens Hackathon
Students at work @ HoloLens Hackathon

Microsoft HoloLens & Bosch XDK Use Case

In contrast to these ideas, our concept for the hackathon was a „real-time use case“, focusing on the sensor device Cross Domain Development Kit (XDK), which measures data from multiple sensors, e.g. temperature, humidity, pressure, vibration, light intensity, noise and more. The recorded data can be sent out via Bluetooth or WLAN to the receiving device.

Therefore, our main task of the hackathon was to visualize the live object and to generate the connection between the XDK and the HoloLens. In this case, we used a refrigerator as a live object for XDK data acquisition.

In the physical world, we applied the first Bosch XDK at the door of the fridge to detect its movement, the temperature and if the lights inside are turned on/off. At the back of the fridge, we applied the second XDK to measure its current power consumption and the vibration and temperature of the compressor. In the next step, the XDK sensor data was visualized with the help of the HoloLens. In the cyber-physical world, you could now see a digital twin of the refrigerator as an animated 3D model. It shows if someone opens or closes the door and the measured live sensor data. Although this is only a small use case, it shows what is already possible within a hackathon of 24 hours. Moreover, this use case is applicable to any other remote or live object.

iot augmented reality use case
Illustration of the Augmented Reality & IoT Use Case at the Hackathon (Desktop only)

In the video above, you can see two different screens. On the large screen, you see Tharani Tharan Muthukumaran getting a cold drink from the fridge. In the synchronized, smaller video, you see the view from the HoloLens’ perspective. It shows the live movement of the refrigerator door and the changing sensor data. The inside temperature and power consumption increased when Tharani opened the door. Moreover, the HoloLens showed a change in light and animated the door while it opened. Unfortunately, HoloLens has the small disadvantage that the quality of the camera does not allow high-resolution images or videos to be shot. For this reason, the text in the small video is a little blurry. This shortcoming, however, is going to be remedied with the release of HoloLens2. Although there is still some room for improvement in the AR technology, it has great potential for the future.

Potential of Augemented Reality and IoT

In combination with IoT sensor devices, such as the XDK from Bosch, users are now able to visualize a digital twin with live data. Therefore, concepts like Condition Monitoring, Machine Monitoring, Process Monitoring and Predictive & Remote Maintenance will have completely new possibilities. One possibility would be, to maintain a machine remotely and project it into the cyber-physical world without being physically present. The virtual commissioning of entire plants may also be tested before a production chain is ultimately put into operation. This is a great potential to reduce costs and to save money and time.

Many other areas of application are possible in the interplay of these two technologies. These include, for example, visualizing car diagnosis or the enhancement of the user experience during a project demonstration and many more.

Retrospective of the HoloLens Hackathon

The combination of the HoloLens and the XDK at the Hackathon was very revealing for us. On one hand, the XDK with its strong in-built algorithms, has enormous sensor accuracy, and is applicable in almost every use case, protocol and vendor. On the other hand, the HoloLens, offers the possibility to see the world from a new perspective. This interplay opens up many new fields of application. We will definitely keep an eye on this field and are eager about the future.

To sum it up, we can say that the hackathon was a great experience for us. We achieved an awesome team effort and had the chance to gain insights in a new programming framework (Unity 3D). We would like to thank every person involved for this great opportunity. Special thanks to our four talented friends and team members for their input, motivation and determination to transform our concepts to a working solution. We are looking forward to stay in touch with you. Another special thanks to the organization team (IHK, Create 3D, M&M software, Esskreation and many more).

To find out more about the HoloLens hackathon, visit this website (german only):

Share this on:


Do you want to find out more? Contact us.

Bosch Connected Devices and Solutions GmbH