Catalina HoloLens Experience [1:37]

Developing the Catalina HoloLens Experience

Imagine a museum where objects come to life as you walk amongst them. With augmented reality technology, that’s becoming possible—and museums are starting to combine virtual and physical displays within their galleries, enabling visitors to explore exhibits and artefacts in new ways.

Augmented reality (AR) or mixed reality (MR) both describe visual experiences that merge the virtual with the real world – presenting 3D models, holograms and virtual objects within a physical environment.

MAAS and CSIRO

In 2017 the MAAS Digital Studio team partnered with CSIRO’s Data61 Immersive Environments Lab to develop a ‘mixed reality’ experience using the Microsoft HoloLens, a self-contained, holographic computer. The Digital Studio team is responsible for the Museum’s website, online collection and in-gallery interactive experiences.

This collaborative project aimed to prototype what mixed reality technologies might offer in designing future museum visitor experiences. The subject was a celebrated aircraft from the MAAS transport collection – the Catalina flying boat Frigate Bird II.

Thinking 3D

One objective of the project was to explore the sorts of applications that may be possible if the MAAS collection were to be digitised in 3D.

Imagine being able to browse our online collection, and then download a 3D hologram of an object to place within your room.  Imagine a classroom where a teacher could use any of our objects in their lesson, or print them out on a 3D printer for students to handle. These sorts of uses of 3D are quickly entering the realm of reality.

These technologies could offer virtual access to our entire collection, not subject to the constraints of available gallery space.

The Catalina’s 1951 Voyage

The project team had a great story to work with. In 1951, the swashbuckling Australian aviator Captain PJ Taylor flew across the South Pacific from Australia to Chile, via Tahiti and Easter Island, in the Catalina Frigate Bird II. His mission was to find the shortest air route from Australia to Chile, and both Taylor and the Frigate Bird II had some amazing adventures along the way.

MAAS Curator Damien McDonald went through the archives to find a set of fascinating photographs and documents from the historic expedition.

Developing the application

The next step was to create holographic assets – a map of the voyage that can be displayed on a table, a gallery of images from each stop on the journey, and a holographic 3D model of the plane itself – complete with whirling propellers and the sound of the powerful Pratt & Whitney engines at work.

The result is a mixed reality application delivered on the HoloLens platform that feels like a virtual diorama, designed for visitor groups.

Viewers can navigate a holographic map of the expedition, moving through waypoints on the map using gestures. Archival photos with informative captions accompany each stop. The model of the Catalina hangs in mid-flight, and viewers can walk around it, exploring significant aspects of the aircraft’s design.

Technology and learnings

The Catalina HoloLens application was developed with Unity Engine and Microsoft Mixed Reality Toolkit, an engine and code for developing virtual games. The team used a 3D model of the Catalina sourced from a WW2 game developer. Custom textures and paint were added to match the model to the real Frigate Bird II. The application was designed as a networked experience, with a ‘host’ leading a number of ‘guests’ through the presentation. The system uses a technology called ‘Spatial Anchor Sharing’ to create the shared experience so all participants view the same objects. Another significant feature was custom Bluetooth peer-to-peer networking without requiring a server.

While mixed reality technology is yet to see widespread uptake at a consumer level, developing this prototype was an opportunity to learn about 3D modelling, mixed reality, voice interfaces and storytelling using these emerging technologies.

The prototype Catalina application has since been used to demonstrate mixed reality and holographic interfaces to small groups within and outside the museum. Reactions have been very positive, with many viewers expressing a sense of wonder at trying out a holographic experience for the first time, and curiosity about the potential for these technologies in museums.

In May 2018 MAAS and CSIRO were awarded an iAward for Innovation by the Australian Information Industry Association for the Catalina HoloLens project. The award was presented by the Premier of the ACT Andrew Barr at a ceremony in Canberra.