Augmented, Virtual Reality 101 for Geospatial Pros
Augmented reality (AR) and virtual reality (VR) are trendy topics as of late. Harvard Innovation Lab deems them “the next frontier for technological innovation.” In a 2016 report, Goldman Sachs suggested AR and VR “have the potential to become the next big computing platform,” comparing the technologies to the rise of the PC and smartphone. In one scenario, the report estimates a whopping $80 billion in AR/VR revenue by 2025 ($45 billion in hardware, $35 billion in software).
So is there a place for AR and/or VR in the geospatial vertical? And is there a feasible AR/VR business model that geospatial professionals can get behind? POB interviewed a few individuals from the private sector and academia who make a living examining this relationship. They say yes.
One of these individuals, Alexander Klippel, professor at the Penn State Department of Geography, says geospatial professionals can play a crucial role in AR and VR. Looking back at the geospatial advancements made in sensing and modeling over the years, he sees AR and VR as next steps in data delivery, a part of a widespread effort to move from static to dynamic through immersive experiences.
“Whether this is the floor of the ocean to mountain tops to other planets, we can make this information available now in an actual experience, and that’s the important difference, in an experience that actually allows us to be there,” Klippel says.
He views geospatial opportunities in AR and VR as twofold: 1. Providing content that makes AR and VR possible, including 3D and location data. 2. Acquiring the skills necessary to actually create AR and VR deliverables, as is done now with 3D point clouds.
What are Augmented and Virtual Reality?
While commonly grouped together due to their immersive digital nature, AR and VR are not the same. AR overlays digital information onto the real world that exists around the viewer. VR generates digital imagery of an imaginary or replicated world. Popular AR hardware includes Microsoft HoloLens, Google Glass and Magic Leap. Popular VR hardware includes Oculus, HTC Vive and Samsung Gear VR.
“Virtual reality is all about being there. It’s about immersion; putting the person inside the experience so they can look around, move and see things. Augmented reality is interacting with the real world, but with an overlay on top of that real world,” says Eric Wittner, product manager for CityEngine, 360VR and procedural technology at Esri.
A lot of people think of head-mounted hardware when they think of AR and VR, but AR can be displayed on a mobile phone or tablet using a camera to capture the real world on its screen. In addition, users can experience VR through a computer screen using their mouse or keypad to traverse. VR can also be displayed on a television screen using a gaming remote to navigate, or a mobile phone or tablet that responds to body movement.
Augmented reality requires a lot more data accuracy and precision than virtual reality does, according to Chris Andrews, product manager for 3D across the Esri ArcGIS platform. “If you’re looking at proposed pipe locations in virtual reality, you’re not going to get a sense of if they’re in the right spot or not against the real-world data like you would get if you were looking at the same proposed pipes superimposed on a view of the real world that you’re actually looking at. … Augmented reality absolutely relies on the fact that you have accurate data of the location of the things that are right in front of you.”
Which is More Relevant to Geospatial?
It is hard for Klippel to say that AR or VR is more important for geospatial professionals. He can see them evolving through a combination, a continuum, a spectrum, that doesn’t have a fixed boundary.
Andrews and Wittner say AR has far more potential applications to allow geospatial data and tools to be applied to real-world problems. With more than petabytes of information available, and mobile devices able to interact with that data and the world around them, AR and geospatial technology are already actively used, they say. Andrews says that pulling up Google Maps on a smartphone and looking up a restaurant nearby, in some sense, is a blend of the two. After all, it involves sitting in the real world and pulling up live digital geospatial data to gain more insight into that world that cannot be so conveniently accessed through the naked eye.
Another example of AR’s relevance to geospatial data in practice today is the PeakFinder app, Wittner says. It shows the landscape into the distance using USGS terrain data to identify peaks in the U.S. “And even if you’re sitting in the base of a valley and you can’t see anything, you can hold it up and, in essence, look through the mountain to see where the next peak is behind you. It’s integrated with a compass, so as you spin around you see the different peaks,” he explains. “That’s augmented reality that’s running, working and distributed broadly as an app right now, and one of my favorite examples.”
Over time, Andrews and Wittner expect to see more solutions that blend geospatial tech with augmented reality.
Powering the Platforms
In general, the geospatial industry is not a primary driver of data going into AR or VR experiences, according to Andrews. This is because the majority of AR and VR experiences — an estimated $11.6 billion worth of revenue in 2025, according to Goldman Sachs — are tied to video games. He says any time consumer technology is applied to the commercial industry, geospatial technology and other data that are required for such experiences pose a much higher complexity problem set. But, he says, “Geospatial expertise and technology is essential when the project that AR or VR is being applied to has to do with real-world spatial problems or structures, assets, things like that.”
Along with AR and VR, Klippel researches 3D modeling. He says 3D data and the LiDAR technology used to acquire it are becoming more accessible, both in price point and abundance. Geospatial professionals are leaders in the gathering, processing and delivery of 3D data, and Klippel views that expertise as invaluable for the creation of AR and VR solutions.
He highlights an important observation made in the 2016 Goldman Sachs report, suggesting content developers are reluctant to create AR and VR content without an installed base, while consumers are reluctant to buy technology without content to experience. With more data than they often know what to do with, geospatial professionals have a wealth of content stored away that could power AR and VR presentations. In cases where data is not on file, they can use their data acquisition expertise to deliver it.
“I couldn’t imagine a better content provider than geospatial technologies. So I think it’s an absolutely incredible time to see how these two areas — the sensing of the environment, the modeling of the environment and then also to make this information available in this completely new way — is coming together,” Klippel says.
Geospatial professionals don’t just have the opportunity to supply the data that makes AR and VR possible on the back end. They can be involved downstream with actual AR and VR deliverables. “I would not, for a second, say the main role of geospatial professionals is just to provide content and leave it at that,” Klippel says. “Most of our students are geographers, so they are creating these experiences. They are not just providing content so someone else can do whatever they need to do with it.”
While data is number one, in this case, Andrews says access to that data in a way that can facilitate a quality AR or VR experience is also important. “So if you’re trying to show a proposed urban plan, then you absolutely need not only the access to the geospatial information for the city or environment around it and the proposed plan, but you also need some way to translate the data from a geodatabase or wherever it is in geospatial land into formats and tools that can be used inside an augmented reality or virtual reality experience.”
He says the landscape of tools available to make that data translation is a heterogeneous one, consisting of many different game engines and non-game engines. Wittner says CityEngine is a content creation tool for 3D, and his team is working on streamlining workflows to Unity and Unreal, both gaming engines commonly used to build immersive experiences. 360VR, which does not utilize gaming technology, is a proprietary VR experience from Esri that works with mobile devices. The ArcGIS Runtime team is working with 3D Runtime, which is a lightweight application for mobile devices and desktop. A VR experience can be delivered directly from it, or the experience can be augmented by connecting Runtime to Unity to consume GIS data across services online and then display that directly into a VR experience.
“We’re kind of at the beginning of this. There are so many possibilities of different patterns for the ways people can consume spatial and virtual reality and augmented reality. There probably won’t ever be a single dominant paradigm because they all serve different needs and different purposes,” Wittner says.
There are a number of solutions that already exist for the creation of AR and VR experiences, Klippel says, and he considers them very accessible. He finds that it depends on what the available data looks like. “If you’re happy with just 360-degree images, that’s already extremely accessible. If your’e looking for more fidelity with respect to the presentation and interactivity, then you would need to go into the realm of game engines.”
The Unity gaming engine has proved flexible for his research, allowing the incorporation of LiDAR data to create an immersive experience. He expects it to become mainstream for augmented reality as well, and notes that Apple has released a development kit specifically for AR, ARKit.
So what applications combine geospatial technology and expertise with AR and VR?
One example falls under the smart city category. Wittner draws up a situation where a city manager is evaluating the proposals of a group of developers participating in a design challenge for a particular property. Those designs can be entered into software and published as separate 3D experiences consumed with the use of a VR device.
The manager can then walk around inside the different designs and view them from a human scale; not just a static 2D drawing. One developer’s proposal of a four-and-a-half story mixed-use downtown might feel cramped. Another proposal with setbacks at the second and third floors might offer a more open feel. Such important characteristics might not be noticeable without the advantage of actually traversing the downtown as it would appear in real life.
For augmented reality, Wittner offers up the example of a facilities manager walking the halls of their building. With the use of a mobile device or headset display device, they can see a virtual overlay of room conditions across the structure. “In the device you’ve got your camera looking at the real world, and then overlaid on top you have color-coded indications saying this room is vacant and it has a readiness for occupancy of 75 percent. So you can see things beyond the normal vision,” he says.
He tested the building case for VR as well using the Samsung Gear VR headset. Using a building information model (BIM), Wittner built a VR experience that allows users to go in and strip away the walls so they can see the electrical, water and HVAC characteristics. He points out that this remote ability to assess buildings in such an immersive way is valuable for employees who want to view building components from somewhere else in the world. “Then the idea of being able to actually have that in hand in a device, to be able to look through a wall virtually; most of the people I know from the facilities world would dream of having that. So those are the spaces I’m most excited about.”
Augmented and virtual experiences also call on geospatial tools for natural environment cases, like making climate change more accessible. Klippel is a part of a National Science Foundation project where VR provides model outputs about what a particular area might look like in 2050 in terms of tree species. He says the project is still in the abstract stages, but geospatial technology like CityEngine is being used to create an immersive experience.
“So whether this is climate change, the city of the future, the inside of a volcano — places that you cannot be for whatever reason — this is something that is becoming accessible to us,” Klippel says.
According to the 2017 POB 3D Trends study, demand remains strong for 3D services, with 73 percent of respondents indicating growing demand. New technology with improved capabilities was listed as a driver of increased 3D demand by 44 percent of respondents. AR and VR serve as innovative ways to deliver 3D data, moving farther from static and closer to dynamic, away from presentation and toward experience.
“Do I expect everybody to be wearing a head-mounted display, doing their whole job through virtual reality in a year or two? Probably not,” Andrews says. “But do I already hear the need for the geospatial group in large cities and large engineering firms to have some expertise in how to bring 3D or even 2D content into AR and VR experiences? I think there’s already demand for that. … I think it is important for [geospatial professionals] to be aware of these technologies so that they’re not surprised when they get asked about applying them in their day jobs.”
In the meantime, Wittner recommends geospatial professionals start to look at how their organization is collecting, consuming and distributing 3D geospatial content to prepare for AR and VR.