Dealing with 70,000 Miles of LiDAR Data
Geospatial data has been shared by government entities for decades, but just within the past eight to 10 years, the process of freely sharing information with the public has been formalized and more commonly referred to as “open data,” says Andrew Stauffer, product manager for civic technology at Esri. He works with Esri’s research and development center on the company’s open data product. While the movement of democratizing data is not necessarily new, he points out that it has definitely evolved and improved over the years.
One example of open data advancement is a project mandated by the Washington State Legislature in 2015, which involves the collection, processing and sharing of LiDAR data with the public by the Washington Department of Natural Resources (DNR). The goal is to cover the entire state. “The LiDAR data collection, I find fascinating,” Stauffer says. “It’s an expensive dataset to collect, it’s very rich and it’s very dense. So you definitely need to know what you’re doing in this space.”
The initial point of the program was to foster a better understanding of geologic hazards, like landslides, but the DNR’s LiDAR team realizes the wide-ranging applicability of LiDAR data acquisition. “We’re also conscientiously collecting LiDAR data at a high quality so it can serve forestry, wildfire mitigation, precision agriculture, flood mapping. So even though our particular focus is on geologic hazards, we understand that other groups are going to be looking at this data and downloading it and using it for their own application,” says Abigail Gleason, LiDAR manager with the Washington State DNR. She is one of two officially responsible for overseeing the LiDAR project.
As of Feb. 10, Gleason says at least 41 percent of the state had been accounted for. That includes 6-foot resolution and 3-foot resolution, 30.3 percent at 3-foot resolution or higher. Gleason says about 23.8 percent of the data is high enough quality for geologic hazard assessment. The plan is to eventually re-scan areas for the sake of analyzing change over time and benefiting from technological advancements that will inevitably occur.
So far, the project has consisted of specific sub-projects covering one area at a time. The management of collection starts off with establishing funding contracts with partners including the U.S. Geological Survey (USGS) 3D Elevation Program (3DEP), counties, cities, utilities and private companies. The Washington State DNR LiDAR team does not acquire data on its own, so partners are extremely important in helping pay private LiDAR specialists to carry out the work, usually by plane and sometimes by helicopter. A partner is typically a stakeholder in a given sub-project; the area they are helping fund data collection of is usually one they would benefit from having 3D data on.
“We like to use our local partners and get an understanding of how they’re going to use the data, what they’re going to use the data for, any special requirements that we should put against the collection like low tide or certain times of the year; try to work with our local group to satisfy their requirements,” Gleason says.
Then, her team meets with the contractor responsible for data acquisition and processing. They lay out project cost, when collection will be carried out, when they expect to start processing and when they expect to deliver — a basic timeline of the project. Gleason says her team is very hands-on throughout the time the contractor is working to create a deliverable, keeping abreast of schedule changes, unexpected limitations, etc.
Once the data is delivered, it comes to Gleason’s office where she conducts quality assurance. She determines if all expected files are included, if they are the agreed upon naming convention, whether they are readable and if they are in the correct projection. “Can we read them? Is there something in the processing that can say this needs to be reprocessed? There’s a lot of back and forth with the vendor at that point in case anything needs to be reprocessed, hopefully not re-flown, but that is a possibility,” Gleason says.
The LiDAR team uses ArcGIS quite often to index and create the database. They are in the process of getting Quick Terrain Modeler. Gleason says it is their preferred LiDAR point cloud visualization tool, and that she can see herself mostly using that in the future to visualize the point cloud, quality assure it and drape DEMs on top of it.
Once the LiDAR team decides the data is good to go, they have two separate processes. For the DNR team, they put things like slope aspects on the internal network. For the general public, they use a different external process that places the data on the LiDAR portal. “With that, we’re making sure that all of our products … are the same tiling scheme. It’s a standard tiling scheme where we make sure they’re all in the same projection, that there’s a database that tracks each of these tiles. We work with our portal partners to deliver that data and then load it up onto the LiDAR portal.”
Those who access the portal can check boxes next to the areas they are interested in, which are named according to location and date. Then a window pops up indicating the number of files selected and their approximate size. Once the user selects the download option, the files are zipped up and saved to their computer. Right now, digital surface models, digital terrain models and hill shade options are available. Gleason says the plan moving forward is to offer LAVs. “That might change in the future. We’re learning more about how to do it, how is the best way to do it, but mostly the data … really just sits on a server and there’s a URL that points to them, and when a user clicks on that tile, there’s a database in the background that says, OK, that user clicked here, so that’s this tile; I’m going to grab this URL and I’ll put it in my bucket and it gets zipped up. That’s kind of the back end of it,” she says.
The aerial LiDAR data collected is usually topographic, but Gleason says if partners want bathymetric and contractors have the capability, the Washington State DNR LiDAR team is open to it and can write specifications. In the case of bathymetric LiDAR, Gleason points out the value in looking at when a lake, for example, has historically been drawn down to cover it then and get as much of the shoreline and its submerged elevations as possible.
Such considerations are not special to the bathymetric approach. With topographic data collection, Gleason says it is important to take the state’s heavy presence of deciduous trees into account. She says the trees are a large part of why they like collection to be done at 8 ppsm or better; below that, penetrating leaves becomes more difficult. They are very conscientious about collecting data in leaf-off conditions, so they tend to conduct fly-overs of densely vegetated areas in fall, winter and early spring. “That just increases the chances that your LiDAR pulse is going to hit the ground,” Gleason says.
At higher elevations, she says there are mostly coniferous trees, so there really is no leaf-off period of time. In these areas, snow becomes the factor to consider. They simply avoid acquisition during the presence of snow if possible. “We don’t want to collect when there’s snow on the ground because that, of course, doesn’t give you an accurate measurement of the ground.”
One of the biggest challenges Gleason says the LiDAR program faces is making sure the public is aware of what is going on. She says she isn’t sure her team has met that challenge yet, but that they continue to work toward it. “Educating people about LiDAR data is something that’s very important. … You definitely run into different perceptions of what it is and what you can do with it and what it means.”
Data with a Purpose
While LiDAR data collection is fascinating to Stauffer, especially for the purpose of open data sharing, he says he can most see it fueling innovation within the state from tech talent that resides there and academic researchers looking to advance geographic studies. “LiDAR isn’t really for the general citizen, so hopefully this is going to spur research projects helping with a startup and different things like that,” he says.
That said, Stauffer and his colleagues at Esri have been spending a lot of time thinking about everyday end users who aren’t necessarily experts in LiDAR, but may benefit from practical information made possible by geographic information systems (GIS). He says he has seen a lot of GIS data shared that never really gets used. “The current status quo right now is you go to an open data site, you search, and you download the data.”
Stauffer’s problem with the status quo is that the majority of his neighbors have no idea what a CSV file, let alone a Shapefile is. On the other hand, he says, those same neighbors are very interested in the location of schools and neighborhoods. One of the things Esri is working toward is providing more practical tools to everyday citizens and getting cities to not just share data, but focus on initiatives. “It’s not just about 500 or 600 datasets; good luck and go find it and do what you want,” he says. “It’s really about we have this education; please look at all of our schools, grade level, neighborhoods, district boundaries, bus stops. Then, if somebody wants to go in there and annotate on top of that, how can we enable the other 90-plus percent of my neighbors to actually make an informative map that they can bring to a PTA meeting or another school board or city council meeting? They’re not going to go use developer tools to do that.”
With mobile devices now a standard, Stauffer says location data is critical information and that governments stand to benefit from sharing data into the screens of citizens. He says it can improve commute times, lower carbon emissions and help improve quality of life if there is less traffic congestion. “In addition to that, other datasets like emergency shelters are critical during a time of disaster. Big tech companies have the ability to help citizens by automatically routing them to these locations.”
A big part of being more thoughtful with respect to the effectiveness of open geospatial data is tying in the real-time aspect, which Stauffer says is one of the bigger challenges. He points out that the existence of geospatial data is great and the fact that it is freely available to the public is too, but, “If it’s rapidly changing, what do we do?” He says he has seen intriguing examples of open data sites that include parking information.
With real-time indicators being so key, Esri is working with WAZE, a traffic navigation application that depends largely on real-time, crowdsourced information from users. Stauffer says that if cities work with it, if, for example, a permit is issued to close a road for construction for a set period of time and it is mapped, an application like WAZE could help connect cities to citizens by helping automatically reroute them as they navigate to avoid the road closure and resulting congestion. “They’re actually meeting me in my screen or in my dashboard, helping me route a little bit more effectively to work. Those are some interesting and fun challenges we’re working on and I think could be a big thing in the future of geospatial data,” Stauffer says.
As someone who works closely with open geospatial data as a concept, he sees it on the rise and anticipates it to continue at a rapid pace. “The problem in the past is that there was open data for the sake of it, and now we’re targeting it on open data with a purpose.”