Point of Beginning

A Dynamic Coastline

August 31, 2011
A colorized point cloud of the Penobscot River near Bangor, Maine.

The idea started simply enough. In 2008, Fay Rubin, director of the New Hampshire statewide GIS known as the GRANIT System, began exploring the possibility of collecting LiDAR data on the Piscataqua River and several other watersheds that cross the boundary between Maine and New Hampshire. Collaboration with other agencies is a key tenet of the NH GRANIT program, so Rubin reached out to Michael Smith, director of the Maine Office of GIS, to find out whether Maine might be interested in participating in a joint effort. The idea began to take root.

Then, in February 2009, the U.S. Congress passed the American Recovery and Reinvestment Act (ARRA) of 2009, making $275 billion available for federal contracts, grants and loans. The move eventually injected $17 million into the USGS National Geospatial Program (NGP), $14 million of which was allocated for LiDAR and elevation data. As the potential impact of this funding was absorbed by various state agencies, the idea that originated in New Hampshire and Maine began to grow. “I was talking with Dan Walters, the USGS geospatial liaison for Maine, and we just started brainstorming,” Smith says. “We knew the stimulus funding was coming down the pipe, and we knew there were states with more clout than Maine and New Hampshire. I thought if we teamed up with the other New England states, then we would be able to bring something to the table that none of the other states would have, which was regional cooperation and coordination.”

The 8,000 mi2 project area included 2,893 mi2 in Maine, 902 mi2 in New Hampshire, 2,022 mi2 in Massachusetts, 1,074 mi2 in Rhode Island, 511 mi2 in Connecticut and 568 mi2 in New York.

Smith and Walters became increasingly enthusiastic as they began shaping a plan to bring millions of dollars in LiDAR funding to the region. But could multiple states effectively collaborate on such a complex project? Smith was determined to find out. “The problem we’re all facing in the coastal states is sea level rise and its impact on flooding and on development in general,” he says. “Everyone understood that LiDAR data would be invaluable in helping us plan for and mitigate sea level rise in our coastal communities. The question was whether we could work together as a six-state team and get the large amount of money we were asking for.”

The GIS coordinators in the small states of New Hampshire and Rhode Island immediately saw the benefits of joining with the other states to request ARRA funding for LiDAR data. Connecticut and Vermont also indicated that they were onboard. Massachusetts, however, which has the highest population of the New England states, was understandably a bit skeptical. What was the likelihood that USGS would award the requested funding to the six-state collaborative? And even if the ARRA funding was awarded, a substantial amount of matching funds would have to come from the states. How would each state ensure a return equal to the amount of its investment? Would the states be better off pursuing individual grants?

After challenging Smith and the other state GIS coordinators with these and other critical questions, Massachusetts agreed to join the collaborative effort. But then the group hit another roadblock: The USGS announced it would prioritize projects that focused on the coastline. This meant Vermont couldn’t participate. The team quickly formulated an alternate plan. “We realized it would be politically expedient to include New York since the state has so many people in the coastal area,” Smith says, “and there was interest from New York in being part of the project.”

Photo Science used TerraModeler and its own proprietary software to generate hydro break lines, assign elevation values, flatten water surfaces, and assure drainage monotonisity.

Following months of phone calls, meetings and heated negotiations, the six states, led by Smith, submitted a $2.7 million budget proposal on Nov. 30, 2009, requesting $1.4 million in ARRA funding. It was a staggering figure. “There were definitely times when I wondered whether all of our efforts would pay off,” Smith admits.

Within the USGS, the proposal generated a mix of excitement and trepidation. “Their goal of getting continuous LiDAR coverage from Maine to New York fit beautifully with our objective to obtain data along the coastal areas, and we believed the collaboration would lead to cost-effectiveness and consistent datasets across a large area,” says Teresa Dean, a geographer and ARRA coordinator for the USGS National Geospatial Program. “But working with a large group like this can be very challenging in terms of getting the group to agree on a data specification, and then working the financial end. We have had previous projects where the more partners that are involved, the harder it is because of all the complexities of it.”

Despite these concerns, USGS decided to move forward. “This project has so much value that we wanted to do it,” Dean says.

In January 2010, USGS awarded the requested $1.4 million in ARRA funding to the six-state collaborative to pursue the LiDAR for the North East project. It was the highest amount awarded to any individual project. Smith and the other state GIS coordinators were elated; however, the real work had only just begun.

Photo Science developed an innovative "bare water" classification to retain the actual water elevation surface while eliminating noise and outliers.

One of the first challenges the team had to face was that of timing. For the project to move forward, the matching funds from the states had to be committed. This meant that all the various details, from the project scope and specifications to the contract documents, had to be in place. “Each state has a different timeline and schedule for its budget cycles that doesn’t necessarily match up with the federal cycles,” Dean says, “so it was a challenge to make sure we met the rules and regulations for each entity to make the funding work.”

In Massachusetts, for example, all of the state-contributed funding had to be committed by July 1, 2010. The clock was ticking.

As Smith and the other state GIS coordinators began studying the USGS Geospatial Products and Services Contract (GPSC, referred to as “gypsy”) under which the task order would be issued, they soon realized they had another problem: Their original cost estimates had been too low. Instead of the anticipated $200 per square mile, the project was going to cost nearly $280 per square mile. The agreement that the six-states had so carefully crafted would have to be reshaped. Further complicating the situation was that each state was contributing a different amount of of matching funds, and some states wanted the data collected at higher point densities (1-meter nominal post spacing vs. 2-meter spacing), increased vertical accuracies (9.25 cm vertical root mean square error [RMSEz] vs. 15 cm RMSEz), and tide coordinated LiDAR acquisition (predicted mean low tide +/- 90 minutes) compared to that of the USGS base LiDAR specification (version 12) being used for the rest of the project.

Fortunately, Smith had an idea to keep the situation fair. “I applied a mathematical formula to show how each state could get an amount of data proportionate to their match,” he says. “When you start taking stuff away from people, they tend to get grumpy, but there wasn’t any of that on this project. We all saw that there wasn’t enough money. Communication and transparency were essential.”

The final project footprint covered 8,000 square miles along the Northeast coastline, with Massachusetts and Rhode Island putting in more funding to obtain a higher point density and improved surface definition compared to the rest of the states.

In June 2010, just prior to the Massachusetts funding deadline, the USGS issued the first task order to Lexington, KY-based Photo Science, a GPSC contractor, to begin work on the ARRA-funded portion of the project. A second task order for all other funding portions was issued to Photo Science in September 2010. “We were running right up to the wire getting all the details squared away,” Smith says.

Once the contracts were all in place, the next step was planning the LiDAR data collection effort. The data collection flights would need to take place in leaf-off conditions without any snow or flooding. Additionally, the region would need to be free of clouds and fog between the aircraft sensors and the ground. Data voids were unacceptable unless caused by water absorption or low infrared reflectivity due to asphalt, composition roofing, or filling in another flight line swath. With these parameters in mind, the Photo Science team identified the fall of 2010 and winter of 2011 as the optimum collection window.

Photo Science combined smoothed, best estimated trajectory (SBET) and laser point data to create unclassified return point clouds to LAS format v1.2 by flight line.

In a typical year, the team would have had plenty of time to gather most of the data before the snow arrived. This wasn’t a typical year. Rain and fog persisted for most of the fall. After watching the weather carefully, Photo Science assigned three aircraft to begin flying the region on Oct. 23. However, progress was slow. Over the next 51 days, there were only 21 possible flight days due to continued poor weather in the region. Then, on Dec. 21, 2010, a major snowstorm shut down almost the entire New England region, making data acquisition flights impossible. “We had unprecedented cold temperatures and unprecedented snow amounts, and it just lingered and lingered and lingered--well into March in most areas,” says Michael Shillenn, Photo Science vice president and project manager. “Then, once the snow melted, a weather pattern set in where we either had a cloud deck that was too low that we couldn’t fly under, or we had severe turbulence that impeded our ability to stay online and collect the data. It was a perfect storm.”

A storm was brewing in the state GIS offices, as well. Although everyone understood the unpredictability of the weather, promises had been made, and money had already been spent. “We had proposed and expected to collect at least 80 percent of the data in the autumn window of 2010,” Smith says. “In reality, only one area of Maine was collected, and nothing basically south of Bar Harbor was done in the fall except maybe some flights along the Hudson River in New York. I don’t think any part of New Hampshire, Massachusetts or Rhode Island was collected during that time. That created quite a bit of frustration among those partners. They were concerned that they had sold a bill of goods to their respective partners and weren’t going to be able to deliver on it.”

Digital elevation models (DEMs) and digital surface models (DSMs) created by fusing the LiDAR data with imagery provide substantial value. Shown here is imagery of Maine from the USDA National Agriculture Imagery Program (NAIP) draped on a DSM with line of sight (LOS) from a 20-meter tower placed toward the center (this picture) and draped on a DSM with a 10-meter raised water profile (next picture).

Once again, communication was key. “We tried really hard to keep everyone informed as to exactly what challenges we were encountering, where we were working and where we didn’t think we could work, and we solicited their support and help,” Shillenn says, noting that Smith and the USGS liaison Dan Walters were crucial links in the communication chain.

Ultimately, these efforts paid off. “It required a lot of understanding on everybody’s part--each state had to get through whatever issues they were looking at and come up with some alternative plans and expectations,” Smith says. “In the end we were able to diplomatically make it all work.”

Despite unfavorable weather patterns that persisted throughout the spring, Photo Science was able to resume data collection efforts in mid-March and finish by May 30, just two months behind the anticipated schedule. The firm allocated additional resources to the project to speed its completion; at one point, six aircraft were assigned to gather LiDAR data. This resulted in a flood of data that had to be analyzed and processed on the back end. “This was a very large project, but more than the physical size, it was a very challenging project in terms of its geographic and topographic diversity,” Shillenn says. “It went from Long Island, a major urban center in terms of ground features with one of the most tightly controlled airspaces in the country, to downeast Maine, which has mountains and a tremendous amount of relief, and just about everything in between. Those factors posed a bigger challenge to getting it all collected and processed than just the square mileage itself. And the weather only added to that.”

Drawing from experience gained through other challenging projects, such as the Humboldt Bay Ecosystem Mapping Project in California, the team was able to keep everything in perspective. “Our past experiences really helped us internally calibrate our own expectations as well as articulate and communicate with our clients what they should expect and give them a level of confidence that we had been down this path before,” Shillenn says. “Although this was probably the severest broad-area challenge we’d ever had in terms of the weather and snow conditions, I always believed that we would prevail. We had the right approach, the right attitude and the right resources.”

In all, the Photo Science team flew more than 2,000 flight lines and more than 23,000 flight line miles to collect nearly 13,000 1500 x 1500-meter tiles. Each tile is being converted to the LAS v1.2 format and is being sorted according to standard LiDAR classes, including unclassified/nonground (Class 1), ground/bare-earth with extra points (Class 2), low or high noise (Class 7), water (Class 9) and ignored ground near breaklines (Class 10).

The team also developed a new classification--bare water (Class 19)--to address challenges in processing the coastal water data. Temporal and tide differences in the missions resulted in large differences in coastal water levels. Using routine hydro flattening in an effort to assign uniform elevation would have resulted in a loss of near-shore and island bare-earth data. “By using a bare water classification, the team is able to retain the actual water elevation surface while eliminating noise and outliers,” Shillenn explains. “The classification also enhances islands smaller than one acre, wave action and near-shore shallow water definition.”

The final deliverables will include all return data in the raw point cloud (delivered by flight swath in LAS v1.2), a fully classified point cloud, a georeferenced bare-earth digital elevation model (DEM) raster delivered in an ERDAS IMG format, and shapefiles of hydro breaklines, as well as project reports and metadata. Substantial portions of the data have already been delivered, and the remainder is expected to be delivered by the end of the year.

The resulting dataset will be the largest continuous piece of LiDAR data from a single project and will provide substantial taxpayer savings as the value of the data is extracted over the long term. “The beauty of the project is that we’ll have continuous coverage from Maine to New York,” Dean says.

Photo Science is a MAPPS member firm.

Smith believes most of the data will be consumed as digital elevation models for applications such as identifying coastal erosion, determining flood risk, conducting studies on ecosystems, and mitigating the impact of sea level rise. The DEMs have a broad appeal, he says, because they’re instantly and widely available to a number of people without further training and can be consumed in any GIS package. However, both Smith and Dean believe that the raw LiDAR data will become increasingly valuable in the future. “For the USGS, LiDAR data is primarily being used to create digital elevation model data,” Dean says. “But the value of LiDAR data itself is becoming widely recognized. There are many more applications that can be accomplished with it. I don’t think we’ve begun to scratch the surface.”

Shillenn agrees. “Within the last year, there has been an explosion in the popularity, versatility and utility of LiDAR data to support a whole host of applications,” he says “The software manufacturers are working hard to catch up, and they’re building the tools to directly ingest, analyze and edit LiDAR data in its native format, the robust LAS format, which carries rich information about each point in the point cloud. Even Esri is involved--the latest version of ArcGIS has a very robust set of LiDAR tools that are going to really mainstream the use of LiDAR data.”

As the demand for that data increases, other states and other consumers will undoubtedly look to the trailblazers for ideas on collecting and applying that data as well as collaborating to fund LiDAR projects. For his part, Smith is excited to serve in that role. “I’m most proud of the fact that we got six states to work together,” he says. “It could have been a feeding frenzy where we were all competing against each other for the available USGS funding. Instead, the New England states banded together and were able as a unit to come up with a much better proposal and a much better model for funding geospatial assets. I think that was the most beautiful part of the whole project--even better than getting data for the entire Maine coast.”

Aircraft/Sensor Configuration

• Cessna U-206s

• Four Leica ALS 50 II & ALS 60 with a max pulse rate of 200 KHz, multi-pulse

• Three Optech Gemini ALTM with a max pulse rate of 167 KHz, multi-pulse

• Applanix POS/AV DG position and orientation systems

• Trimble 7500 dual frequency GPS receivers (base stationing)

• Calibration – home base and local

• Horizontal: UTM Zones 18 and 19, NAD83, meters

• Vertical: NAVD88, meters (most recent Geoid model approved by the NGS)

Managing LiDAR Data

In addition to collecting and processing the data, Photo Science is also playing another valuable role in this project--that of data management consultant. For example, Shillenn recently teamed with Michael Umansky of Applied Imagery, developer of the Quick Terrain Modeler software, to present a session at the Maine GIS User Group summer meeting on “What to Do with LiDAR Data.” According to Smith, the insights were invaluable. “I’ve been involved in this project right from the start,” he says, “but in those two and a half hours I learned a ton.”

Shillenn says Photo Science is increasingly involved in data education and exploitation. “We are data developers--that’s our legacy and heritage. But clearly, it’s leveraging the intelligence and value, and extracting the derivative information from really rich datasets like this one with high-resolution imagery that allows us to support clients at all levels of government and do things like change detection, impervious surface analysis, land cover mapping, support environmental/green infrastructure development. A lot of good work can be done by going beyond simply developing data to helping our clients leverage those great datasets to solve problems. That’s really where the opportunity is and where there’s a lot of excitement at Photo Science.”

LiDAR post processing

Automated and manual filtering of the bare earth surface model during the post processing effort results in the classification of 90 percent of artifacts, 95 percent of outliers, 95 percent of vegetation and 98 percent of buildings. Photo Science uses Applied Imagery’s Quick Terrain Modeler (QT Modeler) to check the bare earth dataset at the tile level. GeoCue’s LP360EQC and Esri’s ArcGIS software are used to assess the quality of multiple tiles at the macro level. Two hundred fifteen supplemental GPS ground control points obtained by James W. Sewall & Co. survey crews are also being used to check and validate ground classification during the post-processing phase. (An additional 120 GPS quality assurance check points established in open terrain are being used by USGS quality assurance staff to validate fundamental vertical accuracies but are not being used by Photo Science in its workflow.)