AIG HYMAP USA GROUPSHOOT

Tutorial - Getting Started With Your HyMap Data

What you've got
You probably received a whole box of CDs and are saying to yourself, "what have I gotten into!” Lets make it a little easier to start with. First, you have two types of data for your site, 1. Radiance, 2. Apparent Reflectance (Effort-Corrected).  The Radiance data can be recognized by examining the “Processing:” label on each CD. This is the 2nd line of text from the bottom of the CD. The radiance data say “Processing: Radiance”, the apparent reflectance data are labeled “Processing Reflectance”.  Unless you're into the do-it-yourself mode, you can save yourself a lot of trouble by taking all of the Radiance CDs, putting a rubber band around them, and setting them aside. You've reduced your data volume by 2X. We'll be adding an additional tutorial later for those customers who want to start with the radiance data and do everything themselves.

Sponsor Frequently Asked Questions (FAQ)

Take a look at what's on the Reflectance CD
Each Reflectance Data CD contains the Apparent Reflectance data (Calculated from the Radiance data using ATREM, then "Effort" Corrected for residual atmospheric and instrument effects), HyMap data products, sensor information, and/or aircraft ephemeris data. All the files have the same root name and an  underscore and additional descriptors, if needed. The root name indicates the sensor, date of acquisition, and flight, run, and scene numbers. For example, HyMap data scene 1 acquired on the second run of the first flight September 29, 1999 has the following root name:

  HY19990929f01r02s01
  where

  HY            Hymap sensor ID            
  19990929      Date of acquisition
  fxx           Flight number
  rxx           Flight run number
  sxx           Scene number (run broken into multiple files,
                numbered sequentially).

The following files are provided on the Reflectance Product CD. Image files also have ENVI headers with a final extension of ".hdr".

File Extension Description Details
_effort Effort Data: BIL apparent reflectance data
_effort_gainoff.txt Effort Gains and Offsets: ASCII text file used to generate effort data from ATREM data
atrem_input.txt ATREM Input File: Parameters used for ATREM processing
_vapor ATREM Water Vapor Image: Extracted from the AVIRIS data using ATREM
_trans.txt ATREM Gas Transmission: ASCII File of Gaseous Transmission Spectra from ATREM
_atrem_wav.txt ATREM Wavelength file: Band Number, Wavelength, FWHM
_geo Geometrically Corrected Image:  North-oriented True Color Quicklook image in UTM coordinates (typically on second apparent reflectance CD in multiple CD sets)
_geo_igm Input Geometry File: UTM Easting and Northing for each original pixel
_geo_glt Geometric Lookup Table File:  Contains information about where each input pixel goes in a geocorrected output image
_report.txt Output Report File: Describes processing parameters
.log Flight Acquisition Log: HyMap acquisition parameters
_ephemeris.txt Aircraft Ephemeris:

Line number, UTM X, UTM Y, altitude, pitch, roll, and heading

 

Verify Site Coverage
Probably the most important file on each CD to get started is the “geo” file. This is a quicklook of the data in georeferenced form.  It can be found by looking for the file with "_geo" at the end of the name.  Open this file and display the bands labeled "Band 14", "Band 8", and "Band 2" (RGB) to see the spatial coverage of your data as a true color image.  An ENVI header is provided for convenience when opening and displaying in ENVI, however, these data can easily be displayed using other software.  The key image information is contained in a text file with "_geo.hdr" at the end of the name.

One sponsor received their data and pointed out "I opened several quick look images that came with the data. With the bands set to 14, 8 and 2, these "true color" images are very blue and I am wondering if this is something I did wrong or the way it supposed to be?".  This occurs because of the mask values on the image inserted by the
geocorrection. To get a good contrast stretch, make your main window small enough so that it fits inside the image itself, then select Functions->Display Enhancements->Default (quick) Stretches->[Image] Quick 2%. This should produce a properly stretched image.

I ordered 5m data, why does the CD say 4m?  How do I determine true pixel resolution?
The pixel spacing of the geometrically corrected image is marked on the CD, labeled for example as "GSD: 4m" at the end of the 3rd line from the bottom of the CD.  This pixel size is set by AIG during the geometric correction.  You can find the true pixel resolution of the uncorrected data by examining the "_report.txt"  file on the CD by printing or using a text editor.  The "mean downtrack line spacing (m):" parameter indicates the mean GSD in the flight direction.  The "nadir across track spacing (m):" indicates the actual GSD at nadir.

Examine the Apparent Reflectance Data
Open the "_effort" file using ENVI or another display/analysis program. Display a grayscale image (Band 14 is a good one), a true color image (Bands 14, 8, 2),  or a False Color Infrared image (Bands 28, 14, 8).  Spatially/Spectrally browse the image by panning the display with a spectral profile displayed. Familiarize yourself with the spatial and spectral characteristics of your data.

I notice some spikes in the reflectance data plot.
Why are these in the data and what do I do about them? (Mark bad bands)
You may notice  "spikes" in the reflectance data around bands 62-63 and bands 94-95.  These bands are on the edges of the 1.4 and 1.9 mm atmospheric bands, where most of the signal is obscured by the atmosphere.  The best thing to do is to mask these out as "bad bands" prior to processing.  Depending on your actual atmospheric conditions, you may have to mask several bands on either side of these bands as well to totally eliminate the spikes.  This can be accomplished by selecting "Edit Header" from the ENVI File menu, choosing Edit Attributes->Bad Bands List, then using "control click" (depressing the Ctrl key and clicking with the left mouse button) on the bands to be excluded from the plots and future analysis. Be sure to save your results (you may have to change file protection on the header if you copied it from CD).  The next time you plot a spectral profile, these bands will not be displayed.  Additionally, they will be removed from the dataset the first time you complete an ENVI processing procedure that creates a new output file.

Why do there seem to be two or more apparent reflectance CDs for each flightline?
Because we've elected to distribute the HyMap data on CD, we've been forced to break flightlines into segments that will fit within the 650mb CD limitation.  Both the radiance data and the apparent reflectance data have been broken up this way.  You can tell which segments go together and how by looking at the "Flightline Id:" on each CD.  The last 3 characters of the Flightline Id indicate the scene number.  Match the flight number and run number, then look for the sequential scene numbers.  For example, a HyMap data set broken into 3 scenes would have the sequential designations of:

  HY19990929f01r02s01
  HY19990929f01r02s02
  HY19990929f01r02s03

Only the CD with the first HyMap data segment has all of the ancillary HyMap data files.  Subsequent CDs with additional segments contain only the effort data.

How do I reconstruct multiple segments into a single file for analysis?
It may be easiest just to use the virtual mosaic to do the processing, however, this treats the data as BSQ and therefore programs that work best on BIL data (e.g. MNF Transform, PPI, etc) may be somewhat slower than using a BIL file.  You can convert the data to BIL by either converting the virtual mosaic on-the-fly, or by first making an output mosaic, and then doing a BSQ-BIL conversion.  Our recommendation is to convert the virtual mosaic directly to BIL before doing any additional processing. Here are the options, the decision is yours.

To make the conversion, first copy each segment from the multiple CDs to your disk.  Be sure to copy the "_effort" and the "_effort.hdr" files for each segment. If you want to just look at the data in one piece, the best thing to do is to create a virtual mosaic using ENVI.  If you want to process the data as a whole, you really need to mosaic the data to new output file.  Unfortunately, ENVI 3.2's default mosaic output is a BSQ file, so you should also convert the BSQ file to BIL (optimized for spatial/spectral processing), otherwise all of the spectral processing will be painfully slow.   The next version of ENVI will allow you to directly create a BIL mosaic from BIL data, but for now you've got to do the manual conversion.

Creating the Virtual Mosaic:  Open each segment of the dataset using ENVI.  Select Register->Mosaic Images->Pixel Based Images from the ENVI main menu.  Click on the Import menu item at the top of the Pixel Based Mosaic dialog and choose "Import file without feathering".  Select scene 01 from the Mosaic Input File dialog and click OK. Click again on the Import menu item at the top of the Pixel Based Mosaic dialog, choose "Import file without feathering", and select scene 02 from the Mosaic Input File dialog and click OK. Repeat for as many scenes as you have for the complete flightline. Now enter the "Y Size" for the output mosaic in the appropriate box at the top of the Pixel Based Mosaic dialog.  This can be determined by adding together the number of lines for each image included in the mosaic, or by examining the "_report.txt" file and looking at the input file size. For example, a line in the "_report.txt" file will say something like "Input File: 512 x 5512 x 128 [Integer]".  The 5512 dimension will be your output mosaic Y Size. Now click on each image in the mosaic and enter the starting line for that image in the "YO" box in the Pixel Based Mosaic dialog. The starting line of each segment can be determined as "number of lines in the preceding image(s)+1". Once all of the images have been imported and positioned, select "File->Save Template" to create a virtual mosaic.  Give the template a file like rootname.mos, click OK to build the mosaic template, then go to the ENVI File menu and open the template file using File->Open Image File.  The virtual mosaic can be viewed and processed as if all of the data were in one input file.

Converting the Virtual Mosaic directly to BIL
Once the virtual mosaic is created and opened as above, then the simplest, and quickest way to convert to BIL is to convert directly from the virtual mosaic file.  Select "Utilities->File Utilities->Convert Data (BIL, BSQ, BIP)".  Choose the virtual mosaic image and click OK.  In the File Convert Parameters dialog, click on the "BIL" radio button to select BIL output, enter an output filename, and click OK to start the processing.  The converted data will appear in the Available Bands List when processing is completed. This step may a moderate amount of processing time (around an hour for an average flightline), but will be well worth it because of the time saved later when using the BIL data for spectral processing.

Creating a mosaicked output file and converting to BIL (in-place):  Alternatively, follow the procedure above, then once all the images have been imported and positioned, select File->Apply, enter an output file name, and click OK to create the BSQ mosaic file. The individual bands will appear in the Available Bands List when the processing is completed.  Now, convert the mosaic into BIL by selecting "Utilities->File Utilities->Convert Data (BIL, BSQ, BIP)".  Choose the mosaic image and click OK.  In the File Convert Parameters dialog, click on the "BIL" radio button to select BIL output,  click on the toggle arrow to change "Convert in Place" to "Yes", and click OK to start the processing.  The converted data will appear in the Available Bands List when processing is completed. These two steps may take substantial processing time (several hours), but can be used if the direct convert option above is not an option.

Where do I go from here? What kind of processing do I need to do?
AIG recommends using ENVI and a standard processing methodology to achieve best results.  This typically consists of the following:

MNF Transform to reduce spectral dimensionality
Pixel Purity Index (PPI) to reduce spatial dimensionality
Thresholding of PPI to further reduce spatial dimensionality to size that can be interactively visualized
n-D Visualization to select endmembers
Mixture-Tuned-Matched Filtering (MTMF) to map spatial occurrences and abundances
Geometric Correction of results and map output

We typically analyze the 0.4 ~ 1.3 mm and the 2.0 - 2.5 mm regions separately, then combine the results at the end.  Please see the ENVI on-line or printed tutorials for additional details.

How do I georeference my data or results?
We've provided an ENVI routine that you can install to do the georeferencing.  This uses the information in the "_geo_glt" file provided with the data to allow precision georeferencing based on the HyMap ephemeris data.  If you're already running ENVI, you need to exit the program (and IDL) to install the new functionality.  
Place the file "hymap_geocorrect_from_glt.sav" in the "save_add" directory of your ENVI installation and then restart ENVI. When you restart ENVI a new option will automatically appear on the ENVI menu under "Utilities->Data Specific Utilities->Hymap->HyMap Geocorrect from GLT Lookup File"

Select the new option and open/select the "_geo_glt" file in the HyMap GLT Input File dialog.  Open/Select the "_effort" data (the mosaicked BIL data of the full flightline) or an uncorrected output product (e.g.: a MTMF or other mapping result) in the Uncorrected Data Input File dialog. Spectrally subset the data in this dialog if desired.  Enter -9999 as the Background Value enter an output filename, and click OK to perform the geometric correction.

While these images are visually pleasing and map-correct, they do have several practical drawbacks. First they have null values (-9999) around their edges that must be  masked in processing. Secondly they are often inflated in size by replicated pixels as indicated in the "_glt" files. These two disadvantages lead to our suggestion to acquire and process the HyMap imagery in its raw spatial format, then apply the geocorrection to the derived final products.  We do not recommend geocorrecting the entire reflectance data cube (_effort).

How about a quick mosaic of overlapping flightlines?
You can make a quick mosaic of overlapping flightlines by opening all of the georeferenced (_geo) images, choosing Register->Mosaic Images->Georeferenced Images from the ENVI main menu, and importing each of the overlapping images sequentially, with the images you want on top following the underlying images.  Be sure to use "Import With Feathering" and enter "-9999" as the "Background Data Value to Ignore" in the Enter Parameters dialog.  When all of the images to be mosaicked have been imported, select File->Apply, enter the output filename, enter -9999 into the Background Value text box., and click OK to create the mosaic.

How accurate is the georeferencing?  When I look at the mosaic,
I notice some offsets across image boundaries and apparent repeat pixels.
The geometric transformation and georeferenced images provided are the best that can be accomplished without use of a detailed digital elevation model and orthorectification.  The correction is "precise" based upon what we know about the sensor geometry and aircraft attitude from the ephemeris data.  There may be some absolute accuracy offsets, however, introduced by the topography.  This may be result in slight offsets in absolute position from a map and also may be visible as offsets across image boundaries and apparent repeat pixels when putting together a mosaic.  These problems can typically be corrected by selecting a few ground control points (GCPs) to tie the images to a map, and/or using GCPs to match mosaicked images in overlap regions.

Other Frequently Asked Questions - Click Here

AIG offers several options for clients needing further assistance:

Hyperspectral Data Analysis and Image Processing Workshops
To maximize your hyperspectral processing capabilities plan on attending one of AIG's hyperspectral workshops. The next workshop is offered November 16-19, 1999.  Participants gain in-depth, hands-on instruction in hyperspectral data processing  analysis, and interpretation. Additionally, the fundamentals and principles of imaging spectrometry are presented and advanced methodologies are discussed and demonstrated. Classes are offered in Boulder, Colorado.

End-to-end spectral processing and analysis
With over 60 combined years of hyperspectral analysis experience we can help you extract the most information from your HyMap data. Some of the spectral  processing options offered by AIG include target identification and mapping; endmember extraction, identification and quantification; surface material maps; production processing; data merging and resampling; as well as many other services. Contact AIG for more information.

For Additional Help or Information about the HyMap Groupshoot data contact:
Analytical Imaging and Geophysics LLC
4450 Arapahoe Ave., Suite 100
Boulder, CO 80303 USA
Phone: 303-604-2844
FAX:   303-665-6090
WWW: http://www.aigllc.com
   E-Mail: services@aigllc.com

Copyright 1999, © AIG Limited Liability Company, 4450 Arapahoe Ave, Suite 100, Boulder, Colorado 80303, Phone: 303-604-2844, FAX: 303-665-6090, Email: info@aigllc.com, (Updated 21 June, 2000 ).
 

Send mail to webmaster@aigllc.com with questions or comments about this web site.
Copyright © 2000 Analytical Imaging and Geophysics, LLC
Last modified: June 21, 2000