[ g e o m o d s ]
Geomods is a set of Python modules and console scripts for utilizing elevation data and Digital Elevation Models.
==============================================
Python Dependencies:
==============================================
- GDAL *
- numpy *
- scipy
- lxml *
- requests *
- matplotlib
==============================================
External Dependencies for DEM generation and data processing:
==============================================
- GDAL *
- GMT
- MBSystem
- LASTools
- VDatum 4.x
* required
==============================================
Install via git and pip:
==============================================
% git clone https:/github.com/ciresdem/geomods.git
% cd geomods
% pip install --user --upgrade ./
==============================================
Install via git and setup.py:
==============================================
% git clone https://github.com/ciresdem/geomods.git
% cd geomods
% python setup.py build
% sudo python setup.py install --record geomods.install
==============================================
Install via zip and setup.py:
==============================================
% wget https://github.com/ciresdem/geomods/archive/master.zip
% unzip master.zip
% cd master
% python setup.py build
% sudo python setup.py install --record geomods.install
==============================================
Uninstall from setup.py:
==============================================
% sudo python setup.py clean
% sudo xargs rm -r < geomods.install
==============================================
Update with setup.py:
==============================================
% git pull
% sudo python setup.py clean
% sudo xargs rm -r < geomods.install
% python setup.py build
% sudo python setup.py install --record geomods.install
==============================================
Python Modules:
==============================================
- fetches - Data fetching
- waffles - DEM generation and processing
- datalists - datalist processing
- regions - region processing
- xyzfun - xyz data functions
- gdalfun - gdal functions
- gmtfun - gmt functions
- mbsfun - mb-system functions
- vdatumfun - vdatum functions
- utils - general utilities and processing
==============================================
Console Scripts:
==============================================
- waffles - Generate a Digital Elevation Model and derivatives.
- fetches - Fetch elevation data from various sources.
- datalists - process and analyze datalists.
- gdal_chunk.py - Split a gdal grid into chunks.
- gdal_crop.py - Crop a gdal grid by it's nodata value.
- gdal_clip.py - Clip a gdal grid to an ogr vector.
- gdal_null.py - Generate a null gdal grid.
- gdal_mask.py - Apply a raster mask to a gdal grid.
- gdal_split.py - Split a gdal grid based on z value.
- gdal_query.py - Query the z value of a gdal grid with x/y coordinates.
- vdatum_cmd.py - Use NOAA's VDatum via command-line.
- hillshade.sh - Generate an ETOPO1 style hillshade image.
==============================================
Examples:
==============================================
- fetch and process lidar data from NOAA's Digital Coast for all the regions found in tiles.shp
% fetches dc -R tiles.shp -p --where 'Datatype LIKE "lidar%"'
- list all the NOS hydrographic data urls in the given region
% fetches nos -R -90/-89/29/30 -l > nos.urls
- generate a DEM for each region found in tiles.shp using GMT surface and the data found in input_data.datalist
% waffles -M surface -R tiles.shp -E .00000925925 input_data.datalist --verbose
- generate spatial-metadata from the datalist input_data.datalist
% waffles -R tiles.shp -E .00000925925 input_data.datalist --verbose -M spat-meta
- generate a vertical datum conversion grid:
% waffles -R -90/-89/29/30 -E 0.00083333 -M vdatum:ivert=navd88:o_vert=mhw
- generate a 'bathymetry surface' DEM:
% waffles -R region.shp -E 1s:.3333333s -X6 -M surface:upper_limit=0 -C coast_ply.shp:invert=True
- generate a nearest-neighbor DEM using all the .tif grid files in the current directory:
% waffles -R region.shp -E1s -Mnearneighbor:radius=3s *.tif
- split grid file tmp.tif into chunks of 1000x1000 cells:
% gdal_chunk.py tmp.tif -c 1000
- transform the vertical datum of an xyz data file:
% vdatum_cmd.py elev_data.xyz -i lmsl:ft:sounding -o navd88:ft:height
==============================================
Waffles CLI:
==============================================
waffles [OPTIONS] <datalist/entry>
Generate DEMs and derivatives and process datalists.
General Options:
-R, --region Specifies the desired REGION;
This can either be a GMT-style region ( -R xmin/xmax/ymin/ymax )
or an OGR-compatible vector file with regional polygons.
If a vector file is supplied it will search each region found therein.
If omitted, use the region gathered from the data in DATALIST.
-E, --increment Gridding CELL-SIZE in native units or GMT-style increments.
append :<inc> to resample the output to the given <inc>: -E.3333333s:.1111111s
-F, --format Output grid FORMAT. [GTiff]
-M, --module Desired DEM MODULE and options. (see available Modules below)
syntax is -M module:mod_opt=mod_val:mod_opt1=mod_val1:...
-O, --output-name BASENAME for all outputs.
-P, --epsg Horizontal projection of data as EPSG code [4326]
-X, --extend Number of cells with which to EXTEND the REGION.
append :<num> to extend the processing region: -X6:12
-T, --filter FILTER the output DEM using one or multiple filters. <fltr:fltr_val:split_value=z>
Available filters:
1: perform a Gaussian filter at -T1:<factor>.
2: use a Cosine Arch Filter at -T2:<dist(km)> search distance.
3: Spike Filter at -T3:<stand-dev. threshhold>.
The -T switch may be set multiple times to perform multiple filters.
Append :split_value=<num> to only filter values below z-value <num>.
e.g. -T1:10:split_value=0 to smooth bathymetry using Gaussian filter
-Z, --z-region Restrict data processing to records that fall within the z-region
Use '-' to indicate no bounding range; e.g. -Z-/0 will restrict processing to data
records whose z value is below zero.
-C, --clip CLIP the output to the clip polygon -C<clip_ply.shp:invert=False>
-K, --chunk Process the region in CHUNKs. -K<chunk-level>
-W, --w-region Restrict data processing to records that fall within the w-region (weight).
Use '-' to indicate no bounding range; e.g. -W1/- will restrict processing to data
records whose weight value is at least 1.
-G, --wg-config A waffles config JSON file. If supplied, will overwrite all other options.
generate a waffles_config JSON file using the --config flag.
-p, --prefix Set BASENAME to PREFIX (append inc/region/year info to output BASENAME).
-r, --grid-node Use grid-node registration, default is pixel-node
-w, --weights Use weights provided in the datalist to weight overlapping data.
-a, --archive Archive the datalist to the given region.
-m, --mask Generate a data mask raster.
-u, --uncert Generate an associated uncertainty grid.
-c, --continue Don't clobber existing files.
-q, --quiet Lower verbosity to a quiet. (overrides --verbose)
--help Print the usage text
--config Save the waffles config JSON and major datalist
--modules Disply the module descriptions and usage
--version Print the version information
--verbose Increase the verbosity
Datalists and data formats:
A datalist is a file that contains a number of datalist entries, while an entry is a space-delineated line:
`/path/to/data format weight data,meta,data`
Supported datalist formats:
-1 ['datalist', 'mb-1']
-2 ['zip', 'gz']
-4 ['nos', 'dc', 'gmrt', 'srtm_cgiar', 'srtm_plus', 'mar_grav', 'charts', 'mb', 'tnm', 'emodnet', 'chs', 'hrdem', 'cudem']
168 ['xyz', 'csv', 'dat', 'ascii']
200 ['tif', 'img', 'grd', 'nc', 'vrt', 'bag']
300 ['las', 'laz']
Modules (see waffles --modules <module-name> for more info):
surface, triangulate, cudem, nearest, num, vdatum, mbgrid, invdst, average, linear, spat-meta, coastline, uncertainty
==============================================
Fetches CLI:
==============================================
fetches [OPTIONS] <module[:parameter=value]* ...>
Fetch geographic elevation data.
General Options:
-R, --region Specifies the desired region to search;
This can either be a GMT-style region ( -R xmin/xmax/ymin/ymax )
or an OGR-compatible vector file with regional polygons.
If a vector file is supplied it will search each region found therein.
-W, --where restricted_where: Attribute query (like SQL WHERE)
-E, --increment Blockmedian/mean CELL-SIZE in native units or GMT-style increments.
-Z, --z-region Restrict data processing to records that fall within the z-region
Use '-' to indicate no bounding range; e.g. -Z-/0 will restrict processing to data
records whose z value is below zero.
-F, --fg-config A fetches config JSON file. If supplied, will overwrite all other options.
generate a fetches_config JSON file using the --config flag.
-l, --list Return a list of fetch URLs in the given region.
-d, --dump Dump the XYZ elevation data in WGS84 to stdout.
-p, --process Process fetched elevation data to ASCII XYZ format in WGS84.
If -E or -Z are set, processing will use those switches in data processing.
-u, --update Update the Fetches Remote Elevation Datalist.
-i, --index Print the fetch FRED results in the given region.
--help Print the usage text
--config Save the fetches config JSON
--version Print the version information
Modules (see fetches --modules <module-name> for more info):
dc nos charts ncei_thredds usace tnm gmrt mb mar_grav srtm_plus emodnet ngs chs hrdem
Examples:
% fetches -R -90.75/-88.1/28.7/31.25 nos --where "Date > 2000"
% fetches -R region.shp -p dc nos:datatype=bag charts:datatype=enc
% fetches -R region.shp dc:datatype=lidar -l > dc_lidar.urls
% fetches -R -89.75/-89.5/30.25/30.5 tnm:ds=4:formats=IMG gmrt:res=max:fmt=geotiff
