$Header$ -*-text-*-

The netCDF Operators NCO version 4.9.8 have arrived.

http://nco.sf.net (Homepage, Mailing lists, Help)
http://github.com/nco (Source Code, Issues, Releases, Developers)

What's new?
Version 4.9.8 contains a few cool features.
These include support for unpacking sparse 1D (S1D) data and restart
files used for plant functional types and multiple elevation classes,
splitting monthly resolution timeseries that do not begin/end in
Jan/Dec, seamless climos for the E3SM ice-sheet model (MALI), and
corner-case bugfixes for inferring grid files and handling
sub-gridscale data.

Work on NCO 4.9.9 has commenced and will continue improving support
for analysis of land surface datasets packed into sparse-1D formats,
splitting daily resolution datasets along monthly boundaries,
and begin ncremap support for the MOAB regridding package. 

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. Windows builds should now contain the nces.exe and ncrcat.exe
   executables. Previously, Windows users had to run nces and ncrcat
   commands indirectly, using ncra.exe -Y nces ..., or manually copy
   ncra.exe to nces.exe, etc. The executables are identical, and
   behave differently depending on their invocation name. 
   ncrcat             ... # These all do the same thing
   ncrcat.exe         ... # "
   ncra.exe -Y ncrcat ... # "
   http://nco.sf.net/nco.html#ncrcat
   http://nco.sf.net/nco.html#nces

B. ncclimo supports new options --mth_srt and --mth_end in splitter
   mode. The arguments to these options specify the (1-based) month in
   which the requested timeseries will begin and end, respectively,
   and these default to 1 (January) and 12 (December). To extract
   14-month timeseries from individual monthly input files one would
   use, e.g.,
   ncclimo --yr_srt=1 --yr_end=2 --mth_srt=4 --mth_end=5 ...
   http://nco.sf.net/nco.html#ncclimo
   http://nco.sf.net/nco.html#mth_srt
   http://nco.sf.net/nco.html#mth_end

C. ncclimo/ncremap support analysis of MALI land ice model output
   when invoked with the new '-P mali' or '--prc_typ=mali' options.
   ncclimo -P mali -s 1 -e 2 -o climo mali.hist*.nc
   ncremap -P mali --map=map.nc in.nc out.nc
   http://nco.sf.net/nco.html#prc_typ
   http://nco.sf.net/nco.html#ncclimo

D. ncks now supports unpacking sparse 1D (S1D) ELM/CLM data and
   restart files for plant functional types (PFTs) and multiple
   elevation classes (MECs).
   ncks --s1d -v cols1d_topoglc --hrz=elm_mali_ig_history.nc \
        elm_mali_restart.nc out.nc
   ncks --s1d -v GPP,pfts1d_wtgcell elm_crop_history.nc out.nc
   After such S1D datasets have been "unpacked", they may easily
   be regridded with, e.g., ncremap.
   This feature is under continued development and user-feedback
   would be appreciated. Documentation does not yet exist.
   http://nco.sf.net/nco.html#ncks

E. ncks now prints more informative messages when users attempt
   to vertically interpolate variables (such as EAM/CAM US, VS) in
   hybrid coordinate data files that contain multiple grids (e.g.,
   the normal and staggered grids) though only one surface pressure
   variable.
   http://nco.sf.net/nco.html#ncremap
   Thanks to Walter Hannah for noticing this.
 
F. As of version 4.9.8 the default ncremap behavior is to omit the
   staggered grid from output files on an cap grid.
   The new flag --stg_grd turns-on outputting the staggered
   grid, and thus recovers the previous default behavior. 

BUG FIXES:

A. ncap2's array() function was rewritten to use a starting value
   plus an index times an increment, rather than adding increments, in
   order to reduce rounding errors. Thanks to Henry Butowsky.

B. The regridder in ncremap no longer attempts to access the
   (non-existent) tally array in SGS-weighted fields without missing
   values. The previous behavior would cause a crash. Fortunately
   this combination does not often occur in practice. Upgrading to
   4.9.8 is the only solution, there is no workaround. 

C. The capability to infer MPAS grids introduced in 4.9.8 introduced
   a bug that requires that horizontal coordinates have a units
   attribute. Many datafiles strongly disagree with this presumption!
   The requirement has been removed. The solution is to upgrade to
   4.9.8, there is no workaround.

D. The ncclimo --xcl_var flag has been fixed. Previously the
   flag had no effect. Now a variable extraction list may be turned
   into a variable exclusion by using --xcl_var --vars=v1,v2,...vn.
   The workaround is to use the option combination
   --nco_opt='-x' -v v1,v2,...vn. The solution is to upgrade.

E. At some unknown point, the behavior of ncclimo in MPI mode on Cori
   seems to have changed. Slurm on Cori does not (never did? no longer
   does?) allow multiple srun commands to execute concurrently on the
   same node by default. Thanks to Noel Keen to guiding me to the
   options (--gres=craynetwork:0 --mem=20000) required to fix this.
   This Slurm issue does not appear to affect any other DOE machines.
   The workaround is to avoid MPI mode on Cori, the solution is to
   upgrade. 

Full release statement at http://nco.sf.net/ANNOUNCE

KNOWN PROBLEMS DUE TO NCO:

   This section of ANNOUNCE reports and reminds users of the
   existence and severity of known, not yet fixed, problems. 
   These problems occur with NCO 4.9.8 built/tested under
   MacOS 11.0.1 with netCDF 4.7.4 on HDF5 1.10.7 and with
   Linux with netCDF 4.8.0-development (2021227) on HDF5 1.8.19.

A. NOT YET FIXED (NCO problem)
   Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments

   Demonstration:
   ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)
   ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl   
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   20190201: Possibly this problem was fixed in netCDF 4.6.2 by https://github.com/Unidata/netcdf-c/pull/1001
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride
   Workaround #3: Compile NCO with netCDF >= 4.6.2

B. NOT YET FIXED (netCDF4 library bug)
   Simultaneously renaming multiple dimensions in netCDF4 file can corrupt output

   Demonstration:
   ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc # Completes but produces unreadable file foo.nc
   ncks -v one ~/foo.nc

   20150922: Confirmed problem reported by Isabelle Dast, reported to Unidata
   20150924: Unidata confirmed problem
   20160212: Verified problem still exists in netCDF library
   20160512: Ditto
   20161028: Verified problem still exists with netCDF 4.4.1
   20170323: Verified problem still exists with netCDF 4.4.2-development
   20170323: https://github.com/Unidata/netcdf-c/issues/381
   20171102: Verified problem still exists with netCDF 4.5.1-development
   20171107: https://github.com/Unidata/netcdf-c/issues/597
   20190202: Progress has recently been made in netCDF 4.6.3-development
   More details: http://nco.sf.net/nco.html#ncrename_crd

C. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable

   20130724: Verified problem still exists. 
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
   NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

D. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
   ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails 

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., 
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
   http://nco.sf.net#bnr

