$Header$ -*-text-*-

The netCDF Operators NCO version 4.9.9 have arrived.

http://nco.sf.net (Homepage, Mailing lists, Help)
http://github.com/nco (Source Code, Issues, Releases, Developers)

What's new?
Version 4.9.9 contains more fixes than features, and can probably be
skipped by those who do not use ncremap or ncclimo. ncclimo now
supports the mth_srt/end options for high resolution timeseries that
do not begin/end in Jan/Dec (previously it only supported those
arguments for monthly resolution). ncremap has crucial fixes for
weight generation and map-checking for masked grids, fixes for
curvilinear grid inferral, and automatic rounding when regridding
integer-valued fields (like masks).  

Work on NCO 5.0.0 has commenced and will continue improving support
for analysis of land surface datasets packed into sparse-1D formats,
and ncremap support for the MOAB regridding package mbtempest. 

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. ncclimo now supports options --mth_srt and --mth_end in
   high-frequency splitter mode. Previously it only supported those 
   arguments for monthly resolution timeseries. The arguments specify
   the (1-based) month in which the requested high-frequency
   timeseries should begin and end, respectively. These default to 1
   (January) and 12 (December). To extract 14-month timeseries from
   high frequency (e.g., daily-resolution) input files one would 
   use, e.g.,
   ncclimo --clm_md=hfs --yr_srt=1 --yr_end=2 --mth_srt=4 --mth_end=5 ...
   http://nco.sf.net/nco.html#ncclimo
   http://nco.sf.net/nco.html#mth_srt
   http://nco.sf.net/nco.html#mth_end

B. ncclimo is now smarter about inferring both the temporal
   resolution of high frequency timeseries, and the number of
   timesteps per file. As a result, explicit invocation of the tpd
   (timesteps-per-day) and dpf (days-per-file) option should no longer
   be necessary. Explicitly specifying these options will, of course,
   prevent ncclimo from trying to infer them.
   ncclimo --clm_md=hfs           ... # This should work fine
   ncclimo --clm_md=hfs --tpd=8   ... # No longer necessary
   ncclimo --clm_md=hfs --dpf=10  ... # No longer necessary
   http://nco.sf.net/nco.html#ncclimo
   http://nco.sf.net/nco.html#tpd
   http://nco.sf.net/nco.html#dpf

C. ncremap now automatically rounds integer-valued fields, such as
   masks. Previously when ncremap regridded an integer-valued field,
   it used implicit type-conversion rules to convert the regridded
   destination value to an integer value for output. However, users
   generally expect a regridded value of, say, 1.9999999, to be
   converted to the integer 2 not 1. Hence, ncremap now automatically
   applies the rint() function to internal floating point values
   before converting them to integers for output. Regrid integer
   valued fields such as masks now generally behave as desired.

D. ncremap now always invokes TempestRemap's GenerateOverlapMesh with
   its --allow_no_overlap option. This option permits TR to generate
   meshes from not-completely-overlapped regional grids. This is often
   useful in post-processing regional data, for example.
   http://nco.sf.net/nco.html#tr
   http://nco.sf.net/nco.html#allow_no_overlap

E. Conda builds should now be more robust, due to extensive testing.
   Thanks to Xylar Asay-Davis for helping us understand and alter the
   nco-feedstock for Conda. As a result of this, OpenMP should work
   on all MacOS Conda installations, and thus make map-file generation
   much faster for high-resolution maps.

BUG FIXES:

A. NCO now treats masks self-consistently. Its internal
   weight-generator adheres to the rules that no masked source
   gridcell shall contribute to a destination gridcell, and that 
   no masked destination gridcell shall receive any contributions.
   Moreover, the ncks map-checker now reports whether, and how often
   map-files violate these rules. Previously, NCO's weight generator
   would incorrectly handle, with obviously detrimental outcomes,
   grids with masked regions. There is no workaround, and the only
   solution is to upgrade. Thanks to Henry Butowsky for fixing the
   weight-generation issues. 
   http://nco.sf.net/nco.html#rgr
   http://nco.sf.net/nco.html#chk_map

B. ncremap now correctly infers grids from curvi-linear data files.
   Previously, the inferral mechanism could inadvertently omit about
   half of the area of each inferred gridcell. Note that this only
   affected inferral of grids from curvilinear data files. Explicitly
   specified curvilinear grids (with full vertice information) have
   always worked correctly. There is no workaround, and the only
   solution is to upgrade. 

C. DAP access to files via the https protocol now works with Windows.
   Previously, only http worked with Windows. There is no workaround,
   and the only solution is to upgrade. Thanks to Pedro Vicente for
   identifying this bug.

Full release statement at http://nco.sf.net/ANNOUNCE

KNOWN PROBLEMS DUE TO NCO:

   This section of ANNOUNCE reports and reminds users of the
   existence and severity of known, not yet fixed, problems. 
   These problems occur with NCO 4.9.8 built/tested under
   MacOS 11.0.1 with netCDF 4.7.4 on HDF5 1.10.7 and with
   Linux with netCDF 4.8.0-development (2021227) on HDF5 1.8.19.

A. NOT YET FIXED (NCO problem)
   Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments

   Demonstration:
   ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)
   ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl   
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   20190201: Possibly this problem was fixed in netCDF 4.6.2 by https://github.com/Unidata/netcdf-c/pull/1001
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride
   Workaround #3: Compile NCO with netCDF >= 4.6.2

B. NOT YET FIXED (netCDF4 library bug)
   Simultaneously renaming multiple dimensions in netCDF4 file can corrupt output

   Demonstration:
   ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc # Completes but produces unreadable file foo.nc
   ncks -v one ~/foo.nc

   20150922: Confirmed problem reported by Isabelle Dast, reported to Unidata
   20150924: Unidata confirmed problem
   20160212: Verified problem still exists in netCDF library
   20160512: Ditto
   20161028: Verified problem still exists with netCDF 4.4.1
   20170323: Verified problem still exists with netCDF 4.4.2-development
   20170323: https://github.com/Unidata/netcdf-c/issues/381
   20171102: Verified problem still exists with netCDF 4.5.1-development
   20171107: https://github.com/Unidata/netcdf-c/issues/597
   20190202: Progress has recently been made in netCDF 4.6.3-development
   More details: http://nco.sf.net/nco.html#ncrename_crd

C. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable

   20130724: Verified problem still exists. 
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
   NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

D. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
   ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails 

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., 
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
   http://nco.sf.net#bnr

