$Header$ -*-text-*-

netCDF Operators NCO version 5.1.9 strut their stuff

http://nco.sf.net (Homepage, Mailing lists, Help)
http://github.com/nco/nco (Source Code, Issues, Releases)

What's new?
Version 5.1.9 updates ncremap to employ new TempestRemap
weight-generation algorithms (bilinear and integrated bilinear),
updates ncremap to recognize new names for existing algorithms,
changes ncremap's default treatment of filling empty areas with
missing values, and fixes a long-standing bug with ncra and ncrcat 
subcycle and interleave options. A notable fix for MS Windows OS
is included as are a few other miscellaneous features and fixes
described below.

Work on NCO 5.2.0 has commenced and aims to add support for Zarr S3 
stores, and to polish support for new codecs..

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. ncremap supports a new flag, --mpt_mss, to control the values
placed in empty unmasked destination gridcells in sub-gridscale (SGS)
mode. SGS mode interprets every gridcell as being fractionally covered 
by an amount contained in sgs_var (e.g., landfrac, seaicefrac).
Empty in this context means unmasked cells where sgs_var is zero,
e.g., no land, or no sea ice. Since ~2020, ncremap has filled empty
SGS cells with the missing value. NCO 5.1.9 changes that behavior so
that, by default, empty SGS cells are filled with zeros. This makes
maps of sea-ice variables, e.g., zero in open ocean and non-zero
where sea ice exists. Users can explicitly request the previous
behavior (missing values instead of zeros) with the --mpt_mss
flag (which stands for "empty missing").
ncremap -P mpasseaice           --map=map.nc in.nc out.nc # Empty = 0.0
ncremap -P mpasseaice --mpt_mss --map=map.nc in.nc out.nc # Empty = _FillValue
http://nco.sf.net/nco.html#mpt_mss

B. ncremap supports new TempestRemap bilinear and integrated bilinear
weight-generation algorithms. The algorithms sport the recommended
names trbilin and trintbilin.
ncremap --alg_typ=trbilin --grd_src=src.nc --grd_dst=dst.nc --map=map.nc
ncremap --alg_typ=trintbilin --grd_src=src.nc --grd_dst=dst.nc --map=map.nc
http://nco.sf.net/nco.html#tr

C. ncremap supports new "standard" names for E3SM v3 regridding
algorithms and for older algorithms. These new names are simply
synonyms for the existing algorithm names. No algorithm names have 
been deprecated (yet) so existing commands will still work.
The new names are of the form ToolAlgorithm where Tool is esmf, nco,
tr, or mbtr and Algorithm is the "classic" algorithm name, aave,
bilin, etc. 
ncremap --alg_typ=esmfaave --grd_src=src.nc --grd_dst=dst.nc --map=map.nc
ncremap --alg_typ=ncoaave  --grd_src=src.nc --grd_dst=dst.nc --map=map.nc
ncremap --alg_typ=traave   --grd_src=src.nc --grd_dst=dst.nc --map=map.nc
https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/1217757434/Mapping+file+algorithms+and+naming+convention

D. ncks supports a new flag, --chk_xtn, that reports whether filename
extensions comply with NASA's Dataset Interoperability Working Group
(DIWG) which recommends ".nc", ".h5", and ".he5", depending on the API
used to write the file. To check a file's compliance with the DIWG
recommendation:  
$ ncks --chk_xtn ~/nco/data/in.nc
$ ncks --chk_xtn ~/nco/data/in.nc4
$
http://nco.sf.net/nco.htlm/chk_mss

BUG FIXES:
   
A. NCO versions from 4.9.4--5.1.8 (September, 2020 through October,
2023) contained a bug that affected the subcycle and interleave (SSC
and ILV) hyperslab features.  The bug was triggered by invoking the
SSC feature without explicitly providing an ILV parameter.  The
software failed to initialize an internal flag that indicated whether
ILV had been invoked.  The resulting behavior was compiler-dependent.
Most compilers set the ILV flag to false (as it should have been), but
some compilers set it to true.  The bug expressed itself by extracting
only a single timestep, rather than the number of timesteps indicated
by the SSC parameter.  This behavior was fixed in NCO version 5.1.9
(November, 2023). There is no workaround to this problem, the solution
is to upgrade. 
http://nco.sf.net/nco.html#bug_ilv

B. ncap2 now successfully parses naked numeric constants with the
value -9223372036854775808LL (i.e., NC_MIN_INT64). Previously, the
C++ standard library functions failed when attempting this, and we
still have no idea why. All hail Henry Butowsky for hacking together
a workable implementation. There is no workaround to this problem, the
solution is to upgrade. 

C. Depending on how they were compiled, some recent NCO versions
could have expected the wrong path separator on MS Windows OS.
Your installation suffered from this bug if NCO on Windows expected
'/' (the UNIX separator) instead of '\' (the Windows separator).
There is no workaround to this problem, the solution is to upgrade.  

Full release statement at http://nco.sf.net/ANNOUNCE
    
KNOWN PROBLEMS DUE TO NCO:

This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems. 
These problems occur with NCO 5.1.9 built/tested under
MacOS 13.5.2 (c) with netCDF 4.9.3-dev on HDF5 1.14.2 and with
Linux FC38 with netCDF 4.9.2 on HDF5 1.14.1.

A. NOT YET FIXED (NCO problem)
   Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments

   Demonstration:
   ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)
   ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl   
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   20190201: Possibly this problem was fixed in netCDF 4.6.2 by https://github.com/Unidata/netcdf-c/pull/1001
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride
   Workaround #3: Compile NCO with netCDF >= 4.6.2

B. NOT YET FIXED (netCDF4 library bug)
   Simultaneously renaming multiple dimensions in netCDF4 file can corrupt output

   Demonstration:
   ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc # Completes but produces unreadable file foo.nc
   ncks -v one ~/foo.nc

   20150922: Confirmed problem reported by Isabelle Dast, reported to Unidata
   20150924: Unidata confirmed problem
   20160212: Verified problem still exists in netCDF library
   20160512: Ditto
   20161028: Verified problem still exists with netCDF 4.4.1
   20170323: Verified problem still exists with netCDF 4.4.2-development
   20170323: https://github.com/Unidata/netcdf-c/issues/381
   20171102: Verified problem still exists with netCDF 4.5.1-development
   20171107: https://github.com/Unidata/netcdf-c/issues/597
   20190202: Progress has recently been made in netCDF 4.6.3-development
   More details: http://nco.sf.net/nco.html#ncrename_crd

C. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable

   20130724: Verified problem still exists. 
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
   NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

D. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
   ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails 

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., 
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
   http://nco.sf.net#bnr

