$Header$ -*-text-*-

The netCDF Operators NCO version 4.8.0 art b'rn

http://nco.sf.net (Homepage, Mailing lists, Help)
http://github.com/nco (Source Code, Issues, Releases, Developers)

What's new?

Version 4.8.0 contains, as usual, numerous enhancements to ncremap
and ncclimo. The most significant is vertical interpolation, which
is useful to create new initial conditions for models, and to convert 
data to/from any desired hybrid coordinate or pure pressure vertical
grid. Also we have changed NCO_PATH_OVERRIDE from opt-out to opt-in
due to the increased maturity and use of the Conda NCO package which
now pulls-in both ESMF_RegridWeightGen and TempestRemap.

ncap2 now allows creating brand-new files without specifying an
existing input file, which eliminates an indefensible and annoying
requirement. ncks can now easily detect and identify NaNs in datasets.
See below for a fuller list of new features and pointers to more info.

Work on NCO 4.8.1 has commenced. This will include plumbing work that
allows simultaneous horizontal and vertical regridding, and that
allows ncclimo to invoke the vertical regridding features.

Work on NCO 5.0.0 has commenced "under the hood". The key leap in that 
release will be support for netCDF4 user-defined types. Printing of
netCDF4 user-defined types ENUM and VLEN is ready now (though
unsupported) with the --udt flag. 5.0.0 will contain the finished
version of that, and progress on native weight generation by ncremap.

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. ncremap now accepts two new Tempest map configurations, "se2se" and 
   "fv2fv". Use them to regrid one spectral element grid to another,
   or one finite volume grid to another with TempestRemap.
   http://nco.sf.net/nco.html#se2se
   http://nco.sf.net/nco.html#fv2fv

B. ncap2 now allows creating brand-new files without specifying an
   input filename. Previously, users had to specify an input file 
   even if the output file had no dependency on it.
   ncap2 -s 'foo=1' ~/foo.nc # Create new or modify existing foo.nc
   ncap2 -O -s 'foo=1' ~/foo.nc # Overwrite old with new foo.nc
   ncap2 -s 'foo=1' ~/foo.nc ~/foo.nc # Add to old foo.nc
   http://nco.sf.net/nco.html#ncap2

C. ncremap and ncclimo have a new flag, --no_stdin, that disables
   checking stdin for input files. This is a flag that takes no
   argument, not a key-value option. Add --no_stdin to any command
   that might run in an environment, like Azure CI or CWL, that uses
   stdin for other purposes. 
   ncclimo --no_stdin --caseid=foo ...
   ncremap --no_stdin --map=map.nc in.nc out.nc
   http://nco.sf.net/nco.html#no_stdin

D. ncremap can now interpolate files in the vertical dimension.
   The --vrt_fl=vrt_fl option points to a vertical grid file
   that specifies the desired vertical grid for the output file.
   Currently supported grids include the hybrid vertical coordinate
   employed by E3SM and CESM, and pure pressure coordinates.
   ncremap --vrt_fl=vrt_fl in.nc out.nc
   http://nco.sf.net/nco.html#vrt

E. ncremap and ncclimo now accept the --xcl_var switch. This switch
   changes the extraction list (specified with -v vars or --vars=vars)
   into an exclusion list. Behavior is identical to the common NCO -x
   option:
   ncremap --xcl_var --vars=excluded,from,output in.nc out.nc
   http://nco.sf.net/nco.html#xcl_var

F. ncremap and ncclimo now alter paths on certain supercomputers at
   DOE HPC centers only when requested to. Previously, these commands
   used pre-defined paths, when available, to ensure users could
   easily run the executables in CZ's directories. Users could opt-out
   of these paths by exporting NCO_PATH_OVERRIDE=No in their
   environment. Thanks to inclusion of external (non-NCO) executables
   such as ESMF_RegridWeightGen and TempestRemap in Anaconda
   environments widely used at DOE labs, this is no longer necessary.
   Now users must opt-in to the preset paths by exporting 
   NCO_PATH_OVERRIDE=Yes in their environments.
   export NCO_PATH_OVERRIDE=Yes # Use CZ's ESMF, TR, and NCO
   ncremap -a conserve --map=map.nc in.nc out.nc
   http://nco.sf.net/nco.html#ncremap   

G. ncclimo adopts a default job_nbr of var_nbr in splitter mode.
   Previusly the default was 2. However, the splitter is extremely
   memory efficient and scales well up to over a hundred variables on
   all HPCs tested. Hence this change.
   http://nco.sf.net/nco.html#job_nbr
   
H. ncks has a new --chk_nan option. This option invokes a special
   checker that looks for NaN of NaNf in double- and single-precision
   floating-point variables, respectively. If a NaN is encountered,
   NCO prints its location and then exits with an error code. Thanks
   to Matthew Thompson of NASA for this suggestion.
   ncks --chk_nan in.nc
   http://nco.sf.net/nco.html#chk_nan

I. New synonyms --xtr_ass_var and --xcl_ass_var, which stand for
   "exclude associated variables" and "extract associated variables",
   have been added to the -C and -c switches, respectively.
   Simultaneously, the --no-coords and --no-crd synonyms have been
   renamed --no_coords and --no_crd because NCO is standardizing on
   underscores not dashes for multi-component option names.
   ncks --xcl_ass_var lat in.nc
   ncks --xtr_ass_var lat in.nc
   http://nco.sf.net/nco.html#xcl_ass_var
   http://nco.sf.net/nco.html#xtr_ass_var

BUG FIXES:

A. ncremap runs 20-30x faster on Cori KNL than before.
   Cori KNL has an unusual default setting for OMP_PROC_BIND that
   causes hardware thread contention with NCO. Thanks to Noel Keen of
   NERSC for help diagnosing this problem.

Full release statement at http://nco.sf.net/ANNOUNCE

KNOWN PROBLEMS DUE TO NCO:

   This section of ANNOUNCE reports and reminds users of the
   existence and severity of known, not yet fixed, problems. 
   These problems occur with NCO 4.8.0 built/tested under
   MacOS 10.14.1 with netCDF 4.6.1 on HDF5 1.10.2 and with
   Linux with netCDF 4.6.2-development (20180515) on HDF5 1.8.19.

A. NOT YET FIXED (NCO problem)
   Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments

   Demonstration:
   ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)
   ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl   
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   20190201: Possibly this problem was fixed in netCDF 4.6.2 by https://github.com/Unidata/netcdf-c/pull/1001
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride
   Workaround #3: Compile NCO with netCDF >= 4.6.2

B. NOT YET FIXED (netCDF4 library bug)
   Simultaneously renaming multiple dimensions in netCDF4 file can corrupt output

   Demonstration:
   ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc # Completes but produces unreadable file foo.nc
   ncks -v one ~/foo.nc

   20150922: Confirmed problem reported by Isabelle Dast, reported to Unidata
   20150924: Unidata confirmed problem
   20160212: Verified problem still exists in netCDF library
   20160512: Ditto
   20161028: Verified problem still exists with netCDF 4.4.1
   20170323: Verified problem still exists with netCDF 4.4.2-development
   20170323: https://github.com/Unidata/netcdf-c/issues/381
   20171102: Verified problem still exists with netCDF 4.5.1-development
   20171107: https://github.com/Unidata/netcdf-c/issues/597
   20190202: Progress has recently been made in netCDF 4.6.3-development
   More details: http://nco.sf.net/nco.html#ncrename_crd

C. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable

   20130724: Verified problem still exists. 
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
   NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

D. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
   ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails 

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., 
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
   http://nco.sf.net#bnr

