$Header: /cvsroot/nco/nco/doc/ANNOUNCE,v 1.678 2014/01/09 19:13:35 zender Exp $ -*-text-*-

The netCDF Operators NCO version 4.4.1 are ready. 

http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")

This release focuses on stability and speed. 
It also addresses previous omissions so that full path names are
accepted by all appropriate options, such as the ncwa weights and
masks (-w and -m), and ncks auxiliary coordinates (-X).
See below for bugfixes include ncra and ncrcat use-cases involving
strides with superfluous files and/or multi-record output (MRO).

Other significant new features include major improvements to
conversion of HDF4, HDF5, and netCDF4 files to netCDF3 and netCDF4
files. Most of this work is an offshoot of writing an NCO and
CFchecker-based solution to the problem of checking CF-compliance
for of datasets in HDF4, HDF5, and netCDF4 formats. The solution
is a script called ncdismember that now works well with most
NASA stewarded datasets I've thrown at it.

We postponed the name-change of ncra and ncwa to ncrs and ncws,
respectively, until the next version for stability. 
Work on NCO 4.4.1 is underway, focused on stability and speed.   
There will be more netCDF4 mop-up (-X and --cnk) and, possibly,
improved HDF4 support, and cache manipulation for chunking.  

Enjoy,
Charlie

"New stuff" in 4.4.1 summary (full details always in ChangeLog):

NEW FEATURES:

As indicated in the last release, NCO 4.4.1 renamed ncra as ncrs and
ncwa as ncws (Record Statistics and Weighted Statistics,
respectively).  This is to more accurately reflect their purpose.
The same technique is used to guarantee backwards compatibility.
That is, NCO installs by default with symbolic links from ncra->ncrs
and ncwa->ncws so calling either operator by its old name still works,
and calling either operator by its new name produces the same results.
However, ncra and ncwa are deprecated and will eventually be removed.
Please change your scripts/habits accordingly within a few years...

A. The -X option for auxiliary coordinates now works fully when
   latitude and longitude coordinates are nested in group hierarchies.
   http://nco.sf.net/nco.html#aux
   fxm verify

J. The --cnk_byt option was introduced to allow users to manually
   specify the total desired chunksize (in Bytes). In the absence
   of this parameter, NCO sets the chunksize to the filesystem
   blocksize of the output file (if obtainable via stat()), or else  
   to 4096 B, the Linux default blocksize. 
   ncks -4 --cnk_byt=8192 in.nc out.nc
   Note that --cnk_dmn arguments are still in elements, not bytes. 
   Should we use bytes instead of elements for all chunk arguments?
   Send us your preference to help the decisions for 4.4.1.
   http://nco.sf.net/nco.html#cnk

BUG FIXES:

A. 

KNOWN ISSUES NOT YET FIXED:

   This section of ANNOUNCE reports and reminds users of the
   existence and severity of known, not yet fixed, problems. 
   These problems occur with NCO 4.4.1 built/tested with netCDF
   4.3.1-rc7 snapshot 20131222 on top of HDF5 hdf5-1.8.9 with these methods: 

   cd ~/nco;./configure --enable-netcdf4  # Configure mechanism -or-
   cd ~/nco/bld;make dir;make allinone # Old Makefile mechanism

A. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable

   20130724: Verified problem still exists. 
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
   NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.

   Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47

B. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
   ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails 

C. NOT YET FIXED (NCO problem)
   Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments

   Demonstration:
   ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

D. NOT YET FIXED (netCDF library problem)
   Probe hidden attributes (chunking, compression) of HDF4 files

   Demonstration:
   ncdump -h -s ~/nco/data/hdf.hdf # (dies)
   ncks -m ~/nco/data/hdf.hdf # (works by avoiding fatal calls)

   20131230: Verified problem still exists

   Cause: some libnetCDF library functions fail on HDF4 file inquiries.
   Bug report filed: netCDF #HZY-708311 ncdump/netCDF4 segfaults probing HDF4 file
   Tracking tickets NCF-272, NCF-273

E. [FIXED in netCDF 4.3.1-rc5 ... please upgrade] 
   netCDF4 library fails when renaming dimension and variable using
   that dimension, in either order. Works fine with netCDF3.
   Also library causes var rename to imply dimension rename, and visa versa.
   Hence coordinate renaming does not work with netCDF4 files.
   Problem with netCDF4 library implementation.

   Demonstration:
   ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc
   ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc # Breaks with "NetCDF: HDF error"
   ncks -m ~/foo.nc

   20130724: FIXED in netCDF 4.3.1-rc5 in 201212. Will be in netCDF 4.3.1.
   Bug report filed: netCDF #YQN-334036: problem renaming dimension and coordinate in netCDF4 file 
   Workaround: Use ncrename twice; first rename the variable, then rename the dimension. 
   More Info: http://nco.sf.net/nco.html#ncrename_crd

"Sticky" reminders:

A. Pre-built, up-to-date Debian Sid & Ubuntu packages:
   http://nco.sf.net#debian

B. Pre-built Fedora and CentOS RPMs:
   http://nco.sf.net#rpm

C. Pre-built Windows (native) and Cygwin binaries:
   http://nco.sf.net#windows

D. Pre-built AIX binaries:
   http://nco.sf.net#aix

E. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)?
   SWAMP efficiently schedules/executes NCO scripts on remote servers:  

   http://swamp.googlecode.com

   SWAMP can work command-line operator analysis scripts besides NCO. 
   If you must transfer lots of data from a server to your client
   before you analyze it, then SWAMP will likely speed things up. 

F. NCO support for netCDF4 features is tracked at

   http://nco.sf.net/nco.html#nco4

   NCO supports netCDF4 atomic data types, compression, chunking, and groups.

G. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., 
   NASA AURA HIRDLS HDF-EOS5
   NASA ICESat GLAS HDF5
   NASA MERRA HDF4
   NASA MODIS HDF4
   NASA SBUV HDF5...

