/* -*- mode: html -*- */
/**
\if NOT DEVELOPER_SECTION
\mainpage NDS Client Library
\endif

\tableofcontents

\section Abstract
<center>NDS Developers</center>
<center>LIGO Laboratory, California Institute of Technology, LIGO Scientific Collaboration</center>
<center><a href="mailto:nds-discuss@ligo.org">nds-discuss@ligo.org</a></center>

\par
This document explains how to use the NDS (Network Data Server) client interface in
<a href="http://python.org/">Python</a>, <a href="http://octave.org/">Octave</a>,
and <a href="http://mathworks.com/">MATLAB</a> programs.
NDS is a TCP/IP protocol for retrieving online or archived data from thousands of instrument channels at
<a href="http://ligo.org/">LIGO</a>
(Laser Interferometer Gravitational-Wave Observatory) sites and data analysis clusters.
Language modules for Python, JAVA,
MATLAB, and Octave.
are provided through <a href="http://swig.org/">SWIG</a>.

\par
This document is for the NDS2 client @PROJECT_VERSION@.

<!-- ################################################################ -->

\section intro Introduction

<p>
  This document explains how to use the NDS (Network Data Server) client interface
  in
  <a href="http://python.org/">Python</a>,
  <a href="http://www.java.com/">Java</a>,
  <a href="http://mathworks.com/">MATLAB</a>,
  and
  <a href="http://octave.org/">Octave</a>
  programs.
  NDS is a TCP/IP protocol for retrieving online or archived data from thousands
  of instrument channels at <a href="http://ligo.org/">LIGO</a>
  (Laser Interferometer Gravitational-Wave Observatory) sites and data analysis clusters.
  Language modules for
  Python,
  Java,

  MATLAB,
  and
  Octave
  are provided through <a href="http://swig.org/">SWIG</a>.
</p>

\subsection intro_features Features
<p>
  The SWIG language interface provides some special features:
</p>
<dl>
  <dt>Uniform API across multiple languages.</dt>
  <dd>
    The NDS programming interface is the same in Python, Java, MATLAB, and Octave,
    up to syntax and coding convention differences in each of the languages.
  </dd>

  <dt>Works in all (modern) versions of MATLAB.</dt>
  <dd>
    The MATLAB/Java interface is compatible with all versions of MATLAB going
    back to the early 2000s.
  </dd>

  <dt>Supports NDS2 epochs.</dt>
  <dd>
    NDS2 servers have the concept of epochs of work, these are ranges of start/stop times that requests
    may be constrained to.  This allows requests for data availability and channel lists to be limited
    to the channels available during the specified epoch, as opposed to the entire history of NDS2.
  </dd>

  <dt>Server protocols are cached.</dt>
  <dd>
    When you connect to a new server, the protocol version of the server is saved on disk to speed up
    the process of connecting to that server in the future.
  </dd>

  <dt>Users can specify how to handle data that resides on tape.</dt>
  <dd>
    The NDS2 servers have access to large stores of data that are backed by archival quality tape systems.
    This means that data that has not been recently accessed may require retrieval from tape.
    This is very slow operation which can take from minutes to an hour depending on the load on the storage system.
    To help users gauge the length of time a request will take the system can raise a new error when
    data is seen to reside on tape storage.
  </dd>

  <dt>Iterate and Fetch requests that work with gaps.</dt>
  <dd>
    The old interface will return a data not found error if there is a gap in the data.  The new interface
    allows the application to select the response to gaps.
    This can be aborting (as in the old code) or filling in the gap with some value.
  </dd>
  
  <dt>Channel availability information is exposed.</dt>
  <dd>
    A new api is introduced which allows for the retreival of availablility information from
    NDS2 servers on a per channel basis.
  </dd>
</dl>

<!-- ---------------------------------------------------------------- -->

\subsection intro_issues Known issues
<p>
  The following known issues exist:
</p>
<dl>

  <dt>One transfer at a time with any given connection.</dt>
  <dd>
    The SWIG interface permits reusing an NDS connection for multiple requests.
    However, while one request is in progress over a given connection,
    the server will refuse most other requests.
    As a consequence, some care must be taken when using the online data
    retrieval functions <!-- link linkend="conn.iterate" --><b>iterate()</b> and <b>next()</b></a>.
    For example, consider the following MATLAB transcript in which <b>iterate()</b>
    is called twice on the same <b>connection</b> object:
    \code{.sh}
      conn = nds2.connection('nds.ligo-la.caltech.edu');
      conn.iterate(1037022904, 1037023024, {'L1:PSL-ISS_PDA_OUT_DQ'});
      conn.iterate(1037022904, 1037023024, {'L1:PSL-ISS_PDA_OUT_DQ'});

      Error using nds2.connection/iterate
      Java exception occurred:
      java.lang.RuntimeException: Another transfer is already in progress. Complete
      the transfer or retry on a new connection object.
	  at nds2.nds2JNI.connection_iterate__SWIG_0(Native Method)
	  at nds2.connection.iterate(connection.java:87)
    \endcode
    The above example could be fixed by
    (a) creating a second <b>connection</b> object for the second call to <b>iterate()</b> or
    (b) completing the first request by consuming all of the data with <b>next()</b>.
    <p>Note that the <a name="conn.find_channels"><b>find_channels()</b></a>
    and <a name="conn.fetch"><b>fetch()</b></a> methods always complete their transfers before returning,
    so it is safe to issue them repeatedly in any combination.</p>
  </dd>

  <dt>Windows installer by default does not add NDS libraries to the system PATH.</dt>
  <dd>
    The Windows installer by default has the option
    "Do not add nds2-client to the system PATH"
    checked.
    Be sure to choose one of the other two options,
    "Add nds2-client to the system PATH for all users" or
    "Add nds2-client to the system PATH for current user",
    so the proper dll files can be loaded at runtime.
  </dd>

  <dt>Windows installer includes only the MATLAB/Java language interface.</dt>
  <dd>
    The Windows installer does not include the Python and Octave language interfaces.
  </dd>

  <dt>Requesting unfiltered channel lists from servers that advertise millions of channels is slow and memory intensive.</dt>
  <dd>
    Some NDS servers (notably <b>nds.ligo.caltech.edu</b>) advertise millions of channels.
    Attempting to retrieve the full channel list from such a server may involve transferring as much as a hundred megabytes of channel metadata over the network.
    Moreover, preparing the representation of millions of channels as objects in Python, Octave, or MATLAB is memory-intensive, consuming a few hundred bytes per object.
    As a result, requesting the full, un-filtered list of all channels with <a name="conn.find_channels"><b>find_channels()</b></a> can consume hundreds of megabytes of RAM when working with such NDS servers.
    (Note that the legacy MATLAB interface also has this issue.)
    A remedy would be to have the <b>find_channels()</b> method return an iterator rather than a list.
  </dd>

  <dt>Capitalization of method names in Java.</dt>
  <dd>
    According to prevailing code conventions, in C, Python, and Octave, function names are all lowercase with underscores (<b>_</b>) separating words, as in <a name="conn.find_channels"><b>find_channels</b></a>.
    The MATLAB interface, on the other hand, follows the "camel case" convention of Java, as in <a name="conn.find_channels"><b>findChannels</b></a>.
  </dd>

  <dt>Indexing of arrays.</dt>
  <dd>
    In C and Python, arrays are indexed starting with <b>0</b>, whereas in Octave and MATLAB arrays are indexed starting with <b>1</b>.
    (On the other hand, if you use the MATLAB/Java bindings directly from Java, <b>0</b>-based indexing applies.)
    Also, the syntax for accessing elements of sequences returned by <a name="conn.find_channels"><b>find_channels()</b></a>,  <a name="conn.fetch"><b>fetch()</b></a>, and  <a name="conn.iterate"><b>next()</b></a> are slightly different in the target languages: use square brackets <b>[]</b> in C, Python, and Java; curly braces <b>{}</b> in Octave; and parentheses <b>()</b> in MATLAB.
  </dd>

  <dt>Java strings and MATLAB.</dt>
  <dd>
    MATLAB does not automatically convert
    <a href="http://docs.oracle.com/javase/6/docs/api/java/lang/String.html">Java strings</a>
    to MATLAB char arrays in all circumstances, so some methods in the Java interface have return types of <b>char [][]</b> instead of <b>String</b>.
    This subtlety does not affect MATLAB users; it is only a concern for those who wish to use the NDS interface in Java source code.
  </dd>

  <dt>Cyrus SASL 2.1.25 and MacPorts.</dt>
  <dd>
    NDS2 authentication currently does not work the MacPorts package <i>cyrus-sasl2</i> version 2.1.25.
    If you are compiling <i>nds2-client</i> from source, build against either Apple's own distribution of cyrus-sasl2 (which comes preinstalled on every Mac) or build against the MIT Kerberos GSSAPI.
  </dd>
</dl>

<!-- ################################################################ -->

\section install Installation

<p>
  It is easy to install the NDS client on Mac OS X using the \ref install_macports "MacPorts package"
  and on Windows using the \ref install_windows_installer "Windows installer",
  or from <a name="install_source">source</a> on Linux, UNIX, or Mac OS X.
</p>

\subsection install_binaroies Installing from packages
<p>
  The recommended way to install the nds2 client software is from the LSC soft package repositories.
  You should configure your box to use the LSC soft repositories
  (the details of this are out of scope of this document,
  but you may consult <a href="http://software.ligo.org/">software.ligo.org</a> for information.
</p>

If you are using a Debian you can install the packages using <b>apt-get</b>:
\code{.sh}
  $ apt-get install
\endcode

If you are using a Scientific Linux 7 system you can install the packages using <b>yum</b>:
\code{.sh}
  $ sudo yum install nds2-client-all
\endcode

<!-- ---------------------------------------------------------------- -->

\subsection install_source Installing from source

\subsubsection install_source_dependencies Dependencies

\par
Before you begin installing from source, you should make sure that you have the following dependencies installed.

\li <a href="http://sqlite.org/">Sqlite</a>: required only for the SWIG bindings.
\li <a href="http://swig.org/">SWIG</a>: required only for building from source.
\li <a href="http://python.org/">Python</a>: required only for Python interface.
\li <a href="http://octave.org/">Octave</a>: required only for Octave interface.
\li <a href="http://www.java.com/">Java</a>: required only for MATLAB/Java interface.
\li Either <a href="http://cyrusimap.web.cmu.edu/">Cyrus SASL</a>
or <a href="http://web.mit.edu/kerberos/">MIT Kerberos</a>:
required only for connecting to Kerberos-protected NDS protocol version 2 servers.


\par
If you are on Mac OS X and have the
<a href="http://macports.org/">MacPorts</a> package manager installed,
then you can install all of the dependencies for building from source with
all language interfaces enabled by using the <b>port</b> command:
\code{.sh}
$ sudo port install swig-java swig-octave swig-python pkgconfig kerberos5 sqlite3 cmake
\endcode

\par
On Debian, install the dependencies using <b>apt-get</b>:
\code{.sh}
$ apt-get install swig libsqlite3-dev octave octave-headers default-jdk python libsasl2-dev krb5-user libsasl2-modules-gssapi-mit cmake
\endcode

\par
And on Red Hat or Scientific Linux, install the dependencies using <b>yum</b>:
\code{.sh}
$ yum install swig sqlite-devel octave octave-devel java-1.6.0-openjdk-devel cyrus-sasl-devel cmake
\endcode

\subsubsection install_source_obtain Obtaining the source code

First, download a source release tarball, unarchive it,
and change directories into the downloaded source folder,
using the following commands,
where <b><b>X.X.X</b></b> is the version of <i>nds2-client</i> that you want:

\code{.sh}
$ wget http://www.lsc-group.phys.uwm.edu/daswg/download/software/source/nds2-client-<b>X.X.X</b>.tar.gz
$ tar -xzf nds2-client-<b>X.X.X</b>.tar.gz
$ mkdir nds2-client-<b>X.X.X</b>-build
$ cd nds2-client-<b>X.X.X</b>-build
\endcode

\subsubsection install_source_building Building

Next, configure, build, and install the package.
This is done with the cmake tool.
The table below lists essential command line arguments for enabling or disabling optional features.

<table>
  <caption>Essential command line options for <b>cmake</b></caption>
  <tr>
	<th>Option</th>
	<th>Purpose</th>
  </tr>
  <tr>
	<td><b>-DCMAKE_INSTALL_PREFIX=/path/to/local</b></td>
	<td>Install files into <i><b>/path/to/local</b>/bin</i>, <i><b>/path/to/local</b>/lib</i>, etc.</td>
  </tr>
  <tr>
    <td><b>-DPROG_MATLAB_=/path/to/matlab</b></td>
    <td>Use this to make sure the build finds a matlab installation.</td>
  </tr>
</table>

\note
If you are on a Mac and you use MacPorts, you should add the following options to cmake:

By default, files will be installed into <i>/usr/bin</i>, <i>/usr/lib</i>, <i>/usr/share</i>, etc.
If you would like to install into another location,
such as your home directory,
you should provide the <b>-DCMAKE_INSTALL_PREFIX</b> option when you call <b>cmake</b>.

For example, to enable all available features,
and install into the directory <i><b>/path/to/local</b></i>,
run the following commands:

\code{.sh}
$ cmake ../nds2-client-X.Y.Z -DCMAKE_INSTALL_PREFIX=/path/to/local
$ make
$ make install
\endcode

\subsubsection install_source_environment Environment script

If you are going to install the NDS client in your own home directory,
then you will need to set some environment variables every time that you log in.

The client installs a script to help setup environment variables for the specific install of the nds2-client.
Assuming that you are installing into the directory <i><b>/path/to/local</b></i>, run the following command.

\par
<b>Example environments script</b>
\code{.sh}
$ source <b>/path/to/local</b>/etc/nds2-client-user-env.sh
\endcode

<!-- ---------------------------------------------------------------- -->

\subsection install_macports Installing with MacPorts
<p>
  On Mac OS X, you can use the following instructions to install the latest version of
  <i>nds2-client</i> with the <a href="http://macports.org/">MacPorts package manager</a>.
</p>

\li If you do not already have MacPorts,
<a href="http://www.macports.org/install.php">download and install the latest version of MacPorts for your version of Mac OS</a>.
\li Make sure that your MacPorts installation is up to date by running:
\code{.sh}
$ sudo port selfupdate
\endcode
\li Finally, install the <i>nds2-client</i> port with the MEX interface for Octave and SWIG interfaces for all supported languages with:
\code{.sh}
$ sudo port install nds2-client
\endcode
Alternatively, you can manually choose which features to enable by manually selecting specific
<a href="http://guide.macports.org/#using.variants">MacPorts variants</a>. For example, the above is equivalent to:
\code{.sh}
$ sudo port install nds2-client <b>+mex_octave</b> <b>+swig_python27</b> <b>+swig_octave</b> <b>+swig_java</b>
$ sudo port install py27-nds2-client
\endcode
You can replace <b><b>+swig_python27</b></b> with <b><b>+swig_python25</b></b>, <b><b>+swig_python26</b></b>, etc., to enable the Python language interface for different versions of the Python interpreter. You can also use <b><b>-swig_python27</b></b>, <b><b>-swig_octave</b></b>, or <b><b>-swig_java</b></b> to disable the Python, Octave, or Java/MATALB interfaces, respectively.
If you have MATLAB installed, you can also build the legacy MEX interface for MATLAB using MacPorts by enabling the <b>mex_matlab</b> variant:
\code{.sh}
$ sudo port install nds2-client +mex_matlab
\endcode
\li <em>Optional:</em> If you installed the Python language interface, you may want to make the Python interpreter supplied by MacPorts the default interpreter, using the following command:
\code{.sh}
$ sudo port select --set python python27
\endcode
This should print the message:
\code{.sh}
Selecting 'python27' for 'python' succeeded. 'python27' is now active.
\endcode

<!-- ---------------------------------------------------------------- -->

\subsection install_windows_installer Installing with Windows installer

\note
Currently the nds2-client Windows installer is only available for 64-bit Windows.

\par
The nds2 client (version @PROJECT_VERSION@) installer for Windows 7 available at the moment is only 64 bits.
You can download it from
<a href="https://software.ligo.org/windows/nds2-client-@PROJECT_VERSION@-win64.exe">
https://software.ligo.org/windows/nds2-client-@PROJECT_VERSION@-win64.exe
</a>.

\par
Run the installer.
The default options are reasonable if you don't know of a reason to change them.

\par
An important dependency of the nds2 client is the authentification libraries.
If you do not have them installed then the client will complain of missing gssapi64.dll.
The recomended authentification package for Windows is MIT Kerberos.
It is important that you get the 64 bit version.
MIT does not have an installer for 64 bit Windows so it is advised to use the one provided by Secure Endpoints through their
<a href="http://www.secure-endpoints.com/#heimdal">Heimdal release</a>.
Notice that they recomend installing first the
<a href="http://www.secure-endpoints.com/binaries/mit-kfw-3-2-2/kfw-amd64-3-2-2.msi">64 bit installer</a>
and then install the
<a href="http://www.secure-endpoints.com/binaries/mit-kfw-3-2-2/kfw-i386-3-2-2.msi">32 bit installer</a>
on top of the previous one.
I can confirm that the 2nd step was not needed, just installing the 64 bit installer was enough.

\par
In order to test the nds2 client you would need to use the previously
installed Kerberos (from the Start menu the executable is called "Network Identity Manager".
Notice that you can set it up to start authomatically when you log in) to authenticate to LIGO.ORG,
using the albert.einstein naming convention.
From the menu 'Credential' choose 'Obtain new Credentials'.
In the field 'Realm' insert: LIGO.ORG, in Username insert: your equivalent albert.einstein,
then fill the Password field and you are ready to press OK.

\par
Notice that you should only have this Credential active,
if you have any other credential active at the same time causes issues with 'daq_connect' of the nds2 client.
This error is obtained:
error in gss_init_sec_contextError in daq_connect: Request SASL authentication protocol

\par
Other reason to obtain a similar error message is by having the date
and time on your computer considerably different to that of the nds2 server.

\par
At this point we can confirm that the nds2 client is properly installed
and communicating with the Ticket Manager by opening a command window and entering:n
\code{.sh}
  cd C:\Program Files (x86)\nds2-client @PROJECT_VERSION@\bin
  nds2_query -l -n nds.ligo.caltech.edu
\endcode
Check if you see the above error messages.
If everything is working you will get a (long) list of available channels. 
If you see the dialog box complaining of a missing MSVCR100.dll library then you will need to install the Visual C redistribution libraries which can be downloaded for free from
<a href="http://www.secure-endpoints.com/binaries/mit-kfw-3-2-2/kfw-i386-3-2-2.msi">Microsoft</a>.
They come as an installer, and need to be run from an account with administrator privileges.

\par
You may also want to add the nds2 client bin folder to the system variable PATH.
In order to do this right click in 'Computer' and select 'Properties'.
There click in 'Advanced system settings' for which you will need administrator privileges.
On the tab 'Advanced' click on the button 'Environment Variables' and under 'System variables' choose the variable 'Path' then click 'Edit...' and in the 'Variable value' field ADD the folder: C:\Program Files (x86)\nds2-client @PROJECT_VERSION@\bin.

<!-- ################################################################ -->

\section getting_started Getting started

This chapter explains how to get started with the NDS client programming interface,
including how to sign in to access authenticated NDS protocol 2 servers
and how to complete the configuration of your MATLAB environment.
It concludes with transcripts of very simple 'Hello, world'-like
sessions in Python, Octave, and MATLAB.

<!-- ---------------------------------------------------------------- -->

\subsection> getting_started_kerberos Obtaining a Kerberos ticket

<p>
  If you are going to work with an authenticated NDS Protocol 2 server,
  you will first need to establish a valid Kerberos ticket by running <b>kinit</b>.
  If you have never used <b>kinit</b> before,
  you will need to pass your LIGO.ORG username to <b>kinit</b>, like this:
  \code{.sh}
    $ kinit <b>albert.einstein</b>@LIGO.ORG
    Password for <b>albert.einstein</b>@LIGO.ORG:
  \endcode
  replacing <b>albert.einstein</b> with your own user name. The <b>@LIGO.ORG</b> part <em>must</em> be in all-caps.
  At the password prompt, provide your Kerberos password.
  <em>Probably need to remind users here how and when they would have signed up for a Kerberos password.</em>
</p>

<p>
  If you have previously signed in, then you can renew your existing
  Kerberos ticket by running <b>kinit</b> without a username, like this:
  \code{.sh}
    $ kinit
    Password for <b>albert.einstein</b>@LIGO.ORG:
  \endcode
  Again, type your Kerberos password when prompted.
</p>

\subsection getting_started_envirnment Setting up your MATLAB environment

<p>
  If you are going to use the MATLAB/Java interface,
  one additional configuration step is required:
  you must add the NDS client to
  <a href="http://www.mathworks.com/help/techdoc/ref/javaclasspath.html">MATLAB's Java classpath</a>.
  You can either modify the 'dynamic classpath' for your current MATLAB session,
  or permanently add NDS to MATLAB's classpath.
</p>

<!-- ---------------------------------------------------------------- -->

\subsection getting_started_classpath Adding NDS to the classpath for the current MATLAB session

<p>
  You can modify the 'dynamic classpath' for the current
  MATLAB session by using the MATLAB command
  <a href="http://www.mathworks.com/help/techdoc/ref/javaaddpath.html"><b>javaaddpath</b></a>,
  like this:
</p>
\code{.sh}
  > javaaddpath /path/to/local/share/java/nds2/nds2.jar
\endcode
\note
For example: if you are on a Mac and you installed from MacPorts, the appropriate path is:
\code{.sh}
  > javaaddpath /opt/local/share/java/nds2/nds2.jar
\endcode

<!-- ---------------------------------------------------------------- -->

\subsection getting_started_add_classpath Permanently adding NDS to the classpath

<p>
  Alternatively, you can <em>permanently</em> add NDS to the
  MATLAB classpath by editing the MATLAB configuration file
  <i>classpath.txt</i>.
  Inside MATLAB, open this file by issuing the <b>edit</b> command:

  \code{.sh}
    > edit classpath.txt
  \endcode

  This command opens <i>classpath.txt</i> in an editor.
  Scroll to the end of the file, which might look something like this:
  \code{.sh}
    ...
    # Java classpath entries for webintegration
    $matlabroot/java/jar/webintegration.jar
    # Java classpath entries for ws_clients
    $matlabroot/java/jarext/mwaws_client.jar
    $matlabroot/java/jarext/dws_client.jar
    $matlabroot/java/jarext/webservices/loginws_client.jar
    $matlabroot/java/jarext/webservices/service_request_client.jar
  \endcode

  Add the following line:
  \code{.sh}
    /path/to/local/share/java/nds2/nds2.jar
  \endcode

  Finally, save <i>classpath.txt</i> and restart MATLAB.
</p>

<!-- ---------------------------------------------------------------- -->

\subsection getting_started_java Note on java paths, jar files versus loose classes

<p>
  In previous versions the java class path was configured to point
  to a library directory and not to a specific file.
  This is a change implemented starting in NDS client 0.13.0.
  For the first few 0.13.x releases both the old method of referring
  to a directory and the new of refering to a specific file will be made available.
  However going forward users should configure their
  MATLAB installs by directly referencing the nds2.jar file.
</p>

<p>
  Some explaination for the changs.
  The MATLAB NDS client is implemented in the java programming language.
  Java packages are typically distributed in .jar files (Java ARchives).
  In the past the java/MATLAB bindings had distributed the
  NDS client as a loose set of java '.class' files.
  This is changing to bring the NDS bindings more
  in line with how Java is typically used.
  To help users during a transition period the NDS client will be
  distributed with both the '.jar' file and the loose java '.class'
  files for the next few releases.
  However the end state will involve distributing only the '.jar' files.
</p>

<!-- ---------------------------------------------------------------- -->

\subsection getting_started_script Your first script with NDS

<dt>Getting started: your first script with NDS in Python</dt>
\code{.sh}
>>> import nds2
>>> conn = nds2.connection('nds.ligo-wa.caltech.edu', 31200)
>>> print(conn)
<nds.ligo-wa.caltech.edu:31200 (protocol version 2)>
>>> conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')
>>> buffers = conn.fetch(1024417918, 1024417928, ['H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'])
>>> print(buffers)
'<H0:PEM-EY_SEISX (GPS time 953618740.000000000, 15360 samples)>', '<H0:PEM-EY_SEISY (GPS time 953618740.000000000, 15360 samples)>'
>>> buffers[0].data
2.39337158,  2.39943528,  2.38724732, ...,  2.39846468, 2.39515495,  2.39528942
\endcode

<td>Getting started: your first script with NDS in Octave</td>
\code{.sh}
> conn = nds2.connection('nds.ligo-wa.caltech.edu', 31200)
conn = <nds2.ligo-wa.caltech.edu:31200 (protocol version 2)>
> conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')
> buffers = conn.fetch(1024417918, 1024417928, {'H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'})
  [1] =
<H1:PSL-ISS_PDA_OUT_DQ (GPS time 1024417918.000000000, 327680 samples)>
  [2] =
<H1:PSL-ISS_PDB_OUT_DQ (GPS time 1024417918.000000000, 327680 samples)>
> buffers{1}.data
   2.3934
   2.3994
   2.3872
   2.3891
  ...
\endcode

<td>Getting started: your first script with NDS in MATLAB</td>
\code{.sh}
> conn = nds2.connection('nds.ligo-wa.caltech.edu', 31200)
conn =

<nds.ligo-wa.caltech.edu:31200 (protocol version 2)>
> conn.setParameter('ALLOW_DATA_ON_TAPE', '1')
> buffers = conn.fetch(1024417918, 1024417928, {'H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'})

buffers =

nds2.buffer[]:
    [nds2.buffer]
    [nds2.buffer]

<![CDATA[>>]]> buffers(1).getData()

ans =

   2.3934
   2.3994
   2.3872
   2.3891
  ...
\endcode

\subsection channel_list_cache About the channel list cache

<p>
  NDS servers typically advertise tens to hundreds of thousands of channels.
  Under the NDS1 protocol, the channel list must be downloaded before fetching data so that the data types of the channels are known.
  NDS2 does not suffer from this limitation because the data types of requested channels are always sent along with the binary data.
  However, when you are using the NDS2 client library interactively you may want to be able to search through the channel list at any time.
</p>

<p>
  The SWIG interface deals with the channel lists is several ways.
  For NDS1 connections the SWIG interface caches the channel lists for all servers to which you have connected in the past 24 hours.
  After querying the channel list for a given server, for the next 24 hours any time you call <a name="conn.find_channels"><b>find_channels()</b></a>, instead of downloading the channel list from the server, the NDS client library searches through your local cache.
  After 24 hours, the next time you call <a name="conn.find_channels"><b>find_channels()</b></a> for that server, the client library will clear the cache and download that server's channel list again.
</p>

<p>
  You can discard the contents of the channel cache for a given server at any time
  using the <b>connection.clear_cache()</b> method (in Java, <b>connection.clearCache()</b>).
  This command drops all the tables in the database, leaving the database empty.
</p>

<p>
  Each server's cache is stored on your computer in its own <a href="http://sqlite.org/">Sqlite</a> database.
    \li On UNIX, Linux, and Mac OS, the databases are stored in
        <i>/var/tmp/nds2-client_<b>VERSION</b>_<b>UID</b>_<b>HOST</b>_<b>PORT</b>.sqlite</i>.
    \li On Microsoft Windows, the databases are stored in your "local application data folder"
	under <i>&quot;C:\\Documents and Settings\\<b>USER</b>\\Local Settings\\Application Data\\nds2-client_<b>VERSION</b>_<b>HOST</b>_<b>PORT</b>.sqlite&quot;</i>.
</p>

<p>
  For NDS2 servers the list is not cached.
  However you can reduce the size of the channel list that is returned by setting an epoch (a [start,stop) time range).
  By setting an epoch you receive only the channels that were active during that [start, stop) time frame.
</p>

<!-- ################################################################ -->

\section recipies Recipes

This chapter explains the NDS client programming interface by example with recipes for common tasks.

\subsection recipies_conn_new Opening a connection

<p>
  To open a new connection, create a new connection object with <b>nds2.connection()</b>.
  The arguments are the server's hostname, port number, and the protocol version.
  If the port number is omitted, then the default port is 31200.
  If the protocol version is omitted,
  then the protocol version is automatically detected by first attempting
  an NDS2 connection and then falling back to NDS1.
</p>

<td>Opening a connection in Python</td>
\code{.sh}
# Open a connection, discovering the server's protocol version automatically.
conn = nds2.connection('example.edu')
# Equivalent to:
# conn = nds2.connection('example.edu', 31200)
# Equivalent to:
# conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_TRY)
# And also equivalent to:
# conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_ONE | nds2.connection.PROTOCOL_TWO)

# Open a connection to an NDS protocol 1 server.
conn = nds2.connection('example.edu', 31200, 1)
# Equivalent to:
# conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_ONE)

# Open a connection to an NDS protocol 2 server.
conn = nds2.connection('example.edu', 31200, 2)
# Equivalent to:
# conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_TWO)
\endcode

<td>Opening a connection in Octave</td>
\code{.sh}
#Open a connection, discovering the server's protocol version automatically.
conn = nds2.connection('example.edu')
# Equivalent to:
# conn = nds2.connection('example.edu', 31200)
# Equivalent to:
# conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_TRY)
# And also equivalent to:
# conn = nds2.connection('example.edu', 31200, bitor(nds2.connection.PROTOCOL_ONE, nds2.connection.PROTOCOL_TWO))

# Open a connection to an NDS protocol 1 server.
conn = nds2.connection('example.edu', 31200, 1)
# Equivalent to:
# conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_ONE)

# Open a connection to an NDS protocol 2 server.
conn = nds2.connection('example.edu', 31200, 2)
# Equivalent to:
# conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_TWO)
\endcode

<td>Opening a connection in MATLAB</td>
\code{.sh}
% Open a connection, discovering the server's protocol version automatically.
conn = nds2.connection('example.edu')
% Equivalent to:
% conn = nds2.connection('example.edu', 31200)
% Equivalent to:<
% conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_TRY)
% And also equivalent to:
% conn = nds2.connection('example.edu', 31200, bitor(nds2.connection.PROTOCOL_ONE, nds2.connection.PROTOCOL_TWO))

% Open a connection to an NDS protocol 1 server.
conn = nds2.connection('example.edu', 31200, 1)
% Equivalent to:
% conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_ONE)

% Open a connection to an NDS protocol 2 server.
conn = nds2.connection('example.edu', 31200, 2)
% Equivalent to:
% conn = nds2.connection('example.edu', 31200, nds2.connection.PROTOCOL_TWO)
\endcode

\subsection conn_set_parameters Setting connection parameters

<p>
  The connection object has a series of parameters that can be set.
  Currently the parameters that can be set are
  "ALLOW_DATA_ON_TAPE", "GAP_HANDLER", and "ITERATE_USE_GAP_HANDLERS".
</p>

<dl>

  <dt>ALLOW_DATA_ON_TAPE</dt>
  <dd>
    NDS2 only.
    The NDS2 server may serve data that resides on a high latency storage layer, such as a tape system.
    This may lead to data requests taking minutes or hours to complete, depending on the load on the storage system.
    As of version 0.12 of the client the default is to raise an error when accessing data that is on a high latency storage layer.
    This allows the application to provide feedback (if needed) to a users regarding amount of time that a request may take.
    If this parameter is set to a  true value ("True", "1", "yes") then an error will not be raised when requesting data on a high latency storage.

    As of client 0.12.0 "ALLOW_DATA_ON_TAPE" is set to False by default.
    You MUST enable data on tape if you want it.
  </dd>

  <dt>GAP_HANDLER</dt>
  <dd>
    For a given request there may not be be data available to fill the request completely.
    This happens due to issues upstream of the NDS server.
    How this is handled is application specific.
    Setting the "GAP_HANDLER" parameter allows the application to specify what to do.
    This includes options such as abort, zero fill the data, ...

    <table>
      <caption>Available handlers:</caption>
      <tr>
	<th>"ABORT_HANDLER"</th>
	<td>This aborts the request when a gap is found in the data.</td>
      </tr>
      <tr>
	<th>"STATIC_HANDLER_ZERO"</th>
	<td>This zero fills any missing data.</td>
      </tr>
      <tr>
	<th>"STATIC_HANDLER_ONE"</th>
	<td>This fills any missing data with ones.</td>
      </tr>
      <tr>
	<th>"STATIC_HANDLER_NAN"</th>
	<td>This fills any missing data with NaN values (or zero for integer channels).</td>
      </tr>
      <tr>
	<th>"STATIC_HANDLER_POS_INF"</th>
	<td>This fills any missing data with +infinity (or the maximum integer value for integer channels).</td>
      </tr>
      <tr>
	<th>"STATIC_HANDLER_NEG_INF"</th>
	<td>This fills any missing data with -infinity (or the minimum integer value for integer channels).</td>
      </tr>
    </table>
  </dd>

  <dt>ITERATE_USE_GAP_HANDLERS</dt>
  <dd>
    The iterate methods have a special case.  Unlike fetch operations which work on a single block, the iterate methods retrieve chunks of data that
    may not need to be contigous.  Setting ITERATE_USE_GAP_HANDLERS to "false" configures the connection to simply skip any gaps in the data and only return
    the data that is available.
	
    Please note that if you are asking for multiple channels that do not have identical gaps the NDS servers will return a data not found error if ITERATE_USE_GAP_HANDLERS is set to false.
  </dd>
</dl>

<p>
  Parameters can also be given default values in the system environment.
  The names of the environment variables are derived by prefixing them with "NDS2_CLIENT_".
  So the available enviornment variables are "NDS2_CLIENT_ALLOW_DATA_ON_TAPE",
  "NDS2_CLIENT_GAP_HANDLER", and "NDS2_CLIENT_ITERATE_USE_GAP_HANDLERS".
</p>

<!-- Python Example -->
<dt>Setting parameters in Python with connection.set_parameter()</dt>
\code{.sh}
# Allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')
# Equivalent to:
# conn.set_parameter('ALLOW_DATA_ON_TAPE', 'True')
    
# Set a gap handler:
conn.set_parameter('GAP_HANDLER', 'STATIC_HANDLER_ZERO')
\endcode

<!-- Octavwe Example -->
<dt>Setting parameters in Octave with connection.set_parameter()</dt>
\code{.sh}
# Allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')
# Equivalent to:
# conn.set_parameter('ALLOW_DATA_ON_TAPE', 'True')

# Set a gap handler:
conn.set_parameter('GAP_HANDLER', 'STATIC_HANDLER_ZERO')
\endcode

<!-- MATLAB Example -->
<td>Setting parameters in MATLAB with connection.set_parameter()</td>
\code{.sh}
% Allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')
% Equivalent to:
% conn.set_parameter('ALLOW_DATA_ON_TAPE', 'True')

% Set a gap handler:
conn.set_parameter('GAP_HANDLER', 'STATIC_HANDLER_ZERO')
\endcode

\subsection conn_find_channels Looking up and counting channels

You can look up information (channel metadata or simple channel counts) about available channels using the versatile <b>connection.find_channels()</b> and <b>connection.count_channels()</b> methods. You can look up a single channel by its name, or you can look up multiple channels with
<a href="https://github.com/lpsinger/bash_pattern"><b>bash</b>-compatible pattern matching</a>.
Channel name patterns support the following syntax:

<ul>
  <li>The <b>?</b> symbol matches any single character.</li>
  <li>The <b>*</b> symbol matches any number (zero or more) of characters.</li>
  <li>Comma-separated lists of alternatives inside curly braces such as <b>{FOO,BAR,XYZZY}</b> match any of <b>FOO</b>, <b>BAR</b>, or <b>XYZZY</b>.</li>
  <li>Lists of alternatives can contain wildcards or nested lists of alternatives.</l>
</ul>

As an example containing all of the above features, the pattern <b>?????, * {l{on,ov}e}ly {world,planet}</b> matches the text <b>Hello, very lovely world</b>.

<p>You can optionally limit the search by channel type: <b>ONLINE</b>, <b>RAW</b>, <b>RDS</b>, <b>STREND</b>, <b>MTREND</b>, <b>TEST_POINT</b>, <b>STATIC</b>, or any bitwise combination of these flags.</p>

<p>You can also limit the search by data type: <b>INT16</b>, <b>INT32</b>, and <b>INT64</b> for signed 16-, 32-, and 64-bit integers respectively; <b>FLOAT32</b> and <b>FLOAT64</b> for 32- and 64-bit floating point, <b>COMPLEX32</b> for floating point complex; or any bitwise combination of these flags.</p>

<p>Finally, you can restrict the search by minimum and maximum sample rate in Hz.</p>

<p>When requesting channel counts there are a few items to concider.  On NDS1 all counts are done against the locally cached channel database.  On NDS2 simple requests (filtering on name, and channel type [<b>RAW, ONLINE, ...</b>]) are simple requests to the servers, more detailed requests will require the server to send a partial channel list for client side filtering.  For NDS2 the channel count information is also constrained to the currently set NDS2.</p>

<!-- Python Example -->
<dt>Looking up channel information in Python with <b>connection.find_channels()</b></dt>
\code{.sh}
# List all channels.
conn.find_channels()
# Equivalent to:
# conn.find_channels('*')
<('<G1:ASC_BDO3_EP-ROT (512Hz, RAW, INT16)>', '<G1:ASC_BDO3_EP-TILT (512Hz, RAW, INT16)>', ...]]>

# Look up a single channel by name.
conn.find_channels('H0:PEM-EY_SEISX')
('<H0:PEM-EY_SEISX (256Hz, RAW, INT16)>',)

# Look up multiple channels by name with a <b>?</b> wildcard, standing for exactly one free character.
conn.find_channels('H0:PEM-EY_SEIS?')
('<H0:PEM-EY_SEISX (256Hz, RAW, INT16)>', '<H0:PEM-EY_SEISY (256Hz, RAW, INT16)>', '<H0:PEM-EY_SEISZ (256Hz, RAW, INT16)>')

# Look up multiple channels by name with a <b>?</b> wildcard, standing for any number (zero or more) free characters.
conn.find_channels('H0:PEM-EY_*')
'<H0:PEM-EY_BPO5 (16Hz, RAW, FLOAT32)>', '<H0:PEM-EY_BPO5.max (0.0166667Hz, MTREND, FLOAT32)>', '<H0:PEM-EY_BPO5.mean (0.0166667Hz, MTREND, FLOAT64)>', ...

# Look up all ONLINE-type channels with any name.
conn.find_channels('*', nds2.channel.CHANNEL_TYPE_ONLINE)

# Look up all 16-bit and 32-bit integer, ONLINE-type channels with names ending in 'SEISY'.
conn.find_channels('*SEISY', nds2.channel.CHANNEL_TYPE_ONLINE, nds2.channel.DATA_TYPE_INT16 | nds2.channel.DATA_TYPE_INT32)

# Look up all 16-bit integer, ONLINE-type channels with names ending in 'SEISY' and whose sample rates are between 128 Hz and 512 Hz.
conn.find_channels('*SEISY', nds2.channel.CHANNEL_TYPE_ONLINE, nds2.channel.DATA_TYPE_INT16, 128, 512)
\endcode

<!-- Python Example -->
<dt>Counting available channels in Python with <b>connection.count_channels()</b></dt>
\code{.sh}
# Count all channels.
conn.count_channels()
9686715L
# Equivalent to:
# conn.count_channels('*')
9686715L

# Count all L1 online channels (connected to nds.ligo-la.caltech.edu)
conn.count_channels('L1:*', nds.channel.CHANNEL_TYPE_ONLINE)
1036L
\endcode

<!-- Octave Example -->
<dt>Looking up channel information in Octave with <b>connection.find_channels()</b></dt>
\code{.sh}
# List all channels.
conn.find_channels()
# Equivalent to:
# conn.find_channels('*')
('<G1:ASC_BDO3_EP-ROT (512Hz, RAW, INT16)>', '<G1:ASC_BDO3_EP-TILT (512Hz, RAW, INT16)>', ...)

# Look up a single channel by name.
conn.find_channels('H0:PEM-EY_SEISX')
('<H0:PEM-EY_SEISX (256Hz, RAW, INT16)>',)

# Look up multiple channels by name with a <b>?</b> wildcard, standing for exactly one free character.
conn.find_channels('H0:PEM-EY_SEIS?')
('<H0:PEM-EY_SEISX (256Hz, RAW, INT16)>', '<H0:PEM-EY_SEISY (256Hz, RAW, INT16)>', '<H0:PEM-EY_SEISZ (256Hz, RAW, INT16)>')

# Look up multiple channels by name with a <b>?</b> wildcard, standing for any number (zero or more) free characters.
conn.find_channels('H0:PEM-EY_*')
('<H0:PEM-EY_BPO5 (16Hz, RAW, FLOAT32)>', '<H0:PEM-EY_BPO5.max (0.0166667Hz, MTREND, FLOAT32)>', '<H0:PEM-EY_BPO5.mean (0.0166667Hz, MTREND, FLOAT64)>', ...)

# Look up all ONLINE-type channels with any name.
conn.find_channels('*', nds2.channel.CHANNEL_TYPE_ONLINE)

# Look up all 16-bit and 32-bit integer, ONLINE-type channels with names ending in 'SEISY'.
conn.find_channels('*SEISY', nds2.channel.CHANNEL_TYPE_ONLINE, bitor(nds2.channel.DATA_TYPE_INT16, nds2.channel.DATA_TYPE_INT32))

# Look up all 16-bit integer, ONLINE-type channels with names ending in 'SEISY' and whose sample rates are between 128 Hz and 512 Hz.
conn.find_channels('*SEISY', nds2.channel.CHANNEL_TYPE_ONLINE, nds2.channel.DATA_TYPE_INT16, 128, 512)
\endcode

<!-- Octave Example -->
<dt>Counting available channels in Octave with <b>connection.count_channels()</b></dt>
\code{.sh}
# Count all channels.
conn.count_channels()
9686715L
# Equivalent to:
conn.count_channels('*')
9686715L

# Count all L1 online channels (connected to nds.ligo-la.caltech.edu)
conn.count_channels('L1:*', nds.channel.CHANNEL_TYPE_ONLINE)
1036L
\endcode

<!-- MATLAB Example -->
<dt>Looking up channel information in MATLAB with <b>connection.findChannels()</b></dt>
\code{.sh}
% List all channels.
conn.findChannels()
% Equivalent to:
% conn.findChannels('*')
'<G1:ASC_BDO3_EP-ROT (512Hz, RAW, INT16)>', '<G1:ASC_BDO3_EP-TILT (512Hz, RAW, INT16)>', ...

% Look up a single channel by name.
conn.findChannels('H0:PEM-EY_SEISX')
('<H0:PEM-EY_SEISX (256Hz, RAW, INT16)>',)

% Look up multiple channels by name with a <b>?</b> wildcard, standing for exactly one free character.
conn.findChannels('H0:PEM-EY_SEIS?')
('<H0:PEM-EY_SEISX (256Hz, RAW, INT16)>', '<H0:PEM-EY_SEISY (256Hz, RAW, INT16)>', '<H0:PEM-EY_SEISZ (256Hz, RAW, INT16)>')

% Look up multiple channels by name with a <b>?</b> wildcard, standing for any number (zero or more) free characters.
conn.findChannels('H0:PEM-EY_*')
('<H0:PEM-EY_BPO5 (16Hz, RAW, FLOAT32)>', '<H0:PEM-EY_BPO5.max (0.0166667Hz, MTREND, FLOAT32)>', '<H0:PEM-EY_BPO5.mean (0.0166667Hz, MTREND, FLOAT64)>', ...

% Look up all ONLINE-type channels with any name.
conn.findChannels('*', nds2.channel.CHANNEL_TYPE_ONLINE)

% Look up all 16-bit and 32-bit integer, ONLINE-type channels with names ending in 'SEISY'.
conn.findChannels('*SEISY', nds2.channel.CHANNEL_TYPE_ONLINE, bitor(nds2.channel.DATA_TYPE_INT16, nds2.channel.DATA_TYPE_INT32))

% Look up all 16-bit integer, ONLINE-type channels with names ending in 'SEISY' and whose sample rates are between 128 Hz and 512 Hz.
conn.findChannels('*SEISY', nds2.channel.CHANNEL_TYPE_ONLINE, nds2.channel.DATA_TYPE_INT16, 128, 512)
\endcode

<!-- MATLAB Example -->
<dt>Counting available channels in Matlab with <b>connection.count_channels()</b></dt>
\code{.sh}
% Count all channels.
conn.count_channels()
9686715L
% Equivalent to:
conn.count_channels('*')
9686715L

% Count all L1 online channels (connected to nds.ligo-la.caltech.edu)
conn.count_channels('L1:*', nds.channel.CHANNEL_TYPE_ONLINE)
1036L
\endcode

\subsection conn_fetch Fetching offline data

<p>
  Use the <b>connection.fetch()</b> method to retrieve data spanning a range of GPS times and a list of channel names.
</p>

<p>
  If the requested data range comprises multiple buffers, <b>connection.fetch()</b> will concatenate and trim the buffers as required to match the requested time range.
</p>

\warning
If your request includes one or more minute-trend channels, then the GPS start and stop time arguments <em>must be divisible by 60</em>.

<!-- Python Eample -->
<dt>Fetching offline data in Python</dt>
\code{.sh}

# As these requests are for older data explicitly allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')

# Arguments are start and stop times in GPS seconds and a list of channel names.
buffers = conn.fetch(1024417918, 1024417928, ['H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'])

# The return value of <b>fetch()</b> is a sequence of <b>nds2.buffer</b> objects.
print buffers
('<H1:PSL-ISS_PDA_OUT_DQ (GPS time 1024417918.000000000, 327680 samples)>', '<H1:PSL-ISS_PDB_OUT_DQ (GPS time 1024417918.000000000, 327680 samples)>')

# Each buffer object has a <b>channel</b> property which provides information about the channel.
print buffers[0].channel
<H1:PSL-ISS_PDA_OUT_DQ (32768Hz, RAW, FLOAT32)>
print buffers[0].channel.sample_rate
32768.0

# Each buffer object also has properties called <b>gps_seconds</b> and <b>gps_nanoseconds</b> that relate the buffer's timestamp.
print buffers[0].gps_seconds, buffers[0].gps_nanoseconds
1024417918 0

# Finally, each buffer's numerical content is in the property called <b>data</b>.
buffers[0].data
array([ 2.39337158,  2.39943528,  2.38724732, ...,  2.39846468,
        2.39515495,  2.39528942], dtype=float32)
\endcode
</example>

<!-- Octave Example -->
<dt>Fetching offline data in Octave</dt>
\code{.sh}
# As these requests are for older data explicitly allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')

# Arguments are start and stop times in GPS seconds and a list of channel names.
buffers = conn.fetch(1024417918, 1024417928, {'H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'})

# The return value of <b>fetch()</b> is a cell array of <b>nds2.buffer</b> objects.
buffers
buffers =
(
  [1] =

<H1:PSL-ISS_PDA_OUT_DQ (GPS time 1024417918.000000000, 327680 samples)>
  [2] =

<H1:PSL-ISS_PDB_OUT_DQ (GPS time 1024417918.000000000, 327680 samples)>
)

# Each buffer object has a <b>channel</b> property which provides information about the channel.
buffers{1}.channel
ans =

<H1:PSL-ISS_PDA_OUT_DQ (32768Hz, RAW, FLOAT32)>
buffers{1}.channel.sample_rate
ans =  32768

# Each buffer object also has properties called <b>gps_seconds</b> and <b>gps_nanoseconds</b> that relate the buffer's timestamp.
buffers{1}.gps_seconds
ans =  1.0244e+09
buffers{1}.gps_nanoseconds
ans = 0

# Finally, each buffer's numerical content is in the property called <b>data</b>.
buffers{1}.data
ans =

   2.3934
   2.3994
   2.3872
   2.3891
   ...
\endcode

<!-- MATLAB Example -->
<dt>Fetching offline data in MATLAB</dt>
\code{.sh}
% As these requests are for older data explicitly allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')

% Arguments are start and stop times in GPS seconds and a list of channel names.
buffers = conn.fetch(1024417918, 1024417928, {'H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'})

% The return value of <b>fetch()</b> is a cell array of <b>nds2.buffer</b> objects.
buffers

buffers =

nds2.buffer[]:
    [nds2.buffer]
    [nds2.buffer]

% Each buffer object has a <b>channel</b> property which provides information about the channel.
buffers(1).getChannel()

ans =

<H1:PSL-ISS_PDA_OUT_DQ (32768Hz, RAW, FLOAT32)>

buffers(1).getChannel().getSampleRate()

ans =

       32768

% Each buffer object also has properties called <b>gps_seconds</b> and <b>gps_nanoseconds</b> that relate the buffer's timestamp.
buffers(1).getGpsSeconds()

ans =

   1.0244e+09

buffers(1).getGpsNanoseconds()

ans =

     0

% Finally, each buffer's numerical content is in the property called <b>data</b>.
buffers(1).getData()

ans =

    2.4013
    2.3873
    2.3949
    2.3995
    ...
\endcode

\subsection conn_iterate Grabbing online or offline data progressively

<p>Use the method <b>connection.iterate()</b> to retrieve data progressively or online. Use <b>connection.next()</b> to grab each chunk as it becomes available. <b>connection.iterate()</b> supports the following arguments:
  <table>
    <tr>
      <th>Offline, manual step size</th>
      <td>
	connection.iterate(<b>start</b>, <b>stop</b>, <b>step</b>, <b>list_of_channel_names</b>) Request the channels <b>list_of_channel_names</b>, spanning GPS times in seconds from <b>start</b> to <b>stop</b>, in blocks of <b>step</b> seconds.
      </td>
    </tr>
    <tr>
      <th>Offline, automatic step size</th>
      <td>
	connection.iterate(<b>start</b>, <b>stop</b>, <b>list_of_channel_names</b>) Same as above, but choose an appropriate <b>step</b> automatically to keep each buffer smaller than 128 MB.
      </td>
    </tr>
    <tr>
      <th>Online, manual step size</th>
      <td>
	connection.iterate(<b>step</b>, <b>list_of_channel_names</b>) Request online data for channels <b>list_of_channel_names</b> starting from the current GPS time and continuing indefinitely. Transfer data in steps of <b>step</b> seconds.
      </td>
    </tr>
    <tr>
      <th>Online, autoamtic step size</th>
      <td>
	connection.iterate(<b>list_of_channel_names</b>) Request online data, but use the shortest possible <b>step</b>.
      </td>
    </tr>
  </table>
</p>

\warning
If your request includes one or more minute-trend channels,
then <b>start</b>, <b>stop</b>, and <b>step</b> must be divisible by 60.

\warning
If you specify a GPS stop time,
the returned buffers may not exactly end on that GPS stop time,
because <b>connection.next()</b> makes no attempts to trim the received buffers to size.

<!-- Python Example -->
<p>In Python, <b>connection.iterate()</b> returns the original <b>nds2.connection</b> object, which implements the
<a href="http://docs.python.org/2/library/stdtypes.html#iterator-types">Python iterator protocol</a>.
As a result, you can use <b>connection.iterate()</b> as the iterator of a
<a href="http://docs.python.org/2/reference/compound_stmts.html#for"><b>for...in</b> loop</a>.
<dt>Grabbing online or offline data progressively in Python</dt>
\code{.sh}
# As these requests are for older data explicitly allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')

# To progressively download offline data in small chunks,
# call connection.iterate() with the same arguments as connection.fetch():
# GPS start and stop times, and list of channel names.
for bufs in conn.iterate(1024417918, 1024417928, ['H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ']):
    print bufs


('<H1:PSL-ISS_PDA_OUT_DQ (GPS time 1024417918.000000000, 32768 samples)>', '<H1:PSL-ISS_PDB_OUT_DQ (GPS time 1024417918.000000000, 32768 samples)>'
('<H1:PSL-ISS_PDA_OUT_DQ (GPS time 1024417919.000000000, 32768 samples)>', '<H1:PSL-ISS_PDB_OUT_DQ (GPS time 1024417919.000000000, 32768 samples)>')
...

# Alternatively, to initiate online data retrieval starting from the current time,
# omit the GPS start and stop time when calling connection.iterate().
for bufs in conn.iterate(['H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ']):
    print bufs
...
\endcode
</example>

<!-- Example -->
In Octave, there is no <b>for...in</b> loop, but you can retrieve the buffers in a <b>while</b> loop, checking whether there is more data remaining with <b>connection.has_next()</b>.
<dt>Grabbing online or offline data progressively in Octave</dt>
\code{.sh}
# As these requests are for older data explicitly allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')

# To progressively download offline data in small chunks,
# call connection.iterate() with the same arguments as connection.fetch():
# GPS start and stop times, and list of channel names.
conn.iterate(1024417918, 1024417928, {'H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'});

# After calling connection.iterate(), call connection.next() repeatedly to retrieve bufferes repeatedly.
while conn.has_next()
  bufs = conn.next();
  disp(bufs);
end
(
  [1] =

<H1:PSL-ISS_PDA_OUT_DQ (GPS time 1024417918.000000000, 32768 samples)>
  [2] =

<H1:PSL-ISS_PDB_OUT_DQ (GPS time 1024417918.000000000, 32768 samples)>
)
(
  [1] =

<H1:PSL-ISS_PDA_OUT_DQ (GPS time 1024417919.000000000, 32768 samples)>
  [2] =

<H1:PSL-ISS_PDB_OUT_DQ (GPS time 1024417919.000000000, 32768 samples)>
)
...

# Alternatively, to initiate online data retrieval starting from the current time,
# omit the GPS start and stop time when calling connection.iterate().
conn.iterate({'H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'});
while conn.has_next()
  bufs = conn.next();
...
\endcode

<!-- MATLAB Example -->
You can also use a <b>while conn.hasNext()</b> loop in MATLAB.
<dt>Grabbing online or offline data progressively in MATLAB</dt>
\code{.sh}
% As these requests are for older data explicitly allow data on tape.
conn.set_parameter('ALLOW_DATA_ON_TAPE', '1')

% To progressively download offline data in small chunks,
% call connection.iterate() with the same arguments as connection.fetch():
% GPS start and stop times, and list of channel names.
conn.iterate(1024417918, 1024417928, {'H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'});

% After calling connection.iterate(), call connection.next() repeatedly to retrieve bufferes repeatedly.
while conn.hasNext()
  bufs = conn.next();
  disp(bufs);
end
nds2.buffer[]:
    [nds2.buffer]
    [nds2.buffer]



nds2.buffer[]:
    [nds2.buffer]
    [nds2.buffer]

...

% Alternatively, to initiate online data retrieval starting from the current time,
% omit the GPS start and stop time when calling connection.iterate().
conn.iterate({'H1:PSL-ISS_PDA_OUT_DQ', 'H1:PSL-ISS_PDB_OUT_DQ'});
while conn.hasNext()
  bufs = conn.next();
...
\endcode

\subsection conn_get_availability Getting channel availability

Channels are not always available.
The channel may have a gap in the data for many reasons,
including events such as computer and IFO downtime at the observatories
(planned and unplanned) and IFO model changes (renaming, adding/removing channges, ...).

NDS2 servers have a listing of when each channel is available
and can provide that data to users.
NDS1 servers do not currently export this information.
As such this section only applies to NDS2 connections.

The <b>connection.get_availability()</b> method is used to retrieve timespans when a channel is available.
The call operates on a list of channel names,
and returns detailed information regarding the availability of each channel requested.
The information includes the start and stop times of segments of data as well
as the frame types of the files these segments of data apply to.
The detailed list often contains overlapping segments, often science and commissioning frames.

If all you need is a list of timespans that the data is available during,
you can convert the detailed availability list to a simpler list that is only gps_start, gps_stop pairs.

<!-- Python Example -->
<dt>Retrieving H1:GDS-CALIB_STRAIN availability during O1 in python</dt>
\code{.sh}

import nds

conn = nds.connection('nds.ligo.caltech.edu', 31200)

# only view the O1 time spans
# we could also do this as
# conn.set_epoch('1126621184-1137258496')
# or
# conn.set_epoch(1126621184, 1137258496)
#
conn.set_epoch('O1')

avail = conn.get_availability(['H1:GDS-CALIB_STRAIN'])

print "Detailed availability - %s" % avail[0].name
for entry in avail[0].data:
	print "%s %d %d" % (entry.frame_type, entry.gps_start, entry.gps_stop)

# convert the detailed list to a simplified list
simple_avail = avail.simple_list()

print "Simplified availability"
for entry in simple_avail[0]:
	print "%d %d" % (entry.gps_start, entry.gps_stop)

# This produces the following output (with the output truncated):
Detailed availability - H1:GDS-CALIB_STRAIN
H-H1_HOFT_C00 1126621184 1126988756
H-H1_HOFT_C00 1126989824 1127581128
H-H1_HOFT_C00 1127581340 1127581364
H-H1_HOFT_C00 1127581452 1127581772
H-H1_HOFT_C00 1127581796 1129383148
H-H1_HOFT_C00 1129396152 1129999948
H-H1_HOFT_C00 1130000036 1130014756
...
Simplified availability
1126621184 1126988756
1126989824 1127581128
1127581340 1127581364
1127581452 1127581772
1127581796 1129383148
1129396152 1129999948
1130000036 1130014756
...

\endcode

<!-- ################################################################ -->

\section common_problems Common Problems
<p>
  There are a few common issues that have been seen when working with NDS1 and NDS2 using the SWIG clients.
</p>

<dl>
  <dt>Slow data retrieval when data is on tape.</dt>
  <dd>
    It is simply a slow process to read back data from tape.
    To help manage expectations the SWIG client will
    now default to an error state when asked for data that resides on tape.
    This is not done to prevent access to the data,
    but to allow the user to resubmit the request
    with the understanding that data will take a while (you should expect to wait at least several minutes,
    possibly much longer depending on the load on the tape system).

    You can automatically ignore errors caused by data being on tape by setting the 'ALLOW_DATA_ON_TAPE' parameter.
    <ul>
      <li>Either by setting the NDS2_CLIENT_ALLOW_DATA_ON_TAPE environment variable to 1 and restarting your analysis job (restarting python, matlab, or octave).  On a UNIX type system with bash it can be set as <b>export NDS2_CLIENT_ALLOW_DATA_ON_TAPE=1</b>, remembering to start the analysis job from that shell.</li>
      <li>Or by calling the set_parameter('ALLOW_DATA_ON_TAPE', '1') method on your connection object.</li>
    </ul>
  </dd>


  <dt>Data not found due to a gap.</dt>
  <dd>
    This problem is mitigated with the new SWIG interface which
    allows fetch and iterate requests to survive gaps in the data.
    In addition the connection object has ability
    to return availability information.
  </dd>

        
  <dt>NDS2 does not recognize my channel.</dt>
  <dd>
    If the data exists, it may be that the NDS2 server is mapping your request to the wrong channel.
    When a channel is addressed only by its base name it may not be resolved to proper channel.
    This can happen when there has been a rate or data type change in the epoch you currently using.
    You may request a channel 'A' expecting it to be at 2kHz, but NDS2 sees two versions of 'A',
    one at 2kHz and one at 8kHz and NDS2 could pick the wrong one.

    There are two ways to help mitigate this.
    First set an epoch.
    This can reduce the channel list that NDS2 has to choose from.
    Use the connection.set_epoch() method.

    Second, you can more fully qualify the channel name.
    Fully qualified names are of the form <b>basename,rate,type</b> or <b>basename,type,rate</b>.
    So you could ask for the 'H1:GDS-CALIB_STRAIN' channel
    by requesting 'H1:GDS-CALIB_STRAIN,raw,16384' or 'H1:GDS-CALIB_STRAIN,16384,raw'.
    By qualifying your channel requests you can help reduce ambiguity.
    In this example there happen to be two H1:GDS-CALIB_STRAIN channels,
    one 'raw' version which is the archived version for historical data,
    and the 'online' data which is only useful for streaming live,
    no history is retained.
  </dd>
</dl>

<!-- ################################################################ -->

\section other_doc Other Documentation
<p>
  The nds developerment group is migrating the documentation from this
  manual to a different format of documentation that will be easier to
  maintain and update as time goes forward.
  The new documentation is available at
  <a href="https://www.lsc-group.phys.uwm.edu/daswg/projects/nds-client/doc/doxygen/">
    https://www.lsc-group.phys.uwm.edu/daswg/projects/nds-client/doc/doxygen/
  </a>.
</p>

*/
