Octcdf

From GHER

(Redirected from NetCDF toolbox for Octave)
Jump to: navigation, search

The package octcdf is deprecated. Please use the octave-netcdf package to read and write NetCDF files in octave.




The page below is just kept for reference.


NetCDF is a machine-independent file format for scientific data sets. Octcdf is a NetCDF toolbox for Octave which aims to be compatible with the "original" matlab toolbox. The matlab NetCDF toolbox is a collection of matlab scripts written in an object-oriented approach. This octave toolbox is written in C calling directly the NetCDF library. With the octcdf toolbox, it is therefore possible to write m-files using NetCDF which can run in matlab and octave.

Octcdf was developped before matlab implemented its own NetCDF interface (which is different from the "original" matlab toolbox). Since 2013, a matlab-compatible toolbox for octave has been released. For new projects, I recommended to use the later toolbox instead of octcdf. If an object-oriented interface is desired then ncArray (which works on matlab and octave) can be used.


With octcdf, it is also possible to access a dataset from an OPeNDAP server (Open-source Project for a Network Data Access Protocol, formerly known as DODS). OPeNDAP allows you to download efficiently a subset of large datasets such as e.g. numerical model results. The data is imported directly in octave, regardless in which format the data is stored on the server.

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

Contents

Requirements

  1. NetCDF libraries and headers (version 3.6.0 or later). Detailed installing instructions are available from the NetCDF site.
  2. Octave and headers (version 3.3.51 or higher)
  3. gcc and g++ (the same version used to build octave)

octcdf works with octave version 3.4.0 (and later), but not with octave version 3.2.x (any version of octcdf).

There are different ways to compile octcdf:

Library How is this library detected? Functionalities
netCDF 3.6.x netcdf.h and libnetcdf.{a,so} must be at default location (or CPPFLAGS or LDFLAGS must be defined) "classic" and "64-bit offset" netCDF files
netCDF 4.1 (or later) nc-config must be in PATH "classic", "64-bit offset", and "NETCDF4 with classical model" netCDF files (but no general NETCDF4 files)
OPeNDAP NetCDF Client Library (version 3.5.2 or higher) requiring the DAP SDK (version 3.5.2 or higher) ncdap-config must be in PATH "classic" and "64-bit offset" netCDF files and OPeNDAP
netCDF 4.1 (or later) with OPeNDAP support (--enable-opendap) nc-config must be in PATH "classic", "64-bit offset" and "NETCDF4 with classical model" netCDF files (but no general NETCDF4) and OPeNDAP


Note: Octave and NetCDF have to use the same HDF5 library.

Installation

Installation from your Linux repository

Some Linux distributions include already octcdf. In this case you can install octcdf and its dependencies using your distributions' package manager. Under Debian, Ubuntu or Mint this can be done from a terminal via

sudo apt-get install octave-octcdf

Users of Fedora can install octcdf by issuing

yum install octave-octcdf

This works also on Red Hat and Centos after enabling the Extra Packages for Enterprise Linux repositories.

Installation with octave's package manager

Recent version of octave (tested with 3.6.2 and later) can automatically download packages and install them in one command:

pkg install -forge -verbose -auto octcdf

The verbose option allows you to verify that the installer use the netcdf library that you are expecting to use. Or you can also download it yourself from http://octave.sourceforge.net/octcdf/index.html. Do not decompress the tar file. In octave, issue the following command:

pkg install -verbose octcdf-x.y.z.tar.gz

Per default, the NetCDF libraries are installed in /usr/local/lib and include files in /usr/local/include where octcdf will find them. The installer use the following approach to determine the location of the netcdf libraries:

  1. if nc-config is available (present in your PATH), then the library location is extracted using nc-config --libs and nc-config --cflags
  2. otherwise check if ncdap-config is present and use the information accordingly
  3. otherwise check rely on the information in the environment variables CPPFLAGS and LDFLAGS. If you have the NetCDF libraries (libnetcdf.a) and include files (netcdf.h) in a non-standard location, then you need to set the environment variables CPPFLAGS and LDFLAGS. For example, if you have installed NetCDF in /usr/local/netcdf (using ./configure --prefix /usr/local/netcdf), the following variables have to be defined before starting octave:
export CPPFLAGS=-I/usr/local/netcdf/include
export LDFLAGS=-L/usr/local/netcdf/lib

The directory /usr/local/netcdf/lib should contain at least the file libnetcdf.a (or libnetcdf.so) and the directory /usr/local/netcdf/include should contain at least the file netcdf.h

Manual installation

If the previous approach does not work for you, you can install octcdf without the package manager:

tar zxvf octcdf-x.y.z.tar.gz
cd octcdf-x.y.z/src
./configure
make
mkdir $HOME/octcdf-x.y.z
cp *.oct ../inst/*.m ../PKG_ADD $HOME/octcdf-x.y.z

You can choose any directory instead of "$HOME/octcdf-x.y.z". You need to tell octave about octcdf by using addpath:

octave:1> addpath ~/octcdf-x.y.z

You can put the addpath line in $HOME/.octaverc if you want to load octcdf when octave starts.

Loading octcdf

As any octave-forge package, the package is loaded with:

pkg load octcdf

This command add octcdf to the search path. You can add this command to your .octaverc file.

Testing the toolbox

If you have octave-forge installed, you can test the toolbox by typing "nctest" in octave. All tests should succeed.

octave:1> nctest
writing test output to nctest.log
>>>>> /home/johndoe/octave/octcdf-x.y.z/nctest.m
PASSES 23 out of 23 tests

Using the toolbox

This package was developed to be compatible with the (original) netcdf toolbox for Matlab. Most of its documentation (see https://sourceforge.net/p/mexcdf/svn/HEAD/tree/netcdf_toolbox/trunk/netcdf/@netcdf/netcdf.m) is also applicable to octcdf.

The toolbox uses an operator syntax to manipulate dimensions, variables and attributes. The toolbox provides the following functions: ncbyte, ncchar, ncclose, ncdouble, ncfloat, ncint, ncshort and netcdf. To learn more about these function type "help function_name" in octave (or visit http://octave.sourceforge.net/functions_by_package.php).

The file example_netcdf.m illustrates a typical usage of the NetCDF toolbox.

% Example for creating and reading a netcdf file


% create some variables to store them in a netcdf file

latitude = -90:1:90;
longitude = -179:1:180;
[y,x] = meshgrid(pi/180 * latitude,pi/180 * longitude);
temp = cos(2*x) .* cos(y);

%---------------------------------------%
%                                       %
% write data to a netcdf file           %
%                                       %
%---------------------------------------%

% create netcdf file called example.nc

nc = netcdf('example.nc','c');

% define the dimension longitude and latitude of size
% 360 and 181 respectively.

nc('longitude') = 360;
nc('latitude') = 181;

% coordinate variable longitude

nc{'longitude'} = ncdouble('longitude');              % create a variable longitude of type double with 
                                                      % 360 elements (dimension longitude).
nc{'longitude'}(:) = longitude;                       % store the octave variable longitude in the netcdf file
nc{'longitude'}.units = 'degrees_east';                % define a string attribute of the variable longitude

% coordinate variable latitude

nc{'latitude'} = ncdouble('latitude');;               % create a variable latitude of type double with 
                                                      % 181 elements (dimension latitude).                
nc{'latitude'}(:) = latitude;                         % store the octave variable latitude in the netcdf file
nc{'latitude'}.units = 'degrees_north';                % define a string attribute of the variable latitude

% variable temp

nc{'temp'} = ncdouble('latitude','longitude');        % create a variable temp of type double of the size 360x181 
                                                      % (dimension longitude and latitude).    
nc{'temp'}(:) = temp;                                 % store the octave variable temp in the netcdf file
nc{'temp'}.long_name = 'Temperature';                 
nc{'temp'}.units = 'degree Celsius';                  % define a string attribute of the variable
nc{'temp'}.valid_range = [-10 40];                    % define a vector of doubles attribute of the variable

                                                      % define a global string attribute
nc.history = 'netcdf file created by example_netcdf.m in octave';
nc.title = 'sample file';

close(nc)                                             % close netcdf file and all changes are written to disk


disp(['example.nc file created. You might now inspect this file with the shell command "ncdump -h example.nc"']);


%---------------------------------------%
%                                       %
% read data from a netcdf file          %
%                                       %
%---------------------------------------%

nc = netcdf('example.nc','r');                       % open netcdf file example.nc in read-only

n = nc('longitude');                                 % get the length of the dimension longitude

temp = nc{'temp'}(:);                                % retrieve the netcdf variable temp
temp_units = nc{'temp'}.units;                       % retrieve the attribute units of variable temp
temp_valid_range = nc{'temp'}.valid_range;           % retrieve the attribute valid_range of variable temp

global_history = nc.history;                         % retrieve the global attribute history


Record dimensions

To define a "record" dimension (i.e. a dimension which can grow) you can set it to zero which has a special meaning in this case:

nc('time) = 0;                                       % record dimension
nc{'time'} = ncdouble('time');                       % decare variable
nc{'time'}(1:length(time_values)) = time_values;     % store variable 

Note that the following does not work for record dimensions:

nc{'time'}(:) = time_values;  % this fails !

autonan and autoscale

If autonan is turned on, then NaN values translated to _FillValue during reading (vice versa during writing). If autoscale is turned on, then variables are scaled using the scale_factor (default 1) and add_offset (default 0) attribute:

data_in_memory = scale_factor * data_on_disk + add_offset 

The following example shows how to enable autonan and autoscale:

nc = netcdf('myncfile.nc','c');
nc('dim1') = 5;
nc{'var1'} = ncdouble('dim1');
nv = nc{'var1'};
nv = autonan(nv,1);
nv = autoscale(nv,1);

String variables

Here is an example how to write a list of string variables "loc_short_name" into a netcdf file.

% data to write
loc_short_name(1,1:30) = "PONTA_DELGADA                 ";
loc_short_name(2,1:30) = "HORTA                         ";
loc_short_name(3,1:30) = "GIBRALTAR                     ";
loc_short_name(4,1:30) = "LISBOA                        ";
loc_short_name(5,1:30) = "AKUREYRI                      ";
loc_short_name(6,1:30) = "REYKJAVIK                     ";
loc_short_name(7,1:30) = "STYKKISHOLMUR                 ";

% creating netcdf file
nc = netcdf('filename','c');

% define dimensions
nc('loc') = 7;
nc('str') = 30;

% define and store variable loc_short_name
nc{'loc_short_name'} = ncchar('loc','str');                                    
nc{'loc_short_name'}(:) = loc_short_name;  

% close netcdf file
close(nc)

OPeNDAP

The file example_opendap.m shows how to use octcdf with a OPeNDAP server. Instead of using a NetCDF file name, you use a OPeNDAP URL. The following code is a minimal example showing how to download a subset of a variable from a OPeNDAP server.

% open a opendap URL
nc = netcdf('http://asterix.rsmas.miami.edu/thredds/dodsC/atl-ops-forecast/temp','r');

% download a subset of the variable temp
temp = nc{'temp'}(end,1,661:996,77:588);

% download the attribute "missing_value" and replace every
% occurrences by NaN in our local copy
temp(temp == nc{'temp'}.missing_value) = NaN;

% close the connection
close(nc);

Troubleshooting

recompile with -fPIC

On Linux with AMD 64 bit you are seeing:

`a local symbol' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-linux-gnu/3.4.6/../../../../lib/libnetcdf.a: could not read symbols: Bad value

well, this means that your have to recompile NetCDF with the option -fPIC:

./configure --with-pic


HDF5 library version mismatched error

Error while reading a netcdf 4 file:

Warning! ***HDF5 library version mismatched error***
The HDF5 header files used to compile this application do not match
the version used by the HDF5 library to which this application is linked.
Data corruption or segmentation faults may occur if the application continues.
This can happen when an application was compiled by one version of HDF5 but
linked with a different version of static or shared HDF5 library.
You should recompile the application or check your shared library related
settings such as 'LD_LIBRARY_PATH'.
You can, at your own risk, disable this warning by setting the environment
variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'. 
Setting it to 2 or higher will suppress the warning messages totally.
Headers are 1.8.8, library is 1.8.5
	    SUMMARY OF THE HDF5 CONFIGURATION
	    =================================

General Information:
-------------------
		   HDF5 Version: 1.8.5-patch1
[...]

make sure that octave and netcdf was compiled with the same HDF5 library.

Segmentation fault on opening any OPENDAP URL

Segmentation fault on opening any OPENDAP URL. For example

ncdump('http://test.opendap.org/dap/data/nc/sst.mnmean.nc.gz');
...
Segmentation fault (core dumped)

This issue was observed on Centos 6.4 with octave 3.4.3, octcdf 1.1.2 and netcdf 4.1.1 (from EPEL). The issue seems to come from the fact that netcdf and ncurse have both a function called attrset and the wrong function is called from netcdf. As a workaround one can start octave as the following:

LD_PRELOAD=/usr/lib64/libnetcdf.so.6 octave

The function attrset from netcdf was renamed dap_attrset in version netcdf 4.1.2. So later version of netcdf should not have this issue.


Multiple NetCDF libaries or libaries at non-standard location

Check the output of the shell commands:

which nc-config 
nc-conflig --libs

Does this correspond to the library that you want to use?

If not, define your PATH and LD_LIBRARY_PATH (in your .bashrc and in your current shell)

export PATH="/path/to/the/right/bin/nc-config:$PATH"
export LD_LIBRARY_PATH="/path/to/the/right/lib:$LD_LIBRARY_PATH"

Report Bugs

If you want to write a bug report, please include the following information:


  1. A minimal code which reproduces the error.
  2. Name of your architecture (for example Intel Celeron or AMD Opteron)
  3. Version of octave and from where you got it (for example octave 3.6.0 from http://www.gnu.org/software/octave/download.html or octave 3.6.0 from the cygwin repository)
  4. Version of octcdf and from where you got it (for example octcdf 1.0.11 from sourceforge or octcdf 1.0.11 from Fedora's octave-forge package)
  5. Version of your NetCDF library
  6. The file nctest.log created by nctest in octave
  7. Name and version of your operating system. For Linux, please specify the distribution (for example Windows 8 or Linux Fedora 19). If you use cygwin, please sent also the list of installed packages by running the command:
cygcheck -c > all_packages.txt

and sent the file "all_packages.txt".

Comments and feedback

Any suggestions, comments, bug fixes, ... are of course very appreciated. Send them to the help@octave.org.

Alexander Barth

Personal tools