DINEOF

From GHER

(Difference between revisions)
Jump to: navigation, search
 
(47 intermediate revisions not shown)
Line 4: Line 4:
<html>
<html>
-
<head>
 
-
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
 
-
 
-
 
-
<style type="text/css">
 
-
.DownloadButton {
 
-
/*  display: block;*/
 
-
  display: none;
 
-
  text-decoration: none;
 
-
  width: 330px;
 
-
  background-color: green;
 
-
  text-align: center;
 
-
 
-
  border-radius: 6px 6px 6px 6px;
 
-
  -moz-border-radius: 6px 6px 6px 6px;
 
-
 
-
  background-color: #489615; /* fallback color if gradients are not supported */   
 
-
  background-image: -webkit-gradient(linear, left top, left bottom, from(#84C63C), to(#489615)); 
 
-
  background-image: -webkit-linear-gradient(top, #84C63C, #489615); 
 
-
  background-image:    -moz-linear-gradient(top, #84C63C, #489615); 
 
-
  background-image:    -ms-linear-gradient(top, #84C63C, #489615); 
 
-
  background-image:      -o-linear-gradient(top, #84C63C, #489615); 
 
-
  background-image: linear-gradient(to bottom, #84C63C, #489615); /* current standard, but unimplemented and subject to change */ 
 
-
  color: white;
 
-
  font-size: 26px;
 
-
padding: 25px 5px;
 
-
  margin: 20px;
 
-
}
 
-
</style>
 
-
 
-
</head>
 
-
<body>
 
-
 
<center>
<center>
<a href="/mediawiki/upload/DINEOF/Package-3.0/dineof-3.0.zip" class="DownloadButton DownloadButtonUnknownOS">Download DINEOF now!<br/>
<a href="/mediawiki/upload/DINEOF/Package-3.0/dineof-3.0.zip" class="DownloadButton DownloadButtonUnknownOS">Download DINEOF now!<br/>
Line 43: Line 10:
</a>
</a>
</center>
</center>
 +
</html>
-
<script>
+
For more information on how DINEOF works, please refer to [[DINEOF_references | Alvera-Azcarate et al (2005)]] and [[DINEOF_references | Beckers and Rixen (2003)]] The multivariate application of DINEOF is explained in Alvera-Azcarate et al (2007), and in [[DINEOF_references | Beckers et al (2006)]] the error calculation using an optimal interpolation approach is explained. If you need a copy of any of these papers, don't hesitate to contact us! For more information about the Lanczos solver, see Toumazou and Cretaux (2001).
-
// download button for users OS visible
+
<center>
-
document.getElementsByClassName('DownloadButton' )[0].style.display = 'block';
+
If you want to try DINEOF without installation on standard satellite data sets:
-
</script>
+
</center>
 +
<center>
 +
http://web.dineof.net/
 +
</center>
-
</body>
 
-
</html>
 
-
For more information on how DINEOF works, please refer to [[DINEOF_references | Alvera-Azcarate et al (2005)]] and [[DINEOF_references | Beckers and Rixen (2003)]] The multivariate application of DINEOF is explained in Alvera-Azcarate et al (2007), and in [[DINEOF_references | Beckers et al (2006)]] the error calculation using an optimal interpolation approach is explained. If you need a copy of any of these papers, don't hesitate to contact us! For more information about the Lanczos solver, see Toumazou and Cretaux (2001).
+
== Daily reconstruction in the western Mediterranean Sea ==
-
== An example ==
+
<!--[[Image:Dineof_onweb_small2.png|frame|Daily Mediterranean Sea SST reconstruction using DINEOF at http://www.dineof.net/DINEOF]]-->
 +
 
 +
[[Image:Dineof_logo.png|link=DINEOF|left|60px]]
 +
<html>
 +
<div style="float: right; border: 1px solid gray; padding:10px; margin:10px">
 +
<a href="http://www.dineof.net/DINEOF/dineof_cloudy.html"> <img src="http://www.dineof.net/DINEOF/WebRec/Cloudy/DINEOF2_10.png" width="397" height="142" alt="DINEOF reconstruction" title="DINEOF reconstruction"/></a>
 +
<p><center><a href="http://www.dineof.net/DINEOF/dineof_cloudy.html">Daily Mediterranean Sea SST reconstruction using DINEOF</a></center></p>
 +
</div>
 +
</html>
-
[[Image:Dineof_onweb_small2.png|frame|Daily Mediterranean Sea SST reconstruction using DINEOF at http://gher-diva.phys.ulg.ac.be/DINEOF/dineof.html]]
 
-
You can see [http://gher-diva.phys.ulg.ac.be/DINEOF/dineof_all.html here] a daily reconstruction of the Sea Surface Temperature of the Western Mediterranean Sea, and [http://gher-diva.phys.ulg.ac.be/DINEOF/dineof_allCAN.html here] for the Canary-Madeira region.
+
You can see [http://www.dineof.net/DINEOF/dineof_all.html here] a daily reconstruction of the Sea Surface Temperature of the Western Mediterranean Sea.
Line 65: Line 41:
* Download cloudy data  
* Download cloudy data  
* Run DINEOF for outlier detection
* Run DINEOF for outlier detection
-
* Remove outliers from initial data set
+
* Calculate and remove outliers from initial data set
* Re-run DINEOF on the cleaned data set
* Re-run DINEOF on the cleaned data set
-
* Calculate outliers
+
* Calculate error maps
* Make plots, and display on web page
* Make plots, and display on web page
This product is updated daily with the latest SST data from the Mediterranean Sea.
This product is updated daily with the latest SST data from the Mediterranean Sea.
 +
 +
== Daily reconstruction in the Black Sea ==
 +
 +
<!--[[Image:Dineof_onweb_small2.png|frame|Daily Mediterranean Sea SST reconstruction using DINEOF at http://www.dineof.net/DINEOF]]-->
 +
 +
[[Image:Dineof_logo.png|link=DINEOF|left|60px]]
 +
<html>
 +
<div style="float: right; border: 1px solid gray; padding:10px; margin:10px">
 +
<a href="http://www.dineof.net/DINEOF/dineof_Black.html"> <img src="http://www.dineof.net/DINEOF/WebRecBlack/Cloudy/DINEOF2_10.png" width="397" height="142" alt="DINEOF reconstruction" title="DINEOF reconstruction"/></a>
 +
<p><center><a href="http://www.dineof.net/DINEOF/dineof_Black.html">Daily Black Sea SST reconstruction using DINEOF</a></center></p>
 +
</div>
 +
</html>
 +
 +
You can see [http://www.dineof.net/DINEOF/dineof_Black.html here] a daily reconstruction of the Sea Surface Temperature of the Black Sea.
 +
 +
The same procedure as in the western Mediterranean Sea is applied to Black Sea SST (download cloudy data, run DINEOF for outlier detection, calculate and remove outliers from initial data set, re-run DINEOF on the cleaned data set, make plots, and display on the web page).
 +
 +
Other applications include [http://www.dineof.net/DINEOF/dineof_Corsica.html the north of Corsica], and [http://www.dineof.net/DINEOF/dineof_allCAN.html the Canary-Madeira region].
 +
 +
These images are generated automatically each day and show the capability of DINEOF to work in operational mode.
== Getting started ==
== Getting started ==
Line 83: Line 79:
* [[Output files | Output files]]
* [[Output files | Output files]]
* [[DINEOF output | The text output ]]
* [[DINEOF output | The text output ]]
 +
* [[Troubleshooting | Troubleshooting ]]
== List of DINEOF References ==
== List of DINEOF References ==
Line 88: Line 85:
[[DINEOF_references | A list of publications]] made by our group about DINEOF and papers related to its development. There is also a list of papers by other people is using DINEOF for their research
[[DINEOF_references | A list of publications]] made by our group about DINEOF and papers related to its development. There is also a list of papers by other people is using DINEOF for their research
-
== Troubleshooting ==
 
-
There are some recurrent errors that people encounter when compiling/working with DINEOF. Here is a little list (with solutions):
 
-
=== "Segmentation fault" ===
 
-
When compiling with ifort, you may encounter a "segmentation fault" error when running DINEOF. Something like this:
 
-
  ****************************************************
+
== Problems/Feedback ==
-
  Numerical data read
+
-
  You entered the values:
+
-
  number of EOF modes you want to compute          2
+
-
  maximal size for the Krylov subspace          8
+
-
 
+
-
  You asked not to  normalise of the input matrices
+
-
 
+
-
  The right and left EOFs will be written in directory Output/
+
-
 
+
-
  ****************************************************
+
-
  You entered filenames seacoos2005.avhrr
+
-
                      seacoos2005.avhrr.mask
+
-
  Data error in UINQUIRE, not a conform file: "seacoos2005.avhrr".
+
-
  Segmentation fault
+
-
The problem appears to be that ifort allocates too few stack memory by default. If you type the following command in a terminal:
+
If you have problems to install or use DINEOF, we would be glad to help you. We have created a mailing list where you can ask your questions, and check the questions already posted by others. You can join as a member by going to: http://groups.google.com/group/dineof. Once you are a member, you can ask your question by sending an email to: '''dineof at googlegroups dot com''' (with obvious substitutions). Please join our discussion group! See also [[Troubleshooting]]
-
ulimit -s 1000000
 
-
That may solve the problem (the number needs to be a very high number, whatever you choose)
+
In order for us to help you, please provide the following information when reporting your problem through the mailing list:
-
=== "Error  = -3"===
+
# Which version of DINEOF is used. Did you compile it yourself or did you use the binaries we provide?
-
Another common problem is when you start to see this message appear in the screen when you run DINEOF:
+
# Which operating system and which version are you working on? (for example: Windows 8.1, Ubuntu 14.04 ...)
 +
# Are you working on a 32-bit or 64 bit CPU?
 +
# Error message (the exact screen output, please)
-
  Error with _saupd, info =          -3
 
-
  Check documentation in _saupd
 
-
This one is also simple to solve: you need to specify in dineof.init the number of modes to be calculated (nev), and the Kryvlov space (ncv), which need to be smaller than the temporal size of your data, AND, ncv > nev+5. If one or both of these conditions is not met, the above message will appear.
+
If you are compiling DINEOF from source, please make sure to give us also the following information:
 +
# Which Fortran compiler (and which version) are you using (for example, gfortran 4.8.0)
 +
# Compiler flags used to compile ARPACK (If you did not compile ARPACK by yourself, then you can just specify from where you got the compiled library, e.g. Ubuntu software repository or MacPorts).
 +
# Your Makefile and the corresponding file in Compilers/ (e.g. Linux-gfortran.mk if you use gfortran on Linux).
 +
# Compile log (make.log) generated by compiling DINEOF:
 +
make clean all > make.log
-
=== "Error  = -8"===
+
If you think there is a bug, please provide a stripped-down version of your problem which still exhibits the problem.
-
A less common problem:
+
-
Error with _saupd, info =          -8                                     
+
'''Please do not omit any of those points. The more information you provide, the more likely it will be that we can help you.'''
-
Check documentation in _saupd         
+
-
This problem is related to the filtering of the time covariance matrix: if the time vector is not well written (i.e., the values are not all distinct), the filtering cannot be made properly. As the time is in simple precision, even if the values are distinct, they can be read as identical values.
+
== FAQ ==
-
Here is an example of a time vector which lead to the problem:
+
In construction!!
-
ncdump -v time bstot_mumm.nc
+
-
data:                                                             
+
-
time = 731582, 731582.020833333, 731582.041666667, 731582.0625,
+
-
Solutions:
+
<div style="float: left;text-align:justify; width: 55%">
-
* Deactivate the filtering (B_DIFF option in file ppdef.h)
+
-
* Write the time in double precision
+
-
=== netCDF compilation error ===
+
* '''Are there other ways to use DINEOF than with the compiled Fortran code?'''
-
In some systems, the fortran part of netCDF is in a separate netcdf library, called "netcdff". In those, it is needed to add -lnetcdff (two f's) before the -lnetcdf option, in the compilation command:
 
-
-lnetcdff -lnetcdf
+
If you are not needing to exploit all features of the compiled Fortran code, some wrappers or implementations in popular analysis softwares can be exploited:
-
If you use the DINEOF Makefile, this means to add this option to the <nowiki>"DINEOF_LIBRARIES"</nowiki> variable:
+
There is an R code available but not controlled, validated or maintained by our group:  
 +
http://menugget.blogspot.be/2012/10/dineof-data-interpolating-empirical.html#uds-search-results
-
DINEOF_LIBRARIES =  -L/usr/lib -larpack  -llapack  -lblas  -lnetcdff -lnetcdf
+
Then there is an online version for use with standard satellite data: http://www.dineof.net/
-
of the compilation file (Linux-gfortran.mk, etc)
+
A matlab wrapper can be found here https://svn.oss.deltares.nl/repos/openearthtools/trunk/matlab/applications/+dineof/
-
=== DINEOF compilation error due to ARPACK ===
+
There also seems to be an ArcGIS implementation: http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=1897232
-
If, when compiling DINEOF, you find the following error:
+
Finally an IDL wrapper was probably prepared : https://groups.google.com/forum/#!topic/dineof/PbkQwu8t7Z4
-
/home/user/ARPACK/libarpack.a(second.o): In function <nowiki>`second_'</nowiki>:
 
-
second.f:(.text+0x16): undefined reference to <nowiki>`etime_'</nowiki>
 
-
collect2: ld returned 1 exit status
 
-
make: *** [dineof] Error 1
 
-
you will need to edit file <nowiki>'second.f'</nowiki> located in your $HOME/ARPACK folder. There, you should comment the following line by adding an asterisk at the beginning:
+
* '''Does the reconstruction take into account the time proximity of the images, i.e., does yesterday's image influence more than an image 6 months ago?'''
-
EXTERNAL          ETIME
+
All the images have the same weight in the reconstruction. But in the recent versions, a filter is applied on the time-covariance matrix. This filtering provides a better consistence between successive images. Also note that even if there is no '''explicit''' information on time closeness, if the process is more correlated to recent events, the EOF decomposition will detect it and provide naturally a reconstruction which uses recent information rather than older ones.
-
becomes:
+
* '''How many days/months/years of data are necessary for a reconstruction?'''
-
* EXTERNAL          ETIME
+
As a general rule, the longest time series of images will provide the best results, as the principal modes of variability will be well reproduced. However, if the memory usage is limited, then the time series can be shortened.
-
== Problems/Feedback ==
+
* '''Is it possible to reconstruct images with almost 100% of missing data?'''
-
If you have problems to install or use DINEOF, we would be glad to help you. We have created a mailing list where you can ask your questions, and check the questions already posted by others. You can join as a member by going to: http://groups.google.com/group/dineof. Once you are a member, you can ask your question by sending an email to: '''dineof at googlegroups dot com''' (with obvious substitutions). Please join our discussion group!
+
Yes, it is possible, but the result may not be reliable. In general, images containing less than 5% of data do not provide useful information
 +
and might affect the final result.  
 +
* '''How to deal with clouds that are not well detected by the algorithms?'''
-
In order for us to help you, please provide the following information when reporting your problem through the mailing list:
+
There is now a quality control implemented in DINEOF: it provides a list of suspect pixel values according to:
 +
# the proximity with detected clouds
 +
# the distance to the local median value
 +
# the spatial coherence with the reconstruction
-
# Which version of DINEOF is used. Did you compile it yourself or did you use the binaries we provide?
+
* '''Why the reconstruction does partially remove gradients or variability?'''
-
# Which operating system and which version are you working on? (for example: Windows 7, Ubuntu 10.04 ...)
+
-
# Are you working on a 32-bit or 64 bit CPU?
+
-
# Error message (exact screen output)
+
 +
Because the reconstruction with DINEOF is obtained as a truncated EOF decomposition. There are higher-order modes that are not considered for the final reconstruction, as the reconstruction with this number of modes does not provide the minimal cross-validation error.
-
If you are compiling DINEOF from source, please make sure to give us also the following information:
+
It is also possible to keep the original pixels and only fill in the clouds. Doing so, the gradients and the variability is conserved but there might be artefacts at the edges of the filled regions.
-
# Which Fortran compiler (and which version) are you using (for example, gfortran 4.3.0)
+
-
# Compiler flags used to compile ARPACK
+
-
# Your Makefile and the corresponding file in Compilers/ (e.g. Linux-gfortran.mk if you use gfortran on Linux).
+
-
# Compile log (make.log) generated by compiling DINEOF:
+
-
make clean all > make.log
+
* ''' Why isn't there any graphical user interface yet ?'''
 +
DINEOF is intended for users who need to repeat calculation a number of times (be it because they are scientists who will need to test different parameter values on reconstuctions, or operational users aiming a recalculating regularly new analysis). In this case, rather then click every time across a series of menues, a command line approach with batch files is much more efficient. For the occasional user, a GUI is under development.
 +
* ''' Can I use DINEOF with 1D or 3D fields instead of 2D images?'''
-
'''Please do not omit any of those points'''.
+
Yes, as DINEOF does not know anything about spatial position and even in the 2D case simply stacks all values of a given moment into a column vector. The same can be done for 1D and 3D very easily.
 +
</div><div style="float: left; width: 45%">
 +
[[File:DINEOF faq.png|frameless|super|right|350px]]
 +
</div>

Latest revision as of 06:58, 13 September 2016

DINEOF is an EOF-based method to fill in missing data from geophysical fields, such as clouds in sea surface temperature. This page contains basic instructions to download and compile DINEOF. The source code and binaries are freely available for download right here!:


Download DINEOF now!

For more information on how DINEOF works, please refer to Alvera-Azcarate et al (2005) and Beckers and Rixen (2003) The multivariate application of DINEOF is explained in Alvera-Azcarate et al (2007), and in Beckers et al (2006) the error calculation using an optimal interpolation approach is explained. If you need a copy of any of these papers, don't hesitate to contact us! For more information about the Lanczos solver, see Toumazou and Cretaux (2001).

If you want to try DINEOF without installation on standard satellite data sets:

http://web.dineof.net/


Contents

Daily reconstruction in the western Mediterranean Sea

DINEOF reconstruction

Daily Mediterranean Sea SST reconstruction using DINEOF


You can see here a daily reconstruction of the Sea Surface Temperature of the Western Mediterranean Sea.


Daily filled fields, error maps and outliers for the past ten days are shown in the mentioned web page. The reconstruction is done on 6 months of data, and the whole procedure takes about two hours. The steps taken are the following:

  • Download cloudy data
  • Run DINEOF for outlier detection
  • Calculate and remove outliers from initial data set
  • Re-run DINEOF on the cleaned data set
  • Calculate error maps
  • Make plots, and display on web page


This product is updated daily with the latest SST data from the Mediterranean Sea.

Daily reconstruction in the Black Sea

DINEOF reconstruction

Daily Black Sea SST reconstruction using DINEOF

You can see here a daily reconstruction of the Sea Surface Temperature of the Black Sea.

The same procedure as in the western Mediterranean Sea is applied to Black Sea SST (download cloudy data, run DINEOF for outlier detection, calculate and remove outliers from initial data set, re-run DINEOF on the cleaned data set, make plots, and display on the web page).

Other applications include the north of Corsica, and the Canary-Madeira region.

These images are generated automatically each day and show the capability of DINEOF to work in operational mode.

Getting started

List of DINEOF References

A list of publications made by our group about DINEOF and papers related to its development. There is also a list of papers by other people is using DINEOF for their research


Problems/Feedback

If you have problems to install or use DINEOF, we would be glad to help you. We have created a mailing list where you can ask your questions, and check the questions already posted by others. You can join as a member by going to: http://groups.google.com/group/dineof. Once you are a member, you can ask your question by sending an email to: dineof at googlegroups dot com (with obvious substitutions). Please join our discussion group! See also Troubleshooting


In order for us to help you, please provide the following information when reporting your problem through the mailing list:

  1. Which version of DINEOF is used. Did you compile it yourself or did you use the binaries we provide?
  2. Which operating system and which version are you working on? (for example: Windows 8.1, Ubuntu 14.04 ...)
  3. Are you working on a 32-bit or 64 bit CPU?
  4. Error message (the exact screen output, please)


If you are compiling DINEOF from source, please make sure to give us also the following information:

  1. Which Fortran compiler (and which version) are you using (for example, gfortran 4.8.0)
  2. Compiler flags used to compile ARPACK (If you did not compile ARPACK by yourself, then you can just specify from where you got the compiled library, e.g. Ubuntu software repository or MacPorts).
  3. Your Makefile and the corresponding file in Compilers/ (e.g. Linux-gfortran.mk if you use gfortran on Linux).
  4. Compile log (make.log) generated by compiling DINEOF:
make clean all > make.log

If you think there is a bug, please provide a stripped-down version of your problem which still exhibits the problem.

Please do not omit any of those points. The more information you provide, the more likely it will be that we can help you.

FAQ

In construction!!

  • Are there other ways to use DINEOF than with the compiled Fortran code?


If you are not needing to exploit all features of the compiled Fortran code, some wrappers or implementations in popular analysis softwares can be exploited:

There is an R code available but not controlled, validated or maintained by our group: http://menugget.blogspot.be/2012/10/dineof-data-interpolating-empirical.html#uds-search-results

Then there is an online version for use with standard satellite data: http://www.dineof.net/

A matlab wrapper can be found here https://svn.oss.deltares.nl/repos/openearthtools/trunk/matlab/applications/+dineof/

There also seems to be an ArcGIS implementation: http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=1897232

Finally an IDL wrapper was probably prepared : https://groups.google.com/forum/#!topic/dineof/PbkQwu8t7Z4


  • Does the reconstruction take into account the time proximity of the images, i.e., does yesterday's image influence more than an image 6 months ago?

All the images have the same weight in the reconstruction. But in the recent versions, a filter is applied on the time-covariance matrix. This filtering provides a better consistence between successive images. Also note that even if there is no explicit information on time closeness, if the process is more correlated to recent events, the EOF decomposition will detect it and provide naturally a reconstruction which uses recent information rather than older ones.

  • How many days/months/years of data are necessary for a reconstruction?

As a general rule, the longest time series of images will provide the best results, as the principal modes of variability will be well reproduced. However, if the memory usage is limited, then the time series can be shortened.

  • Is it possible to reconstruct images with almost 100% of missing data?

Yes, it is possible, but the result may not be reliable. In general, images containing less than 5% of data do not provide useful information and might affect the final result.

  • How to deal with clouds that are not well detected by the algorithms?

There is now a quality control implemented in DINEOF: it provides a list of suspect pixel values according to:

  1. the proximity with detected clouds
  2. the distance to the local median value
  3. the spatial coherence with the reconstruction
  • Why the reconstruction does partially remove gradients or variability?

Because the reconstruction with DINEOF is obtained as a truncated EOF decomposition. There are higher-order modes that are not considered for the final reconstruction, as the reconstruction with this number of modes does not provide the minimal cross-validation error.

It is also possible to keep the original pixels and only fill in the clouds. Doing so, the gradients and the variability is conserved but there might be artefacts at the edges of the filled regions.

  • Why isn't there any graphical user interface yet ?

DINEOF is intended for users who need to repeat calculation a number of times (be it because they are scientists who will need to test different parameter values on reconstuctions, or operational users aiming a recalculating regularly new analysis). In this case, rather then click every time across a series of menues, a command line approach with batch files is much more efficient. For the occasional user, a GUI is under development.

  • Can I use DINEOF with 1D or 3D fields instead of 2D images?

Yes, as DINEOF does not know anything about spatial position and even in the 2D case simply stacks all values of a given moment into a column vector. The same can be done for 1D and 3D very easily.

Personal tools