DINEOF

(Difference between revisions)
Jump to: navigation, search
 Revision as of 20:12, 8 February 2013 (view source)Jmb (Talk | contribs) (→FAQ)← Older edit Revision as of 20:14, 8 February 2013 (view source)Jmb (Talk | contribs) (→FAQ)Newer edit → Line 106: Line 106: # the distance to the local median value # the distance to the local median value # the spatial coherence with the reconstruction # the spatial coherence with the reconstruction - * '''Why the reconstruction does partially remove gradients or variability?''' * '''Why the reconstruction does partially remove gradients or variability?''' Line 114: Line 113: It is also possible to keep the original pixels and only fill in the clouds. Doing so, the gradients and the variability is conserved but there might be artefacts at the edges of the filled regions. It is also possible to keep the original pixels and only fill in the clouds. Doing so, the gradients and the variability is conserved but there might be artefacts at the edges of the filled regions. + * ''' Can I use DINEOF with 1D or 3D fields instead of 2D images?''' + + Yes, as DINEOF does not know anything about spatial position and even in the 2D case simply stacks all values of a given moment into a column vector. The same can be done for 1D and 3D very easily.
[[File:DINEOF faq.png|frameless|super|right|350px]] [[File:DINEOF faq.png|frameless|super|right|350px]]

Revision as of 20:14, 8 February 2013

DINEOF is an EOF-based method to fill in missing data from geophysical fields, such as clouds in sea surface temperature. This page contains basic instructions to download and compile DINEOF. The source code and binaries are freely available for download right here!:

Download DINEOF now!

For more information on how DINEOF works, please refer to Alvera-Azcarate et al (2005) and Beckers and Rixen (2003) The multivariate application of DINEOF is explained in Alvera-Azcarate et al (2007), and in Beckers et al (2006) the error calculation using an optimal interpolation approach is explained. If you need a copy of any of these papers, don't hesitate to contact us! For more information about the Lanczos solver, see Toumazou and Cretaux (2001).

An example

Go directly to the DINEOF home page

You can see here a daily reconstruction of the Sea Surface Temperature of the Western Mediterranean Sea, and here for the Canary-Madeira region.

Daily filled fields, error maps and outliers for the past ten days are shown in the mentioned web page. The reconstruction is done on 6 months of data, and the whole procedure takes about two hours. The steps taken are the following:

• Download cloudy data
• Run DINEOF for outlier detection
• Remove outliers from initial data set
• Re-run DINEOF on the cleaned data set
• Calculate outliers
• Make plots, and display on web page

This product is updated daily with the latest SST data from the Mediterranean Sea.

List of DINEOF References

A list of publications made by our group about DINEOF and papers related to its development. There is also a list of papers by other people is using DINEOF for their research

Problems/Feedback

If you have problems to install or use DINEOF, we would be glad to help you. We have created a mailing list where you can ask your questions, and check the questions already posted by others. You can join as a member by going to: http://groups.google.com/group/dineof. Once you are a member, you can ask your question by sending an email to: dineof at googlegroups dot com (with obvious substitutions). Please join our discussion group! See also Troubleshooting

In order for us to help you, please provide the following information when reporting your problem through the mailing list:

1. Which version of DINEOF is used. Did you compile it yourself or did you use the binaries we provide?
2. Which operating system and which version are you working on? (for example: Windows 7, Ubuntu 10.04 ...)
3. Are you working on a 32-bit or 64 bit CPU?
4. Error message (exact screen output)

If you are compiling DINEOF from source, please make sure to give us also the following information:

1. Which Fortran compiler (and which version) are you using (for example, gfortran 4.3.0)
2. Compiler flags used to compile ARPACK
3. Your Makefile and the corresponding file in Compilers/ (e.g. Linux-gfortran.mk if you use gfortran on Linux).
4. Compile log (make.log) generated by compiling DINEOF:
```make clean all > make.log
```

Please do not omit any of those points.

FAQ

In construction!!

• Does the reconstruction take into account the time proximity of the images, i.e., does yesterday's image influence more than an image 6 months ago?

All the images have the same weight in the reconstruction. But in the recent versions, a filter is applied on the time-covariance matrix. This filtering provides a better consistence between successive images. Also note that even if there is no explicit information on time closeness, if the process is more correlated to recent events, the EOF decomposition will detect it and provide naturally a reconstruction which uses recent information rather than older ones.

• How many days/months/years of data are necessary for a reconstruction?

As a general rule, the longest time series of images will provide the best results, as the principal modes of variability will be well reproduced. However, if the memory usage is limited, then the time series can be shortened.

• Is it possible to reconstruct images with almost 100% of missing data?

Yes, it is possible, but the result may not be reliable. In general, images containing less than 5% of data do not provide useful information and might aﬀect the final result.

• How to deal with clouds that are not well detected by the algorithms?

There is now a quality control implemented in DINEOF: it provides a list of suspect pixel values according to:

1. the proximity with detected clouds
2. the distance to the local median value
3. the spatial coherence with the reconstruction
• Why the reconstruction does partially remove gradients or variability?

Because the reconstruction with DINEOF is obtained as a truncated EOF decomposition. There are higher-order modes that are not considered for the final reconstruction, as the reconstruction with this number of modes does not provide the minimal cross-validation error.

It is also possible to keep the original pixels and only fill in the clouds. Doing so, the gradients and the variability is conserved but there might be artefacts at the edges of the filled regions.

• Can I use DINEOF with 1D or 3D fields instead of 2D images?

Yes, as DINEOF does not know anything about spatial position and even in the 2D case simply stacks all values of a given moment into a column vector. The same can be done for 1D and 3D very easily.