Brief documentation of RCPM
installation and configuration
Installation
RCPM comes as a zip archive that contains a fully operational
working directory.
To install RCPM you have
just to extract the whole content of the zip-file to an arbitrary
working directory (RCPMWD).
drwx------+ 4
anib Domain
U 0 Nov 10 13:49 cfg
drwx------+ 2 anib Domain
U 0 Nov 10 13:49 src
-rwx------+ 1 anib Domain
U 106496 Nov 10 13:43 rcpm.exe
-rwx------+ 1 anib Domain
U 970 Nov 10 13:43 rcpm_Bariri.cfg
-rwx------+ 1 anib Domain
U 1012 Nov 10 13:43 rcpm_SollingF1.cfg
-rwx------+ 1 anib Domain
U 1005 Nov 10 13:43 rcpm_SoroeEuroflux.cfg
rcpm.exe is the
executable file. The other files in RCPMWD are configuration files
for specific application runs.
There are two sub-directories in the subdirectory cfg:
drwx------+ 8
anib Domain
U 0 Nov 10 13:49 rcpm
drwx------+ 5
anib Domain
U 0 Nov 10 13:49 runs
- The sub-directory rcpm
contains information for RCPM
to work properly. This entire directory tree must not be changed,
because file names, contents and locations are hard coded in the
programme.
- The sub-directory runs
contains configurable files. You will note quickly that you are free to
use whichever file names you want, as long they match those in the file
of the application in RCPMWD
, e.g., compare the file names in rcpm_SollingF1.cfg with those
in /runs/SollingF1 (see
below for further explanation).
A typical directory with user defined configuration
contains 5 files, e.g.:
-rwx------+ 1
anib Domain U 292
Nov 10 13:43 rcpm_GeneralProgrammeControl.cfg
-rwx------+ 1
anib Domain U 421
Nov 10 13:43 rcpm_SollingF1_Despiking.cfg
-rwx------+ 1
anib Domain U 2139 Nov
10 13:43 rcpm_SollingF1_FastSensorsList.cfg
-rwx------+ 1
anib Domain U 803
Nov 10 13:43 rcpm_SollingF1_FindSteps.cfg
-rwx------+ 1
anib Domain U 584
Nov 10 13:43 rcpm_SollingF1_SiteInfo.cfg
+ a file that would be like
-rwx------+ 1
anib Domain U 2139 Nov
10 13:43 rcpm_SollingF1_Filtered_SensorsList.cfg
These will be further described in the following section.
Configuration
Before we go into details on the content of the files, the following
syntax is worth to be noted.
syntax of the configuration files
The routines for reading the configurations and thereby the syntax
have been developed by Dirk Gros (1997). There is always a first
column that has no other function as leaving
space for important information, it is not considered at all by RCPM. The following lines
contain the configuration information between two delimiters (e.g. ':'
or '|'). Which delimiter is to be used is hard coded in the read
routine and may not be changed. So you need to follow exactly the
examples given with the programme. However everything what happens
outside the two delimiters is being ignored by the programme. The
maximal length of a record in a configuration file is
512 characters.
# FIRST RECORD FREE EDITABLE
BECAUSE NOT USED BY THE PROGRAMME, e.g. File for ... the
delimiter is '|'.
SECOND RECORD COMMENTS
ILLUSTRATIONSEXPLANATIONS NOT USED BY THE PROGRAMME
|dir /s /b|
third Record ......
|777|
4th RECORD COMMENTS
ILLUSTRATIONSEXPLANATIONS NOT USED BY THE PROGRAMME
|9.56212343|
...
Do you know which is the information used by RCPM? It is dir /s /b, 777, and 9.56212343.
The sequence of the
records is also hard coded and must therefore be identical with the
example files. With only very few exceptions there is no check,
whether the contents in the files is reasonable.
Configuration file for files and folders
(FiFo.cfg)
#
rcpm configuration file for Lonzee. The delimiter is '|'
path where to find cfg
file
|W:\rcpm\cfg\runs\Lonzee\|
path from where the raw data archives are
copied
|W:\data\raw\Lonzee\2006\06\|
path to which the new raw data are written (optional)
|W:\data\rcpm_output\Lonzee\BA\raw\|
path to which the statistical data are
written
|W:\data\rcpm_output\Lonzee\BA\stat\|
path to which the spectra are
written
|W:\data\rcpm_output\Lonzee\BA\spectra\2006\06\|
name of the fast sensor configuration
file
|rcpm_Lonzee_FastSensorsList.cfg|
name of the filtered sensor configuration
file
|rcpm_generic_FilteredSensorsList.cfg|
name of the Site information configuration
file
|rcpm_Lonzee_SiteInfo_BA.cfg|
name of the general programme control
file
|rcpm_GeneralProgrammeControl_BA.cfg|
name of the despiking control
file
|rcpm_generic_Despiking.cfg|
name of the FindSteps control
file
|rcpm_generic_FindSteps.cfg|
Delimiter is '|', the
explanations are given above.
Fast sensor's list file
/*FastSensorsNames.cfg_Syntax:
begin with x y z t the sequence may differ from the sequence in the raw
data )*/
number of fast
sensors
: 6:
name of x (case
sensitive)
:x:
window of time lag
estimation starts (int, in samples) : 0:no lag
window of time lag
estimation ends (int, in samples) : 0:no lag
default lag
time
(int, in samples) : 0:no default
LowPassFilterConstant
(0<=const<=1)
(float)
: 0:
name of
y
:y:
window of time lag
estimation starts (int, in samples) : 0:
window of time lag
estimation ends (int, in samples) : 0:
default lag
time
(int, in samples) : 0:
LowPassFilterConstant
(0<=const<=1)
(float)
: 0:
name of
z
:z:
window of time lag
estimation starts (int, in samples) : 0:
window of time lag
estimation ends (int, in samples) : 0:
default lag
time
(int, in samples) : 0:
LowPassFilterConstant
(0<=const<=1)
(float)
: 0:
name of
t
:Ts:
window of time lag
estimation starts (int, in samples) : 0:
window of time lag
estimation ends (int, in samples) : 20:
default lag
time
(int, in samples) : 0:
LowPassFilterConstant
(0<=const<=1)
(float)
: 0.0:
name of
CO2
:CO2:
window of time lag
estimation starts (int, in samples) : 0 :lag is being accepted
window of time lag
estimation ends (int, in samples) :100:between 60 and
140 samples
default lag
time
(int,samples) :0:
LowPassFilterConstant
(float)
:0.0:
name of
H2O
:H2O:
window of time lag
estimation starts (int, in samples) :0:
window of time lag
estimation ends (int, in samples) :200:
default lag
time
(int, in samples) :0:
LowPassFilterConstant
(float) :0.0:
This file allows you a little more flexibility as the read routine is
itself configured by the first numerical entry (=second row) in the
file (hier it's '6'). It is
interpreted such that the five records sequence (name, window Start,
window End, default time lag LowPassFilterConstant)
will be read for 6 times. As noted in the first record RCPM
expects always the first entries to regard to the channels of the sonic
anemometer. The names of the fast sensors must match exactly (case
sensitive)
the ones
provided in the header of the raw data file. It is however not
necessary to include all fast sensors in the post-processing. The
variable names will later be used to form the headers of the RCPM
output. The numerical parameters following the variable name are
only relevant for such eddy
correlation systems that
suffer for whichever reason, e.g. long tubes, from low pass filtering.
The correction can be switched off by entering 0 for the low pass filter
constant. The file rcpm_Soroe_FastSensorsList.cfg
gives an example where the low pass filter correction is used.
Filtered sensor's list file
/*FastSensorsNames.cfg_Syntax:
begin with x y z t the sequence may differ from the sequence in the raw
data )*/
number of fileterd
sensors
:9:
name of unfiltered
sensor
:Ts:
name of filtered
sensor
:t0004:
cut off
frequency
:0.004:
window of time lag
estimation starts (int, in samples) : 0:
window of time lag
estimation ends (int, in samples) :250:
default lag
time
(int,samples) :60:
name of unfiltered
sensor
:Ts:
name of filtered
sensor
:t0008:
cut off
frequency
:0.008:
window of time lag
estimation starts (int, in samples) : 0:
window of time lag
estimation ends (int, in samples) :150:
default lag
time
(int,samples) :48:
name of unfiltered
sensor
:Ts:
name of filtered
sensor
:t0016:
cut off
frequency
:0.016:
window of time lag
estimation starts (int, in samples) : 0:
window of time lag
estimation ends (int, in samples) :100:
default lag
time
(int,samples) :29:
name of unfiltered
sensor
:Ts:
name of filtered
sensor
:t0032:
cut off
frequency
:0.032:
window of time lag
estimation starts (int, in samples) : 0:
window of time lag
estimation ends (int, in samples) :75:
default lag
time
(int,samples) :29:
name of unfiltered
sensor
:Ts:
name of filtered
sensor
:t0065:
cut off
frequency
:0.065:
window of time lag
estimation starts (int, in samples) : 0:
window of time lag
estimation ends (int, in samples) :50:
default lag
time
(int,samples) :18:
name of unfiltered
sensor
:Ts:
name of filtered
sensor
:t0133:
cut off
frequency
:0.133:
window of time lag
estimation starts (int, in samples) : 0:
window of time lag
estimation ends (int, in samples) :50:
default lag
time
(int,samples) :10:
name of unfiltered
sensor
:Ts:
name of filtered
sensor
:t0277:
cut off
frequency
:0.277:
window of time lag
estimation starts (int, in samples) : 0:
window of time lag
estimation ends (int, in samples) :50:
default lag
time
(int,samples) : 7:
name of unfiltered
sensor
:Ts:
name of filtered
sensor
:t0614:
cut off
frequency
:0.614:
window of time lag
estimation starts (int, in samples) : 0:
window of time lag
estimation ends (int, in samples) :20:
default lag
time
(int,samples) : 5:
name of unfiltered
sensor
:Ts:
name of filtered
sensor
:t1626:
cut off
frequency
:1.626:
window of time lag
estimation starts (int, in samples) : 0:
window of time lag
estimation ends (int, in samples) :10:
default lag
time
(int,samples) : 2:
This configuration file allows you to generate new variables from
measured data (e.g. T), by filtering them with an autoregressive filter
with a given cut-off frequecy. These data are helpful to empirically
investigate the relationship beteen low-pass filtering correction
factor and cut-off frequency, windspeed and stratification (see. Ibrom,
A., E. Dellwik, N.O. Jensen, H. Flyvbjerg and K. Pilegaard. 2007.
Strong low-pass filtering effects on water vapour flux measurements
with closed-path eddy correlation systems. Agric. For. Meteorol.
147:140 -156. equation 9 and figures 5 and 6).
Passing site
specific information to the programme
#
Solling site ID for rcpm. The delimiter is '|'.
Sonic type
(1-4)
|3|
which usat correction (0: none 1:IR 2:WK
3:IS)
|0|
Sonic orientation (upward
= 1
upside down =
-1)
|-1|
Sonic coordinate system (right handed = 1 left handed =
-1) |-1|
measurement height (z-d in
m
(float)
|15.0|
sampling frequency
(Hz)
|10|
averaging intervall
(s)
|1800|
raw data file extension
(without
dot <=8 charcters
)
|e32|
raw data format (e32=1 Risoe
ASCII=2)
|1|
name (no extension) of the
statsitical output
file
|SollingF1Debug|
minimum number of observations in
a
file
|9000|
It is crucial to be sure to have entered the right number in the second
record, which lets RCPM identify the sonic type. The currently
considered
sonic types are:
- Meteks USAT-3
- Meteks USAT-3 with a USA-1 sensor head
- Meteks USA-1
- Solent R2
In the 3. record you specify, if you want to apply a correction to the
USATs. This is only valid for METEK sonics and the automatic sensor
heat correction must be switched of during the measurements. I use IS
for the USA-1. This methods still has to be published. If using a Gill
set this correction to 0. However the so-called angle of attack
correction will be aplied anyway. The 4. record is meant for those keen
researchers, who decided to use
their sonic in the upside-down mode. All sonics (Gill and USats) have a
left handed coordinate system (record 5) .
The measuremement height is used for the
normalistaion of spectra.
The extension of the raw data is used to
identify raw data files in the
file lists (see below). There is no way to run RCPM
with a wrong raw data format. You will soon be sure after
starting the programme as if to whether having entered a wrong value in
record 9.
To organise the statsitical RCPM
output to similar looking files you enter a run specific
name in record 10.
The minimum requirement for observations
in a raw data file is to be entered to record No 11. It makes RCPM reject
too small raw data files. If the number of observations gets
smaller than the
various window sizes in the spike and step finding routines RCPM will crash hoplessly.
General programme
control file
Whatever name you will give to the file it must contain three entries,
e.g.:
#
general programme control for rcpm. The delimiter is '|'.
command to generate a directory
listing
|dir /s /b|
want fast data output after any
step of post-processing? yes=1
no=0
|1|
want spectral output? yes=1
no=0
|1|
want linear detrend? yes=1
no=0 > 1 use number as time constant for recursive
filter |1|
- Call of a system commad that generates a ASCII list with the raw
data including their path. This should enable you to use the same
programme under UNIX and DOS.
- A number out of 0 and
1 for
configuring the fast data output
after every post processing step. This helps you to better understand
how the programme works and whether the set of parameter values, e.g.
for the despiking routines, let them perform well.
- Here you may select 1
for providing spectral output or 0 for ommiting it ,
which makes the programme only
slightly faster and prevents RCPM to flood the hard disk with one new
file per raw data file.
- Linear detrending may be switched off (modern view point, I'm
using it still, because it clearly excludes outliers). The output
variables are just the same, with the only difference that covariances
in step 5 are calculated from data after step 3, rather than 4.
Parameters of the linear detrend are still being output.
Configuration
of the despiking routine
#
de-spiking control for rcpm. The delimiter is '|'.
generate a file out put (yes=1
no=0)
|0|
window size in samples
(e.g.200,
int)
|200|
maxlength of a spike
(int)
|4|
threshold as manifolds (float) of
local standard
deviations |4|
number of steps (int)for
segmented_standard_deviation
|6|
In order to get rid of outliers RCPM contains a despiking routine, that
is based on a comparison of the difference of two neighbouring
values with a local standard deviation. You can tune this routine
to your needs by installing a certain window
and threshold factor. The maximum length of a spike is also given in
the 4th record. If the difference between the two neighbours does
still persist
being too high, such occurrence will be counted as a 'drop',
which is not very much different from a 'step' (see below). At
the moment this routine has still to be tested. As mentioned
above
the output of the raw data after despiking can be a valuable tool to
find the right parameter value combination. The parameter to entered in
the last record is the length of a moving
window in which the standard deviation is being calculated (so-called LP_sdev). This feature
allows you to filter out high frequency
variance in the time series and to compare the low pass filtered
standard deviation with the
un-filtered one.
Configuration of the
step find algorithm
de-spiking
control for rcpm. The delimiter is '|'.
generate a file output (yes=1
no=0)
|0|
smooth window size in samples (e.g.200,
int)
|10|
window size in samples (e.g.200,
int)
|400|
threshold as manifolds (float) of local standard
deviations |4.0|
absolute threshold sensor 0 (x)(same as in
FastSensorsList) |5.0|
absolute threshold sensor 1 (y)(same as in
FastSensorsList) |5.0|
absolute threshold sensor 2 (z)(same as in
FastSensorsList) |5.0|
absolute threshold sensor 3 (Ts)(same as in
FastSensorsList) |5.0|
absolute threshold sensor 4 (CO2)(same as in
FastSensorsList) |0.15|
absolute threshold sensor 5 (H2O)(same as in
FastSensorsList) |50.0|
The step find algorithm bases on the comparison of the difference
between two moving averages that are calculated from two neighbouring
data windows (of SmoothWindow size) that are moving through the raw
data array. It is the
intention that a step will cause a large pulse in this
difference and can therefore be easily detected by the despiking
algorithm, which in turn uses the given parameters window and threshold
factor in the same maner as above described. This feature is in
particular important, when using an open
path sensor
during rain. The find step algorithm provides also the feature to
search for absolute steps, which are configured for each channel
individually. The
algorithm is in a beta testing phase.
© Andreas Ibrom
Author Andreas Ibrom Date: 11.06.2008
Risø National Laboratory
BIO-ECO Byg. 309
Frederiksborgvej 399
DK-4000 Roskilde