Previous page (GUI) <------- Back to index -------> Next page (Advanced)


Recording

Generated files contents and their naming conventions

Configuration files

The configuration files are text files that can be created by pressing the SaveAs button and giving a new name for the file. These files contain setup data in format key=value one per row, split in [groups]. If needed, they can be patched by hand with a text editor. In case of missing fields, Echoes takes a silently a default hardcoded value.


Log file

The file echoes.log is a system log created in the working directory, overwriting a previous one if present. It could be interesting only for debugging purposes, to track what the program were doing just before a crash. With -n parameter at command line, its verbosity can be incremented when more details are needed. The defined levels are:

n Mnemonic Meaning
0 NONE The log file is never created
1 FATAL Logs only fatal errors that cause immediate crash
2 CRITICAL Logs also critical errors causing loss of functionalities
3 WARNING Logs also warnings about anomalous situations that could cause future loss of functionalities
4 INFO Logs also some details about the internal state of the program that are not presented on GUI
5 DEBUG Logs also cryptic and verbose messages for development/debugging purposes

On Linux platforms, Echoes uses the system logging daemon facility syslog for logging purposes, so echoes.log won't be created.

The text sent to the logger can be sent to the standard output too by specifying the parameter -v in the command line. In this way, the log messages can be watched even in real time in a console or terminal window while the program runs.


Automatic screenshots

Automatic screenshots are .png images captured on waterfall window while acquisition runs in automatic or periodic acquisition modes. Their file names are coded as follows:

autoshot_<configuration name>_<acquisition mode>_<date/time in ISO-8601 format>_<nnnnn>.png

for instance:

autoshot_AUTO_GRAVES_automatic_2021-01-04_00067.png

tells us that it's an image captured automatically on Jan 04, 2021 when the program was running in automatic acquisition mode after loading a configuration file named AUTO_GRAVES.rts and it is the 67th image captured since 00:00 UTC.

The last 5 digits are the event/screenshot reference number (column 1 in statistic CSV).

The screenshots caught while running in periodic acquisition mode differ only for the presence of the UTC Hour besides the date and progressive number:

autoshot_default_periodic_2021-01-04T14-37-06Z_00003.png

The hour is needed because in automatic mode the event numbering is zeroed by postprocessing thread while archiving the data at midnight UTC, while this doesn't apply to files generated in other acquisition modes; the progressive number is zeroed only by stopping the acquisition, and if periodic acquisition runs for many days, the numbering covers the entire period.


Manual screenshots

The screenshots can be generated even manually in any moment, regardless the acquisition mode selected, by pressing the button Manual shot at bottom/right corner of the waterfall.

Manual shots numbering follow a different progression (mmmmm) than automatic ones (nnnnn) and are named as follows:

manshot_<configuration name>_<acquisition mode>_<datetime in ISO-8601 format>_<mmmmm>.png

Manual shots are never considered for statistic purposes.


Dump files

The files containing data to be plotted are called dump files. There are two types of dump files: power dumps and spectral dumps.

Power dumps

These ASCII files contains all the information needed to plot a 2D power vs. time diagram, without information about frequencies. They are produced when Plot type is power 2D regardless the acquisition mode chosen.

Each line of the file contains four numbers:

  1. Unix timestamp (seconds elapsed since midnight UTC, Jan 1, 1970)

  2. average signal level in the scan (N).

  3. maximum signal level in the scan (S).

  4. difference (S-N) in the scan.

for instance:

1609774415.473    -1.03   -16.35   -15.32
1609774425.472    -4.36   -16.35   -11.98
1609774435.474    -7.75   -16.35   -8.59
1609774445.474    -11.12   -16.35   -5.23
1609774455.474    -14.49   -16.35   -1.85
1609774465.473    -17.85   -16.35   1.50
1609774475.472    -21.18   -16.35   4.84
1609774485.474    -24.54   -16.35   8.19 
...

The resulting plot looks similar to the power history graph at waterfall's right side.

2Dpower

When running in Periodic or Automatic modes, the power dumps files comply with the following naming convention:

power_<configuration name>_<acquisition mode>_<date in ISO-8601 format>_<nnnnn>.dat

like

power_default_periodic_2021-01-04T21-42-15Z_00001.dat

The power dumps can be generated even in Continuous mode, that produces a continuous dump file that grows until acquisition stops. The power dumps follow the same naming convention except for the progressive number that is not needed:

power_default_periodic_2021-01-04T15-32-29Z.dat
Spectral dumps

These ASCII files contains all the information needed to plot a waterfall-equivalent diagram. The spectral dump files can be plotted on a 2D surface like a waterfall (Plot type = color mapped 2D )

2Dplot

or in a 3D perspective (Plot type = perspective 3D )

3Dplot

the format and contents of these dump files is the same, only the GNUplot command file (described later) differs.

Each line contains three numbers:

  1. Unix timestamp (seconds elapsed since midnight UTC, Jan 1, 1970)

  2. FFT point frequency [Hz]

  3. FFT point power [dBfs]

for instance:

1609794049.307    2158   -22.12
1609794049.307    4316   -20.67
1609794049.307    6474   -20.66
1609794049.307    8633   -21.07
1609794049.307    10791   -20.77
1609794049.307    12949   -21.47
1609794049.307    15107   -20.59
1609794049.307    17266   -20.63
1609794049.307    19424   -21.01
1609794049.307    21582   -20.64
1609794049.307    23741   -21.01
1609794049.307    25899   -20.72
...

When a scan line is completed, the last row contains further 4 numbers:

  1. a repetition of first number: Unix timestamp in seconds.

  2. average signal level in the scan (N).

  3. maximum signal level in the scan (S).

  4. difference (S-N) in the scan.

like this:

1609794049.307    1500000   -20.96    1609794049.307   -22.67   -16.35   6.32 

the purpose of these additional numbers is to make still possible to plot a Power 2D graph from a spectral dump, ignoring the three-numbers rows.

After the last row, an empty row marks the end of a scan line. This is required by GNUplot to plot surfaces.

The number of scan lines present in a file depends of the Refresh interval measured value and the Data dump coverage set. For this reason the spectral dumps are bulky to manage. The dump generation should be enabled only on machines having plenty of free disk space to store them.

When running in Periodic or Automatic modes, the spectral dumps files comply with the following naming convention:

spectra_<configuration name>_<acquisition mode>_<date in ISO-8601 format>_<nnnnn>.dat

like

spectra_default_periodic_2021-01-04T21-42-15Z_00003.dat

The spectral dumps can be generated even in Continuous mode, that produces a continuous dump file that grows until acquisition stops. The spectral dumps follow the same naming convention except for the progressive number that is not needed:

spectra_default_continuous_2021-01-04T21-01-55Z.dat

When no more space is available on the drive (except the space reserved with Reserved storage in Output tab) the acquisition stops automatically.


GNUplot command file

This is an ASCII file containing commands that GNUplot executes to plot all the dump files produced in the acquisition session just ended. It is created when the GNUplot output checkbox is set, regardless the acquisition mode. Depending of the Plot type requested, the commands in the file change, in order to plot the data in the required format.

The naming convention for these files is:

<plot type>_<configuration name>_<acquisition mode>_<date in ISO-8601 format>.plt

where <plot_type> can be:

for instance:

power_default_continuous_2021-01-04T15-32-29Z.plt

spectra_default_periodic_2021-01-04T21-42-15Z.plt

spec3D_default_automatic_2021-01-04T23-07-35Z.plt

The command files are simple ASCII text files, so they can be edited later, to better suit user's needs. There are many examples at gnuplot.info site to learn how to personalize the plots. It's an universe in itself.

When opening the command file with GNUplot, it starts a sequential presentation of all the plots generated during the session. A small dialog box (gnuplot pause) appears to display the next plot, till the end. The box can be dragged in a corner with the mouse if covers the plot. The plotting window supports some keyboard/mouse commands to zoom / shift / rotate the plot. Plots can also be saved as image files.


Statistic CSV table file

These files are always generated regardless the acquisition mode set; anyway their content changes depending of it. The number of columns is always the same, unused columns are kept empty. In automatic acquisition mode, all the columns are filled.

The file naming convention is similar to that defined for screenshots:

scan_<configuration name>_<acquisition mode>_<date/time in ISO-8601 format>.csv

except the prefix scan_, the other fields have the same meanings. In automatic acquisition mode one files covers exactly one day while in other modes the file is created when the acquisition starts so the date and hour embedded in file name are related to the moment when acquisition started.

The statistic CSV is a textual semicolon-separated values table, 34 columns wide. The table below describes the content of each column. The letter refers to the column ID when the file is opened in a spreadsheet :

Column nr. Column ID Title Description Automatic mode only
1 A Event/scan number While working in continuous or periodic mode, this is a progressive scan number and each row gets a different number. Numbering starts from 1 at each acquisition session. Under automatic mode instead, the program gets the capability of events detection so this column keeps the progressive event number. Since always 3 lines are produced for each event (the raising front, the peak and the falling front) these lines will be marked with the same event number.
2 B Date Date UTC of scan/event in local short format.
3 C UTC time UTC time including milliseconds
4 D Unix timestamp Unix timestamp in seconds from January 1, 1970 midnight UTC
5 E Tune [Hz] Dongle tuning frequency
6 F Downsampled BW [Hz] Downsampled bandwidth. If Downsampler bypass has been checked, it matches the hardware bandwidth or the sample rate.
7 G Central Hz Frequency centered in waterfall, it's also the center of the peak detection range
8 H Lowest Hz Lowest frequency covered by waterfall at full bandwidth.
9 I Highest Hz Highest frequency covered by waterfall at full bandwidth.
10 J Zoomed BW Hz Bandwidth displayed on waterfall after applying Freq. Zoom. If zoom is set to 1.0X, it matches the full bandwidth.
11 K Step Hz Resolution in Hz of a single FFT point
12 L Threshold mode Threshold mode: Absolute, Differential and Automatic x
13 M Up threshold [dBfs] An event peak happens when the reference data value exceeds this threshold. This value matches the one entered in GUI, except if Threshold mode (column L) is automatic; in that case, the value is calculated dynamically at each scan on the basis of the percentage given at column P x
14 N Dn threshold [dBfs] An event starts when the reference data value exceeds this threshold and terminates when the value falls under it. This value matches the one entered in GUI, except if Threshold mode (column L) is automatic; in that case, the value is calculated dynamically at each scan on the basis of the percentage given at column P x
15 O Auto up threshold [%] Upper threshold percentage in automatic mode, filled only if column (L) is Automatic x
16 P Auto down threshold [%] Lower threshold percentage in automatic mode, filled only if column (L) is Automatic x
17 Q Range low Hz Lowest frequency in event detection range
18 R Range high Hz Highest frequency in event detection range
19 S Top peak (S) [dBfs] Power of the highest signal got in the event detection range.
20 T Average peak [dBfs] Average of the last n peaks detected. n is the number entered in Averaged scans for N calculation, Output tab.
21 U Average noise (N) [dBfs] Filtered average value in the zoomed BW. This is in fact an average of average: the average values of the last n scans are averaged to obtain this value. n is the number entered in Averaged scans for N calculation, Output tab.
22 V Difference (S-N) [dBfs] Instantaneous difference between columns (S) and (U). S-N
23 W Average difference (S-N) [dBfs] Averaged difference of the latest n S-N values. n is the number entered in Averaged scans for N calculation, Output tab.
24 X Top peak [Hz] Frequency of the peak signal (S)
25 Y reserved for future use always zero
26 Z Standard deviation Standard deviation of S versus N
27 AA Lasting [mS] Time difference between the falling and raising fronts of the event. This value is nonzero only in falling front rows. x
28 AB Frequency shift [Hz] Difference between the peak frequency (X) and the central frequency (G) x
29 AC Echo area [FFT points] Number of contiguous FFT points which power value exceeds the column (N) forming the shape of the echo. This data is available only on falling edges. x
30 AD Interval area [FFT points] Number of all the FFT points included in event detection range multiplied for the number of scans covered by the event, forming a rectangle including the shape of the echo. This data is available only on falling edges. x
31 AE Peaks count Number of the FFT points included in event detection range that exceed the upper threshold (M) counted on all the scans covering the event. This data is available only on falling edges. x
32 AF LOS speed [m/s] Apparent speed (Line Of Sight). This data is available only on falling edges. x
33 AG Event status This column marks the event fronts: raise, peak, fall. x
34 AH Shot name File name of the screenshot representing this scan or event.

A note about LOS column: that speed is calculated as follows:

frequency shift at raising edge  x  Speed of light 
-------------------------------------------------------
                central frequency

Calculating this value requires the measured echo to be a head echo. Otherwise the content of this column will be unreliable.

Another note is about the areas columns:

The image below shows what is intended for areas: the Interval area is the yellow box including the echo and covering the echo lasting and the event detection interval, while Echo area is the area covered by the echo itself, here in black.

areas

Areas are expressed in FFT points, not pixels, so resizing the waterfall window doesn't affect areas. These areas are used (with other columns too) to discriminate false positives (fakes) in post-processing.

In order to avoid problems while importing the statistic CSV with office suites, the decimal point used in files could be a point or a comma, depending of the operating system localization (the language selected with -l option at startup is irrelevant).


Daily report

The data contained in report files are related to archived data (the archive is explained in Archiving section), that means only data recorded in automatic mode. One daily report file is created for each configuration rts archived. It's a CSV table file that resumes the counts of all the captured events contained in the statistic CSV files found in working directory and related to the same configuration.

There is one row per each day covered by those files; the events found are counted separately hour by hour and discriminated in underdense, overdense and fakes. Once opened in a spreadsheet, the file looks as follows:

daily_report

For representation reasons (the complete table has 176 columns) the columns related to hours 02h to 22h have been cut away.

The file is created (if not yet existing) in the archive, under the folder named as the active configuration and, every midnight UTC (midnight swap, see Archiving), a new row with the data related to the day just terminated is appended to it.

Besides the counts, the table includes the values of S and N by hours. These values plotted in a graph can give an idea of the variation of background noise along the days.

The report files follow this naming convention:

daily_<configuration name>.csv

for instance:

daily_AUTO_GRAVES.csv

Full report

The full report is a single HTML page divided in 7 sections that can be included or not in the report by checking the appropriate boxes in the Report contents group, Reporting tab. It can be created by request when pressing the Generate report button, or automatically after midnight swap if the box Automatic report is checked and if the To date in Report coverage group matches the day just terminated.

The file is created in the archive, in the subfolder named as the to date, under the root folder of the active configuration (see also the Archiving section below).

Since it contains relative links to the external images (hourly counts, screenshots and dump images) when copying the report to another support the images present in the root folder and those in subfolders should be copied too maintaining the same subfolders structure.

As alternative, you can print a pdf document from the browser to get everything bundled together.

Note: When generating a full report any previously existing html file present in the destination folder will be erased.

It is possible to generate a full report even without having screenshots available, but only dump files. While generating the report, the dumps are plotted then saved on png files making the same function of a screenshot, with the difference that there are three different plot types.

However, these automatic conversions take time, so the time it takes to generate the entire report, if you decide to include plots instead of including screenshots, could grow by a factor of 4.

The full report file name follows this naming convention:

<configuration name>-<start date>_to_<end date>.html

for instance:

AUTO_GRAVES_2020-12-18_to_2020-12-27.html

Please keep present that the start and ending dates embedded in filename are the dates entered in report coverage controls, that could not match with the dates of the data reported. This happens, for instance in case the report coverage interval includes a To date currently in the future and the report is generated now by pressing the pushbutton. The report name will include the future date even if it didn't come yet and the data and images included in the report are necessarily in the past.

You can find the report files in the date subfolder related to the last day covered by the report. The links to images and css are relative, so the report can be copied to any other place if you copy the covered date subfolders too.


Archiving

Midnight swap

If not yet existing, at midnight UTC, a 5-levels folders hierarchy is created under the working directory, where all the files produced in the past days are classified then moved. When this happens, the acquisition stops and is restarted automatically when the moving terminates. This operation is called midnight swap.

Generally speaking, the midnight swap takes a couple of minutes or even less if no plots are involved and there is only the past day to archive. Nevertheless, if the files in working directory have been produced in different days and never archived before (because the acquisition never ran on midnight UTC) Echoes will archive everyone in a proper day subfolder, increasing the total time.

The last operation executed in the swap is the deletion of expired data. This depends of the number of days specified in Data lasting, in Preferences tab; once created and filled up the folder of the past day, the oldest folders are deleted.

Archive folders hierarchy

Level I

At first level is a folder named as the configuration file active; it is called the root folder. This folder contains:

For instance, this is the archive subfolders structure on my Windows10 laptop, below the working directory, here called meteore that is placed on an external USB HDD drive:

root_folder_tree

and these are the files in AUTO_GRAVES, the root folder created to store the data collected when Echoes was running after loading the configuration AUTO_GRAVES.rts.

root_folder_content

Daily folders could contain:

As example, this is the daily subfolders structure related to December 29, 2020 on my laptop:

For instance, this is the archive subfolders structure on my Windows10 laptop, below the working directory, here called meteore that is placed on an external USB HDD drive:

date_subfolder_tree

and these are the files in AUTO_GRAVES, the root folder created to store the data collected when Echoes was running after loading the configuration AUTO_GRAVES.rts.

date_subfolder_contents

Level III

At the third level, under each day folder, three further subfolders are created. The first contains screenshots, the second contains the dumps and related GNUplot command files, while the third (stats) contains the statistic csv table of that day.

Level IV and V

The fourth level folders are created only for shots and plots, to subdivide the related files by category: underdense, overdense and false positives (fakes).
The plots have another subdivision by power and spectra (2D or 3D doesn't matter) before subdivision by category, that is at fifth level.

When the midnight swap terminates, the working directory contains only the configuration files, the manual shots and the program's log (unless other configurations have been running having the same working directory).

This is a condensed schema of the archive directories:

daily_archive


Previous page (GUI) <------- Back to index -------> Next page (Advanced)