README.md

# PermaSense Data Manager: Code for management and processing data
Authors: Samuel Weber, Jan Beutel, Matthias Meyer  
Copyright: ETH Zurich, 2021  
Version: 3.0, March 31, 2021  

## Used for following publication(s)
Weber, S., Beutel, J., Da Forno, R., Geiger, A., Gruber, S., Gsell, T., Hasler, A., Keller, M., Lim, R., Limpach, P., Meyer, M., Talzi, I., Thiele, L., Tschudin, C., Vieli, A., Vonder Mühll, D., and Yücel, M.: A decade of detailed observations (2008–2018) in steep bedrock permafrost at the Matterhorn Hörnligrat (Zermatt, CH), Earth Syst. Sci. Data, 11, 1203–1237, https://doi.org/10.5194/essd-11-1203-2019, 2019.

and dataset

Weber, Samuel; Beutel, Jan; Da Forno, Reto; Geiger, Alain; Gruber, Stephan; Gsell, Tonio; Hasler, Andreas; Keller, Matthias; Lim, Roman; Limpach, Philippe; Meyer, Matthias; Talzi, Igor; Thiele, Lothar; Tschudin, Christian; Vieli, Andreas; Vonder Mühll, Daniel; Yücel, Mustafa (2020): In-situ measurements in steep bedrock permafrost in an Alpine environment on the Matterhorn Hörnligrat, Zermatt Switzerland; 2008-2019. PANGAEA, https://doi.org/10.1594/PANGAEA.916193

## Aim
The Python3 `permasense_datamgr` toolbox allows to:

* Query data from PermaSense GSN~server and save it locally as csv-files,
* Load locally stored csv-files,
* Filter according to reference values if available,
* Clean data manually if needed using a metadata lookup table,
* Generate 60-minute aggregates using in principle arithmetic mean (exceptions for weather data, see ESSD paper Weber et al., 2019),
* Export yearly csv-files for each position/location,
* Generate standard plots for all positions/locations as sanity check and
* Query images from PermaSense GSN~server and save it locally as jpg-files.

## Requirements

Python and additional modules are required. Using anaconda you can install the requirements by executing the following command from this directory.


    conda env create -f condaEnvironment.yaml
    conda activate permasense_datamgr

Note: Before you can use the function `get_GSNimg` in `manage_GSNdata.py`, you have to download NConvert and copy it in the directory `gsn_data_management`.
* NConvert is a powerful command line multi-platform batch image processor with more than 80 commands. Compatible with 500 image formats.
* **Download:** https://www.xnview.com/en/nconvert/

Note2: This software will run on any computer with an appropriate python installation. But since it's a lot of data that needs to be treaded it runs best (fastest) on a multicore linux server using the parallel download options you can switch from command line.

## Getting Started

Run:

    python manage_GSNdata.py

By default data are generated in a subdirectory `./data`


## Advanced GPS Analysis

A jupyter notebook for analyzing and plotting GNSS data is available as well:

    analyse_gps.ipynb

This can be used after populating a local database (see above) in a local subdirectory, e.g. `./data`

## Structure: Folders and Files

.  
├── condaEnvironment.yaml  
├── examples  
│   └── run_figures4paper_ESSD2020.py  
├── manage_GSNdata.py  
├── metadata  
│   ├── clean_metadata.csv  
│   ├── filter_metadata.csv  
│   └── vsensor_metadata  
├── NConvert  
│   ├── Formats.txt  
│   ├── license.txt  
│   ├── nconvert  
│   ├── Plugins.txt  
│   ├── ReadMe.txt  
│   ├── Usage.txt  
│   └── WhatsNew.txt  
├── permasense  
│   ├── GSNdata.py  
│   ├── plotting.py  
└── README.md