Skip to content
Snippets Groups Projects
README.md 2.58 KiB
Newer Older
perma's avatar
perma committed
# PERMASENSE DATA MANAGER: Code for management and processing PermaSense data
Authors: Samuel Weber, Jan Beutel, Matthias Meyer  
Copyright: ETH Zurich, 2020  
jbeutel's avatar
jbeutel committed
Version: 2.0, March 31, 2020  
perma's avatar
perma committed

## Used for following publication(s)
perma's avatar
perma committed
Weber, S., Beutel, J., Da Forno, R., Geiger, A., Gruber, S., Gsell, T., Hasler, A., Keller, M., Lim, R., Limpach, P., Meyer, M., Talzi, I., Thiele, L., Tschudin, C., Vieli, A., Vonder Mühll, D., and Yücel, M.: A decade of detailed observations (2008–2018) in steep bedrock permafrost at the Matterhorn Hörnligrat (Zermatt, CH), Earth Syst. Sci. Data, 11, 1203–1237, https://doi.org/10.5194/essd-11-1203-2019, 2019.
perma's avatar
perma committed


perma's avatar
perma committed
## Aim
The Python3 `permasense_datamgr` toolbox allows to:

perma's avatar
perma committed
* Query data from PermaSense GSN~server and save it locally as csv-files,
* Load locally stored csv-files,
* Filter according to reference values if available,
* Clean data manually if needed using a metadata lookup table,
perma's avatar
perma committed
* Generate 60-minute aggregates using in principle arithmetic mean (exceptions for weather data, see ESSD paper Weber et al., 2019),
* Export yearly csv-files for each position/location,
* Generate standard plots for all positions/locations as sanity check and
* Query images from PermaSense GSN~server and save it locally as jpg-files.

## Requirements

Python and additional modules are required. Using anaconda you can install the requirements by executing the following command from this directory.

perma's avatar
perma committed

    conda env create -f condaEnvironment.yml
    conda activate permasense_datamgr

perma's avatar
perma committed
Note: Before you can use the function `get_GSNimg` in `manage_GSNdata.py`, you have to download NConvert and copy it in the directory `gsn_data_management`.
* NConvert is a powerful command line multi-platform batch image processor with more than 80 commands. Compatible with 500 image formats.
* **Download:** https://www.xnview.com/en/nconvert/


## Getting Started

Run:

perma's avatar
perma committed
    python manage_GSNdata.py
perma's avatar
perma committed

By default data are generated in a subdirectory `./data`

perma's avatar
perma committed

## Structure: Folders and Files

.  
├── condaEnvironment.yaml  
├── examples  
│   └── run_figures4paper_ESSD2020.py  
├── manage_GSNdata.py  
├── metadata  
│   ├── clean_metadata.csv  
│   ├── filter_metadata.csv  
│   └── vsensor_metadata  
├── NConvert  
│   ├── Formats.txt  
│   ├── license.txt  
│   ├── nconvert  
│   ├── Plugins.txt  
│   ├── ReadMe.txt  
│   ├── Usage.txt  
│   └── WhatsNew.txt  
├── permasense  
│   ├── GSNdata.py  
│   ├── plotting.py  
└── README.md