# PEM Toolkit

Post-processing and diagnostics utilities for PEM outputs.

This folder contains small tools to:
- inspect NetCDF files,
- merge NetCDF time chunks,
- visualize fields (maps, sections, profiles),
- inspect stratification layers in restart files,
- edit `startfi.nc` orbital/variable settings,
- orchestrate repeated script execution over multiple run directories,
- compute quick Fortran code statistics.

## Contents

| File | Purpose | Mode |
|---|---|---|
| `analyse_netcdf.py` | Print min/max/mean and anomaly flags for numeric NetCDF variables | CLI + interactive |
| `concat_pem.py` | Concatenate `{basename}#.nc` files along `Time` | CLI + interactive prompts |
| `display_netcdf.py` | Generic NetCDF variable plotting tool | Interactive + partial CLI |
| `ini_pem_orbit.sh` | Set `startfi.nc` orbital fields from `run_pem.def` date and `obl_ecc_lsp.asc` lookup/interpolation | CLI (script-header parameters) |
| `modify_startfi_orbit.sh` | Set fixed orbital values in `startfi.nc` and derive related fields | CLI (script-header parameters) |
| `modify_startfi_var.sh` | Set any NetCDF variable to a chosen value in `startfi.nc` | CLI (script-header parameters) |
| `multiple_exec.sh` | Execute one selected script in each subdirectory (except template dir) | CLI (script-header parameters) |
| `visu_layering.py` | Single-file stratification diagnostics from `restartevo#.nc` | Interactive |
| `visu_layering_evo.py` | Multi-file stratification evolution over time + orbital diagnostics | Interactive |
| `fortran_code_stats.sh` | Fortran source metrics per file and per directory | CLI |
| `display_netcdf.yml` | Conda environment for visualization scripts | Configuration |

## Prerequisites

### Python environment

A ready-to-use Conda file is provided:

```bash
cd PEM/toolkit
conda env create -f display_netcdf.yml
conda activate display_netcdf
```

Core Python packages used by these tools include:
- netCDF4
- xarray
- numpy
- matplotlib
- cartopy
- pandas
- scipy
- vedo (3D display in `display_netcdf.py`)

### Shell tools

`fortran_code_stats.sh` uses standard Unix tools:
- bash
- find
- awk
- sed
- grep
- sort
- cut
- wc

### Extra system dependencies (startfi utilities)

The three `startfi.nc` editing scripts require:
- `ncap2` (NCO package)
- `bc`

Install example (Debian/Ubuntu):

```bash
sudo apt-get install nco bc
```

## Quick start

### 1) Analyse a NetCDF file

```bash
python analyse_netcdf.py path/to/file.nc
```

Interactive mode:

```bash
python analyse_netcdf.py
```

What it does:
- loops over all numeric variables,
- prints shape and dimensions,
- prints min/max/mean with NaN-safe operations,
- reports NaN and negative-value anomalies.

### 2) Concatenate PEM NetCDF outputs

```bash
python concat_pem.py --folder diags --basename diagevo --start 1 --end 10 --output merged.nc
```

Interactive mode:

```bash
python concat_pem.py
```

Notes:
- expected input naming is `{basename}<index>.nc` (example: `diagevo1.nc`),
- each file must expose a `Time` coordinate,
- output is written with cumulative `Time` values.

### 3) Visualize variables from NetCDF

Interactive mode (recommended with current script state):

```bash
python display_netcdf.py
```

CLI mode (available but less robust than interactive mode):

```bash
python display_netcdf.py path/to/file.nc --variable var_name --cmap turbo --output out.png
```

Supported plot situations in the current code:
- scalar values,
- 1D profiles/series,
- 2D slices,
- `physical_points` remapping to lat/lon map,
- optional polar and 3D display prompts in interactive sessions.

### 4) Inspect one restart stratification file

```bash
python visu_layering.py
```

Or provide file path directly:

```bash
python visu_layering.py starts/restartevo1.nc
```

The script asks for:
- grid point index,
- slope index,
- whether subsurface layers should be shown,
- stratum index for detailed composition.

Generated files:
- `layering_simple_profiles.png`
- `layering_overlaid_profiles.png`
- `layering_stacked_profiles.png`
- `stratum_<k>_composition.png`

### 5) Plot stratification evolution over many files

```bash
python visu_layering_evo.py
```

Default interactive inputs:
- folder: `starts`
- base NetCDF name: `restartevo`
- workflow info file: `pem_workflow.sts`
- orbital file: `obl_ecc_lsp.asc`

Generated files include:
- `layering_evolution_igX_isY.png`
- `layering_rgb_evolution_igX_isY.png`
- `dust_ice_obliquity_gridX_slopeY.png`
- `orbital_parameters_simu.png`
- `sin_ecc_times_Lsp.png`

### 6) Edit `startfi.nc` variables or orbital parameters

#### 6a) Set one variable in `startfi.nc`

Edit these parameters in `modify_startfi_var.sh`:

```bash
name_file="startfi.nc"
var_name="co2"
new_var=8.
```

Then run:

```bash
./modify_startfi_var.sh
```

#### 6b) Set fixed orbital values in `startfi.nc`

Edit these parameters in `modify_startfi_orbit.sh`:

```bash
name_file="startfi.nc"
new_obl=25.18941689
new_ecc=0.09340902
new_Lsp=251.06881714
new_iniLs=0.
```

Then run:

```bash
./modify_startfi_orbit.sh
```

This script updates orbital-related `controle(...)` entries in `startfi.nc` (obliquity, perihelion/aphelion distances, perihelion day, initial sol).

#### 6c) Set orbital values from Earth date in `run_pem.def`

Required files in working directory:
- `startfi.nc`
- `run_pem.def` with `pem_ini_earth_date=...`
- `obl_ecc_lsp.asc`

Then run:

```bash
./ini_pem_orbit.sh
```

The script reads `pem_ini_earth_date`, interpolates orbital parameters from `obl_ecc_lsp.asc`, and writes updated values into `startfi.nc`.

### 7) Run one script in multiple subdirectories

Edit in `multiple_exec.sh`:

```bash
script_name="pem_workflow.sh"
tempdir_name="Template"
```

Then run:

```bash
./multiple_exec.sh
```

It executes the selected script in each direct subdirectory where that script exists, skipping the template directory.

### 8) Compute Fortran code statistics

From the root of a Fortran source tree:

```bash
/path/to/PEM/toolkit/fortran_code_stats.sh
```

With CSV export:

```bash
/path/to/PEM/toolkit/fortran_code_stats.sh --csv
```

CSV outputs:
- `code_filestats.csv`
- `code_dirstats.csv`

## Data conventions expected by the scripts

### Required files for selected tools

- `startfi.nc`: required by `modify_startfi_var.sh`, `modify_startfi_orbit.sh`, `ini_pem_orbit.sh`
- `run_pem.def`: required by `ini_pem_orbit.sh`
- `obl_ecc_lsp.asc`: required by `ini_pem_orbit.sh`; used by `visu_layering_evo.py`
- `pem_workflow.sts`: default workflow status input for `visu_layering_evo.py`

### NetCDF dimensions and coordinates

The plotting and stratification scripts expect common PEM conventions such as:
- time-like dimensions: `Time`, `time`, or `time_counter`
- latitude-like dimensions: `latitude` or `lat`
- longitude-like dimensions: `longitude` or `lon`
- stratification dimensions: `nb_str_max`
- slope dimensions: `nslope`
- grid dimensions: `physical_points` (or aliases in some functions)

### Stratification variable families

The stratification tools look for variables like:
- `stratif_slope##_top_elevation` (or `stratif_top_elevation`)
- `stratif_slope##_h_co2ice` (or `stratif_h_co2ice`)
- `stratif_slope##_h_h2oice` (or `stratif_h_h2oice`)
- `stratif_slope##_h_dust` (or `stratif_h_dust`)
- `stratif_slope##_h_pore` (or `stratif_h_pore`)
- `stratif_slope##_poreice_coef1..4` (or `stratif_poreice_coef1..4`)

### Optional assets for map rendering

`display_netcdf.py` optionally uses local files in the working directory:
- `MOLA_1px_per_deg.npy` for topography contours
- `molaTeam_contour_31rgb_steps.csv` for some 3D color workflows

If these files are missing, map overlays/3D color options are reduced.

## Known limitations in current scripts

- `display_netcdf.py` currently has a mismatch between declared CLI parameters and the internal `plot_variable(...)` call signature. Interactive mode is the safest path.
- In `display_netcdf.py`, parser flags `--show-polar` and `--show-3d` are declared, but polar/3D behavior is mainly controlled by interactive prompts in plotting paths.
- `concat_pem.py` expects a `Time` coordinate and uses a simple cumulative offset strategy between files.
- `analyse_netcdf.py` reads full arrays in memory for each variable, which can be heavy on large datasets.
- In `visu_layering_evo.py`, an orbital ASCII filename is requested interactively, but the default plotting path currently uses orbital data read from NetCDF restart files.
- `multiple_exec.sh` traverses only direct subdirectories (one level).

## Related docs

- Main PEM documentation: `../README.md`
- Workflow templates/examples: `../deftank/`
- Workflow run prerequisites and required input files: `../deftank/README.md`
- Additional PEM docs: `../doc/README.md`