/
WP-M.1(Task 2: Report on performance indicators) - D-M.1.2 Part 2

WP-M.1(Task 2: Report on performance indicators) - D-M.1.2 Part 2

Summary:

This is a set of standardized metrics that inform on the performance in module M that can be used as a proxy for evaluating success in ITMS-M and thus, the overall success of the ITMS project. For ITMS-M, performance means how well the forward modelling, data assimilation, and inversion is working, hence benefiting the model quality and trust. Selected based on predefined requirements, we consolidated a set of quantifiable key performance indicators (KPIs) across all ITMS-M partners and designed the possible metrics for scoring each KPI. The current KPIs considered for assessment in Phase 1 include:

  1. The use of multiple GHG data streams used for inversion and diagnostics,

  2. Timeliness in the delivery of inversion products,

  3. Uncertainty reporting,

  4. Number of species and sectors treated,

  5. Model evaluation against external inversion systems,

  6. Analysis of different settings, that is, different locations, networks/data streams, tracers, and

  7. User interaction and links to other projects and stakeholders

The applicable scoring metrics are KPI specific and have been carefully considered with foreseen overlaps with ITMS-B and ITMS-Q&S, and thus feed into the performance-oriented traceability matrix being developed in module K to measure overall ITMS success against expected achievements.

KPIs assessment leading to improvement in model quality and trust will benefit the entire envisioned phases of the project. To further facilitate trust for our inversion result, a set of additional diagnostics indicators for the forward model is considered:

  1. Diagnostics of the model performance using atmospheric observations, and

  2. The use of Ensemble uncertainty w.r.t. a-priori dataset.

These indicators however require the establishment of a reference case for the performance rating for which the established translation of statistical results within the ITMS is only foreseen in the next years.

Additionally, a KPI that measures the:

  1. Spatial resolution of the inversion result is useful but finding scores that work across different inversion systems will be only possible after gaining information on sectoral resolution as previously highlighted.

For further information on the current analysis and plans for future development, see:

 

Go back to ITMS Module M or ITMS Evaluation Report_Home