.. _config:

.. currentmodule:: nomenclature

General project configuration
=============================

The following features can be accessed via a general project configuration file:

* Import codelists and mappings from public GitHub repositories
* Add all countries to the region codelist (details: :ref:`countries`)

Configuration file format
-------------------------

The configuration file **must be** located on the same level as the *mappings* and
*definitions* directories.

The file **must be** in yaml format and named *nomenclature.yaml*.

Importing from an external repository
-------------------------------------

In order to import from an external repository, the configuration file must define the
repository and the repositories key.

The repository has a **name** (in the example below *common-definitions*) and a **url**. Multiple repositories can be used in a single configuration file:

.. code:: yaml

  repositories:
    common-definitions:
      url: https://github.com/IAMconsortium/common-definitions.git/
    legacy-definitions:
      url: https://github.com/IAMconsortium/legacy-definitions.git/

In order for the import to work the url must be given in the above format, i.e. with the
leading *https://* and the trailing *.git/*.

Information from an external repository can either be used for codelists ("definitions")
or model mappings, or both. For each definition dimension, i.e. *region* or *variable*
multiple external repositories can be used as the example below illustrates for
*variable*:

.. code:: yaml

  repositories:
    common-definitions:
      url: https://github.com/IAMconsortium/common-definitions.git/
    legacy-definitions:
      url: https://github.com/IAMconsortium/legacy-definitions.git/
  definitions:
    region:
      repository: common-definitions
    variable:
      repositories:
        - common-definitions
        - legacy-definitions
  mappings:
    repository: common-definitions

The value in *definitions.region.repository* can be a list or a single value.

For model mappings the process is analogous using *mappings.repository*.

Filter code lists imported from external repositories
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Since importing the entirety of, for example, common-definitions is too much for most
projects, the list can be filtered using ``include`` and ``exclude`` keywords. Under
these keywords, lists of filters can be given that will be applied to the code list from
the given repository.

The filtering can be done by any attribute:

.. code:: yaml

  repositories:
    common-definitions:
      url: https://github.com/IAMconsortium/common-definitions.git/
  definitions:
    variable:
      repository:
        name: common-definitions
        include:
          - name: [Primary Energy*, Final Energy*]
          - name: "Population*"
            tier: 1
        exclude:
          - name: "Final Energy|Industry*"
            depth: 2

If a filter is being used for repositories, the *name* attribute **must be used**
for the repository.

In the example above we are including:
1. All variables starting with *Primary Energy* or *Final Energy*
2. All variables starting with *Population* **and** with the tier attribute equal to 1

From this list we are then **excluding** all variables that match "Final
Energy|Industry\*" and have a depth of 2 (meaning that they contain two pipe "|"
characters).

Adding countries to the region codelist
---------------------------------------

By setting *definitions.region.country* as *true* in the configuration file:

.. code:: yaml

  definitions:
    region:
      country: true

the nomenclature package will add all countries to the *region* codelist.

More details on the list of countries can be found here: :ref:`countries`.

.. _adding-countries:

Adding NUTS to the region codelist
----------------------------------

By setting *definitions.region.nuts* (optional) in the configuration file:

.. code:: yaml

  definitions:
    region:
      nuts:
        nuts-1: [ AT, BE, CZ ]
        nuts-2: [ AT ]
        nuts-3: true

the nomenclature package will add the selected NUTS regions to the *region* codelist.

In the example above, the package will add: NUTS 1 regions for Austria, Belgium
and Czechia, NUTS 2 regions for Austria, NUTS 3 regions for all EU countries.

More details on the list of NUTS regions can be found here: :ref:`nuts`.

Specify dimensions to be read
-----------------------------

The configuration file offers the possibility to set the dimensions which will be read
by *DataStructureDefinition*, overriding the dimensions from the *definitions*
sub-directories. If no sub-directories exist (e.g.: when importing dimensions
from external repositories), setting dimensions in the configuration file is mandatory.

In the below case we specify *region*, *variable* and *scenario* to be read and used for
validation:

.. code:: yaml

  dimensions:
    - region
    - variable
    - scenario


Filter model mappings from external repositories
------------------------------------------------

We often only want to use a subset of models in a particular project (and not import all mappings), so
there is an option to filter for specific model mappings. This works very similarly to
the filtering for definitions.

.. code:: yaml

  repositories:
  common-definitions:
    url: https://github.com/IAMconsortium/common-definitions.git/
  mappings:
    repository:
      name: common-definitions
      include:
        - MESSAGEix-GLOBIOM 2.1-M-R12

The above example retrieves only the model mapping for *MESSAGEix-GLOBIOM 2.1-M-R12*
from the common-definitions repository.

Configuring processors
----------------------

The ``processors`` section of *nomenclature.yaml* allows processors to be declared
directly in the configuration file, so they are applied automatically when calling
:func:`process` without passing an explicit ``processor`` argument.

Region processor
^^^^^^^^^^^^^^^^

Setting *processors.region-processor* as *true* will automatically create a
:class:`RegionProcessor` from the project's default ``mappings/`` directory:

.. code:: yaml

  processors:
    region-processor: true

This is equivalent to calling:

.. code:: python

  import pyam
  from nomenclature.processor import RegionProcessor
  
  df = pyam.IamDataFrame(data="path/to/file.csv")
  dsd = DataStructureDefinition("definitions")
  processor = RegionProcessor.from_directory("mappings", dsd)
  aggregated_data = process(df, dsd)

NUTS processor
^^^^^^^^^^^^^^

Setting *processors.nuts* to a list of model names will automatically create a
:class:`NutsProcessor` and apply NUTS hierarchical aggregation (NUTS3 → NUTS2 →
NUTS1 → Country → EU27) for those models:

.. code:: yaml

  processors:
    nuts-processor: [ Model A, Model B ]

More details on NUTS aggregation can be found here: :ref:`nuts`.

Country processor
^^^^^^^^^^^^^^^^^

Setting *processors.country-processor* to a list of model names will
automatically create a :class:`CountryProcessor` and aggregate country-level
data to common regional definitions such as R5, R9 and R10 for those models:

.. code:: yaml

  repositories:
    common-definitions:
      url: https://github.com/IAMconsortium/common-definitions.git/

  definitions:
    region:
      country: true
      repository:
        name: common-definitions
        include:
          - hierarchy: [R5, R9, R10]

  processors:
    country-processor: [ Model A, Model B ]

The country processor reads the imported R5/R9/R10 definitions from the region
codelist and aggregates directly from countries to those regions. If one of
those hierarchy levels is not present in the codelist, it is skipped.

More details on the processor API can be found here: :doc:`../api/countryprocessor`.