Reporting

Warning

ixmp.reporting is experimental in ixmp 0.2, and only supports Python 3. The API and functionality may change without advance notice or a deprecation period in subsequent releases.

Top-level methods and classes:

configure([path]) Configure reporting globally.
Reporter(**kwargs) Class for generating reports on ixmp.Scenario objects.
Key(name[, dims, tag]) A hashable key for a quantity that includes its dimensionality.

Others:

reporting.configure(**config)

Configure reporting globally.

Modifies global variables that affect the behaviour of all Reporters and computations, namely RENAME_DIMS, REPLACE_UNITS, and UNITS.

Valid configuration keys—passed as config keyword arguments—include:

Other Parameters:
 
  • units (mapping) – Configuration for handling of units. Valid sub-keys include:
    • replace (mapping of str -> str): replace units before they are parsed by pint. Added to REPLACE_UNITS.
    • define (str): block of unit definitions, added to UNITS so that units are recognized by pint. See the pint documentation.
  • rename_dims (mapping of str -> str) – Update RENAME_DIMS.
Warns:

UserWarning – If config contains unrecognized keys.
class ixmp.reporting.Reporter(**kwargs)

Class for generating reports on ixmp.Scenario objects.

A Reporter is used to postprocess data from from one or more ixmp.Scenario objects. The get() method can be used to:

  • Retrieve individual quantities. A quantity has zero or more dimensions and optional units. Quantities include the ‘parameters’, ‘variables’, ‘equations’, and ‘scalars’ available in an ixmp.Scenario.

  • Generate an entire report composed of multiple quantities. A report may:

    • Read in non-model or exogenous data,
    • Trigger output to files(s) or a database, or
    • Execute user-defined methods.

Every report and quantity (including the results of intermediate steps) is identified by a utils.Key; all the keys in a Reporter can be listed with keys().

Reporter uses a graph data structure to keep track of computations, the atomic steps in postprocessing: for example, a single calculation that multiplies two quantities to create a third. The graph allows get() to perform only the requested computations. Advanced users may manipulate the graph directly; but common reporting tasks can be handled by using Reporter methods:

add(key, computation[, strict, index, sums]) Add computation to the Reporter under key.
add_file(path[, key]) Add exogenous quantities from path.
aggregate(qty, tag, dims_or_groups[, …]) Add a computation that aggregates qty.
apply(generator, *keys) Add computations from generator applied to key.
configure([path]) Configure the Reporter.
describe([key]) Return a string describing the computations that produce key.
disaggregate(qty, new_dim[, method, args]) Add a computation that disaggregates var using method.
finalize(scenario) Prepare the Reporter to act on scenario.
full_key(name_or_key) Return the full-dimensionality key for name_or_key.
get([key]) Execute and return the result of the computation key.
read_config(path) Configure the Reporter with information from a YAML file at path.
visualize(filename, **kwargs) Generate an image describing the reporting structure.
write(key, path) Write the report key to the file path.
graph = {'filters': None}

A dask-format graph.

add(key, computation, strict=False, index=False, sums=False)

Add computation to the Reporter under key.

Parameters:
  • key (hashable) – A string, Key, or other value identifying the output of task.
  • computation (object) –

    One of:

    1. any existing key in the Reporter.
    2. any other literal value or constant.
    3. a task, i.e. a tuple with a callable followed by one or more computations.
    4. A list containing one or more of #1, #2, and/or #3.
  • strict (bool, optional) – If True, key must not already exist in the Reporter, and any keys referred to by computation must exist.
  • index (bool, optional) – If True, key is added to the index as a full-resolution key, so it can be later retrieved with full_key().
  • sums (bool, optional) – If True, all partial sums of key are also added to the Reporter.
Raises:

KeyError – If key is already in the Reporter; any key referred to by computation does not exist; or sums=True and the key for one of the partial sums of key is already in the Reporter.

add() may be used to:

  • Provide an alias from one key to another:

    >>> r.add('aliased name', 'original name')

  • Define an arbitrarily complex computation in a Python function that operates directly on the ixmp.Scenario:

    >>> def my_report(scenario):
>>>     # many lines of code
>>>     return 'foo'
>>> r.add('my report', (my_report, 'scenario'))
>>> r.finalize(scenario)
>>> r.get('my report')
foo

Note

Use care when adding literal str values (2); these may conflict with keys that identify the results of other computations.

add_file(path, key=None)

Add exogenous quantities from path.

A file at a path like ‘/path/to/foo.ext’ is added at the key 'file:foo.ext'.

See also

ixmp.reporting.computations.load_file()

add_product(name, *quantities, sums=True)

Add a computation that takes the product of quantities.

Parameters:
  • name (str) – Name of the new quantity.
  • sums (bool, optional) – If True, all partial sums of the new quantity are also added.
Returns:

The full key of the new quantity.
Return type:

Key
aggregate(qty, tag, dims_or_groups, weights=None, keep=True)

Add a computation that aggregates qty.

Parameters:
  • qty (Key or str) – Key of the quantity to be disaggregated.
  • tag (str) – Additional string to add to the end the key for the aggregated quantity.
  • dims_or_groups (str or iterable of str or dict) – Name(s) of the dimension(s) to sum over, or nested dict.
  • weights (xr.DataArray) – Weights for weighted aggregation.
Returns:

The key of the newly-added node.
Return type:

Key
apply(generator, *keys)

Add computations from generator applied to key.

Parameters:
  • generator (callable) – Function to apply to keys. Must yield a sequence (0 or more) of (key, computation), which are added to the graph.
  • keys (hashable) – The starting key(s)
check_keys(*keys)

Check that keys are in the Reporter.

If any of keys is not in the Reporter, KeyError is raised. Otherwise, a list is returned with either the key from keys, or the corresponding full_key().

configure(path=None, **config)

Configure the Reporter.

Accepts a path to a configuration file and/or keyword arguments. Configuration keys loaded from file are replaced by keyword arguments.

Valid configuration keys include:

  • default: the default reporting key; sets default_key.
  • filters: a dict, passed to set_filters().
  • files: a dict mapping keys to file paths.
  • alias: a dict mapping aliases to original keys.
Warns:UserWarning – If config contains unrecognized keys.
default_key = None

The default reporting key.

describe(key=None)

Return a string describing the computations that produce key.

If key is not provided, all keys in the Reporter are described.

disaggregate(qty, new_dim, method='shares', args=[])

Add a computation that disaggregates var using method.

Parameters:
  • var (hashable) – Key of the variable to be disaggregated.
  • new_dim (str) – Name of the new dimension of the disaggregated variable.
  • method (callable or str) – Disaggregation method. If a callable, then it is applied to var with any extra args. If then a method named ‘disaggregate_{method}’ is used.
  • args (list, optional) – Additional arguments to the method. The first element should be the key for a quantity giving shares for disaggregation.
Returns:

The key of the newly-added node.
Return type:

Key
finalize(scenario)

Prepare the Reporter to act on scenario.

The Scenario object scenario is associated with the key 'scenario'. All subsequent processing will act on data from this scenario.

classmethod from_scenario(scenario, **kwargs)

Create a Reporter by introspecting scenario.

Parameters:
  • scenario (ixmp.Scenario) – Scenario to introspect in creating the Reporter.
  • kwargs (optional) – Passed to Scenario.configure().
Returns:

A Reporter instance containing:

  • A ‘scenario’ key referring to the scenario object.
  • Each parameter, equation, and variable in the scenario.
  • All possible aggregations across different sets of dimensions.
  • Each set in the scenario.
Return type:

Reporter
full_key(name_or_key)

Return the full-dimensionality key for name_or_key.

An ixmp variable ‘foo’ with dimensions (a, c, n, q, x) is available in the Reporter as 'foo:a-c-n-q-x'. This Key can be retrieved with:

rep.full_key('foo')
rep.full_key('foo:c')
# etc.
get(key=None)

Execute and return the result of the computation key.

Only key and its dependencies are computed.

Parameters:key (str, optional) – If not provided, default_key is used.
Raises:ValueError – If key and default_key are both None.
read_config(path)

Configure the Reporter with information from a YAML file at path.

See configure().

set_filters(**filters)

Apply filters ex ante (before computations occur).

filters has the same form as the argument of the same name to ixmp.Scenario.par() and analogous methods. A value of None will clear the filter for the named dimension.

visualize(filename, **kwargs)

Generate an image describing the reporting structure.

This is a shorthand for dask.visualize(). Requires graphviz.

write(key, path)

Write the report key to the file path.

class ixmp.reporting.Key(name, dims=[], tag=None)

A hashable key for a quantity that includes its dimensionality.

Quantities in a Scenario can be indexed by one or more dimensions. For example, a parameter with three dimensions can be initialized with:

>>> scenario.init_par('foo', ['a', 'b', 'c'], ['apple', 'bird', 'car'])

Computations for this scenario might use the quantity foo in different ways:

  1. in its full resolution, i.e. indexed by a, b, and c;
  2. aggregated (e.g. summed) over any one dimension, e.g. aggregated over c and thus indexed by a and b;
  3. aggregated over any two dimensions; etc.

A Key for (1) will hash, display, and evaluate as equal to 'foo:a-b-c'. A Key for (2) corresponds to 'foo:a-b', and so forth.

Keys may be generated concisely by defining a convenience method:

>>> def foo(dims):
>>>     return Key('foo', dims.split(''))
>>> foo('a b')
foo:a-b
add_tag(tag)

Return a new Key with tag appended.

append(*dims)

Return a new Key with additional dimensions dims.

dims

Dimensions of the quantity, tuple of str.

drop(*dims)

Return a new Key with dims dropped.

classmethod from_str_or_key(value, drop=[], append=[], tag=None)

Return a new Key from value.

Parameters:
  • value (str or Key) – Value to use to generate a new Key.
  • drop (list of str, optional) – Existing dimensions of value to drop. See drop().
  • append (list of str, optional.) – New dimensions to append to the returned Key. See append().
  • tag (str, optional) – Tag for returned Key. If value has a tag, the two are joined using a ‘+’ character. See add_tag().
Returns:
Return type:

Key
iter_sums()

Generate (key, task) for all possible partial sums of the Key.

name

Name of the quantity, str.

classmethod product(new_name, *keys)

Return a new Key that has the union of dimensions on keys.

Dimensions are ordered by their first appearance:

  1. First, the dimensions of the first of the keys.
  2. Next, any additional dimensions in the second of the keys that were not already added in step 1.
  3. etc.
Parameters:new_name (str) – Name for the new Key. The names of keys are discarded.
tag

Quantity tag, str.

Computations

Elementary computations for reporting.

Unless otherwise specified, these methods accept and return Quantity objects for data arguments/return values.

Calculations:

aggregate(quantity, groups, keep) Aggregate quantity by groups.
disaggregate_shares(quantity, shares) Disaggregate quantity by shares.
product(*quantities[, drop]) Return the product of any number of quantities.
ratio(numerator, denominator[, drop]) Return the ratio numerator / denominator.
sum(quantity[, weights, dimensions]) Sum quantity over dimensions, with optional weights.

Input and output:

load_file(path) Read the file at path and return its contents.
write_report(quantity, path) Write a quantity to a file.

Conversion:

make_dataframe(*quantities) Concatenate quantities into a single pd.DataFrame.
ixmp.reporting.computations.aggregate(quantity, groups, keep)

Aggregate quantity by groups.

Parameters:
  • quantity (Quantity) –
  • groups (dict of dict) – Top-level keys are the names of dimensions in quantity. Second-level keys are group names; second-level values are lists of labels along the dimension to sum into a group.
  • keep (bool) – If True, the members that are aggregated into a group are returned with the group sums. If False, they are discarded.
Returns:

Same dimensionality as quantity.
Return type:

Quantity
ixmp.reporting.computations.disaggregate_shares(quantity, shares)

Disaggregate quantity by shares.

ixmp.reporting.computations.make_dataframe(*quantities)

Concatenate quantities into a single pd.DataFrame.

ixmp.reporting.computations.load_file(path)

Read the file at path and return its contents.

Some file formats are automatically converted into objects for direct use in reporting code:

  • csv: converted to xarray.DataArray. CSV files must have a ‘value’ column; all others are treated as indices.
ixmp.reporting.computations.sum(quantity, weights=None, dimensions=None)

Sum quantity over dimensions, with optional weights.

ixmp.reporting.computations.write_report(quantity, path)

Write a quantity to a file.

Parameters:path (str or Path) – Path to the file to be written.

Utilities

class ixmp.reporting.attrseries.AttrSeries(*args, **kwargs)

pandas.Series subclass imitating xarray.DataArray.

Future versions of ixmp.reporting will use xarray.DataArray as Quantity; however, because xarray currently lacks sparse matrix support, ixmp quantities may be too large for available memory.

The AttrSeries class provides similar methods and behaviour to xarray.DataArray, such as an attrs dictionary for metadata, so that ixmp.reporting.computations methods can use xarray-like syntax.

ixmp.reporting.utils.RENAME_DIMS = {}

Dimensions to rename when extracting raw data from Scenario objects. Mapping from Scenario dimension name -> preferred dimension name.

ixmp.reporting.utils.REPLACE_UNITS = {'%': 'percent'}

Replacements to apply to quantity units before parsing by pint. Mapping from original unit -> preferred unit.

ixmp.reporting.utils.UNITS = <pint.registry.UnitRegistry object>

pint unit registry for processing quantity units. All units handled by imxp.reporting must be either standard SI units, or added to this registry.

ixmp.reporting.utils.clean_units(input_string)

Tolerate messy strings for units.

Handles two specific cases found in MESSAGEix test cases:

  • Dimensions enclosed in ‘[]’ have these characters stripped.
  • The ‘%’ symbol cannot be supported by pint, because it is a Python operator; it is translated to ‘percent’.
ixmp.reporting.utils.collect_units(*args)

Return an list of ‘_unit’ attributes for args.

ixmp.reporting.utils.data_for_quantity(ix_type, name, column, scenario, filters=None)

Retrieve data from scenario.

Parameters:
  • ix_type ('equ' or 'par' or 'var') – Type of the ixmp object.
  • name (str) – Name of the ixmp object.
  • column ('mrg' or 'lvl' or 'value') – Data to retrieve. ‘mrg’ and ‘lvl’ are valid only for ix_type='equ', and ‘level’ otherwise.
  • scenario (ixmp.Scenario) – Scenario containing data to be retrieved.
  • filters (dict, optional) – Mapping from dimensions to iterables of allowed values along each dimension.
Returns:

Data for name.
Return type:

Quantity
ixmp.reporting.utils.dims_for_qty(data)

Return the list of dimensions for data.

If data is a pandas.DataFrame, its columns are processed; otherwise it must be a list.

ixmp.reporting.RENAME_DIMS is used to rename dimensions.

ixmp.reporting.utils.keys_for_quantity(ix_type, name, scenario)

Iterate over keys for name in scenario.