Storage back ends (ixmp.backend)

By default, the ix modeling platform is installed with ixmp.backend.jdbc.JDBCBackend, which can store data in many types of relational database management systems (RDBMS) that have Java DataBase Connector (JDBC) interfaces—hence its name.

However, ix modeling platform is extensible to support other methods of storing data: in non-JDBC RDBMS, non-relational databases, local files, memory, or other ways. Developers wishing to add such capabilities may subclass ixmp.backend.base.Backend and implement its methods.

Provided backends

ixmp.backend.BACKENDS = {'jdbc': <class 'ixmp.backend.jdbc.JDBCBackend'>}

Mapping from names to available backends. To register additional backends, add elements to this variable.

class ixmp.backend.jdbc.JDBCBackend(jvmargs=None, **kwargs)

Backend using JPype/JDBC to connect to Oracle and HyperSQLDB instances.

Parameters:
  • dbtype ('HSQLDB', optional) – Database type to use. If None, a remote database is accessed. If ‘HSQLDB’, a local database is created and used at the path given by dbprops.
  • dbprops (path-like, optional) –

    If dbtype is None, the name of a database properties file (default: default.properties) in the properties file directory (see Config) or a path to a properties file.

    If dbtype is ‘HSQLDB’, the path of a local database, (default: ``$HOME/.local/ixmp/localdb/default`) or name of a database file in the local database directory (default: $HOME/.local/ixmp/localdb/).

  • jvmargs (str, optional) – Java Virtual Machine arguments. See start_jvm().

JDBCBackend supports:

  • dbtype='HSQLDB': HyperSQL databases in local files.

  • Remote databases. This is accomplished by creating a ixmp.Platform with the dbprops argument pointing a file that specifies JDBC information. For instance:

    jdbc.driver = oracle.jdbc.driver.OracleDriver
jdbc.url = jdbc:oracle:thin:@database-server.example.com:1234:SCHEMA
jdbc.user = USER
jdbc.pwd = PASSWORD

It has the following methods that are not part of the overall Backend API:

read_gdx Read the Scenario from a GDX file at path.
write_gdx Write the Scenario to a GDX file at path.
read_gdx(s, path, check_solution, comment, equ_list, var_list)

Read the Scenario from a GDX file at path.

Parameters:
  • check_solution (bool) – If True, raise an exception if the GAMS solver did not reach optimality. (Only for MESSAGE-scheme Scenarios.)
  • comment (str) – Comment added to Scenario when importing the solution.
  • equ_list (list of str) – Equations to be imported.
  • var_list (list of str) – Variables to be imported.
write_gdx(s, path)

Write the Scenario to a GDX file at path.

jdbc.start_jvm()

Start the Java Virtual Machine via JPype.

Parameters:jvmargs (str or list of str, optional) –

Additional arguments for launching the JVM, passed to jpype.startJVM().

For instance, to set the maximum heap space to 4 GiB, give jvmargs=['-Xmx4G']. See the JVM documentation for a list of options.

Backend API

  • ixmp.Platform implements a user-friendly API for scientific programming. This means its methods can take many types of arguments, check, and transform them—in a way that provides modeler-users with easy, intuitive workflows.
  • In contrast, Backend has a very simple API that accepts arguments and returns values in basic Python data types and structures.
  • As a result:
    • Platform code is not affected by where and how data is stored; it merely handles user arguments and then makes, usually, a single Backend call.
    • Backend code does not need to perform argument checking; merely store and retrieve data reliably.
ixmp.backend.FIELDS = {'get_nodes': ('region', 'mapped_to', 'parent', 'hierarchy'), 'get_scenarios': ('model', 'scenario', 'scheme', 'is_default', 'is_locked', 'cre_user', 'cre_date', 'upd_user', 'upd_date', 'lock_user', 'lock_date', 'annotation', 'version'), 'ts_get': ('region', 'variable', 'unit', 'year', 'value'), 'ts_get_geo': ('region', 'variable', 'time', 'year', 'value', 'unit', 'meta')}

Lists of field names for tuples returned by Backend API methods.

class ixmp.backend.base.Backend

Abstract base class for backends.

In the following, the bold-face words required, optional, etc. have specific meanings as described in IETF RFC 2119.

Backend is an abstract class; this means it must be subclassed. Most of its methods are decorated with abc.abstractmethod(); this means they are required and must be overridden by subclasses.

Others, marked below with “OPTIONAL:”, are not so decorated. For these methods, the behaviour in the base Backend—often, nothing—is an acceptable default behaviour. Subclasses may extend or replace this behaviour as desired, so long as the methods still perform the actions described in the description.

Backends:

  • must only raise standard Python exceptions.

Methods related to ixmp.Platform:

close_db OPTIONAL: Close database connection(s).
get_auth OPTIONAL: Return user authorization for models.
get_nodes Iterate over all nodes stored on the Platform.
get_scenarios Iterate over TimeSeries stored on the Platform.
get_units Return all registered symbols for units of measurement.
open_db OPTIONAL: (Re-)open database connection(s).
set_log_level OPTIONAL: Set logging level for the backend and other code.
set_node Add a node name to the Platform.
set_unit Add a unit of measurement to the Platform.

Methods related to ixmp.TimeSeries:

  • Each method has an argument ts, a reference to the TimeSeries object being manipulated.
  • ‘Geodata’ is otherwise identical to regular timeseries data, except value are str rather than float.
check_out Check out ts for modification.
commit Commit changes to ts.
delete Remove data values.
delete_geo Remove ‘geodata’ values.
discard_changes Discard changes to ts since the last ts_check_out().
get Retrieve the existing TimeSeries or Scenario ts.
get_data Retrieve time-series data.
get_geo Retrieve time-series ‘geodata’.
init_ts Initialize the TimeSeries ts.
is_default Return True if ts is the default version.
last_update Return the date of the last modification of the ts.
preload OPTIONAL: Load ts data into memory.
run_id Return the run ID for the ts.
set_data Store data.
set_as_default Set the current TimeSeries.version as the default.
set_geo Store time-series ‘geodata’.

Methods related to ixmp.Scenario:

  • Each method has an argument s, a reference to the Scenario object being manipulated.
clone Clone s.
delete_item Remove an item name of type.
get_meta Return all metadata.
has_solution Return True if Scenario s has been solved.
init_s Initialize the Scenario s.
init_item Initialize an item name of type.
item_delete_elements Remove elements of item name.
item_get_elements Return elements of item name.
item_set_elements Add keys or values to item name.
item_index Return the index sets or names of item name.
list_items Return a list of items of type.
set_meta Set a single metadata key.

Methods related to message_ix.Scenario:

  • Each method has an argument ms, a reference to the Scenario object being manipulated.

Warning

These methods may be moved to ixmp in a future release.

cat_get_elements Get elements of a category mapping.
cat_list Return list of categories in mapping name.
cat_set_elements Add elements to category mapping.
cat_get_elements(ms: ixmp.core.Scenario, name, cat)

Get elements of a category mapping.

Parameters:
  • name (str) – Name of the category mapping set.
  • cat (str) – Name of the category within name.
Returns:

All set elements mapped to cat in name.
Return type:

list of str
cat_list(ms: ixmp.core.Scenario, name)

Return list of categories in mapping name.

Parameters:name (str) – Name of the category mapping set.
Returns:All categories in name.
Return type:list of str
cat_set_elements(ms: ixmp.core.Scenario, name, cat, keys, is_unique)

Add elements to category mapping.

Parameters:
  • name (str) – Name of the category mapping set.
  • cat (str) – Name of the category within name.
  • keys (iterable of str or list of str) – Keys to add to cat.
  • is_unique (bool) –

    If True:

    • keys must contain only one key.
    • The Backend must remove any existing member of cat, so that it has only one element.
Returns:
Return type:

None
check_out(ts: ixmp.core.TimeSeries, timeseries_only)

Check out ts for modification.

Parameters:timeseries_only (bool) –

???
Returns:
Return type:None
clear_solution(s: ixmp.core.Scenario, from_year=None)

Remove data associated with a model solution.

Todo

Document.

clone(s: ixmp.core.Scenario, platform_dest, model, scenario, annotation, keep_solution, first_model_year=None)

Clone s.

Parameters:
  • platform_dest (Platform) – Target backend. May be the same as s.platform.
  • model (str) – New model name.
  • scenario (str) – New scenario name.
  • annotation (str) – Description for the creation of the new scenario.
  • keep_solution (bool) – If True, model solution data is also cloned. If False, it is discarded.
  • first_model_year (int or None) – If int, must be greater than the first model year of s.
Returns:

The cloned Scenario.
Return type:

Same class as s
close_db()

OPTIONAL: Close database connection(s).

Close any database connection(s), if open.

commit(ts: ixmp.core.TimeSeries, comment)

Commit changes to ts.

ts_init may modify the version attribute of ts.

Parameters:comment (str) – Description of the changes being committed.
Returns:
Return type:None
delete(ts: ixmp.core.TimeSeries, region, variable, years, unit)

Remove data values.

Parameters:
  • region (str) – Region name.
  • variable (str) – Variable name.
  • years (Iterable of int) – Years.
  • unit (str) – Unit symbol.
Returns:
Return type:

None
delete_geo(ts: ixmp.core.TimeSeries, region, variable, time, years, unit)

Remove ‘geodata’ values.

Parameters:
  • region (str) – Region name.
  • variable (str) – Variable name.
  • time (str) – Time period.
  • years (Iterable of int) – Years.
  • unit (str) – Unit symbol.
Returns:
Return type:

None
delete_item(s: ixmp.core.Scenario, type, name)

Remove an item name of type.

Parameters:
  • type ('set' or 'par' or 'equ') –
  • name (str) – Name of the item to delete.
Returns:
Return type:

None
discard_changes(ts: ixmp.core.TimeSeries)

Discard changes to ts since the last ts_check_out().

Returns:
Return type:None
get(ts: ixmp.core.TimeSeries, version)

Retrieve the existing TimeSeries or Scenario ts.

The TimeSeries is identified based on its (model, scenario) and version.

If ts is a Scenario, get() must set the scheme attribute on it.

Parameters:version (int or None) – If None, the version marked as the default is returned, and ts_get must set TimeSeries.version attribute on ts.
Returns:
Return type:None

See also

ts_set_as_default()

get_auth(user, models, kind)

OPTIONAL: Return user authorization for models.

If the Backend implements access control, this method must indicate whether user has permission kind for each of models.

kind may be ‘read’/’view’, ‘write’/’modify’, or other values; get_auth() should raise exceptions on invalid values.

Parameters:
  • user (str) – User name or identifier.
  • models (list of str) – Model names.
  • kind (str) – Type of permission being requested
Returns:

Mapping of model name (str) → bool; True if the user is authorized for the model.
Return type:

dict
get_data(ts: ixmp.core.TimeSeries, region, variable, unit, year)

Retrieve time-series data.

Parameters:
  • region (list of str) – Region names to filter results.
  • variable (list of str) – Variable names to filter results.
  • unit (list of str) – Unit symbols to filter results.
  • year (list of str) – Years to filter results.
Yields:

tuple – The members of each tuple are:

ID Type Description
region str Region name
variable str Variable name
unit str Unit symbol
year int Year
value float Data value
get_geo(ts: ixmp.core.TimeSeries)

Retrieve time-series ‘geodata’.

Yields:

tuple – The members of each tuple are:

ID Type Description
region str Region name
variable str Variable name
time str Time period
year int Year
value str Value
unit str Unit symbol
meta bool True if the data is marked as metadata
get_meta(s: ixmp.core.Scenario)

Return all metadata.

Returns:Mapping from metadata keys to values.
Return type:dict (str -> any)

See also

s_get_meta()

get_nodes()

Iterate over all nodes stored on the Platform.

Yields:

tuple – The members of each tuple are:

ID Type Description
region str Node name or synonym for node
mapped_to str or None Node name
parent str Parent node name
hierarchy str Node hierarchy ID

See also

set_node()

get_scenarios(default, model, scenario)

Iterate over TimeSeries stored on the Platform.

Scenarios, as subclasses of TimeSeries, are also included.

Parameters:
  • default (bool) – True to include only TimeSeries versions marked as default.
  • model (str or None) – Model name to filter results.
  • scenario (str or None) – Scenario name to filter results.
Yields:

tuple – The members of each tuple are:

ID Type Description
model str Model name
scenario str Scenario name
scheme str Scheme name
is_default bool True if version is the default
is_locked bool True if read-only
cre_user str Name of user who created the TimeSeries
cre_date str Creation datetime
upd_user str Name of user who last modified the TimeSeries
upd_date str Modification datetime
lock_user str Name of user who locked the TimeSeries
lock_date str Lock datetime
annotation str Description of the TimeSeries
version int Version
get_units()

Return all registered symbols for units of measurement.

Returns:
Return type:list of str

See also

set_unit()

has_solution(s: ixmp.core.Scenario)

Return True if Scenario s has been solved.

If True, model solution data is available from the Backend.

init_item(s: ixmp.core.Scenario, type, name)

Initialize an item name of type.

Parameters:
  • type ('set' or 'par' or 'equ') –
  • name (str) – Name for the new item.
Returns:
Return type:

None
init_s(s: ixmp.core.Scenario, scheme, annotation)

Initialize the Scenario s.

s_init may modify the version attribute of s.

Parameters:
  • scheme (str) – Description of the scheme of the Scenario.
  • annotation (str) – Description of the Scenario.
Returns:
Return type:

None
init_ts(ts: ixmp.core.TimeSeries, annotation=None)

Initialize the TimeSeries ts.

ts_init may modify the version attribute of ts.

Parameters:annotation (str) – If ts is newly-created, the Backend must store this annotation with the TimeSeries.
Returns:
Return type:None
is_default(ts: ixmp.core.TimeSeries)

Return True if ts is the default version.

Returns:
Return type:bool

See also

ts_set_as_default(), ts_get(), s_get()

item_delete_elements(s: ixmp.core.Scenario, type, name, keys)

Remove elements of item name.

Parameters:
  • type ('par' or 'set') –
  • name (str) –
  • keys (iterable of iterable of str) – If name is indexed by other set(s), then the number of elements of each key in keys, and their contents, must match the index set(s). If name is a basic set, then each key must be a list containing a single str, which must exist in the set.

See also

s_init_item(), s_item_set_elements()

item_get_elements(s: ixmp.core.Scenario, type, name, filters=None)

Return elements of item name.

Parameters:
  • type ('equ' or 'par' or 'set' or 'var') –
  • name (str) – Name of the item.
  • filters (dict (str -> list of str), optional) – If provided, a mapping from dimension names to allowed values along that dimension.
Returns:

  • pandas.Series – When type is ‘set’ and name an index set (not indexed by other sets).
  • dict – When type is ‘equ’, ‘par’, or ‘set’ and name is scalar (zero- dimensional). The value has the keys ‘value’ and ‘unit’ (for ‘par’) or ‘lvl’ and ‘mrg’ (for ‘equ’ or ‘var’).
  • pandas.DataFrame – For mapping sets, or all 1+-dimensional values. The dataframe has one column per index name with dimension values; plus the columns ‘value’ and ‘unit’ (for ‘par’) or ‘lvl’ and ‘mrg’ (for ‘equ’ or ‘var’).
item_index(s: ixmp.core.Scenario, name, sets_or_names)

Return the index sets or names of item name.

Parameters:sets_or_names ('sets' or 'names') –
Returns:
Return type:list of str
item_set_elements(s: ixmp.core.Scenario, type, name, elements)

Add keys or values to item name.

Parameters:
  • type ('par' or 'set') –
  • name (str) – Name of the items.
  • elements (iterable of 4-tuple) –

    The members of each tuple are:

    ID Type Description
    key str or list of str or None Set elements or value indices
    value float or None Parameter value
    unit str or None Unit symbol
    comment str or None Description of the change

    If name is indexed by other set(s), then the number of elements of each key, and their contents, must match the index set(s). When type is ‘set’, value and unit must be None.
Raises:
  • ValueError – If elements contain invalid values, e.g. key values not in the index set(s).
  • Exception – If the Backend encounters any error adding the elements.

See also

s_init_item(), s_item_delete_elements()

last_update(ts: ixmp.core.TimeSeries)

Return the date of the last modification of the ts.

Returns:
Return type:str
list_items(s: ixmp.core.Scenario, type)

Return a list of items of type.

Parameters:type ('set' or 'par' or 'equ') –
Returns:
Return type:list of str
open_db()

OPTIONAL: (Re-)open database connection(s).

A backend may connect to a database server. This method opens the database connection if it is closed.

preload(ts: ixmp.core.TimeSeries)

OPTIONAL: Load ts data into memory.

run_id(ts: ixmp.core.TimeSeries)

Return the run ID for the ts.

Returns:
Return type:int
set_as_default(ts: ixmp.core.TimeSeries)

Set the current TimeSeries.version as the default.

Returns:
Return type:None

See also

ts_is_default(), ts_get(), s_get()

set_data(ts: ixmp.core.TimeSeries, region, variable, data, unit, meta)

Store data.

Parameters:
  • region (str) – Region name.
  • variable (str) – Variable name.
  • time (str) – Time period.
  • unit (str) – Unit symbol.
  • data (dict (int -> float)) – Mapping from year to value.
  • meta (bool) – True to mark data as metadata.
set_geo(ts: ixmp.core.TimeSeries, region, variable, time, year, value, unit, meta)

Store time-series ‘geodata’.

Parameters:
  • region (str) – Region name.
  • variable (str) – Variable name.
  • time (str) – Time period.
  • year (int) – Year.
  • value (str) – Data value.
  • unit (str) – Unit symbol.
  • meta (bool) – True to mark data as metadata.
set_log_level(level)

OPTIONAL: Set logging level for the backend and other code.

Parameters:level (int or Python logging level) –
set_meta(s: ixmp.core.Scenario, name, value)

Set a single metadata key.

Parameters:
  • name (str) – Metadata key name.
  • value (any) – Value for name.
Returns:
Return type:

None
set_node(name, parent=None, hierarchy=None, synonym=None)

Add a node name to the Platform.

This method must have one of two effects, depending on the arguments:

  • With parent and hierarchy: name is added as a child of parent in the named hierarchy.
  • With synonym: synonym is added as an alias for name.
Parameters:
  • name (str) – Node name.
  • parent (str, optional) – Parent node name.
  • hierarchy (str, optional) – Node hierarchy ID.
  • synonym (str, optional) – Synonym for node.

See also

get_nodes()

set_unit(name, comment)

Add a unit of measurement to the Platform.

Parameters:
  • name (str) – Symbol of the unit.
  • comment (str) – Description of the change or of the unit.

See also

get_units()