Dimensions and relations#

A dimension is an axis of the model, such as snapshot or generator. Declarations are indexed by it, and sum reduces over it.

A relation is a named table between dimensions: a generator's bus, a snapshot's period, or the buses a generator may connect to.

`dimensions`#

dimensions:
  snapshot: { dtype: int }
  generator: { dtype: str }

Every dimension named anywhere in the file is declared here.

Field
`dtype`	`float`, `int`, `str`, `datetime`	default `str`
`description`	free text, never parsed	default `null`

A declaration says that the axis exists and what type its labels have. It never lists the labels. The generators, buses and snapshots arrive with the data, as a table, not as a list somebody keeps in step by hand.

Where the members come from#

The engine that binds the data follows three rules, and every engine follows the same three. So two engines given the same file and the same tables build the same model.

The members come from the key named after the dimension. An engine reads generator from the generator table, and from nowhere else. It reads capacity for its values, never for its list of generators, and it does not treat gen_bus as the list either. If a declaration uses generator and no generator table arrives, the engine raises an error that names generator. It does not build an empty axis, because an empty axis would silently delete every row indexed by it. A declared dimension that no declaration uses needs no table.
The members keep the order the table gives them. The engine does not sort them, whether they are strings, integers or dates. shift, sum_back and position() all count along this order, so an engine that sorted snapshot would give shift(dispatch, along=snapshot, offset=1) a different meaning. To get a particular order, write the table in that order.
A table has each coordinate at most once. Two rows for snapshot == 3 is an error that names 3. The engine does not keep the last, keep the first, or add them. At most once, not exactly once: a coordinate with no row is absence, and absence is how a model masks. A relation's table obeys the same rule.

Every dimension has one list of members, and every parameter is lined up against it when the data binds. So if load has 8760 snapshots and price has 8759, the engine raises an error rather than build a model with one snapshot dropped.

`relations`#

A relation is a table with one column per dimension it relates: a generator's bus, a snapshot's period, or the buses a generator may connect to. It is what makes topology data, so no adjacency matrix and no hand-written join appears anywhere. key: names the columns that identify a row, and values: the columns that key determines. Every relation is keyed: with no values:, the key is every column. The declaration fixes no direction. The operator that walks the table says which column it consumes and which it produces.

dimensions:
  bus: { dtype: str }
  generator: { dtype: str }
  line: { dtype: str }
  snapshot: { dtype: int }
  period: { dtype: int }
relations:
  gen_bus: { key: generator, values: bus } # each generator on one bus
  line_from: { key: line, values: bus } # two relations onto one dimension
  line_to: { key: line, values: bus }
  period_of: { key: snapshot, values: period }
  connection: { key: [generator, bus] } # no values: a generator may connect to several buses

Field
`key`	required — the columns that identify a row (below)
`values`	the columns the key determines; omitted, the table is a bare relation (below)	default none
`description`	free text, never parsed	default `null`

Each side is one dimension, a list of them, or a mapping of column name to dimension where two columns share one (roles).

A relation has at least two columns between the two sides, each over a declared dimension, and each column name is distinct. A column named like a dimension is over that dimension, so values: {bus: line} is refused. A relation name may not shadow a dimension: generator's map onto bus is gen_bus, never a second bus.

A column's values are checked against its dimension's labels when data is bound, so a mistyped bus is refused rather than summed into a group of its own. That is why a label set the model only selects on is still declared as a dimension. Nothing above is indexed by period, but period is declared, so where: "period_of == 1" (where strings) compares against a checked label.

The key is the claim#

key: names the columns that are unique together. key: generator says the generator column holds each label once: the table has one row per generator, so every column under values: is a function of it. key: [generator, period] says the pair holds each combination once. Neither column need be unique on its own: a generator appears once per period, and a period once per generator. The claim is checked at bind, so a generator on two buses is refused (#161). The columns under values: are the relation's value columns. A key that determines a value is read at its dimensions, and no frame carries a dimension twice, so {key: {bus0: bus, bus1: bus}, values: line} is refused. A bare relation may key two columns over one dimension, because nothing reads it.

Each cardinality is one declaration:

to say	write	checked at bind
many-to-one, each generator on one bus	`{key: generator, values: bus}`	one row per generator
one-to-many, a bus and its generators	the same table: `sum(p, by=gen_bus, over=generator, into=bus)` collects a bus's generators, `at(price, by=gen_bus, over=bus, into=generator)` reads a generator's bus	the same
many-to-many, a generator on several buses	`{key: [generator, bus]}`, no `values:`	no row twice
one-to-one	not a claim the language has: a key is one set of columns, and nothing checks the other side

A bare relation is a set of rows: no row twice, and nothing else claimed. sum walks it with both ends named, and a bare where tests it. That is what a many-to-many relation can say, and all it can say.

Walks#

A walk reads a relation from one set of its columns to another. over= names the columns the call consumes, and into= names the columns it produces. Every key column named at neither end is joined on: the operand carries its dimension, and the result keeps it. A value column named at neither end is not read.

sum consumes key columns and produces value columns. at consumes value columns and produces the key.

dimensions:
  generator: { dtype: str }
  zone: { dtype: str }
  period: { dtype: int }
relations:
  zone_of: { key: [generator, period], values: zone } # a generator's zone, per period
parameters:
  demand: { dims: [zone, period] }
  price: { dims: [zone, period] }
variables:
  p: { dims: [generator, period] }
constraints:
  zone_balance: # consumes generator, joins on period, produces zone: [generator, period] -> [zone, period]
    dims: [zone, period]
    expression: sum(p, by=zone_of, over=generator, into=zone) >= demand
  history: # consumes period, joins on generator, produces zone: [generator, period] -> [generator, zone]
    dims: [generator, zone]
    expression: sum(p, by=zone_of, over=period, into=zone) <= 100
  capped_revenue: # consumes zone, joins on period, produces generator: [zone, period] -> [generator, period]
    dims: [generator, period]
    expression: at(price, by=zone_of, over=zone, into=generator) * p <= 1000

The three calls read one table three ways:

call	consumes	joins on	produces	operand	result
`sum(p, by=zone_of, over=generator, into=zone)`	`generator`	`period`	`zone`	`[generator, period]`	`[zone, period]`
`sum(p, by=zone_of, over=period, into=zone)`	`period`	`generator`	`zone`	`[generator, period]`	`[generator, zone]`
`at(price, by=zone_of, over=zone, into=generator)`	`zone`	`period`	`generator`	`[zone, period]`	`[generator, period]`

capped_revenue reads the price of the zone this generator sat in that period. The typesetter prints it as \(\mathrm{price}_{\mathrm{zone\_of}(g,\ e),e}\), and the joined period is the second subscript.

Three things hold for the shape of the call:

Either keyword takes a list. With gen_bt: { key: generator, values: [bus, technology] }, sum(p, by=gen_bt, over=generator, into=[bus, technology]) lands on the product bus x technology in one join. sum(p, by=zone_of, over=[generator, period], into=zone) consumes both key columns at once, so nothing is joined on and period leaves with generator.
into= needs a by=, because a column belongs to a table. over= without a by= names a dimension, as in sum(p, over=period).
One call walks one table. by= names a single relation. To land on columns of two tables at once, declare one relation holding the columns of both; to walk them in turn, write one call each.

The six rules#

Six rules hold for every sum and every at through a relation. They say what a call computes, which of the two operators is legal, and what an edit to the file changes. A change to the notation has to keep all six.

The result is the operand, less the consumed dimensions, plus the produced ones. The operand carries every dimension the call consumes and every one it joins on. In symbols, result = (operand − consumed) ∪ produced, where the joined columns are the key columns named at neither end. The table above is this rule three times.
The result gains a dimension only from the operand, never from an edit to the relation. That is why both ends are written on every call. sum(p, by=gen_bus) is refused, even where gen_bus has one key column and one value column. What a call leaves unsaid, an edit to the relation could change: a table that gained a value column would silently start producing it.
Adding a value column is safe. A relation that gains one changes no existing call, because a value column the call does not name is not read. sum(f, by=ends, over=line, into=bus1) reads bus1 and ignores bus0, and goes on reading bus1 alone when ends gains a third column.
The key is fixed. Every call through a relation joins on the key columns it names at neither end. So a key that gains or loses a column re-aims every call at once, and that is a different table. Declare a new relation rather than edit the key. Calls through the old one keep their meaning, because the old one still says what it said.
An operand may grow. A call means the same when its operand gains a dimension the relation does not name, and that dimension passes through to the result. Give p a scenario dimension, and zone_balance carries scenario too, with the same call. An operand may not grow into a dimension the call lands on. sum(load * p, by=gen_bus, over=generator, into=bus) is refused where load carries bus. The walk would tie the operand's bus to the one it produces rather than add it, and the call reads the same either way. Write load * sum(p, by=gen_bus, over=generator, into=bus). A relation into its own dimension is not this case, because there the dimension landed on is the one just consumed (roles).
sum consumes a key column, and at consumes none. Consume no key column and each coordinate finds one row, so nothing is added up. That is a read, and it is at's. Consume one and a coordinate finds many rows, which is sum's. Each operator is refused in the other's case.

What each end may name follows from rule 6, and from nothing else:

operator	`over=`, consumed	`into=`, produced
`sum`	any columns, at least one a key column	any columns
`at`	value columns only	any columns

An end may not name a column twice, name a column the other end names, or name two columns over one dimension. The operand has one axis per dimension, so nothing would say which column its coordinate is read at.

Five refusals draw the line, and each message names the rewrite:

refused	message
a walk with an end unsaid	`sum() through a relation leaves into= unsaid. A walk names both of its ends, so that a relation may gain a value column without changing what this call means. Write: sum(<expr>), sum(<expr>, over=<dim>) or sum(<expr>, by=<relation>, over=<column>, into=<column>)`
landing on a dimension already carried	`sum(by=gen_bus) lands on ['bus'], which the expression already carries. A walk brings the dims it lands on, so that reading the call tells you what it adds. Move the factor carrying ['bus'] outside the operator, or walk to a column over another dimension.`
`at` on a bare relation	`at(by=connection): at reads one value per coordinate, and 'connection' is not single-valued in ['bus'] at the columns the operand fixes (['generator']) — its key is ['generator', 'bus']. Key the table by columns the read fixes, or read the other way.`
a `sum` that consumes no key column	`sum(by=zone_of): this sum walks to the key ['generator', 'period'], so each coordinate has one term and nothing is added up — that is a read, which is at()'s. Write at(..., by=zone_of, over=['generator'], into=['zone']), or sum toward a value column.`
an operand missing a joined dimension	`at(by=zone_of) joins on ['period'] (columns ['period'] of 'zone_of'), which the expression does not carry (dims ['zone']). A relation is walked between two of its columns and read at the others — index the operand by them, or walk between different columns.`

Partitions#

shift(x, along=d, by=l, within=c), sum_back(x, along=d, by=l, within=c) and position(d, by=l, within=c) walk the one key column over d, join on the other key columns, and group by the value columns within= names. The frame does not change: the group says which rows are neighbours, and nothing lands anywhere.

A partition writes within= whenever it writes by=, for the reason a walk writes both of its ends. shift(x, along=snapshot, by=cal, within=week) walks within weeks of a calendar declared once over [snapshot, day, week], and a value column not named is not read. So a table that gains a column changes no call through it. The group may be two columns over one dimension, such as a line's two buses, because a partition produces no dimension. within= naming a key column is refused, and a bare relation partitions nothing. The refusal for a group left unsaid names the rewrite:

shift() through a relation leaves within= unsaid.
A partition names the value columns it groups by, so that a relation may gain a value column without changing what this call means.
Write: shift(<expr>, along=<dim>, offset=<n>[, edge='wrap'|<number>][, by=<relation>, within=<column>])

Of the six rules, a partition keeps rules 3 and 4. A key column added to the relation is one more column every call joins on, so declare a new relation.

A where string reads a relation too: a value column at its key, two columns of one table compared, or a bare name that tests a row exists (where strings).

Roles#

A bare name or a list names each column after its dimension. Two columns over one dimension need names of their own, and the mapping form gives them:

relations:
  ends: { key: line, values: { bus0: bus, bus1: bus } } # a line's two ends, one table
  rep_of: { key: snapshot, values: { rep: snapshot } } # the representative snapshot

sum(f, by=ends, over=line, into=bus1) - sum(f, by=ends, over=line, into=bus0) is a nodal balance through one table: flow arriving at bus1 less flow leaving bus0. where: "ends.bus0 != ends.bus1" excludes a line whose two ends are one bus.

rep_of relates a dimension to itself. That is how a clustered year runs on a few typical days: every snapshot names the snapshot that stands for it. The rules are the same. sum consumes snapshot and produces rep, both over snapshot, so the frame is [snapshot] before the walk and after it:

constraints:
  representative: # every snapshot takes its representative's value
    dims: [snapshot]
    expression: p == at(p, by=rep_of, over=rep, into=snapshot)
  weighted: # the snapshots a representative stands for, summed onto it
    dims: [snapshot]
    expression: sum(p, by=rep_of, over=snapshot, into=rep) <= 100

The key directs a self-map. key: snapshot makes rep a function of snapshot. at reads each snapshot's representative, and sum collects onto a representative the snapshots it stands for. Two steps along the map are two nested calls. Put both columns under key:, as {key: {from: snapshot, to: snapshot}}, and the table is a neighbour table instead, which sum walks either way and nothing reads. The rows where a snapshot is its own representative cannot be selected with a where, because a value column is never compared to the frame's own coordinate. Declare a bool parameter for them.

How the map is supplied#

The data for gen_bus arrives under the key gen_bus, as a table with one column per declared column, named after it:

sources = {
    'generator': ['g1', 'g2', 'g3'],
    'gen_bus': pl.DataFrame({'generator': ['g1', 'g2'], 'bus': ['north', 'south']}),
}

A partial map is the rows it has. g3 is in no row, so g3 sits on no bus. Absence is the missing row, as it is for a parameter. A null in any column is refused, because a row that is present and empty says both at once. The table holds one row per key tuple. A value that matches no label of its dimension is refused as a typo, never added as a member.

A relation's table stands on its own, so a model gains a relation the way it gains a parameter: one more table, and no change to the others. A column named after a relation inside a dimension's table is refused. That table may carry other extra columns, but this one would be a map read by accident.

Dimension, relation or parameter?#

Every column of data is one of the three. What decides which is what the math does with the column, not what the column holds:

The column…	is declared as	because
is an axis: something is indexed by it, or an aggregation lands terms on it	a `dimension`	its members are the coordinate set every table over it is reindexed onto
has one value per member of a dimension, or per tuple of several — a generator's bus, a line's two ends, a generator's zone by period	a `relation` with that `key`	it is a map every operator walks, and its values are checked against the dimensions they name
relates members of two dimensions many-to-many, with nothing to weigh — which buses a generator may connect to	a bare `relation`, with no `values:`	`sum` walks it with both ends named, and a bare `where` tests it. Nothing reads it, because there is no one value to read
relates members of two dimensions many-to-many, with a weight per pair — a link's efficiency to each bus, a cycle's lines	a `parameter` over both	the weight is the data, its row set is the relation, and the aggregation is `sum(w * x, over=a)`
is a label set the model only selects on or counts within — a period, a season, a zone	a `dimension`, and a `relation` onto it	its labels are checked, at the cost of one line and one table
scales terms — a coefficient, a bound, an offset	a `parameter` (`float` or `int`)	arithmetic is over numbers (dtype)
is a per-row attribute the math only selects on — a fuel, a constraint's sense	a `str` parameter	it names rows rather than scaling them, and no set is declared to check its values against
is a mask	a `bool` parameter	a bare name in a `where` is its own answer

Two rules follow. If b has one value per a, declare b as a relation keyed by a, not as a dimension. Two dimensions that depend on each other, declared as a dims product and cut back with a mask, are one relation.

Everything under dimensions: is an axis. A dimension is never legal where a value belongs, because it is a coordinate space and not data. To use a dimension's labels as data, declare a parameter over it. python -m math_spec check advises on a declared dimension that nothing is indexed by, nothing aggregates into and no relation has a column over (errors).