Declaring

Declaring says what a database will hold before any data arrives: its views, columns, vocabularies and annotation layers, and the size of the shards its items are stored in. Each declare_* method adds a block to the declaration, which the next check() or commit() writes out as schema.toml and checks. The data itself comes later, through insert.

Database

declare_view

declare_view(
    name: str,
    *,
    extent: Any = None,
    projection: str = "none",
    default_label: str | None = "public",
    visibility: Any = "public",
    anchor: bool = False,
    title: str | None = None,
) -> dict

Declare a view, and return its block.

A view is one layout of the items, which a map draws from each item's x and y coordinates in that view. One set of items can have several views, such as two projections of the same embedding. The rows of a view come from insert(name, table, x=, y=, access=).

• name: the view's name.
• extent: the range of coordinates the view covers, as {"x": [min, max], "y": [min, max]}, or as {"lon": [...], "lat": [...]} under a projection. By default the first commit fits it to its rows with a margin. An extent cannot change once built, so a view declared after the first commit must give one.
• projection: "none" (the default) for plain x and y coordinates, or a map projection such as "web_mercator" for longitude and latitude, in which case the view's inserts name their coordinate columns with lon= and lat=.
• default_label: the access label a row takes when its access column is empty, "public" by default. With None there is no default, and a build or request that carries a row without a label is refused.
• visibility: who may see the view at all, "public" by default, or an access label or a list of them.
• anchor: True makes this view's layout decide the order the items are stored in. The first view declared is the anchor by default.
• title: the name shown for the view.

db.declare_view("papers", extent={"x": [0, 100], "y": [0, 100]})

declare_view_group

declare_view_group(
    name: str,
    *,
    metadata: dict | None = None,
    members: str | None = None,
    extent: Any = None,
    projection: str = "none",
    default_label: str | None = "public",
    visibility: Any = "public",
    title: str | None = None,
) -> dict

Declare a view group, and return its block.

A view group is a set of views that share every setting and differ by a key, such as one view per year, and each view in it is named "<group>:<key>". The views and their metadata come from a roster table inserted with insert(name, roster=table, key=), and their rows from insert(name, table, view=), where view= names the column that says which view each row is in.

• name: the group's name.
• metadata: the metadata each view carries, as {name: type}, whose values the roster supplies. A category is written {"type": "category", "vocabulary": vocabulary}, and the roster then holds the value's key. The views carry none by default.
• members: the name of another group whose views this group shares, in which case this group takes no roster and no metadata of its own.
• extent, projection, default_label, visibility, title: as for declare_view, and each applies to every view in the group.

db.declare_view_group("years", metadata={"year": "i32"})

declare_attribute

declare_attribute(
    name: str,
    type: str,
    *,
    render: bool | None = None,
    index: bool | None = None,
    vocabulary: str | None = None,
    analyser: str | None = None,
    scope: Any = "entity",
    title: str | None = None,
    unique: bool | None = None,
) -> dict

Declare a column that the items carry, and return its block.

The column's values come from a view's rows that carry a column of the same name, as insert describes, or from a table inserted on its own with insert(name, table, value=).

• name: the column's name.
• type: "bool", "u8", "u16", "u32", "u64", "i8", "i16", "i32", "i64", "f32", "f64", "timestamp_us", "keyword" (a string matched exactly), "text" (a string searched by word) or "category" (a value from a vocabulary).
• render: True sends the value with every point drawn, so that a map can colour points by it. Any column except a keyword or text column can be rendered, and the choice, False by default, is fixed once the column is declared. A rendered column declared after the first commit draws no value for the points already stored until an insert gives them one.
• index: True makes the column filterable. It is False by default.
• vocabulary: for a category, the vocabulary its values come from.
• analyser: for text, how the text is split into words. "unicode" is the default and only analyser.
• scope: {"group": name} gives the column a separate value in each view of that view group, which must already be declared. By default an item has one value in every view.
• title: the name shown for the column.
• unique: True lets each value be held by one item at most, so that a value names an item and a table can address its rows' items by this column, as insert describes. An insert giving an item a value that another item holds is refused, and a filter {name: {"in": values}} finds the items holding those values whether or not the column is indexed. Only a "keyword", integer or "timestamp_us" column that is not scoped to a group can be unique. It is False by default.

db.declare_attribute("year", type="i32", render=True, index=True)
db.declare_attribute("doi", type="keyword", unique=True)

declare_unique

declare_unique(name: str, unique: bool = True) -> dict

Make a declared attribute unique, or stop it being unique, and return its block.

• name: the attribute.
• unique: False stops the attribute being unique and keeps its values.

After the first commit the change reaches the database at the next commit(). Making a column unique there builds its index over the values the items hold, and the change is refused when two items hold the same value, naming up to ten such values.

db.declare_unique("doi")

declare_vocabulary

declare_vocabulary(
    name: str,
    *,
    closed: bool = False,
    width: str = "u16",
    values: Any = None,
    reserved: Sequence[int] | None = None,
    visibility: str = "public",
    title: str | None = None,
) -> dict

Declare a vocabulary, the set of values a category column may take, and return its block.

Each value in a vocabulary has a key, a small integer code that the items store, and an optional title. The values come from values=, or from insert(name, table, key=, title=, code=).

• name: the vocabulary's name.
• closed: True allows only the values given. An open vocabulary (the default) adds each new value it meets in the inserted rows.
• width: the size of a code, "u8", "u16" (the default) or "u32", which limits how many values the vocabulary can hold.
• values: the values, as a list of keys, or as {key: code} to fix the code of each.
• reserved: codes that are never handed out.
• visibility: "public" (the default) lists every value to every reader, and "derived" lists a value only to a reader who may see at least one item carrying it.
• title: the name shown for the vocabulary.

db.declare_vocabulary("venue", closed=True, values=["neurips", "icml", "iclr"])

declare_shards

declare_shards(size: int) -> dict

Declare the size of the shards the items are stored in, and return the [shards] table.

A shard takes new items until it has handed out size row ids, and is then sealed so that new items go to the next shard. Without this declaration the size is 4,026,531,840.

• size: the number of row ids per shard, as a positive integer.

A first commit whose tables hold more items than the size builds several shards. After the first commit, the next commit() sends the new size to the server, and a size no larger than the number of row ids the open shard has handed out seals that shard at once.

db.declare_shards(50_000_000)

declare_columns

declare_columns(
    frame: Any,
    skip: Sequence[str] = (),
    render: Sequence[str] = (),
    index: Sequence[str] = (),
    keyword: Sequence[str] = (),
    category: Sequence[str] = (),
    unique: Sequence[str] = (),
) -> Declared

Declare every column of a data frame as an attribute, and return what was declared.

Each column is declared with the attribute type that fits its data type, and it is drawn or filterable only when render or index names it. The Declared report returned says what each column it looked at was declared as and why, or why it was not declared when no attribute type fits it. Only the frame's column types are read, and a column already declared is left as it is.

• frame: a pandas or polars data frame or a pyarrow table.
• skip: columns to leave out, such as the id and the coordinates.
• render: columns to send with every point drawn.
• index: columns to make filterable.
• keyword, category: string columns to declare as keywords or as categories, since a string column is otherwise declared as text.
• unique: columns to declare unique, as declare_attribute describes.

Every name in these lists must be a column of the frame. A categorical column, which is a pandas Categorical or an Arrow dictionary column of strings, is declared as a category unless keyword names it. Each category reads the vocabulary of its own name, and when no such vocabulary is declared yet, it is declared open with u16 codes, which limit it to 65,535 values. Because the width cannot change after the first commit, declare a category that needs another width with declare_vocabulary before calling this.

db.declare_columns(frame, skip=["paper_id", "x", "y"], category=["venue"])

declare_layer

declare_layer(
    name: str,
    kind: str,
    *,
    views: Iterable[str] | None = None,
    membership: Any = "enumerated",
    shape: Any = None,
    default_space: str = "view",
    levels: Sequence[Any] | None = None,
    require_member_visibility: Any = "none",
    visibility: Any = "public",
    artifact_visibility: Any = "inherited",
    computed: Sequence[str] = DERIVED,
    supplied: Sequence[Any] | None = None,
    depends_on: Sequence[str] | None = None,
    prune_children: bool = False,
    withdraw_on_member_deletion: bool = False,
    value_set: str | None = None,
    scope: Any = "entity",
    layout: str | None = None,
    artifacts: Any = None,
    title: str | None = None,
) -> dict

Declare an annotation layer, and return its block.

An annotation layer is a set of annotations drawn over the items, such as a clustering, a set of regions or a taxonomy. Each annotation has a key and a set of member items, and both come from insert(name, table, key=) or from separate insert(name, artifacts=...) and insert(name, members=...) calls. Mosaica's names and fields call an annotation an artifact, as in artifacts= and Viewer.artifacts.

• name: the layer's name.
• kind: how the annotations relate: "flat" (no parents), "nested" (a tree), "dag" (several parents), "stacked" (independent levels) or "tiered" (levels in which each coarser annotation contains finer ones).
• views: the views the layer is drawn on, all of them by default.
• membership: how members are decided. "enumerated" (the default) reads them from a table, "spatial" makes each annotation a shape holding the items inside it, and {"attribute": column} makes one annotation for each value of a category column.
• shape: for a spatial layer, "bbox", "circle", "ellipse" or "polygon".
• default_space: for a spatial layer, whether shapes are given in the view's coordinates ("view", the default) or in longitude and latitude ("wgs84").
• levels: the levels of a stacked or tiered layer, each as (level, title) or (level, title, (min_zoom, max_zoom)).
• require_member_visibility: how many of an annotation's members a reader must see before it is shown to them: "none" (the default), "any", "all", {"fraction": 0.1} or {"count": 50}.
• visibility: who may see the layer at all, "public" by default or an access label.
• artifact_visibility: the access label of an annotation that carries none of its own. The default, "inherited", gives it none, so that visibility and require_member_visibility alone decide who sees it. {"field": column, "default": label} reads each annotation's own labels from column, which every insert(name, artifacts=...) must then name with access=. Without a field, the first such insert that names access= sets it.
• computed: which of "centroid", "box" and "hull" the server computes for each reader from the members it may see. All three are computed by default, except that a spatial layer draws its shape in place of a hull and a layer with attribute membership computes none.
• supplied: content you provide for each annotation, as (name, type, requires) entries, where requires takes the values declare_labels takes for content_requires.
• depends_on: the layers this one attaches to, such as the clustering a set of labels names.
• prune_children: True shows only the finest qualifying annotation on each branch of the hierarchy, without its qualifying ancestors.
• withdraw_on_member_deletion: Not built yet: True is refused at check().
• value_set: "closed" when no annotation keys may appear beyond those inserted. The default is closed when an artifacts table, artifacts or an attribute membership says which annotations there are, and "open" otherwise.
• scope: {"group": name} keeps a separate set of annotations for each view of that view group, which must already be declared.
• layout: how the server stores memberships, "rows", "column" or "list". It affects only speed, and the server chooses by default.
• artifacts: annotations written into the declaration itself, as a list of dictionaries or a table.
• title: the name shown for the layer.

db.declare_layer("clusters", kind="flat", require_member_visibility={"count": 20})

declare_labels

declare_labels(
    name: str,
    of: str,
    *,
    content_requires: str = "inherited",
    require_member_visibility: Any = "none",
    artifact_visibility: Any = "inherited",
    type: str = "text",
    title: str | None = None,
) -> dict

Declare a set of text labels for the annotations of another layer, and return its block.

A label set gives annotations of the layer of text to show, such as a topic name for a cluster, and a label is drawn wherever its annotation is drawn. The text comes from insert(name, {key: text}) or insert(name, table, key=, text=). On a layer with levels each label also names the level of its annotation, so the text comes from insert(name, table, key=, text=, attached_level=) and a mapping is refused.

• name: the label set's name.
• of: the declared layer it labels, which must not already carry a label set.
• content_requires: "inherited" (the default) shows a label wherever its annotation is shown, and "all" shows it only to a reader who may see every item the text was written from. Those items come from insert(name, members=table, key=).
• require_member_visibility, title: as for declare_layer.
• artifact_visibility: the access label that every label in the set carries. The default, "inherited", gives the labels none of their own, so that the layer they label and require_member_visibility alone decide who sees them. A field is refused.
• type: the type of each label's text, "text" by default.

db.declare_labels("topics", of="clusters")
db.insert("topics", {"c0": "graph neural networks", "c1": "diffusion models"})

declare

declare(kind: str, block: dict) -> dict

Declare one block written in the keys of the declaration file, and return it.

The declare_* methods build these blocks for you, so use this for a key that none of them takes. The block's contents are checked at the next check() or commit(). Like every declare_* method, this refuses a name already declared for the same kind of thing, and views and view groups count as one kind.

• kind: "view", "view_group", "vocabulary", "attribute" or "layer".
• block: the block as a dictionary, with the keys schema.toml uses.

db.declare("attribute", {"name": "score", "type": "f64", "index": True})

declaration property

declaration: str

The database's declaration, as the TOML text that its schema.toml holds.

write

write() -> dict

Write schema.toml and mosaica.toml, and return the declaration as a dictionary.

Because check() and commit() write the files themselves, call this only to look at them first.

Declared dataclass

What declare_columns declared.

• columns: a DeclaredColumn for each column of the frame that was neither skipped nor already declared, including those no type fits, which were left undeclared.
• vocabularies: the vocabularies this call declared for category columns, leaving out one declared before.

summary

summary() -> list[str]

The lines print() shows: each column with its type, then the vocabularies.

DeclaredColumn dataclass

One column that declare_columns looked at, and the type it declared the column as.

• name: the column's name.
• dtype: the column's Arrow type, as text.
• declared_as: the attribute type the column was declared with, or "not declared" when no type fits it.
• render, index, unique: whether the column was declared rendered, indexed and unique.
• vocabulary: the vocabulary a category column's values come from, or None for any other column.
• why: the reason the column was declared as it was, or was not declared.