Inserting and committing¶

Use insert to hand a table to something you have declared, naming the columns it reads from. Nothing reaches the database until commit, which builds the database and starts its server the first time, and sends what was inserted since on each later call. To see what the next commit would do without sending anything, call check.

Database¶

insert ¶

insert(
    target: str,
    table: Any = None,
    roster: Any = None,
    artifacts: Any = None,
    members: Any = None,
    columns: dict | None = None,
    **named: str,
) -> Insert

Hand a table to something declared, naming the columns to read, and return a record.

Nothing is sent until the next commit(), and a second insert into the same target adds to the first.

A data frame is copied into the database's directory when you call insert, while a Parquet file is read where it lies.

target: the name of a declared view, view group, attribute, layer, label set or vocabulary.
table: a pandas or polars data frame, a pyarrow table, or the path of a Parquet file.
roster, artifacts, members: in place of table, a view group's table of views, a layer's table of annotations or a layer's table of memberships, each in an insert of its own.
columns: {attribute: column} to read an attribute from a column with another name, either a unique attribute that names the rows' items or an attribute that a view's rows fill.
the other keywords: the column that holds each thing the target reads, such as x=, y= and access= for a view. A table in Mosaica's own shape, which is a layer's annotations or memberships, a label set's table read with contents=, a roster or a vocabulary's values, has each column named like one of its keywords read without the keyword, such as key, level, parent or attached_key; a keyword given names the column read in its place. A label set's table given with text= matches only key. The call is refused when it names a column the table lacks, gives a keyword the target does not read, leaves out one the target needs that the table does not carry, or names another column for a keyword whose own column the table carries too, since the build reads that column under its own name.

Each row of a view, attribute or membership table names its item by its mosaica_id column or by its values of the unique attributes, read from each attribute's own column or the one columns= gives. A view's row that names no item makes a new item, and the commit leaves out and reports every other row that does not name exactly one item.

A view's rows also fill each declared attribute whose name a column carries, from the anchor view's rows at the first commit and from any view's rows after it. The Insert returned lists the columns read and the columns ignored, which are those neither named, matched by name nor matching an attribute.

db.insert("papers", frame, x="x", y="y", access="labels")
db.insert("clusters", frame, key="cluster", columns={"paper": "paper_id"})
db.insert("topics", artifacts="clusters.parquet")  # key, level, parent by name

check ¶

check() -> Report | PagedReport

Report what the next commit() would do, without sending anything.

Before the first commit, the check reads the declaration against the column types of the inserted tables and reports each problem it finds. After the first commit, the report lists the requests that the commit would send, and any problem it would stop at before sending them.

db.check()

commit ¶

commit(
    strict: bool = False, progress: bool | Callable[[dict], Any] = True
) -> CommitReport | PagedReport

Make what was inserted part of the database, and return a report of what happened.

The first commit checks the declaration, builds the database from the inserted tables and starts a server over it. After it, no view's extent, column type or choice of rendered columns can change. Each later commit sends what was inserted since the last one to the running server, waits until readers can see it and then forgets those inserts.

Besides the rows that insert says are left out, the commit leaves out a row that names an item, or sets a unique value, that an earlier row of its table did. The rest is applied, and the report counts the rows left out by reason.

strict: True refuses the whole build, or the whole request, that carries such a row.
progress: how the commit shows how far it has gone. True, the default, draws it with tqdm.auto, which is a widget in a notebook and a bar on stderr in a terminal. False draws nothing. A function is called with each record as a dict, in the shape /control/status serves under jobs, and nothing is drawn. The first commit's records are the build's, and a later commit's are an ingest record that counts the rows of its pages sent and then waits for publication. The last record has state finished or failed, except where the build was killed by a signal or the commit was interrupted, which leave the last record running. A later commit with nothing to send makes no record.

A commit that did nothing raises Refusal, with the report as its report, which happens when a problem is found before anything is sent, when the build fails or when the server refuses every request. A commit in which some requests succeeded returns its report with the refusals listed. A request the server answers 429 is sent again as Control describes, and one it gives up on is listed in the report's refusals.

db.commit()

Insert `dataclass` ¶

The record of one insert call, which says what table was handed to which target and which of its columns are read.

target: the name of the declared thing the table was handed to.
kind: what the target is, as "view", "view_group", "attribute", "layer", "labels" or "vocabulary".
role: which of the target's tables this is, as "rows", "roster", "values", "key", "artifacts", "members" or "text".
columns: the column the call named for each thing the target reads, such as {"x": "x", "y": "y", "access": "labels"}.
named_attributes: on a view's rows, the column each attribute is filled from, as {attribute: column}.
items: the column the table carries for each attribute declared unique, as {attribute: column}, which is how each row names its item.
metadata_columns: on a view group's roster, the column each metadata value is read from.
shape, shape_columns: on an annotations table, the kind of shape its rows carry and the columns the shape is read from.
view_key: the one view of a view group that every row is in, when view_key= named it.
read, ignored: the table's columns in its own order, split into those that are read and those that are ignored.
rows: how many rows the table has.
schema: each column's Arrow type.
path: the Parquet file that holds the rows. in_place is True when that is the file the call gave, and False when the package wrote the table into the database's directory.
source, declared_path: the name and the path that the declaration file gives the table before the first commit.

table ¶

table() -> Table

The inserted rows, read back from path as a pyarrow table.

summary ¶

summary() -> list[str]

The lines print() shows, which are one line naming the target, the number of rows, and the columns read and ignored.

Report `dataclass` ¶

The report of a check() before the first commit, which CommitReport extends.

what: "check" or "commit".
ok: True when nothing was refused.
rows: the rows inserted into each view or view group.
frames: each view's name, with its extent in words.
render_columns: the columns sent with every point the map draws.
notes: what each insert read and ignored, and each declared column that no insert fills, with the reason. The first commit's build refuses such a column, although check() accepts it, unless it is unique or scoped to a view group.
findings: the Findings found before anything was built.
log: the declaration check's output, followed on a commit by the build's.

The summary lists every note, finding and warning, and carries the whole log when the report is not ok, because the log says why the check or build failed.

summary ¶

summary() -> list[str]

The lines print() shows: the rows inserted, then the notes, findings and warnings, or the whole log when the check failed.

CommitReport `dataclass` ¶

Bases: Report

The report of the first commit(), which adds to a Report what the build made and where the database is served.

views: how many views the database serves under each name, counting a view group's views under the group.
layers, labels: the annotation layers and the label sets the database declares.
items, minted, unclustered: the build's counts of the items it built, the annotations it made from key columns, and the member rows left in no annotation by a null key. Each is None when the build's log does not give it.
seconds: how long the check, the build and the server's start took.
viewer, session, control: the addresses of the planes where readers read, where tokens are made and where the operator writes.
identity: in words, how the rows name their items.
refused: the rows the build left out, as one dictionary for each file and reason, with the file as source, the block that read it as object, the number of rows as rows, up to ten of their values as values, and reason, one of names_no_item, names_two_items, one_item_twice, one_value_twice and unknown_mosaica_id.

summary ¶

summary() -> list[str]

The lines print() shows: the items built, the views and layers and the address served at, then the notes, findings and warnings.

PagedReport `dataclass` ¶

What a check() or commit() after the first one planned or did.

check() returns it with sent false, with the requests a commit would send in plan and any problem found in findings. commit() returns the same plan with sent true, and counts what the requests did:

rows_accepted: the rows added to each view, each of which created an item or added one to the view. rows is their total.
artifacts_minted, memberships_joined: the annotations added and the memberships joined.
labels_added: the labels added to label sets, which artifacts_minted leaves out.
items_edited: the items already held whose value, label or position a row changed. An edited item keeps its mosaica_id, and placing an item in an annotation counts as a membership instead.
items_unchanged: the rows that named an item and left it as it was, apart from placing it in an annotation.
values_bound, titles_set: the vocabulary values added and the titles replaced, leaving out the values of a derived vocabulary, which a writer without write-all is not told.
parts_filled: the absent parts of held annotations, such as content, that were filled.
already_present: the declarations and vocabulary keys already held, which changed nothing.
without_content: the annotations added without the content their layer declares.
clipped: the rows the view's projection could not place, stored on the view's edge.
clamped: the rows outside a view's extent, moved onto its edge.
refusals: each request the server refused, with its status and answer.
refused: each row or member the server left out while applying the rest of its request. A row is {"target", "view", "row", "reason"}, where row is its position in the table inserted into target, and a member is {"layer", "level", "view", "key", "list", "member", "reason"}, where member is its row as sent. An annotation refused whole for its label has list and member set to None. reason is one of names_no_item, names_two_items, one_item_twice, one_value_twice, unknown_mosaica_id, value_held (a unique value held by an item outside the writer's terms) and label_not_held (a label the writer's terms do not satisfy), and refused_by_reason counts them.
ignored_columns: for each layer, the member columns that were not sent, or that the server ignored, because they are neither mosaica_id nor a unique attribute's column.
mosaica_ids: the id given to each row sent, or None for a refused row.
artifact_ids: the id given to each added annotation, by layer and then by (level, view, key), or None for one refused for its label.
replayed: the requests the server had already carried out.
publication, flush_wait, flush_reached: the publication in which the changes can be read, how many seconds the commit waited for it, and whether the commit saw it arrive.

ok is True when no request was refused and nothing was found before sending, so rows and members in refused leave it True. The summary counts them by reason and lists every finding and refused request.

ok `property` ¶

ok: bool

True when no request was refused and nothing was found before sending.

rows `property` ¶

rows: int

How many rows were added, over every view.

refused_by_reason `property` ¶

refused_by_reason: dict

How many rows and members refused holds for each reason.

summary ¶

summary() -> list[str]

The lines print() shows: the requests planned, or what was added and when it could be read, then the findings and refusals.

Finding `dataclass` ¶

A problem that a check or a commit found, with what to change.

Most findings come from the pre-flight, which runs before anything is sent. A commit() that one stops sends nothing and raises Refusal, which lists the findings, so you correct the data or the declaration and commit again. A later commit whose closing flush is refused, or whose wait runs out, returns its report with a finding instead of raising, because what it sent is durable and can be read after the server's next publication.

what: the problem, in a few words.
detail: what was found and what to change.

A finding shows as its text, which starts refused:.

Inserting and committing¶

Database¶

insert ¶

check ¶

commit ¶

Insert dataclass ¶

table ¶

summary ¶

Report dataclass ¶

summary ¶

CommitReport dataclass ¶

summary ¶

PagedReport dataclass ¶

ok property ¶

rows property ¶

refused_by_reason property ¶

summary ¶

Finding dataclass ¶

Insert `dataclass` ¶

Report `dataclass` ¶

CommitReport `dataclass` ¶

PagedReport `dataclass` ¶

ok `property` ¶

rows `property` ¶

refused_by_reason `property` ¶

Finding `dataclass` ¶