Skip to content

Operating

Use these methods to delete, suppress or withdraw what a committed database serves, and to reach the server it runs under. Because nothing serves a database before its first commit, remove, suppress, unsuppress, leave, drop_layer, drop_view, compact and status are refused until then.

Database

remove 

remove(items: Any, strict: bool = False) -> ChangeReport

Delete items from the database, and return a report of what was deleted.

The items stop being served as soon as the call returns, and their rows are removed from disc at the next compaction, which compact() asks for.

  • items: the items to delete, as one mosaica_id, a list of them or a table. A table is a pandas or polars data frame, a pyarrow table or a dict of columns, and each of its rows names one item by its mosaica_id column or by its values of the unique attributes, which it carries in the columns an insert reads them from.
  • strict: True refuses the whole request when a row names no item, or names two, and the call then raises Refusal.

Without strict, a row that names no item or two is listed in the report as refused and the other rows are applied. A None in the list, or a row whose cells are all null, names no item. A table's columns that name no item are not sent, and the report lists them in ignored_columns, but a table with no column that names items is refused before anything is sent. A mosaica_id may also name an annotation, while a layer's own is refused, because drop_layer is how a layer is removed.

A call in which the server refused every request raises Refusal, with the report as its report.

db.remove([mosaica_id])
db.remove({"paper": ["paper-17", "paper-23"]})

suppress 

suppress(items: Any, strict: bool = False) -> ChangeReport

Hide items from every reader until unsuppress shows them again, and return a report.

A suppressed item stays stored and is left out of every answer to a request that starts after the call returns.

  • items, strict: as for remove, except that a layer's mosaica_id hides the layer and its annotations.

unsuppress 

unsuppress(items: Any, strict: bool = False) -> ChangeReport

Lift the suppression of items, so that their readers see them again, and return a report.

  • items, strict: as for remove.

addresses 

addresses(items: Any) -> list[dict]

The rows that name the items given in a change request, as a list of dictionaries.

Each mosaica_id in a list becomes {"mosaica_id": ...}, and each row of a table becomes its mosaica_id value and its values of the unique attributes, under the attributes' names. A None or a null cell stays None, and every other value is written as text: an integer in decimal digits, a timestamp as the decimal digits of its microseconds since the epoch, and a string as itself.

leave 

leave(
    layer: str,
    key: str,
    items: Any,
    rank: int = 0,
    level: int = 0,
    view: str | None = None,
    strict: bool = False,
) -> ChangeReport

Take items out of the set of items a label's text was written from, and return a report.

A label declared with content_requires="all" is shown only to a reader who may see every item its text was written from, so once an item is taken out of that set, a reader no longer needs to see it to be shown the label.

  • layer: the label set.
  • key: the label's key.
  • items, strict: as for remove.
  • rank: which of the label's texts to change, by its position among them, which is 0 by default. A rank keeps its number when a text before it is withdrawn.
  • level: the level the label is at, which is 0 by default.
  • view: the view the label belongs to, on a layer scoped to a view group.

Taking out every item withdraws the text, and its rank is never used again, so inserting the label with the same texts adds each withdrawn one as a new text at the next free rank. A call the server refuses raises Refusal, with the report as its report.

drop_layer 

drop_layer(name: str, wait: bool = False) -> dict

Remove an annotation layer or a label set, and return the server's answer.

The layer and its uncommitted inserts also leave this database's declaration, so that the name can be declared again, as a new layer that none of the dropped layer's mosaica_ids names.

  • name: the name of the layer or label set.
  • wait: True returns only once readers no longer see the layer, with the time limit that Control.flush describes.

The answer's publication names the publication from which readers no longer see the layer.

drop_view 

drop_view(group: str, key: str, wait: bool = False) -> dict

Remove one view of a view group, and return the server's answer.

Items that the drop leaves in no view are deleted, which cannot be undone, and the answer's deleted says how many there were. Rows sent for the view that readers cannot see yet are dropped with it.

  • group, key: the group and the key of the view "<group>:<key>".
  • wait: as for drop_layer.

compact 

compact(
    wait: bool = False, progress: bool | Callable[[dict], Any] = True
) -> dict

Ask the server to remove the rows of deleted items from disc now.

The server compacts each shard that has something to reclaim, and a request made while a compaction runs waits until that compaction has finished. When no compaction can start until the server is restarted, the call raises Refusal with the reason and requests nothing.

  • wait: False, the default, returns {"request": n} once the server has accepted the request, before the work is done, and control.compaction_request(n) says where the request is. True waits as Control.await_compaction does and returns {"request": n} with what it returned merged in. Where its ok is false, it raises Refusal with the server's detail, from the answer or from the reading that ended the wait, that same dictionary as the refusal's report, and the HTTP status of that reading as its status.

  • progress: how the wait shows the compaction's record, as commit takes it. The default is True.

    db.compact(wait=True)

status 

status() -> dict

The server's report on the state of this database, as a dictionary.

The report says how far writes have got, how deep the server's queues are and how large a page each write request accepts. In it, overlay counts the items deleted or suppressed that no compaction has removed yet, and layer_spaces counts the layers in count, the deleted or suppressed layers and annotations in depth and the deleted ones alone in retirable. shards.records holds one record per shard.

serve 

serve() -> Listening

Start the database's server if it is not running, print the address it serves at, and return the addresses it listens on.

Reads and later commits call this themselves. The server listens on this machine only, on the port stated at create() or open(), or else on the one its mosaica.toml names. To see the map from another machine, forward that port to the same port there, as ssh -L 9141:127.0.0.1:9141 host does for 9141.

control property 

control: Control

A Control client for this database's control plane.

Use it for what Database has no method for. The client sends the operator credential, and reading this property starts the server if it is not running.

ChangeReport dataclass

What a call to remove, suppress, unsuppress or leave did.

  • op: the change, as "delete", "suppress", "unsuppress" or "leave <layer>/<key> rank <rank>".
  • requested, accepted: how many rows the call was given, and how many were applied.
  • refused: each row refused because of how it named its item, as {"row", "item", "reason"}, where row is its position in what was given, item is the row as given, such as {"mosaica_id": "17"}, and reason is names_no_item, names_two_items or unknown_mosaica_id. The other rows were applied, and refused_by_reason counts these.
  • ignored_columns: the columns given that were not sent, or that the server ignored, because they are neither mosaica_id nor a unique attribute's column.
  • refusals: each request the server refused, with its status and answer.

ok is True when every row given was applied.

ok property 

ok: bool

True when every row given was applied.

refused_by_reason property 

refused_by_reason: dict

How many rows refused holds for each reason.

summary 

summary() -> list[str]

The lines print() shows: the rows given and applied, up to fifty refused rows with their reasons, the columns ignored, then the refused requests. The first line ends ok only when every row was applied.

Listening dataclass

The addresses a database's server listens on, each as host:port, which Database.serve() returns. The three are one address unless the database's mosaica.toml gives a plane an address of its own.

  • viewer: the viewer plane, which readers read from with a token.
  • session: the session plane, where the operator credential or an API key holding authorise-as makes and revokes tokens.
  • control: the control plane, where the operator declares and writes.

ServeRefused

Bases: Refusal

The Refusal raised when the server a database starts does not report the addresses it listens on.

Database.serve() raises it, as does any call that starts the server, such as the first commit(). A server that prints no line naming its viewer, session and control addresses within the wait is stopped, and the message holds the last lines it wrote to its output and error streams. A server whose line leaves an address out is stopped too, and the message quotes the line. The wait is 30 seconds unless the MOSAICA_SERVE_TIMEOUT environment variable names another, so raise it on a machine where the server takes longer to start.

Control 

Control(base: str, credential: str | None = None, timeout: float = 300.0)

A client for the control plane of a served database, where the operator declares, writes, deletes and manages who may read.

Database.control returns one for a database you made, and Control(base, credential) makes one for a database someone else runs. Each route method sends one HTTP request and returns its Answer without raising, so a refusal is read from the answer's status and detail. The exceptions are await_compaction, which reads compaction_request until the request is answered and returns how that ended, and compact(wait=True), which calls it. Both raise Refusal for a progress they do not take, and pass on what a progress function raises. When a busy server answers 429, the method waits as long as the server asks, up to 30 seconds, and sends the request again. It returns the 429 once the waits would pass five minutes in all.

  • base: the control plane's address, such as http://127.0.0.1:41234.
  • credential: the operator credential, an API key or an OIDC access token, or None for a server started with mosaica serve --open, which needs none. A method that writes, flushes or compacts needs the write permission, one that declares or drops needs write-all as well, and status and the catalogue methods need admin. A credential that holds write without write-all can name only what its terms let it see.
  • timeout: how long to wait for one answer, in seconds. The default is 300.

The declaring methods take a block of a declaration in JSON, as mosaica check --payloads prints it. Declaring a thing again exactly as it stands answers 200 and changes nothing, so a declaration is safe to send twice. A different declaration under a name in use is refused, except where a method says otherwise.

The catalogue methods manage who may read and write. A catalogue change ends every session whose access it could alter, and its answer counts them in sessions_ended; create_key answers with the new key instead.

status 

status() -> dict

Return the server's status report as a dictionary (GET /control/status).

overlay counts the deletions and suppressions of items that no compaction has yet removed, and compaction reports the requests compact makes. When the server refuses the request, the dictionary is its error body, {"error": ..., "detail": ...}, and it is {} only when no server answered. Call status_answer() to read a refusal's status.

status_answer 

status_answer() -> Answer

Return the server's status report as an Answer (GET /control/status).

Use it to read the status and detail of a refusal, which status() does not give.

limits 

limits() -> dict

Return the limits on what one request to each write route may carry, so that a client can size its requests.

The limits are the limits block of status(), read once and kept for the life of this Control. A request over a limit is refused with 422.

ingest 

ingest(
    body: bytes, batch: str, view: str | None = None, strict: bool = False
) -> Answer

Create items, edit them or add them to a view, from a batch of rows (POST /control/ingest).

  • body: the rows, as the bytes of an Arrow IPC stream.
  • batch: an id for this batch, made fresh for each new one. The server answers the same id with the same bytes as a replay that writes nothing twice, which makes a retry safe, and refuses the same id with other bytes. When no server answers, the method sends the batch once more under its id.
  • view: the view the rows are for, which may be left out when the database has one view.
  • strict: True refuses the whole batch with 409 at its first refused row, so that nothing is written.

Each row names its item by its mosaica_id and its values of the unique columns. A row that names no item creates one at the row's position, while a row that names an item edits it, so that the item keeps its mosaica_id and joins this view if it had no row there. An edit changes only the columns the row carries, and a null clears a stored value.

A row is refused when it names two items, when its mosaica_id names no item, when it names no item and carries no coordinates, or when it names an item or sets a unique value that an earlier row of the batch did. The answer counts the rows applied as created, edited, added and unchanged, lists the refused rows in refused by position and reason, and gives each row's mosaica_id in mosaica_ids, with None for a refused row. A row whose only change places its item in an annotation is counted unchanged, and the membership it adds is counted in joined.

declare_layer 

declare_layer(payload: dict) -> Answer

Declare an annotation layer, a named set of annotations over items, such as clusters with their labels (PUT /control/layers).

Declaring a new layer, whose name may be that of a dropped one, answers 201 with the layer's mosaica_id. When every layer space is in use, or the layer declares more levels than one space holds, the declaration is refused with 409.

declare_view_group 

declare_view_group(name: str, body: dict) -> Answer

Declare a view group, a set of views that share every setting and differ by a key (PUT /control/view_groups/{name}).

The group starts with no views, and create_view adds them.

declare_attribute 

declare_attribute(body: dict) -> Answer

Declare an attribute, a column of values that items carry (PUT /control/attributes).

body names the column, and a new column holds no value for any item until a write gives it one. unique is the one field a declared column may change. Declaring it True builds the column's index over every stored value, and the answer waits for the build. When two items hold the same value, the declaration is refused with 409.

declare_shards 

declare_shards(body: dict) -> Answer

Declare the shard size, the number of row ids a shard hands out to new items before it is sealed (PUT /control/shards).

body is {"size": n}. A new size replaces the old one, and a size no larger than the number of row ids the open shard has handed out seals that shard at once.

declare_vocabulary 

declare_vocabulary(name: str, body: dict) -> Answer

Declare a vocabulary, the set of values a category column takes (PUT /control/vocabularies/{name}).

Declaring a vocabulary again as it stands adds the values it sends, as vocabulary_values does, and changes nothing else.

vocabulary_values 

vocabulary_values(name: str, body: dict) -> Answer

Add a page of values to a vocabulary, or change their titles (PATCH /control/vocabularies/{name}/values).

body holds the page as rows of {key, title?}. A key the vocabulary holds keeps its code and takes the title sent, and a page removes nothing.

declare_view 

declare_view(name: str, body: dict) -> Answer

Declare a view that belongs to no view group (PUT /control/views/{name}).

A view is a coordinate system that items are placed in. This one starts with no rows and takes them at its first flush, and it cannot be dropped.

create_view 

create_view(group: str, key: str, body: dict) -> Answer

Add a view to a view group (PUT /control/views/{group}/{key}).

  • group, key: the group's name and the new view's key, which together name the view <group>:<key>.
  • body: the view's roster record, which holds its metadata and, where it has one, its visibility.

Because the record cannot change once the view exists, sending the record a live view holds answers 200 with that view, and another record for its key is refused with 409. A dropped key may be used again.

publish 

publish(layer: str, body: bytes, strict: bool = False) -> Answer

Publish annotations into one level of a layer (PUT /control/layers/{layer}/artifacts).

  • body: the request's JSON, as bytes, in which each annotation names its members in a member table.
  • strict: True refuses the whole request at its first refused member, with 404 or 409.

A member that names no item, or two, is left out and listed in the answer's refused, and the rest of the request is applied as one change. An annotation whose key the level already holds is not created again, and the members sent join it. The answer gives each annotation's mosaica_id in the order sent. A request that would create more annotations than the level's layer space has ids left for is refused with 409.

grow 

grow(layer: str, body: bytes, strict: bool = False) -> Answer

Change the annotations a level already holds (PATCH /control/layers/{layer}/artifacts).

body is the request's JSON, as bytes. It can add members, fill parts an annotation lacks, and take items out of the set a label's text was written from. Members are refused as for publish, and strict works as it does there. A part that differs from the one the annotation holds is refused with 409, and a key no annotation holds is refused with 422, because this method creates nothing.

changes 

changes(items: list[dict], strict: bool = False) -> Answer

Delete, suppress or unsuppress items, annotations and layers (POST /control/changes).

  • items: one record per change, {"op": ..., "match": {column: value, ...}}. op is "delete", "suppress" or "unsuppress", and match names an item by its mosaica_id and its values of unique columns, as Database.addresses returns them, or an annotation or a layer by its mosaica_id.
  • strict: True refuses the whole request at its first change that names no item, with 404, or names two, with 409.

A change that names no item, or two, is listed in the answer's refused, and the others are applied. A deletion or suppression applies to every request that starts after the server accepts it. The server never answers this route 429, so a busy server does not hold a deletion back. A request that deletes a layer is refused with 422, because drop_layer removes a layer.

drop_layer 

drop_layer(name: str, wait: bool = False) -> Answer

Remove an annotation layer with all its annotations (DELETE /control/layers/{name}).

Readers stop being served the layer at once. Its name may be declared again afterwards, as a new layer that none of the dropped layer's mosaica_ids names. wait holds the answer until the publication carrying the drop is visible, as on flush.

drop_view 

drop_view(group: str, key: str, wait: bool = False) -> Answer

Remove one view of a view group, with the annotations scoped to it (DELETE /control/views/{group}/{key}).

The items the drop leaves in no view are deleted, and the answer's deleted counts them. wait is as for flush.

compact 

compact(
    wait: bool = False, progress: bool | Callable[[dict], Any] = True
) -> Answer

Ask the server for a compaction, which removes the rows of deleted items from disc (POST /control/compact), and wait for its outcome where wait is True.

The server answers 202 with the request's number in body["request"], or 503 where no compaction can start until it restarts, and then compacts every shard with something to reclaim. compaction_request reads where the request is.

  • wait: True passes the number to await_compaction and returns the 202 as the server sent it, with waited set to what await_compaction returned.
  • progress: what await_compaction takes. It is checked before the request is sent.

compaction_request 

compaction_request(request: int) -> Answer

Return where one compaction request is, pending or answered (GET /control/compact/{request}).

request is the number compact answered. The route's description in docs/openapi/mosaica.yaml says what the body holds and when it is 404 or 410.

await_compaction 

await_compaction(
    request: int, progress: bool | Callable[[dict], Any] = True
) -> dict

Wait until the server has answered compaction request number request, reading compaction_request(request) about once a second, and return how the wait ended.

  • progress: how the wait shows the compaction serving the request, the route's job, as Database.commit's progress takes it. The default is True. Once the request is answered, the last record shown is that of the compaction that answered it, where one ran. A compaction shorter than the interval between two readings is shown only by that record.

A reading that no server answered is read again a second later, so a server that restarts while the call waits ends the wait as lost once it is back. The wait has no bound of its own: to stop it, interrupt it, or call compaction_request in a loop of your own instead.

The dictionary returned has ok, which is True only where the request succeeded, and ended, which says how the wait ended:

  • answered: answer and job are the route's, as served, and ok is the answer's.
  • lost: the route answered 404, because the server restarted after the request was made. read is that Answer. Ask again.
  • gone: the route answered 410, because the answer is older than those the server keeps. read is that Answer.
  • unread: the route answered another status outside the 200s, such as 503 once the server's write executor has stopped, and read is that Answer.

flush 

flush(wait: bool = False) -> Answer

Make the writes sent so far visible to readers sooner, by bringing the next publication forward (POST /control/flush).

Writes become visible to readers at a publication, and the answer's publication names the first one certain to include every request sent before this one.

  • wait: True holds the answer until readers see that publication, or until the server's serve.visible_wait_max_secs passes, in which case the answer's visible is false.

list_principals 

list_principals() -> Answer

List every local principal (GET /control/principals).

The answer's principals holds one record per principal, with its name, kind, disabled, has_password, and the terms, permissions and groups granted to it directly. The answer holds no password or key.

show_principal 

show_principal(name: str) -> Answer

Return one principal's record, as list_principals describes it (GET /control/principals/{name}).

The answer's status is 404 when no principal has the name.

create_principal 

create_principal(name: str, kind: str) -> Answer

Create a local principal, a named person or service that can be granted access (POST /control/principals).

  • name: the principal's name. A name already in use is refused with 409.
  • kind: "person" or "service".

The principal starts with no password, key, term or permission.

change_principal 

change_principal(name: str, disabled: bool) -> Answer

Disable or enable a principal (PATCH /control/principals/{name}).

  • disabled: True disables the principal, and False enables it again.

Disabling a principal ends its sessions and the tokens it made for others with authorise, and the server accepts none of its credentials until it is enabled.

delete_principal 

delete_principal(name: str) -> Answer

Delete a principal with its password, API keys, grants and group memberships (DELETE /control/principals/{name}).

set_password 

set_password(name: str, password: str) -> Answer

Set a principal's password, replacing any it had (PUT /control/principals/{name}/password).

A password shorter than the server's catalogue.min_password_length is refused with 422.

clear_password 

clear_password(name: str) -> Answer

Remove a principal's password (DELETE /control/principals/{name}/password).

No password logs the principal in afterwards.

list_keys 

list_keys(principal: str) -> Answer

List a principal's API keys (GET /control/principals/{name}/keys).

The answer holds them under keys, without their secrets.

create_key 

create_key(
    principal: str,
    expires_at: int | None = None,
    permissions: Sequence[str] | None = None,
) -> Answer

Issue an API key for a principal (POST /control/principals/{name}/keys).

  • principal: the principal the key authenticates.
  • expires_at: when the key expires, in seconds since 1970, which also ends every session made with it. Left out, the key does not expire.
  • permissions: the permissions the key is limited to, which never exceed its principal's. Left out, the key holds its principal's.

The answer's key is the whole key, which no later answer shows again, so keep it when you make it. prefix names the key to list_keys and revoke_key.

revoke_key 

revoke_key(prefix: str) -> Answer

Revoke an API key by its prefix (DELETE /control/keys/{prefix}).

Revoking a key ends every session made with it, including the tokens authorise made with it.

list_groups 

list_groups() -> Answer

List every local group, with its terms, permissions and members, under groups in the answer (GET /control/groups).

show_group 

show_group(name: str) -> Answer

Return one group's record (GET /control/groups/{name}).

The answer's status is 404 when no group has the name.

create_group 

create_group(name: str) -> Answer

Create a local group, whose members hold every term and permission granted to it (POST /control/groups).

A name already in use is refused with 409.

delete_group 

delete_group(name: str) -> Answer

Delete a group with its grants and memberships (DELETE /control/groups/{name}).

add_member 

add_member(group: str, principal: str) -> Answer

Add a principal to a group (PUT /control/groups/{name}/members/{principal}).

remove_member 

remove_member(group: str, principal: str) -> Answer

Remove a principal from a group (DELETE /control/groups/{name}/members/{principal}).

grant 

grant(
    principal: str | None = None,
    group: str | None = None,
    term: str | None = None,
    terms: Sequence[str] | None = None,
    permission: str | None = None,
) -> Answer

Grant access terms or a permission to a principal or a group (POST /control/grants).

  • principal, group: the grantee.
  • term, terms: an access term, which says what the grantee may see, or several, which are granted as one change and refused together if one is refused.
  • permission: what the grantee may do, one of "read", "write", "authorise-as", "admin", "read-all" and "write-all".

Name exactly one grantee and exactly one of term, terms and permission. The server refuses any other combination with 422. A grant already held changes nothing.

revoke_grant 

revoke_grant(
    principal: str | None = None,
    group: str | None = None,
    term: str | None = None,
    terms: Sequence[str] | None = None,
    permission: str | None = None,
) -> Answer

Take back terms or a permission that grant gave, with the same arguments (POST /control/grants/revoke).

A grant that is not held changes nothing.

list_providers 

list_providers() -> Answer

List every OIDC provider, whether declared through this API or in mosaica.toml, under providers in the answer (GET /control/providers).

show_provider 

show_provider(name: str) -> Answer

Return one OIDC provider's record (GET /control/providers/{name}).

The answer's status is 404 when no provider has the name.

put_provider 

put_provider(
    name: str,
    issuer: str,
    audience: str,
    jwks_url: str,
    claim_rules: Sequence[dict] | None = None,
    role_mappings: Sequence[dict] | None = None,
) -> Answer

Declare an OIDC provider whose access tokens the database accepts, or replace one whole (PUT /control/providers/{name}).

  • issuer, audience: the iss the provider's tokens carry and the aud they must hold.
  • jwks_url: where the provider publishes its signing keys, over https, or over http to a loopback address.
  • claim_rules: rules that turn a claim's values into access terms, each {"claim": ..., "template": ...}, such as {"claim": "groups[*]", "template": "{value}"}.
  • role_mappings: rules that give an identity a local group's terms and permissions when one of its claims holds a value exactly, each {"claim": ..., "value": ..., "group": ...}.

Replacing a provider ends every session authorised through it. A provider declared in mosaica.toml is refused with 409, because it is changed there.

drop_provider 

drop_provider(name: str) -> Answer

Remove an OIDC provider, which ends every session authorised through it (DELETE /control/providers/{name}).

A provider declared in mosaica.toml is refused with 409, because it is removed there.

list_sessions 

list_sessions(
    principal: str | None = None, provider: str | None = None
) -> Answer

List the live sessions under sessions in the answer (GET /control/sessions).

  • principal: lists only a local principal's sessions, including those authorise made for it.
  • provider: lists only the sessions authorised through an OIDC provider.

Name at most one of them, and leave both out to list every session.

end_sessions 

end_sessions(
    token_id: int | None = None,
    principal: str | None = None,
    provider: str | None = None,
) -> Answer

End one session, or every session of a local principal or an OIDC provider (POST /control/sessions/end).

Name exactly one of token_id, principal and provider. A session's token_id is in list_sessions and on the Token that authorise made, and an ended session's token stops working.

Answer dataclass

The answer to one request a Control sent, whether the server accepted it, refused it or never answered.

  • status: the HTTP status, or 0 when no server answered.
  • body: the answer's JSON object. It is {} when the answer was not JSON, and a JSON value that is not an object is under "value".
  • detail: the answer's text, or why no answer came.
  • attempts: how many times the request was sent, which is more than 1 when the server asked the client to wait with a 429, or when an ingest that got no answer was sent again.
  • seconds: how long the request took, from its first attempt to its answer.
  • waited: for Control.compact(wait=True) where the server accepted the request, how the wait for its outcome ended, as compact describes, and None for every other answer.

ok property 

ok: bool

True when the status is in the 200s, which means the server accepted the request.