Data Security

An overview of data access controls.

Data for a Run is split between the object store (e.g. S3, GCS) and the database.

• Metadata (e.g. name, schema) and aggregate data (e.g. summary statistics, histograms) are stored in the database.

• Raw data is stored in the object store.

All data accesses are mediated by the API ensuring the enforcement of access controls. For more details on permissions, see Users and Permissions.

Database

Database access is always done through the API with the API enforcing access controls to ensure users only access data for which they have permission.

Object Store

Direct object store access is required to upload or download raw Run data using the SDK. Pre-signed URLs are used to provide limited direct access. This access is limited in both time and scope, ensuring only data for a specific Run is accessible and that it is only accessible for a limited time.

When uploading or downloading data for a Run, the SDK first sends a request for a pre-signed upload or download URL to the API. The API enforces access controls, returning an error if the user is missing the necessary permissions. Otherwise, it returns a pre-signed URL which the SDK then uses to upload or download the data.

Data upload

Uploading data to a Run in a given namespace requires write permission to Runs in that namespace. Downloading data from a Run in a given namespace requires read permission to Runs in that namespace.