External drives

External drives writes clean, ready-to-query tables from your Library into storage your organization owns, so your data team can read them in Databricks, Snowflake, Microsoft Fabric or any tool that reads Delta Lake, Apache Iceberg or Parquet.

Who can see this: organization admins, when External drives is turned on for your organization. Open Settings from your avatar menu, then Organization › External drives.

Beta

External drives is in beta, and Querri turns it on one organization at a time. To try it, contact us at support@querri.com. Until it’s on, the page doesn’t appear in Settings.

1. The warning: “A workspace sync exports the workspace’s data without row-level security: anyone who can read the bucket can read every row. That is why only admins can set one up.”
2. Drives: the buckets Querri can write to.
3. Workspace syncs: which workspace goes to which drive, and when.

How it works

There are two parts:

• A drive is a bucket you own: an Amazon S3 bucket, or S3-compatible storage such as Cloudflare R2, MinIO or Wasabi. You connect it once.
• A workspace sync keeps a copy of one workspace’s tables in a folder on a drive, updated on a schedule or whenever the Library finishes updating.

Projects can also send a step’s table to a drive, if an admin opens the drive to project exports. See Export a project step to a drive.

Querri writes into your bucket. You never get access to Querri’s own storage.

Before you start

For an Amazon S3 drive you need:

• An S3 bucket in an AWS account you control. You can give Querri one folder in it rather than the whole bucket.
• Permission to create an IAM role in that account, either through CloudFormation in the AWS console or with Terraform.
• The bucket’s KMS key ARN, if the bucket is encrypted with your own AWS KMS key.

For S3-compatible storage you need the bucket name, the provider’s https:// endpoint URL, and an access key ID and secret that can read, write, delete and list in the bucket. The endpoint has to be a public address.

Add a drive

1. Click Add drive.
2. Fill in Name, for example “Analytics lake”. People choosing a drive for a project export see this name.
3. Under Storage, choose Amazon S3 or S3-compatible.
4. Fill in Bucket. Folder prefix is optional: set it, for example exports/, to keep Querri inside one folder of the bucket.
5. Region is optional for Amazon S3. Leave it blank and Querri detects it when you test the connection.
6. For S3-compatible, also fill in Endpoint URL, Access key ID and Secret access key.
7. Choose Encryption: Bucket default, Amazon S3 managed (SSE-S3) or AWS KMS key (SSE-KMS). For a KMS key, fill in KMS key ARN.
8. Turn on Allow project exports if project members should be able to send step results to this drive. They see the drive’s name only, never its credentials.
9. Click Create and continue (Amazon S3) or Create and test (S3-compatible).

An S3-compatible drive goes straight to the connection test. An Amazon S3 drive goes on to the role setup next.

Connect it to your AWS account

An Amazon S3 drive uses a role in your AWS account, so no keys are stored. After you create the drive, Querri shows Create this role in your AWS account with three steps.

1. The role trusts this principal, with this External ID

Principal to trust is the AWS identity Querri uses. External ID is a value Querri creates for this drive. It never changes, and you can’t choose it. The role you create accepts Querri only when Querri presents this ID, which stops another Querri customer from pointing their drive at your role.

2. Create the role

Use whichever your team prefers:

• Launch CloudFormation opens the AWS console with the stack already filled in: Querri’s principal, the External ID, your bucket, folder prefix and KMS key. Sign in to the account that owns the bucket, acknowledge that the stack creates an IAM role, and create the stack. When it finishes, copy RoleArn from the stack’s Outputs tab.
• Download template saves the same CloudFormation template, to create the stack yourself in the bucket’s account. If there’s no Launch CloudFormation button, use this.
• Prefer Terraform? Show the block shows the same role as Terraform, ready to copy. Apply it and use the querri_export_role_arn output.

The role can write only to this bucket and folder prefix: read, write and delete files there, list that folder, and look up the bucket’s region. It gets KMS access only when the drive uses your KMS key.

3. Paste the role ARN and test

Paste the role’s ARN into Role ARN and click Save and test connection. When every check passes, click Done.

To come back to these steps later, click Edit on the drive, then Show role setup instructions.

Test the connection

Test connection runs six checks in order and stops at the first one that fails:

1. Credentials: Querri signs in with your role or access key.
2. Bucket region: Querri finds which region the bucket is in.
3. Write a test file: Querri writes a small file to _querri/ inside your folder prefix.
4. Read a test file: Querri reads that file back.
5. Delete a test file: Querri deletes it again.
6. List the folder: Querri lists your folder prefix.

When they all pass you see “Connected. Every check passed.” The drive’s status becomes Connected, and the drive row shows when it was last tested. A failed check shows its own message, so you know what to fix. See Troubleshooting the connection test.

Click Test on a drive’s row to run the checks again at any time.

Create a workspace sync

1. Under Workspace syncs, click Add sync.
2. Choose the Workspace and the Drive. Personal workspaces aren’t offered: they hold one member’s private data.
3. Check Folder in the drive. Querri suggests workspaces/ plus the workspace’s name. It sits inside the drive’s folder prefix and can’t overlap another sync’s folder.
4. Under What to sync, choose Views only (the workspace’s curated views, cleaned and ready to query) or All sources and views (raw sources too).
5. Under Formats, choose at least one. See Formats.
6. Under When to sync, choose:
   • When the Library finishes updating: Querri waits until the workspace has been quiet for the Quiet period (minutes), 5 by default, so it doesn’t export halfway through a refresh. It never waits longer than the Longest wait (minutes), 120 by default, so a workspace that never goes quiet still gets exported.
   • On a schedule: Hourly, Daily, Weekly, Monthly or Custom (cron), with a Time and Time zone.
7. Set Keep removed tables for (days), 7 by default. A table that leaves the workspace stays in the bucket this long before it’s purged.
8. Click Create sync.

The workspace, drive and folder can’t change once the sync exists. To move a sync, create a new one.

If any table in the workspace has access policies, the dialog warns you before you save: This export includes tables with access rules. Those tables are still exported, in full. See Security and access control.

Formats

• Delta Lake: read directly by Databricks, Fabric and Snowflake. On by default, and recommended.
• Apache Iceberg: adds Iceberg metadata for engines that prefer Iceberg tables. With Delta also on, both describe the same data files, so the data isn’t stored twice.
• Parquet: one plain Parquet file per table, for tools that read files rather than tables.

Each table gets its own folder, named from the table’s name when it’s first exported. Renaming the table in Querri later doesn’t move its folder, so tables you’ve registered elsewhere keep working. Column names are made safe for every engine, for example lowercase with underscores, and the original name is kept in the column’s comment.

Each sync folder also has _querri/manifest.json, a list of every table with its name, columns, row count and when it was exported.

What each run does

• Tables that haven’t changed are skipped. Querri writes nothing for them.
• Tables that changed are rewritten in full, then switched over in one step, so nobody reads half a refresh. The old files stay for the retention period, so queries already running can finish, then they’re deleted.
• Tables that left the workspace are dropped from the manifest straight away, and their folders are deleted once Keep removed tables for (days) passes.
• Sources with an error aren’t refreshed, and the run says which ones.

Each sync’s row shows the last run and buttons for Run now, History, Use in…, Edit and Delete. Use the switch to pause a sync without deleting it.

History opens Sync history, with each run’s status and every table’s Outcome: Written, Skipped, Failed, Removed or Purged. A run is Partly done when some tables failed and the rest landed. Deferred means the run waited because the Library was still updating, or another run of the same sync was in progress.

Export a project step to a drive

A drive with Allow project exports turned on is offered when someone sets up a step export in a project.

1. In a project, open the result card’s menu and choose Export Settings.
2. Turn on External drive.
3. Choose the Drive.
4. Fill in Folder (inside projects/), or leave it blank to use a folder named after the project. Project exports always land under projects/ on the drive, and can’t overlap a workspace sync’s folder.
5. Choose Formats.
6. Click Configure Export.

The export runs the next time the project runs. After that, the dialog shows when the step was last exported. Only tables can go to a drive: charts and Excel export steps can’t.

The External drive destination only appears once an admin has opened at least one drive to project exports. If a step already exports to a drive that has since been closed, the dialog says “No drive is open to project exports yet. An admin can allow it in Settings › External drives.” so you can turn the export off.

See Results for the result card and its menu.

Read the tables in Databricks, Snowflake or Fabric

After a sync has run, click Use in… on its row. Use these tables shows the folder’s Location and Manifest, then a tab for each platform the sync’s formats support, with the statements already written for your tables. Copy them from there rather than writing your own.

• Databricks (Delta Lake): create a storage credential for an IAM role that can read the bucket, and an external location on the URL shown. Then run the SQL in a SQL warehouse or notebook. It creates a querri schema with one table per export.
• Snowflake (Apache Iceberg): create an external volume for the bucket and a catalog integration, then one Iceberg table per export. After each Querri run, refresh each table with ALTER ICEBERG TABLE ... REFRESH and the metadata path from the manifest. A sync without Iceberg turned on shows a note to turn it on instead.
• Microsoft Fabric (Delta Lake): in a Lakehouse, add an Amazon S3 shortcut with an access key that can read the folder, one shortcut per table folder under Tables.
• Parquet files: the path of each table’s file.

Until the first run lands, the dialog says “There is nothing to read yet. Run the sync first.”

Security and access control

• Querri stays inside your folder. For an Amazon S3 drive, every session Querri opens with your role is narrowed to the drive’s bucket and folder prefix, even if the role itself allows more. For S3-compatible storage, Querri can do whatever the access key allows, so give the key access to that bucket only.
• You never read Querri’s storage. Querri copies the data into your bucket. No credentials to Querri’s storage are handed out.
• Exported tables have no row-level security. Querri’s access policies don’t travel with the data. Anyone who can read the bucket sees every row. Tables with access policies are still exported in full, and Querri flags them: the sync dialog lists them, the sync’s row shows a badge such as “2 tables without access rules”, each run lists them under Exported without access rules, and the manifest marks each one.
• Only admins set up drives and syncs. A sync can export more than any one member may see in Querri.
• Personal workspaces can’t be synced. They hold one member’s private data. See Workspaces and privacy.
• Deleting keeps your data. Deleting a sync stops the updates, and the tables already in the bucket stay. Deleting a drive keeps the data in the bucket too. A drive that a sync or project export still uses can’t be deleted.
• Access key secrets are encrypted when saved and never shown again. Leave the box as it is when editing to keep the stored secret.
• Querri won’t overwrite your own data. Before the first write to a folder, Querri checks it doesn’t already hold data Querri didn’t write, and stops if it does.

Troubleshooting the connection test

Each failed check shows a message. Here’s what each one means and what to change.

Check	Message	What to do
Credentials	”Enter the ARN of the role you created for Querri.”	Paste the role’s ARN into Role ARN.
Credentials	”Your role refused Querri. Check that its trust policy names Querri’s principal and requires exactly the External ID shown on this drive.”	The role’s trust policy doesn’t match this drive. Check Principal to trust and External ID, or create the role again from Launch CloudFormation.
Credentials	”The role ARN was rejected: …”	The ARN isn’t valid. Copy it again from the stack’s outputs. It looks like arn:aws:iam::123456789012:role/name.
Credentials	”Assuming your role failed: …”	AWS returned another error. The message includes it.
Credentials	”Querri could not assume its own external-drives role. This is a Querri configuration problem, not yours.”	Contact support@querri.com.
Credentials	”This drive has no access key saved.” or “The saved secret could not be decrypted; re-enter it.”	Edit the drive and enter the access key ID and secret again.
Bucket region	”Your role does not allow s3:GetBucketLocation.”	Add s3:GetBucketLocation on the bucket to the role.
Bucket region	”The bucket does not exist (or is in a different account than the role).”	Check the bucket name, and that the role is in the bucket’s account.
Bucket region	”The bucket is in …, but this drive is set to …”	Correct Region, or leave it blank so Querri detects it.
Write a test file	”Your role does not allow s3:PutObject on this prefix.”	Allow s3:PutObject on the folder prefix.
Any check	”The bucket’s KMS key does not allow your role to encrypt/decrypt.”	Let the role use the key, in the role’s policy and the key’s policy, and check KMS key ARN.
Read a test file	”Your role does not allow s3:GetObject on this prefix.”	Allow s3:GetObject on the folder prefix.
Read a test file	”The test file read back different bytes.”	Something changed the file between write and read. Check for bucket rules that rewrite objects.
Delete a test file	”Your role does not allow s3:DeleteObject on this prefix (needed to retire old files).”	Allow s3:DeleteObject. Querri needs it to remove old files after the retention period.
List the folder	”Your role does not allow s3:ListBucket for this prefix.”	Allow s3:ListBucket on the bucket for the folder prefix.
Any check	”The bucket is in a different region than this drive is set to.”	Correct Region, or leave it blank.
Any check	”The storage service refused the … step (…).”	The storage service returned another error, named in brackets. For S3-compatible storage, check the endpoint URL and the access key.

The role from Launch CloudFormation or the Terraform block already has every permission the test needs. If a permission check fails with that role, check that the folder prefix and bucket in the stack match the drive.

Related

• Workspaces
• Workspaces and privacy
• Access policies
• Results
• Settings overview