Data Connectors¶
SlideFlow supports seven connector types for chart/replacement data sources:
csvjsondatabricksduckdbredshiftdbt(composable, preferred)databricks_dbt(legacy, still supported)
Use these in any data_source block for charts or replacements.
For a step-by-step migration from legacy databricks_dbt to composable dbt, see DBT Migration Guide.
Installation extras¶
Install connector dependencies based on the connectors you use:
# Base package
pip install slideflow-presentations
# Databricks SQL connector
pip install "slideflow-presentations[databricks]"
# dbt connectors (includes dbt-core adapter stack + Git clone support)
pip install "slideflow-presentations[dbt]"
# Optional dbt warehouse backends
pip install "slideflow-presentations[bigquery]"
pip install "slideflow-presentations[duckdb]"
pip install "slideflow-presentations[redshift]"
Connector Matrix¶
| Type | Best for | Requires network | Required env vars |
|---|---|---|---|
csv | local tabular files | no | none |
json | local API exports/events | no | none |
databricks | direct warehouse SQL | yes | DATABRICKS_HOST, DATABRICKS_HTTP_PATH, DATABRICKS_ACCESS_TOKEN |
duckdb | direct local/in-memory SQL | no | none |
redshift | direct Redshift SQL | yes | REDSHIFT_HOST, REDSHIFT_DATABASE, REDSHIFT_USER, REDSHIFT_PASSWORD or IAM env/config |
dbt | dbt model SQL executed on Databricks, BigQuery, DuckDB, or Redshift (composable config) | yes/no (depends on warehouse and dbt repo) | warehouse-specific env vars (+ Git token env if needed) |
databricks_dbt | dbt model SQL executed on Databricks | yes | same Databricks env vars (+ Git token env if needed) |
CSV¶
Notes:
- Uses
pandas.read_csvwith default parsing behavior. - Relative paths resolve from your current execution directory.
JSON¶
Supported orient values:
splitrecordsindexcolumnsvaluestable
If your JSON shape does not match orient, parsing fails.
Databricks SQL¶
data_source:
type: "databricks"
name: "warehouse_query"
query: |
SELECT month, revenue, target
FROM mart.revenue_summary
WHERE fiscal_quarter = '{quarter}'
# optional connector runtime overrides:
# socket_timeout_s: 300
# retry_max_attempts: 30
# retry_max_duration_s: 900
# retry_delay_min_s: 1
# retry_delay_max_s: 60
Required environment:
export DATABRICKS_HOST="<workspace-hostname>"
export DATABRICKS_HTTP_PATH="<sql-warehouse-http-path>"
export DATABRICKS_ACCESS_TOKEN="<token>"
Optional Databricks connector runtime env tuning:
SLIDEFLOW_DATABRICKS_SOCKET_TIMEOUT_SSLIDEFLOW_DATABRICKS_RETRY_MAX_ATTEMPTSSLIDEFLOW_DATABRICKS_RETRY_MAX_DURATION_SSLIDEFLOW_DATABRICKS_RETRY_DELAY_MIN_SSLIDEFLOW_DATABRICKS_RETRY_DELAY_MAX_S
Databricks request identification:
- SlideFlow sets Databricks SQL
user_agent_entrytoSlideflowso warehouse query history can attribute sessions to SlideFlow runs.
Tips:
- Keep SQL deterministic for reporting workflows.
- Limit columns to what chart/replacement logic needs.
- Prefer validated parameter substitution (
{quarter}from batch params) over string concatenation.
DuckDB SQL¶
data_source:
type: "duckdb"
name: "local_duckdb_query"
database: "/tmp/analytics.duckdb" # optional; defaults to ':memory:'
read_only: true # optional; defaults to true
file_search_path: # optional; used for relative file references in DuckDB
- "/tmp/data"
- "/tmp/snapshots"
query: |
SELECT * FROM sales_summary
Notes:
- Install DuckDB runtime deps:
pip install "slideflow-presentations[duckdb]". file_search_pathcan be a list or a comma-separated string.- If
file_search_pathis omitted, DuckDB uses its default file search behavior.
Redshift SQL¶
data_source:
type: "redshift"
name: "warehouse_query"
host: "example-cluster.abc123.us-east-1.redshift.amazonaws.com"
port: 5439 # optional; defaults to 5439
database: "analytics"
user: "reporting_user"
ssl: true # optional; defaults to true
query: |
SELECT month, revenue, target
FROM mart.revenue_summary
Non-IAM environment fallback:
export REDSHIFT_HOST="<cluster-host>"
export REDSHIFT_PORT="5439"
export REDSHIFT_DATABASE="<database>"
export REDSHIFT_USER="<user>"
export REDSHIFT_PASSWORD="<password>"
IAM auth can be configured in YAML or environment. For provisioned clusters:
data_source:
type: "redshift"
name: "redshift_iam_query"
database: "analytics"
iam: true
cluster_identifier: "prod-analytics"
region: "us-east-1"
db_user: "reporting_user"
query: "SELECT * FROM mart.revenue_summary"
For Redshift Serverless IAM auth, use serverless_acct_id and serverless_work_group; SlideFlow derives is_serverless: true when either field is present.
Useful Redshift env fallbacks:
REDSHIFT_IAMREDSHIFT_DB_USERREDSHIFT_CLUSTER_IDENTIFIERREDSHIFT_REGION,AWS_REGION, orAWS_DEFAULT_REGIONAWS_DEFAULT_REGIONREDSHIFT_PROFILEorAWS_PROFILEREDSHIFT_ACCESS_KEY_IDorAWS_ACCESS_KEY_IDREDSHIFT_SECRET_ACCESS_KEYorAWS_SECRET_ACCESS_KEYREDSHIFT_SESSION_TOKENorAWS_SESSION_TOKENREDSHIFT_SERVERLESS_ACCT_IDREDSHIFT_SERVERLESS_WORK_GROUPREDSHIFT_SSLREDSHIFT_SSLMODEREDSHIFT_TIMEOUT
Notes:
- Install Redshift runtime deps:
pip install "slideflow-presentations[redshift]". - SSL is enabled by default.
- Keep passwords and AWS secrets in environment variables or a secret manager, not in committed YAML.
- SlideFlow sets
application_nametoSlideflowby default for warehouse query attribution.
dbt on Databricks (dbt, preferred)¶
This connector compiles a dbt project, resolves a model's compiled SQL, then executes it on Databricks.
data_source:
type: "dbt"
name: "dbt_model"
model_alias: "monthly_revenue_by_region"
dbt:
package_url: "https://$GIT_TOKEN@github.com/org/analytics-dbt.git"
project_dir: "/tmp/dbt_project_workspace"
branch: "main"
target: "prod"
vars:
start_date: "2026-01-01"
end_date: "2026-01-31"
profiles_dir: "/path/to/profiles"
profile_name: "analytics"
warehouse:
type: "databricks"
Behavior highlights:
- Repositories are cloned under
project_dir/.slideflow_dbt_clones/<key>. - The clone key covers repository, branch, and profile inputs.
targetandvarsdo not trigger another clone ordbt depsrun. - Each unique
target/varscombination writes to an isolatedproject_dir/.slideflow_dbt_targets/<workspace-key>/<variant-key>directory. This sibling tree prevents variants from overwriting each other and keeps generated SQL outside the parsed dbt project, including for projects with broad model paths such asmodel-paths: ['.']. - A workspace-keyed filesystem lock under
project_dir/.slideflow_dbt_lockscoordinates preparation, active artifact use, and cleanup across SlideFlow processes. Threads in one process still share the existing lease and single-flight coordination. - Managed artifact paths must be real contained directories. SlideFlow rejects symlinks, non-directory collisions, and paths that resolve outside their expected workspace. The concrete deps log, variant target, and variant log paths are revalidated immediately before every dbt invocation.
- A private preparation marker is written after clone and
dbt depscomplete. Cache hits require a realdbt_project.yml, the same checked-out Git commit, and the in-memory generation originally recorded for the marker. If any of those values or the clone changes, every cached target/vars variant for that shared workspace is invalidated and rebuilt from one new generation. - Evicted paths remain reserved until lease-safe cleanup finishes, preventing a concurrent request from recreating a directory that cleanup could later delete. Cache bounds are checked again when active leases release.
project_diris treated as a workspace root, not a direct clone target.- Default
compile: truerunsdbt deps, parses the full project, resolves exact requested nodes, and runs one scopeddbt compile --selectfor the normalized model batch. It does not prepend+or build upstream relations. compile: falsenever clones and never runs dbt. In that mode,project_dirmust already be a compiled dbt project containingtarget/manifest.jsonand the compiled SQL files referenced by the manifest.- If
package_urlembeds$TOKEN_NAME, that env var must exist at runtime. - If
profiles_diris provided, SlideFlow copies profiles into the cloned dbt workspace and runs dbt with--profiles-dir <clone_dir>. - If
profiles_diris omitted but the cloned repo containsprofiles.ymlat project root, SlideFlow auto-uses that project-root profiles file. - Clone/dependency work is single-flight across variants. Each variant is parsed once, and all requested analyses/models for that variant are compiled as one sorted, deduplicated selector batch. Cached supersets satisfy later subset requests without recompiling.
- In-process dbt invocations are serialized because dbt-core maintains process globals; warehouse queries remain governed separately by
query_threads. model_aliasresolves a target-invariant manifestnamefirst and retains compiledaliasas a compatibility fallback. If the identifier is ambiguous, set one of:model_unique_idmodel_package_namemodel_selector_nameto avoid ambiguity errors.
dbt on BigQuery (dbt)¶
This connector shape compiles a dbt project, resolves a model's compiled SQL, then executes it on BigQuery.
data_source:
type: "dbt"
name: "dbt_model_bigquery"
model_alias: "monthly_revenue_by_region"
dbt:
package_url: "https://$GIT_TOKEN@github.com/org/analytics-dbt.git"
project_dir: "/tmp/dbt_project_workspace"
branch: "main"
target: "prod"
vars:
start_date: "2026-01-01"
end_date: "2026-01-31"
profiles_dir: "/path/to/profiles"
profile_name: "analytics"
warehouse:
type: "bigquery"
project_id: "my-gcp-project" # optional if BIGQUERY_PROJECT/GOOGLE_CLOUD_PROJECT set
location: "US" # optional
timeout: 120 # optional query/result timeout in seconds
credentials_path: "/path/to/service-account.json" # optional
# credentials_json: '{"type":"service_account",...}' # optional alternative
BigQuery runtime options:
- Install BigQuery runtime dependencies:
pip install "slideflow-presentations[bigquery]". - Set project id via:
warehouse.project_id, orBIGQUERY_PROJECT, orGOOGLE_CLOUD_PROJECT.- Auth options:
warehouse.credentials_path, orwarehouse.credentials_json, or- Application Default Credentials (for example
GOOGLE_APPLICATION_CREDENTIALS). warehouse.timeoutoptionally bounds BigQuery query submission and result download waits. When omitted, SlideFlow leaves the BigQuery SDK timeout behavior unchanged.- SlideFlow initializes the BigQuery client with
client_info.user_agentset toSlideflowfor request attribution in Google-side telemetry.
dbt on Redshift (dbt)¶
This connector shape compiles a dbt project, resolves a model's compiled SQL, then executes it on Redshift.
data_source:
type: "dbt"
name: "dbt_model_redshift"
model_alias: "monthly_revenue_by_region"
dbt:
package_url: "https://$GIT_TOKEN@github.com/org/analytics-dbt.git"
project_dir: "/tmp/dbt_project_workspace"
branch: "main"
target: "prod"
warehouse:
type: "redshift"
host: "example-cluster.abc123.us-east-1.redshift.amazonaws.com"
database: "analytics"
user: "reporting_user"
ssl: true
Redshift warehouse options mirror the direct redshift connector. Omit credential fields from YAML to use the REDSHIFT_* or AWS_* environment fallbacks listed above.
dbt on DuckDB (dbt)¶
This connector shape compiles a dbt project, resolves a model's compiled SQL, then executes it on DuckDB.
data_source:
type: "dbt"
name: "dbt_model_duckdb"
model_alias: "monthly_revenue_by_region"
dbt:
package_url: "https://$GIT_TOKEN@github.com/org/analytics-dbt.git"
project_dir: "/tmp/dbt_project_workspace"
branch: "main"
target: "prod"
warehouse:
type: "duckdb"
database: "/tmp/warehouse.duckdb" # required for dbt+duckdb
read_only: true # optional; defaults to true
file_search_path: # optional
- "/tmp/dbt_project_workspace"
- "/tmp/data"
Legacy dbt on Databricks (databricks_dbt)¶
This legacy shape is still fully supported for backward compatibility.
data_source:
type: "databricks_dbt"
name: "dbt_model_legacy"
model_alias: "monthly_revenue_by_region"
package_url: "https://$GIT_TOKEN@github.com/org/analytics-dbt.git"
project_dir: "/tmp/dbt_project_workspace"
branch: "main"
target: "prod"
vars:
start_date: "2026-01-01"
end_date: "2026-01-31"
Caching and Execution¶
SlideFlow caches connector fetches by config identity, which helps when:
- multiple charts use the same query/file in one run
- multiple replacements reuse one source
Cached DataFrame values are copied on cache write and copied again on cache reads. This keeps table formatting, chart transforms, and custom replacements from mutating shared cached data. The tradeoff is extra memory and copy time for large frames, so prefer selecting only the columns and rows needed for the artifact before handing data to SlideFlow.
Treat connectors as read-only sources during a run for predictable results. clear() and disable() also wake in-flight waiters; a waiter will either reload against the current cache state or run uncached when the cache has been disabled.
Cache/compile tuning env vars:
SLIDEFLOW_DATA_CACHE_MAX_ENTRIES(global source cache cap)SLIDEFLOW_DBT_CACHE_MAX_ENTRIES(bounds compiled variants under.slideflow_dbt_targetsand inactive prepared workspaces under.slideflow_dbt_clones; default from built-in constants)SLIDEFLOW_DBT_COMPILE_FAILURE_BACKOFF_SSLIDEFLOW_DBT_FAILURE_CACHE_MAX_ENTRIES
Recommended Workflow¶
- Start with local
csv/jsonwhile designing charts and replacements. - Move to
duckdb,databricks, orredshiftonce schema and logic are stable. - Move to
dbtwhen business logic should live in dbt models (databricks_dbtremains supported as legacy syntax). - Run
slideflow validatebeforeslideflow buildin CI/CD.
Troubleshooting¶
- File connector errors: check file existence and relative path assumptions.
- Databricks auth errors: verify all three Databricks env vars.
- Redshift auth/config errors: verify database, host/cluster/serverless endpoint, and either password auth or IAM region/identity settings.
- dbt model not found: check
model_alias,branch, andtarget. - dbt alias ambiguity: add
model_unique_id,model_package_name, ormodel_selector_name. - dbt Git clone fails: verify token variable in
package_urland repo access. - dbt
compile:falsefails: ensureproject_dir/target/manifest.jsonexists and every selected manifest node has a compiled SQL file on disk.