Skip to content

Data Connectors

SlideFlow supports seven connector types for chart/replacement data sources:

  • csv
  • json
  • databricks
  • duckdb
  • redshift
  • dbt (composable, preferred)
  • databricks_dbt (legacy, still supported)

Use these in any data_source block for charts or replacements.

For a step-by-step migration from legacy databricks_dbt to composable dbt, see DBT Migration Guide.

Installation extras

Install connector dependencies based on the connectors you use:

# Base package
pip install slideflow-presentations

# Databricks SQL connector
pip install "slideflow-presentations[databricks]"

# dbt connectors (includes dbt-core adapter stack + Git clone support)
pip install "slideflow-presentations[dbt]"

# Optional dbt warehouse backends
pip install "slideflow-presentations[bigquery]"
pip install "slideflow-presentations[duckdb]"
pip install "slideflow-presentations[redshift]"

Connector Matrix

Type Best for Requires network Required env vars
csv local tabular files no none
json local API exports/events no none
databricks direct warehouse SQL yes DATABRICKS_HOST, DATABRICKS_HTTP_PATH, DATABRICKS_ACCESS_TOKEN
duckdb direct local/in-memory SQL no none
redshift direct Redshift SQL yes REDSHIFT_HOST, REDSHIFT_DATABASE, REDSHIFT_USER, REDSHIFT_PASSWORD or IAM env/config
dbt dbt model SQL executed on Databricks, BigQuery, DuckDB, or Redshift (composable config) yes/no (depends on warehouse and dbt repo) warehouse-specific env vars (+ Git token env if needed)
databricks_dbt dbt model SQL executed on Databricks yes same Databricks env vars (+ Git token env if needed)

CSV

data_source:
  type: "csv"
  name: "sales_csv"
  file_path: "./data/sales.csv"

Notes:

  • Uses pandas.read_csv with default parsing behavior.
  • Relative paths resolve from your current execution directory.

JSON

data_source:
  type: "json"
  name: "events_json"
  file_path: "./data/events.json"
  orient: "records"

Supported orient values:

  • split
  • records
  • index
  • columns
  • values
  • table

If your JSON shape does not match orient, parsing fails.

Databricks SQL

data_source:
  type: "databricks"
  name: "warehouse_query"
  query: |
    SELECT month, revenue, target
    FROM mart.revenue_summary
    WHERE fiscal_quarter = '{quarter}'
  # optional connector runtime overrides:
  # socket_timeout_s: 300
  # retry_max_attempts: 30
  # retry_max_duration_s: 900
  # retry_delay_min_s: 1
  # retry_delay_max_s: 60

Required environment:

export DATABRICKS_HOST="<workspace-hostname>"
export DATABRICKS_HTTP_PATH="<sql-warehouse-http-path>"
export DATABRICKS_ACCESS_TOKEN="<token>"

Optional Databricks connector runtime env tuning:

  • SLIDEFLOW_DATABRICKS_SOCKET_TIMEOUT_S
  • SLIDEFLOW_DATABRICKS_RETRY_MAX_ATTEMPTS
  • SLIDEFLOW_DATABRICKS_RETRY_MAX_DURATION_S
  • SLIDEFLOW_DATABRICKS_RETRY_DELAY_MIN_S
  • SLIDEFLOW_DATABRICKS_RETRY_DELAY_MAX_S

Databricks request identification:

  • SlideFlow sets Databricks SQL user_agent_entry to Slideflow so warehouse query history can attribute sessions to SlideFlow runs.

Tips:

  • Keep SQL deterministic for reporting workflows.
  • Limit columns to what chart/replacement logic needs.
  • Prefer validated parameter substitution ({quarter} from batch params) over string concatenation.

DuckDB SQL

data_source:
  type: "duckdb"
  name: "local_duckdb_query"
  database: "/tmp/analytics.duckdb" # optional; defaults to ':memory:'
  read_only: true # optional; defaults to true
  file_search_path: # optional; used for relative file references in DuckDB
    - "/tmp/data"
    - "/tmp/snapshots"
  query: |
    SELECT * FROM sales_summary

Notes:

  • Install DuckDB runtime deps: pip install "slideflow-presentations[duckdb]".
  • file_search_path can be a list or a comma-separated string.
  • If file_search_path is omitted, DuckDB uses its default file search behavior.

Redshift SQL

data_source:
  type: "redshift"
  name: "warehouse_query"
  host: "example-cluster.abc123.us-east-1.redshift.amazonaws.com"
  port: 5439 # optional; defaults to 5439
  database: "analytics"
  user: "reporting_user"
  ssl: true # optional; defaults to true
  query: |
    SELECT month, revenue, target
    FROM mart.revenue_summary

Non-IAM environment fallback:

export REDSHIFT_HOST="<cluster-host>"
export REDSHIFT_PORT="5439"
export REDSHIFT_DATABASE="<database>"
export REDSHIFT_USER="<user>"
export REDSHIFT_PASSWORD="<password>"

IAM auth can be configured in YAML or environment. For provisioned clusters:

data_source:
  type: "redshift"
  name: "redshift_iam_query"
  database: "analytics"
  iam: true
  cluster_identifier: "prod-analytics"
  region: "us-east-1"
  db_user: "reporting_user"
  query: "SELECT * FROM mart.revenue_summary"

For Redshift Serverless IAM auth, use serverless_acct_id and serverless_work_group; SlideFlow derives is_serverless: true when either field is present.

Useful Redshift env fallbacks:

  • REDSHIFT_IAM
  • REDSHIFT_DB_USER
  • REDSHIFT_CLUSTER_IDENTIFIER
  • REDSHIFT_REGION, AWS_REGION, or AWS_DEFAULT_REGION
  • AWS_DEFAULT_REGION
  • REDSHIFT_PROFILE or AWS_PROFILE
  • REDSHIFT_ACCESS_KEY_ID or AWS_ACCESS_KEY_ID
  • REDSHIFT_SECRET_ACCESS_KEY or AWS_SECRET_ACCESS_KEY
  • REDSHIFT_SESSION_TOKEN or AWS_SESSION_TOKEN
  • REDSHIFT_SERVERLESS_ACCT_ID
  • REDSHIFT_SERVERLESS_WORK_GROUP
  • REDSHIFT_SSL
  • REDSHIFT_SSLMODE
  • REDSHIFT_TIMEOUT

Notes:

  • Install Redshift runtime deps: pip install "slideflow-presentations[redshift]".
  • SSL is enabled by default.
  • Keep passwords and AWS secrets in environment variables or a secret manager, not in committed YAML.
  • SlideFlow sets application_name to Slideflow by default for warehouse query attribution.

dbt on Databricks (dbt, preferred)

This connector compiles a dbt project, resolves a model's compiled SQL, then executes it on Databricks.

data_source:
  type: "dbt"
  name: "dbt_model"
  model_alias: "monthly_revenue_by_region"
  dbt:
    package_url: "https://$GIT_TOKEN@github.com/org/analytics-dbt.git"
    project_dir: "/tmp/dbt_project_workspace"
    branch: "main"
    target: "prod"
    vars:
      start_date: "2026-01-01"
      end_date: "2026-01-31"
    profiles_dir: "/path/to/profiles"
    profile_name: "analytics"
  warehouse:
    type: "databricks"

Behavior highlights:

  • Repositories are cloned under project_dir/.slideflow_dbt_clones/<key>.
  • The clone key covers repository, branch, and profile inputs. target and vars do not trigger another clone or dbt deps run.
  • Each unique target/vars combination writes to an isolated project_dir/.slideflow_dbt_targets/<workspace-key>/<variant-key> directory. This sibling tree prevents variants from overwriting each other and keeps generated SQL outside the parsed dbt project, including for projects with broad model paths such as model-paths: ['.'].
  • A workspace-keyed filesystem lock under project_dir/.slideflow_dbt_locks coordinates preparation, active artifact use, and cleanup across SlideFlow processes. Threads in one process still share the existing lease and single-flight coordination.
  • Managed artifact paths must be real contained directories. SlideFlow rejects symlinks, non-directory collisions, and paths that resolve outside their expected workspace. The concrete deps log, variant target, and variant log paths are revalidated immediately before every dbt invocation.
  • A private preparation marker is written after clone and dbt deps complete. Cache hits require a real dbt_project.yml, the same checked-out Git commit, and the in-memory generation originally recorded for the marker. If any of those values or the clone changes, every cached target/vars variant for that shared workspace is invalidated and rebuilt from one new generation.
  • Evicted paths remain reserved until lease-safe cleanup finishes, preventing a concurrent request from recreating a directory that cleanup could later delete. Cache bounds are checked again when active leases release.
  • project_dir is treated as a workspace root, not a direct clone target.
  • Default compile: true runs dbt deps, parses the full project, resolves exact requested nodes, and runs one scoped dbt compile --select for the normalized model batch. It does not prepend + or build upstream relations.
  • compile: false never clones and never runs dbt. In that mode, project_dir must already be a compiled dbt project containing target/manifest.json and the compiled SQL files referenced by the manifest.
  • If package_url embeds $TOKEN_NAME, that env var must exist at runtime.
  • If profiles_dir is provided, SlideFlow copies profiles into the cloned dbt workspace and runs dbt with --profiles-dir <clone_dir>.
  • If profiles_dir is omitted but the cloned repo contains profiles.yml at project root, SlideFlow auto-uses that project-root profiles file.
  • Clone/dependency work is single-flight across variants. Each variant is parsed once, and all requested analyses/models for that variant are compiled as one sorted, deduplicated selector batch. Cached supersets satisfy later subset requests without recompiling.
  • In-process dbt invocations are serialized because dbt-core maintains process globals; warehouse queries remain governed separately by query_threads.
  • model_alias resolves a target-invariant manifest name first and retains compiled alias as a compatibility fallback. If the identifier is ambiguous, set one of:
  • model_unique_id
  • model_package_name
  • model_selector_name to avoid ambiguity errors.

dbt on BigQuery (dbt)

This connector shape compiles a dbt project, resolves a model's compiled SQL, then executes it on BigQuery.

data_source:
  type: "dbt"
  name: "dbt_model_bigquery"
  model_alias: "monthly_revenue_by_region"
  dbt:
    package_url: "https://$GIT_TOKEN@github.com/org/analytics-dbt.git"
    project_dir: "/tmp/dbt_project_workspace"
    branch: "main"
    target: "prod"
    vars:
      start_date: "2026-01-01"
      end_date: "2026-01-31"
    profiles_dir: "/path/to/profiles"
    profile_name: "analytics"
  warehouse:
    type: "bigquery"
    project_id: "my-gcp-project" # optional if BIGQUERY_PROJECT/GOOGLE_CLOUD_PROJECT set
    location: "US" # optional
    timeout: 120 # optional query/result timeout in seconds
    credentials_path: "/path/to/service-account.json" # optional
    # credentials_json: '{"type":"service_account",...}' # optional alternative

BigQuery runtime options:

  • Install BigQuery runtime dependencies: pip install "slideflow-presentations[bigquery]".
  • Set project id via:
  • warehouse.project_id, or
  • BIGQUERY_PROJECT, or
  • GOOGLE_CLOUD_PROJECT.
  • Auth options:
  • warehouse.credentials_path, or
  • warehouse.credentials_json, or
  • Application Default Credentials (for example GOOGLE_APPLICATION_CREDENTIALS).
  • warehouse.timeout optionally bounds BigQuery query submission and result download waits. When omitted, SlideFlow leaves the BigQuery SDK timeout behavior unchanged.
  • SlideFlow initializes the BigQuery client with client_info.user_agent set to Slideflow for request attribution in Google-side telemetry.

dbt on Redshift (dbt)

This connector shape compiles a dbt project, resolves a model's compiled SQL, then executes it on Redshift.

data_source:
  type: "dbt"
  name: "dbt_model_redshift"
  model_alias: "monthly_revenue_by_region"
  dbt:
    package_url: "https://$GIT_TOKEN@github.com/org/analytics-dbt.git"
    project_dir: "/tmp/dbt_project_workspace"
    branch: "main"
    target: "prod"
  warehouse:
    type: "redshift"
    host: "example-cluster.abc123.us-east-1.redshift.amazonaws.com"
    database: "analytics"
    user: "reporting_user"
    ssl: true

Redshift warehouse options mirror the direct redshift connector. Omit credential fields from YAML to use the REDSHIFT_* or AWS_* environment fallbacks listed above.

dbt on DuckDB (dbt)

This connector shape compiles a dbt project, resolves a model's compiled SQL, then executes it on DuckDB.

data_source:
  type: "dbt"
  name: "dbt_model_duckdb"
  model_alias: "monthly_revenue_by_region"
  dbt:
    package_url: "https://$GIT_TOKEN@github.com/org/analytics-dbt.git"
    project_dir: "/tmp/dbt_project_workspace"
    branch: "main"
    target: "prod"
  warehouse:
    type: "duckdb"
    database: "/tmp/warehouse.duckdb" # required for dbt+duckdb
    read_only: true # optional; defaults to true
    file_search_path: # optional
      - "/tmp/dbt_project_workspace"
      - "/tmp/data"

Legacy dbt on Databricks (databricks_dbt)

This legacy shape is still fully supported for backward compatibility.

data_source:
  type: "databricks_dbt"
  name: "dbt_model_legacy"
  model_alias: "monthly_revenue_by_region"
  package_url: "https://$GIT_TOKEN@github.com/org/analytics-dbt.git"
  project_dir: "/tmp/dbt_project_workspace"
  branch: "main"
  target: "prod"
  vars:
    start_date: "2026-01-01"
    end_date: "2026-01-31"

Caching and Execution

SlideFlow caches connector fetches by config identity, which helps when:

  • multiple charts use the same query/file in one run
  • multiple replacements reuse one source

Cached DataFrame values are copied on cache write and copied again on cache reads. This keeps table formatting, chart transforms, and custom replacements from mutating shared cached data. The tradeoff is extra memory and copy time for large frames, so prefer selecting only the columns and rows needed for the artifact before handing data to SlideFlow.

Treat connectors as read-only sources during a run for predictable results. clear() and disable() also wake in-flight waiters; a waiter will either reload against the current cache state or run uncached when the cache has been disabled.

Cache/compile tuning env vars:

  • SLIDEFLOW_DATA_CACHE_MAX_ENTRIES (global source cache cap)
  • SLIDEFLOW_DBT_CACHE_MAX_ENTRIES (bounds compiled variants under .slideflow_dbt_targets and inactive prepared workspaces under .slideflow_dbt_clones; default from built-in constants)
  • SLIDEFLOW_DBT_COMPILE_FAILURE_BACKOFF_S
  • SLIDEFLOW_DBT_FAILURE_CACHE_MAX_ENTRIES
  1. Start with local csv/json while designing charts and replacements.
  2. Move to duckdb, databricks, or redshift once schema and logic are stable.
  3. Move to dbt when business logic should live in dbt models (databricks_dbt remains supported as legacy syntax).
  4. Run slideflow validate before slideflow build in CI/CD.

Troubleshooting

  • File connector errors: check file existence and relative path assumptions.
  • Databricks auth errors: verify all three Databricks env vars.
  • Redshift auth/config errors: verify database, host/cluster/serverless endpoint, and either password auth or IAM region/identity settings.
  • dbt model not found: check model_alias, branch, and target.
  • dbt alias ambiguity: add model_unique_id, model_package_name, or model_selector_name.
  • dbt Git clone fails: verify token variable in package_url and repo access.
  • dbt compile:false fails: ensure project_dir/target/manifest.json exists and every selected manifest node has a compiled SQL file on disk.