Skip to content

Testing

Objectives

  • Prevent regressions for existing users.
  • Validate new behavior before merge/release.
  • Keep local feedback loops fast.

Test layers

  1. Unit tests (tests/)
  2. CLI command behavior
  3. config/registry utilities
  4. chart/template/replacement behavior
  5. property-based invariants for data transforms/formatting edge cases
  6. Integration tests (@pytest.mark.integration)
  7. cross-module workflows with controlled fixtures/mocks
  8. End-to-end tests (@pytest.mark.e2e)
  9. full validate -> build paths on representative configs
  10. Live Google Slides tests (@pytest.mark.live_google, tests/live_tests/)
  11. creates real template presentations via Google API
  12. copies template into a new deck during render
  13. inserts real charts across full coverage:
    • all built-in template charts
    • direct plotly_go charts
    • custom chart functions from registry
  14. verifies live replacements and function-driven behavior:
    • static text replacement
    • dynamic value_fn replacement
    • ai_text replacement via deterministic provider function
    • table replacement placeholders
  15. verifies image elements and placeholder replacement in rendered slides
  16. trashes created files on teardown
  17. Live Google Docs tests (@pytest.mark.live_google_docs, tests/live_tests/)
  18. creates real template documents via Google API
  19. copies template into a new doc during render
  20. validates marker-scoped replacements (slide.id -> {{SECTION:<id>}})
  21. validates inline chart insertion in rendered docs
  22. validates deterministic AI and table replacement behavior
  23. trashes created files on teardown
  24. Live Google Sheets tests (@pytest.mark.live_google_sheets, tests/live_tests/)
  25. creates a real workbook via Google Sheets API
  26. validates replace-mode writes for current snapshot tabs
  27. validates append-mode idempotency using _slideflow_meta run-key tracking
  28. verifies rerun skip behavior prevents duplicate appended rows
  29. trashes created files on teardown

Local commands

uv sync --extra dev --extra ai --locked
source .venv/bin/activate
uv lock --check
uv pip check
uvx --from black==26.3.1 black --check slideflow tests scripts
python -m ruff check slideflow tests scripts
python -m mypy slideflow
pytest -q
pytest -q --cov=slideflow --cov-branch --cov-report=term --cov-report=xml

Run only integration or e2e groups:

pytest -q -m integration
pytest -q -m e2e

Mirror CI marker split locally:

pytest -q -m "not integration and not e2e" --cov=slideflow --cov-branch --cov-report=term --cov-report=xml --cov-fail-under=82
pytest -q -m integration
pytest -q -m e2e

Run live template + chart rendering tests locally:

export SLIDEFLOW_RUN_LIVE=1
export GOOGLE_SLIDEFLOW_CREDENTIALS=/absolute/path/to/service-account.json
export SLIDEFLOW_LIVE_PRESENTATION_FOLDER_ID=<drive-folder-id>
# optional override for chart image uploads:
export SLIDEFLOW_LIVE_DRIVE_FOLDER_ID=<drive-folder-id>
# optional template to copy before test mutations:
export SLIDEFLOW_LIVE_TEMPLATE_ID=<google-slides-template-id>
# optional comma-separated emails to share rendered deck with:
export SLIDEFLOW_LIVE_SHARE_EMAIL=<you@example.com>
# optional permission role for shared deck (reader|writer|commenter):
export SLIDEFLOW_LIVE_SHARE_ROLE=reader
# optional retention toggle; defaults to 1 when sharing is enabled:
export SLIDEFLOW_LIVE_KEEP_ARTIFACTS=1

pytest -q tests/live_tests -m live_google

When SLIDEFLOW_LIVE_SHARE_EMAIL is set, the rendered presentation is shared using the service account and the test prints the deck URL. Artifacts are retained by default in that mode so you can validate the slides visually.

Run live Google Docs tests locally:

export SLIDEFLOW_RUN_LIVE=1
export GOOGLE_DOCS_CREDENTIALS=/absolute/path/to/service-account.json
export SLIDEFLOW_LIVE_DOCUMENT_FOLDER_ID=<drive-folder-id>
# optional override for chart image uploads:
export SLIDEFLOW_LIVE_DRIVE_FOLDER_ID=<drive-folder-id>
# optional seed template document:
export SLIDEFLOW_LIVE_DOC_TEMPLATE_ID=<google-docs-template-id>
# optional comma-separated emails to share rendered doc with:
export SLIDEFLOW_LIVE_SHARE_EMAIL=<you@example.com>
# optional permission role for shared doc (reader|writer|commenter):
export SLIDEFLOW_LIVE_SHARE_ROLE=reader
# optional retention toggle; defaults to 1 when sharing is enabled:
export SLIDEFLOW_LIVE_KEEP_ARTIFACTS=1

pytest -q tests/live_tests -m live_google_docs

Run live Google Sheets tests locally:

export SLIDEFLOW_RUN_LIVE=1
export GOOGLE_SHEETS_CREDENTIALS=/absolute/path/to/service-account.json
export SLIDEFLOW_LIVE_SHEETS_FOLDER_ID=<drive-folder-id>
# optional override:
export SLIDEFLOW_LIVE_DRIVE_FOLDER_ID=<drive-folder-id>
# optional comma-separated emails to share rendered workbook with:
export SLIDEFLOW_LIVE_SHARE_EMAIL=<you@example.com>
# optional permission role for shared workbook (reader|writer|commenter):
export SLIDEFLOW_LIVE_SHARE_ROLE=reader
# optional retention toggle; defaults to 1 when sharing is enabled:
export SLIDEFLOW_LIVE_KEEP_ARTIFACTS=1

pytest -q tests/live_tests -m live_google_sheets

CI quality gates

  • CI enforces version consistency checks.
  • CI enforces dependency consistency via uv lock --check and uv pip check.
  • CI enforces branch-aware coverage floor (--cov-branch --cov-fail-under=82) on unit tests (not integration and not e2e).
  • Coverage gate ratchet policy:
  • current: 82
  • next staged target: 85 after planned test-hardening work lands (roadmap items #196/#197)
  • thresholds should only increase; decreases require explicit maintainer approval
  • CI runs dedicated integration and e2e marker suites in separate steps.
  • Distribution artifacts are built for every CI run.
  • Live Google Slides tests run in a separate workflow (Live Google Slides) so PR CI remains deterministic.
  • Live Google Docs tests run in a separate workflow (Live Google Docs) so PR CI remains deterministic.
  • Live Google Sheets tests run in a separate workflow (Live Google Sheets) so PR CI remains deterministic.

Orchestrated runtime note

  • Chart image export runs through headless Kaleido.
  • For Cloud Run/Databricks/self-hosted runners, ensure a Chrome/Chromium binary is available in the runtime image.
  • On macOS local runs, if desktop Chrome still steals focus, set CHROME_PATH or GOOGLE_CHROME_BIN to a dedicated Chromium/Chrome-for-Testing binary instead of /Applications/Google Chrome.app/....

Compatibility matrix

Compatibility tests assert support remains in place for:

  • CLI commands/options (slideflow build, slideflow validate)
  • connectors (csv, json, databricks, duckdb, dbt, databricks_dbt)
  • replacements (text, table, ai_text)
  • charts (plotly_go, custom, template)
  • template/registry loading paths

Contribution expectations

  • Every bug fix should include a regression test.
  • Every behavior change should update docs and tests in the same PR.
  • Release branches should not introduce untested logic.