Testing & CI

AzurPilot adheres to a layered testing philosophy: low-cost checks execute on every commit, while resource-intensive end-to-end checks run within specific scopes. CI verifies schema consistency, module importability, and regression safety; visual recognition on live hardware requires manual verification.

Testing Layers

Testing Tier	Framework	Scope	Characteristics
Python Unit Tests	unittest	200+ test suites	Device-independent; uses mock frames and temporary configs
Scenario Tests	pytest	tests/shop_event, etc.	Parameterized feature matrices
Frontend Unit Tests	Vitest	Components, hooks, utilities	Executed inside jsdom environments
Frontend E2E Tests	Playwright	Live server and mock suites	Validates core interaction workflows
Import Smoke Tests	Internal script	Scans module/ and deploy/	Subprocess imports with whitelisted failures
Artifact Validation	git diff	Universal	Detects uncommitted schema and contract drift

Running Tests Locally

# Full Python unit test suite
uv run python -m unittest discover -s tests

# Run isolated test suite
uv run python -m unittest tests.test_api

# Pytest scenario tests
uv run pytest tests/shop_event

# Import smoke test (offline)
uv run python -m dev_tools.import_smoke_test

# Fast linter
uv run ruff check . --select E9,F63,F7,F82 --ignore F821,F722

Frontend validation:

npm run typecheck --prefix frontend
npm test --prefix frontend
npm run test:e2e:mock --prefix frontend

Run Targeted Suites First

Project guidelines prioritize running tests directly related to modified modules. Full regression sweeps are reserved for cross-cutting changes affecting three or more subsystems.

GitHub Actions CI Jobs

Configured in .github/workflows/ci.yml, triggered on pushes and PRs:

Job	Environment Matrix	Validation Scope
game-data	Ubuntu / Windows / macOS-14	Identity persistence, stock exchange, auth migration, deployment upgrades
frontend	Ubuntu + Node 24	Lockfile sources, API schema diffs, unit/e2e tests, production builds
ruff	Ubuntu	Evaluates syntax rules E9,F63,F7,F82 (ignoring F821,F722)
button-config-check	Ubuntu	Re-executes button_extract and config_updater, evaluating git diff --exit-code
import-smoke	Ubuntu	Full module imports + tests.test_ci_import

Associated workflows:

• ci-report-placeholder.yml: Posts placeholder PR comments upon submission.
• ci-report.yml: Updates placeholder comments with execution metrics upon CI conclusion.

macOS Platform Matrix

Workflows run on Apple Silicon (macos-14). macos-15-intel is excluded because llvmlite lacks x86_64 prebuilt wheels for that target, causing unavoidable uv sync failures.

API Contract Validation

uv run python -m dev_tools.export_api_schema
git diff --exit-code -- frontend/src/api/generated.ts frontend/src/api/contract.json

Modifying backend Pydantic models without exporting updated contracts fails CI immediately.

Schema Artifact Validation

uv run -m dev_tools.button_extract
uv run -m module.config.config_updater
git diff --exit-code

This check prevents two common errors: modifying YAML definitions without recompiling, and committing manual edits to generated files.

End-to-End Configurations

Command	Configuration	Purpose
npm run test:e2e	Default	Connects to live backend; verifies full operational flows
npm run test:e2e:mock	playwright.mock.config.ts	Runs against mock services; recommended for UI adjustments
npx playwright test --config playwright.mind.config.ts	Mind Calculator	Dedicated dock scanning and calculation verification

Testing Boundaries

Automated Test Coverage	Manual Verification Required
Schema migrations and parameter validation	Live visual recognition accuracy
Scheduler state machines and recovery tiers	Real-world emulator timing quirks
API contracts and WebSocket authentication	Transient in-game event mechanics
Frontend form interactions and edit queues	OEM-specific Android overlay permissions

Static Code Passing Does Not Prove Device Reliability

Documentation must distinguish static code validation from live hardware verification. Passing CI suites does not prove visual recognition functions on live game screens.

Pre-Submission Checklist

• [ ] Unit tests covering modified modules pass
• [ ] ruff lint checks pass cleanly
• [ ] Frontend changes verified via typecheck and vitest
• [ ] YAML changes recompiled via config_updater
• [ ] Button template updates processed via button_extract
• [ ] Backend API adjustments exported via export_api_schema
• [ ] Generated schema files committed alongside source changes
• [ ] PR descriptions document test methodologies and unverified components