Device, Vision & Combat

This subsystem handles visual perception and touch input injection. It acts as the shared foundation for all gameplay modules; coordinate shifts here propagate across all automated tasks.

Device Layer

module/device/ unifies ADB communication, frame capture backends, touch input injection, application lifecycle controls, and emulator management into a unified Device abstraction.

Device
├── Connection        # adb paths, serials, port forwarding
├── Screenshot mixin  # routes to screenshot_* backend per configuration
├── Control mixin     # click / swipe / drag operations
├── AppControl        # start, stop, foreground detection, view hierarchy dump
└── Platform          # emulator discovery and lifecycle per OS

Backend Routing

Screenshot and control backends are implemented as mixin classes. Screenshot and Control route calls via lookup tables:

# Runtime dispatch
func = self.screenshot_methods[self.config.Emulator_ScreenshotMethod]
image = func(self)

Capture and input backends operate independently and can be paired arbitrarily (e.g. nemu_ipc capture paired with MaaTouch control).

Universal Frame Normalization

Frames returned from all backends undergo standardized normalization:

1. Resolution normalization: Frames deviating from 1280×720 are scaled back to reference space.
2. Orientation correction.
3. Optional temporal de-dithering (Emulator.ScreenshotDedithering).
4. Black-frame and resolution validation checks.

Resolution Invariant: 1280×720

All recognition assets, button bounds, and OCR regions are anchored to 1280×720. Altering normalization corrupts visual recognition across all tasks.

Stability Watchdogs

Detection	Mechanism	Raised Exception
Screen Freeze	Compares perceptual hashes across frames with timeout	GameStuckError
Rapid Clicking	Analyzes frequency over trailing 15 clicks	GameTooManyClickError
Button Spam	Consecutive clicks on identical button exceeding 12 times	GameTooManyClickError

The device layer raises exceptions to the scheduler without making recovery decisions; recovery logic resides in alas.py.

Page Graph Navigation

Page Graph

module/ui/page.py registers all known game screens and directional links on import:

page_main.link(button=MAIN_GOTO_CAMPAIGN, destination=page_campaign)

Adding a new screen requires:

1. Instantiating Page(check_button=...).
2. Defining incoming and outgoing edges using link().
3. Verifying that check_button appears reliably on that screen.

Navigation APIs

API	Function
ui_goto(page)	Navigates to target screen via shortest graph path; raises on failure
ui_ensure(page)	Navigates to screen and confirms arrival
ui_back()	Injects back button input
ui_ensure_index()	Scrolls paginated lists to find target index
ui_get_current_page()	Identifies active screen
ui_additional()	Clears transient event popups during navigation

Pathfinding uses breadth-first search (BFS) over the page graph. Avoid hardcoding click sequences; graph navigation adapts to UI updates with simple edge additions.

Unknown Page Recovery

If active screens fail to match any known page node:

1. Taps the top-right HOME button.
2. Checks for dialog popups (maintenance, login, announcements, network drop).
3. Restarts the game to execute full login sequences.

Optical Character Recognition (OCR)

module/ocr/ provides declarative character recognition APIs:

count = self.ocr_quantity(OCR_SHOP_ITEM, amount=1)

Preprocessing Pipeline

Frames undergo deterministic preprocessing rather than raw grayscale inference:

1. Crop bounding box defined by target Button.
2. Extract text colors (defaulting to white), generating high-contrast binarized images.
3. Pass processed tensors into inference models.

This eliminates gradients, outlines, and dynamic background interference.

Model Weights & Backends

• Models: PP-OCRv6 series (lite / standard / pro) utilizing language-specific dictionaries.
• Legacy models: alocr_en_v2_6 and alocr_cn_v3 retained for compatibility.
• Backends: ONNX Runtime (RapidOCR) and NCNN; Windows supports DirectML, QNN, OpenVINO; macOS supports CoreML ANE.
• Standard inference uses recognition-only mode (RecOnlyOCR), bypassing text detection networks on fixed UI regions.

Thread allocation controls:

# Caps intra-op thread pools per OCR session (1..16, bounded by CPU cores)
AZURPILOT_OCR_THREADS=2

Version Benchmarking

Evaluate accuracy and latency with OcrBenchmark before switching models.

Map Perception & Combat

Module	Scope	Entry Point
module/map	Grid perception, perspective transformations, pathfinding	Map
module/map_detection	Map tile recognition and classification	map_detection
module/combat	Combat loops, victory detection, fleet assignment	Combat
module/campaign	Stage selection, battle entry, CampaignRun	CampaignRun
campaign/	Specific stage map scripts and battle definitions	Python source files

Combat Sequence

Detect combat screen → Await post-battle victory screen → Parse combat drops
  ├── Victory → Continue pathfinding / exit
  ├── Defeat → Execute DefeatWithdraw strategy (withdraw / switch fleet / stop)
  └── Freeze → GameStuckError → Trigger recovery

Defeat handling is standardized inside module/defeat_handling.py, coordinating retreats and morale limits across campaign and event stages.

Campaign Stage Scripts

Files under campaign/ are custom Python stage scripts rather than static YAML files:

• Each file models tiles, enemies, puzzle mechanics, and boss behaviors.
• Event stages adapt neighboring event patterns.
• New events require custom map code; the engine does not solve novel mechanics automatically.

Operation Siren Navigation

module/os/ provides specialized coordinate mapping:

OSMap(MapEventResumeMixin, OSFleet, Map, GlobeCamera, StorageHandler, StrategicSearchHandler)

• globe_goto(zone) unifies globe-view navigation and sector exploration.
• OSGrid handles perspective transforms with fixed homography parameters for Operation Siren camera angles.
• Feature routines inherit from OSMap to access navigation primitives.

Review Operation Siren and internal specifications in docs/modules/os/index.md.

Routine Development Workflows

Updating Button Assets

1. Capture a reference screenshot at 1280×720.
2. Save the cropped button asset into assets/<server>/.
3. Reference the template on the Button definition.
4. Extract assets via:

uv run -m dev_tools.button_extract

Adding a UI Page

page_my_feature = Page(check_button=OCR_MY_FEATURE_TITLE)
page_main.link(button=MAIN_GOTO_MY_FEATURE, destination=page_my_feature)
page_my_feature.link(button=BACK_ARROW, destination=page_main)

Adjusting OCR Bounding Boxes

Bounding boxes are governed by Button.area. Validate adjustments against real hardware screenshots.

Testing

Scope	Verification Command
Import Smoke Tests	uv run python -m dev_tools.import_smoke_test
Map & Combat Unit Tests	uv run python -m unittest tests.test_combat_results tests.test_auto_search_defeat tests.test_opsi_map_event_resume
OCR Settings & Threading	uv run python -m unittest tests.test_ocr_settings tests.test_ocr_thread_limits
OCR Benchmarks	WebUI "Tool → OCR Performance Benchmark"

Visual recognition accuracy must be confirmed on live hardware.