OCR & Vision Pipeline

The Android host executes OCR outside the Python environment inside an isolated :ocr process. This guarantees that model crashes do not take down the main user interface and isolates proprietary vendor NPU libraries.

Inference Pipeline

AzurPilot inside PRoot (Python)
  → ONNX session factory inside module/ocr is proxied by android_ocr.py
    → Validates model identity against SHA-256 whitelist
      → Sends preprocessed NCHW tensors with auth token to 127.0.0.1:22302
        → :ocr process executes hardware inference
          → Returns raw output logits
            → Python processes CTC decoding and post-processing

Division of responsibilities:

Stage	Execution Location
ROI cropping, image thresholding	Python (upstream OCR engine)
Neural network inference	Android :ocr service
Dictionary lookup, CTC decoding, error correction	Python
Model weights	Packaged inside the host APK

The inference service binds privately and requires neither Root nor Shizuku.

Model Formats

The application uses Model Format 2:

• Bundles 5 recognition models + 1 detection model converted to LiteRT FP32.
• Maintains 4 MNN models for HiSilicon devices.
• Deprecates legacy ONNX, NCNN, and older CnOCR weights from distribution packages.
• CPU inference uses LiteRT; terminal Softmax operations can execute on GPU.

Upstream weight modifications require an updated APK. The host validates model identity against SHA-256 whitelists and refuses mismatched conversions.

Hardware Acceleration Backends

Hardware Target	Implementation	Requirements
Qualcomm Snapdragon	LiteRT → QNN HTP	Android 12+, ARM64; embeds versioned Skel/Stub binaries
MediaTek Dimensity	LiteRT → NeuroPilot	Android 12+, ARM64; requires compatible vendor APU drivers
HiSilicon Kirin	MNN → HiAI NPU	Android 10+, ARM64; operators must map cleanly to NPU
Samsung Exynos	CPU fallback	Excluded due to AI LiteCore AOT compilation failures
Google Tensor	CPU fallback	Requires external beta SDKs
Xiaomi Custom SoCs	CPU fallback	Lacks compatible toolchains and runtimes
Android 9 / x86_64	LiteRT CPU	Baseline compatibility fallback

Hardware Support Depends on SoCs, Not Brands

NPU acceleration depends on the chipset and driver version rather than device brand. Devices from the same manufacturer may run QNN or fall back to CPU depending on the installed processor.

Vendor Library Isolation

The host creates vendor-specific library directories containing symlinks to application directories, preventing conflicting offline compilers from intercepting inference requests. Symlinks are refreshed following APK updates.

The manifest specifies required=false for vendor RPC, NeuroPilot, and HiAI libraries. Devices lacking these libraries install cleanly and fall back to CPU inference.

Fallback & Recovery

• Initialization errors or inference crashes trigger automatic fallback to LiteRT CPU.
• If a specific model fails on NPU, the host rebinds the service and disables NPU acceleration for that model to prevent crash loops.
• Active backends and operational status are logged and inspectable in Settings.

Threading & Concurrency

• OCR sessions limit intra-op threading by default to avoid thread exhaustion from multi-instance concurrency.
• Configurable via AZURPILOT_OCR_THREADS=1..16, capped by logical CPU cores.
• Applies to new sessions; running workers update upon task restart.

Frame Capture Pipeline

Frame capture operates outside ADB:

Game (Virtual Display)
  → Native bridge acquires framebuffer in privileged process
    → Anonymous SharedMemory (ashmem / memfd)
      → Passes descriptor over Unix domain socket via SCM_RIGHTS
        → Host reads buffer

Metric	Shared Memory	TCP Stream
Average Latency	15.6 ms	47.2 ms
Disk Writes	Zero	Zero

Buffers exist purely within volatile memory. Frame pointers must be released via UnlockPixels immediately after processing.

Benchmarking Tools

Benchmark	Method
OCR Acceleration	In-app benchmark / tools/benchmark_*
Virtual Display Throughput	tools/benchmark_virtual_display.py
Frame Transport Latency	tools/benchmark_capture_transport.py
Template Matching Performance	tools/test_template_matching.py

On-device Python tasks include OcrBenchmark to evaluate comparative latency across CPU, GPU, and NPU.

Implementation Guidelines

Forbidden Pattern	Justification
Instantiating ONNX sessions directly in Python	Attempts to load non-existent model files
Altering models without updating APK	Rejected by SHA-256 whitelist validation
Moving CTC decoding into :ocr	Violates interface boundary with Python
Assuming universal NPU support	Driver and operator support varies significantly
Caching frame pointers across intervals	Pointers dangle immediately upon unlock

Verification Commands

# Compile host OCR bindings
cd app && ./gradlew :app:compileDebugKotlin -x verifyBundledAzurPilotRuntime

# Run display benchmark
uv run python tools/benchmark_virtual_display.py --help

Compilation verifies interface signatures; real-world recognition accuracy requires physical hardware testing with actual game screenshots.