Every model runs on your hardware: Gemma 3 / Phi-4 language models through a llama.cpp sidecar, SAM2 for imagery segmentation, and YOLO-E for object detection in drone inspection footage and still imagery. Natural-language queries over asset data, segmentation masks, and georeferenced detections on poles, crossarms, and conductors — with zero cloud AI calls, because OT networks and storm-zone deployments don't have a cloud.
VoltGrid launches an embedded Python orchestrator (FastAPI) on demand, which in turn drives a llama.cpp server loading Gemma GGUF weights from local disk. Lazy-start by default: the sidecar only spins up when you first use AI.
%APPDATA%/3DMapExplorer/models/; an in-app downloader with progress fetches them once, or you copy them in by hand for air-gapped workstations.VoltGrid (Desktop Host) └─ AiOrchestratorHost → sidecar :8004 ├─ /v1/chat → llama.cpp :8080 (Gemma GGUF) ├─ /nl2sql → NL → DuckDB SQL ├─ /embed → text → vector ├─ /rag/query → vector similarity search ├─ /segment → SAM2 on COGs └─ /health → liveness probe ● all inference local · no cloud AI calls
Language, segmentation, and detection each feed results back into the same map and DuckDB hub — alongside the asset records your utility keeps in GRID State.
Gemma 3 / Phi-4 via llama.cpp answer geospatial questions in the AI Assistant panel — grounded in your local asset data, not a hosted API.
Type "show all poles inspected in the last 30 days" and the LLM drafts the DuckDB SQL with schema context — spatial predicates like ST_DWithin included.
Meta's Segment Anything 2 (Hiera) — click a point or drag a box on any COG window and SAM2 returns polygon boundaries as GeoJSON overlays. Models: sam2-tiny (default, 42 MB) up to sam2-large. Use cases: building extraction, flood extent, ROW vegetation mapping, substation equipment outlines.
Click-to-digitize from any raster — CIELAB delta-E flood fill (port of the Kanahiro/MagicWand QGIS plugin) traces color-similar regions into vector polygons. No model, no GPU, no sidecar — runs on the host CPU in milliseconds. Perfect for fast one-off polygon capture from orthophotos.
Zero-shot detection over the ArcGIS REST ImageServer: COG window → PNG → detect → geo-reference → FeatureSet, consumable by ArcGIS Maps SDK clients.
Frame-level detection in the FMV player as inspection footage rolls, with each detection georeferenced via KLV telemetry, published as a CoT event, drawn on the crew-tracking overlay, and broadcast to TAK peers in the field.
Build a vector index on any DuckDB text column — inspection notes, work descriptions — query for semantically similar rows, and enrich the chatbot's answers with retrieved context.
NL→SQL never executes blind: the generated DuckDB SQL fills the Query Builder editor where you review it, edit it, and click Run yourself. The model drafts; the human decides.
# status + per-feature availability GET /api/ai GET /api/ai/capabilities GET /api/ai/models → installed status POST /api/ai/models/download → SSE progress # inference POST /api/ai/chat POST /api/ai/nl2sql POST /api/ai/segment → SAM2 on a COG POST /api/ai/detect/cog → YOLO-E FeatureSet # sidecar down → 503 · app unaffected
The YOLO-E detection service runs as a separate container; NVIDIA GPU (CUDA 11.8+) is strongly recommended for usable video latency — CPU works but slowly. LLM answers and generated SQL are drafts and should be reviewed before use, which is exactly how the workflow is built.
Drop the weights on disk, flip on the sidecar, and get chat, NL→SQL, segmentation, and detection over your inspection footage and asset data — on hardware you control, on networks that never leave the room.