NVIDIA-AI-Blueprints/video-search-and-summarization

NVIDIA AI Blueprint for video search and summarization (VSS) is a GPU-accelerated reference architecture for building video analytics agents with real-time verified alerts, visual Q&A, and automated reporting. The VSS Blueprint uses vision language models (VLMs) such as NVIDIA Cosmos, LLMs such as NVIDIA Nemotron, RAG, and NVIDIA NIMs.

1,775 ★366 forksC++Updated 21h ago

What you need to know

A reference architecture (AI Blueprint) for building GPU-accelerated video AI agents that search, analyze, summarize, and reason over live or recorded video via natural language, combining vision-language models, RAG, and NVIDIA NIM microservices with an MCP-based agent layer.

Install

Docker Compose deployment on own hardware (see deploy/ compose.yml)
Brev Launchable deployment notebook (deploy/docker/scripts/deploy_vss_launchable.ipynb) on a 2xRTX PRO 6000 SE AWS instance
Helm charts and NIM model configs under deploy/

Usage

  • •Requires NVIDIA AI Enterprise developer license for local NIM hosting plus API catalog keys (build.nvidia.com or NGC)
  • •Five agent workflows: Q&A and report generation (quickstart), Alert Verification, Real-Time Alerts, Video Search (alpha), Long Video Summarization
  • •Agent layer uses the Model Context Protocol (MCP) to expose video analytics, incident records, and vision processing tools to agents
  • •Deploy via Docker Compose on validated GPU topologies (see GPU requirements docs)

Key features

  • ✓Real-time video intelligence microservices (feature extraction, embeddings, stream understanding) publishing to a message broker
  • ✓Downstream analytics enriching metadata into trajectories, incidents, and verified alerts (VLM-based alert verification reduces false positives)
  • ✓Natural-language search across video archives using video embeddings (alpha)
  • ✓Long-video summarization via chunking and aggregation of dense captions; VLM-based visual Q&A and report generation
  • ✓Includes agent skills (agentskills.io-compatible), message broker consumers, spatial AI data utilities, and a Next.js frontend monorepo

Best for

Teams building GPU-accelerated surveillance, smart-space, warehouse automation, or SOP-validation video AI agents.

Caveats

  • ⚠Requires NVIDIA AI Enterprise developer license and API catalog/NGC API keys
  • ⚠Stringent hardware requirements: NVIDIA driver 580.x, NVIDIA Container Toolkit 1.17.8+, Docker Engine 28.3.3 <= version < 29.5.0 (29.5.0+ may fail pulling NGC-hosted images), Docker Compose v2.39.1+, NGC CLI 4.10.0+
  • ⚠OS matrix: x86 Ubuntu 22.04/24.04, DGX-SPARK (DGX OS 7.4.0), IGX-THOR / AGX-THOR (Jetson Linux BSP)
  • ⚠Designed for technical users (video analysts, GenAI/ML engineers); extensive configuration required
Platforms: Linux (Ubuntu x86) · DGX-SPARK · IGX-THOR / AGX-THOR (NVIDIA Jetson)
Documentation ↗

Reviewed 2026-08-07

Topics

computer-visiongenerative-ailong-video-understandingmodel-context-protocolmultimodal-ainatural-language-searchnvidia-nimragreal-time-video-analyticsretrieval-augmented-generationskillsvideo-agentvideo-analyticsvideo-ragvideo-searchvideo-summarizationvideo-understandingvision-agentvision-language-modelvlm
View on GitHub ↗
Stars
1,775★
Forks
366
Language
C++
License
NOASSERTION
Created
2024-10-22
Last push
2026-08-06