NVIDIA

Integrate with NVIDIA using LangChain Python.

LangChain and NVIDIA have partnered across the agent stack:

1. Components
2. Sandboxed agents with OpenShell
3. LangGraph acceleration primitives
4. NeMo Agent Toolkit optimizations
5. Post-training agent workflows
6. Full Stack blueprints

Components

The langchain-nvidia-ai-endpoints package provides LangChain integrations for chat, embeddings, reranking, and retrieval powered by NVIDIA AI—including Nemotron, NVIDIA’s open model family built for agentic AI, and hundreds of community models on the NVIDIA API Catalog.

Models run on NVIDIA NIM microservices: container images that expose a standard OpenAI-compatible API, optimized with TensorRT-LLM for peak throughput on NVIDIA hardware. They can be accessed via the hosted API Catalog or self-hosted on-premises.

Component	Class	Description
Chat	ChatNVIDIA	Chat completions with any NVIDIA-hosted model or local NIM
Chat (Dynamo)	ChatNVIDIADynamo	ChatNVIDIA with KV cache routing hints for Dynamo deployments
Embeddings	NVIDIAEmbeddings	Dense vector embeddings for semantic search and RAG
Reranking	NVIDIARerank	Document reranking by query relevance
Retrieval	NVIDIARAGRetriever	Retrieval from an NVIDIA RAG Blueprint server

Chat: ChatNVIDIA

ChatNVIDIA provides chat completions over NVIDIA-hosted models and local NIM deployments. It supports tool calling, structured output, image inputs, and streaming.

Install

pip install -qU langchain-nvidia-ai-endpoints

Access the NVIDIA API Catalog

1. Create a free account on the NVIDIA API Catalog and log in.
2. Click your profile icon, then API Keys > Generate API Key.
3. Copy and save the key as NVIDIA_API_KEY.

import getpass
import os

if os.environ.get("NVIDIA_API_KEY", "").startswith("nvapi-"):
    print("Valid NVIDIA_API_KEY already in environment. Delete to reset")
else:
    nvapi_key = getpass.getpass("NVAPI Key (starts with nvapi-): ")
    assert nvapi_key.startswith(
        "nvapi-"
    ), f"{nvapi_key[:5]}... is not a valid key"
    os.environ["NVIDIA_API_KEY"] = nvapi_key

Nemotron: featured models for agentic AI

Nemotron is NVIDIA’s open model family designed for agentic AI. The models use a hybrid Mamba-Transformer mixture-of-experts architecture that delivers leading benchmark performance with high throughput and support for up to 1M token context windows. Nemotron model weights, training data, and implementation recipes are published openly under the NVIDIA Open Model License.

from langchain_nvidia_ai_endpoints import ChatNVIDIA

# Nemotron 3 Ultra - frontier reasoning and agentic workflows
llm = ChatNVIDIA(model="nvidia/nemotron-3-ultra-550b-a55b")
result = llm.invoke("Plan a three-step research workflow for competitive analysis.")
print(result.content)

See the ChatNVIDIA integration page for full documentation including tool calling, multimodal inputs, and Nemotron-specific examples.

Chat: ChatNVIDIADynamo

ChatNVIDIADynamo is a drop-in replacement for ChatNVIDIA for use with NVIDIA Dynamo deployments. It automatically injects KV cache routing hints into every request, allowing the Dynamo scheduler to optimize memory allocation, load routing, and request priority.

from langchain_nvidia_ai_endpoints import ChatNVIDIADynamo

llm = ChatNVIDIADynamo(
    base_url="http://localhost:8099/v1",
    model="nvidia/nemotron-3-ultra-550b-a55b",
    osl=512,             # expected output sequence length (tokens)
    iat=250,             # expected inter-arrival time (ms)
    latency_sensitivity=1.0,
    priority=1,
)
result = llm.invoke("Summarize KV cache routing in one sentence.")
print(result.content)

See the ChatNVIDIA integration page for the full ChatNVIDIADynamo reference including per-invocation overrides and streaming.

Embeddings: NVIDIAEmbeddings

NVIDIAEmbeddings generates dense vector embeddings for use in semantic search and RAG pipelines.

from langchain_nvidia_ai_endpoints import NVIDIAEmbeddings

embedder = NVIDIAEmbeddings(model="NV-Embed-QA")
embedder.embed_query("What's the temperature today?")

See the NVIDIAEmbeddings integration page for full documentation.

Reranking: NVIDIARerank

NVIDIARerank reranks a list of documents by relevance to a query using a NeMo Retriever reranking NIM.

from langchain_core.documents import Document
from langchain_nvidia_ai_endpoints import NVIDIARerank

ranker = NVIDIARerank(model="nvidia/llama-3.2-nv-rerankqa-1b-v1")
docs = ranker.compress_documents(
    query="What is GPU memory bandwidth?",
    documents=[Document(page_content=p) for p in passages],
)

Retrieval: NVIDIARAGRetriever

NVIDIARAGRetriever connects LangChain to a running NVIDIA RAG Blueprint server and retrieves relevant documents via the /v1/search endpoint. It supports reranking, query rewriting, and metadata filtering.

from langchain_nvidia_ai_endpoints import NVIDIARAGRetriever

retriever = NVIDIARAGRetriever(base_url="http://localhost:8081", k=4)
docs = retriever.invoke("What is NVIDIA NIM?")

See the NVIDIARAGRetriever integration page for full documentation.

Self-host with NVIDIA NIM Microservices

When you are ready to deploy your AI application, you can self-host models with NVIDIA NIM. For more information, refer to NVIDIA NIM Microservices.

from langchain_nvidia_ai_endpoints import ChatNVIDIA, NVIDIAEmbeddings, NVIDIARerank

# connect to a chat NIM running at localhost:8000, specifying a model
llm = ChatNVIDIA(base_url="http://localhost:8000/v1", model="nvidia/nemotron-3-ultra-550b-a55b")

# connect to an embedding NIM running at localhost:8080
embedder = NVIDIAEmbeddings(base_url="http://localhost:8080/v1")

# connect to a reranking NIM running at localhost:2016
ranker = NVIDIARerank(base_url="http://localhost:2016/v1")

Sandboxed Agents with OpenShell

OpenShell gives LangChain agents a policy-governed Linux sandbox for code execution, file access, process permissions, and network egress. This is useful for deep agents that need to inspect data, write code, run commands, or use tools against local resources while keeping the execution environment isolated from the host.

For LangChain applications, the langchain-nvidia-openshell package adapts OpenShell to the Deep Agents sandbox interface. The agent can run on the host with ChatNVIDIA, while tool execution is dispatched into an OpenShell sandbox through the OpenShellSandbox backend.

Resources:

• LangChain NVIDIA samples includes runnable NVIDIA examples, including an OpenShell incident-analysis agent.
• OpenShell Deep Agent is a reference coding agent that runs inside an OpenShell sandbox, orchestrated by Deep Agents and powered by NVIDIA Nemotron.
• langchain-nvidia-openshell provides the Python adapter, setup notes, policy guidance, and notebook walkthroughs for sandboxed Deep Agents.

import openshell
from deepagents import create_deep_agent
from langchain_nvidia_ai_endpoints import ChatNVIDIA
from langchain_nvidia_openshell import OpenShellSandbox

with openshell.Sandbox() as sandbox:
    backend = OpenShellSandbox(sandbox=sandbox)
    agent = create_deep_agent(
        model=ChatNVIDIA(model="nvidia/nemotron-3-ultra-550b-a55b"),
        system_prompt="You are a careful coding agent.",
        backend=backend,
    )

Accelerate LangGraph with NVIDIA

The langchain-nvidia-langgraph package provides NVIDIA-optimized execution strategies for LangGraph graphs. It offers two complementary optimizations applied at compile time:

• Parallel execution: independent nodes are automatically identified and run concurrently, eliminating unnecessary sequential bottlenecks.
• Speculative execution: both branches of a conditional edge run simultaneously; the wrong branch is discarded once the routing condition resolves.

Neither optimization requires changes to node logic or graph edges.

Install

pip install -qU langchain-nvidia-langgraph

Parallel execution

Replace StateGraph from LangGraph with StateGraph from langchain_nvidia_langgraph.graph. The rest of your graph definition stays the same.

from langchain_nvidia_langgraph.graph import StateGraph, OptimizationConfig
from langgraph.graph import END
from typing import TypedDict

class AgentState(TypedDict):
  ...

graph = StateGraph(AgentState)
app = graph.compile(optimization=OptimizationConfig(enable_parallel=True))

Or wrap an existing StateGraph:

from langgraph.graph import StateGraph as LangGraphStateGraph
graph = LangGraphStateGraph(AgentState)
app = with_app_compile(graph).compile(optimization=OptimizationConfig(enable_parallel=True))

Decorators give explicit control over which nodes participate in optimization:

from langchain_nvidia_langgraph.graph import sequential, depends_on, speculation_unsafe

# Prevent a node from being parallelized (e.g., it writes to shared state)
@sequential
def write_to_db(state):
    ...

# Declare a dependency not expressed in graph edges
@depends_on("write_to_db")
def next_action(state):
    ...

Speculative execution

Enable speculation via OptimizationConfig at compile time. The executor runs conditional branches in parallel and keeps the result that matches the routing decision.

app = graph.compile(optimization=OptimizationConfig(enable_speculation=True))

NeMo Agent Toolkit Optimizations with LangSmith Telemetry

The NVIDIA NeMo Agent Toolkit is an open-source AI toolkit for building, profiling, and optimizing agents. Developers can use LangChain with NeMo Agent Toolkit with minimal code changes to enable profiling, evaluation, GPU capacity plans, and automated optimization. NeMo Agent Toolkit is interoperable with LangSmith.

• Get Started with NeMo Agent Toolkit and LangChain

• Optimize LangChain with NeMo Agent Toolkit and LangSmith

Post-training Agent Workflows

NVIDIA NeMo Gym provides post-training workflows for agentic systems, including examples that pair LangGraph agents with Responses API-style training data and evaluation loops. Use these workflows when you want to improve agent behavior after initial prompting and harness tuning, especially for tool-use and multi-step reasoning tasks.

• NeMo Gym LangGraph agent example

Full Stack Blueprints

NVIDIA and LangChain have collaborated on full stack examples showing how all these components are combined for two enterprise use cases, with focus on production readiness:

• NVIDIA AI-Q is a blueprint for deep research across enterprise data sources using LangChain Deep Agents
• NVIDIA VSS is a blueprint for video search and summarization using LangChain and LangGraph

Additional Resources

• langchain-nvidia-ai-endpoints package README
• langchain-nvidia-langgraph package
• Nemotron model family
• langchain-nvidia-openshell package
• LangChain NVIDIA samples
• OpenShell Deep Agent
• NeMo Gym LangGraph agent example
• Overview of NVIDIA NIM for Large Language Models (LLMs)
• Overview of NeMo Retriever Embedding NIM
• Overview of NeMo Retriever Reranking NIM
• ChatNVIDIA Model
• NVIDIAEmbeddings Model for RAG Workflows
• NVIDIARAGRetriever
• NVIDIA Dynamo

---

Connect these docs to Claude, VSCode, and more via MCP for real-time answers.

Edit this page on GitHub or file an issue.