What Is NVIDIA Cosmos? A Simple Guide to Cosmos 3 World Models

What Is NVIDIA Cosmos? A Simple Guide to NVIDIA/cosmos for Physical AI World Models

Image extracted from the official NVIDIA/cosmos repository, file cosmos-logo-thumbnail.png. This is a PNG image, not SVG.¹

Quick summary

NVIDIA Cosmos is an open platform of world models, datasets and tools for building Physical AI for robots, autonomous vehicles, smart infrastructure and other systems that interact with the physical world. The official repository describes Cosmos as an open platform that enables developers to build Physical AI.²

In plain terms, Cosmos is not a normal chatbot. It is a family of models and tools that help AI understand, simulate and predict the physical world. For example, it can analyze a robot video, reason about what is happening, forecast the next action, or generate simulated video from text, images, videos and action inputs.

The NVIDIA/cosmos repository currently focuses on Cosmos 3, a family of omnimodal world models that jointly process and generate language, images, video, audio and action sequences in a unified Mixture-of-Transformers architecture.³⁴

What is Cosmos used for?

Cosmos targets AI systems that need to understand the real world, not just text.

Example use cases:

• generate a video of a robot moving through a warehouse;
• predict what a robot should do next;
• analyze autonomous-driving video and forecast motion;
• generate synthetic data for robot training;
• check whether a scene is physically plausible;
• roll out a future state from current vision and action inputs;
• caption videos and localize events by time;
• use a reasoner inside embodied agents.

The README describes two runtime surfaces for Cosmos 3: Reasoner and Generator.³

What is a world model?

A world model is an AI model that learns to represent and predict the world. A standard LLM mostly works with text. A world model can work with images, videos, audio, actions, robot trajectories, camera motion and physical scenes.

Example:

• An LLM answers: “What should the robot do?”
• A world model watches a robot video, understands where objects are, predicts plausible motion and then answers or generates a future rollout.

Cosmos 3 extends this into an omnimodal model: it can reason over multiple modalities and generate multiple output types.

What stands out in the NVIDIA/cosmos repository?

The repository contains:

The GitHub page shows a latest release titled Cosmos 3 Launch on June 1, 2026. The README news section says Cosmos 3 was released in the NVIDIA Cosmos 3 Hugging Face collection and Cosmos Framework on May 31, 2026.²⁵

What is Cosmos 3?

Cosmos 3 is the newest model family in the repository. The README describes it as a suite of omnimodal world models designed to process and generate language, images, video, audio and action sequences in one unified Mixture-of-Transformers architecture.³

Important points:

• Omnimodal: works across many modalities.
• World model: focused on the physical world.
• Generator + Reasoner: supports both generation and understanding.
• Physical AI focus: robotics, autonomous systems, embodied agents and simulation.
• Multiple runtime paths: Diffusers, vLLM-Omni, vLLM, NIM and Cosmos Framework.
• OpenMDW-1.1 license: source code and models use a specific model-materials license.⁶

Cosmos 3 architecture in simple terms

The README describes Cosmos 3 as using a Mixture-of-Transformers architecture. In simple terms, it combines two major capabilities:³

Reasoner mode uses causal self-attention for language and visual understanding. Generator mode uses full attention to denoise image, video, audio and action tokens. Both modes share the transformer architecture, multimodal attention layers and a 3D mRoPE representation for spatial and temporal structure.³

Short version:

Reasoner = understand and answer
Generator = simulate and generate

Model family

The README lists these main Cosmos 3 models:³

Beginners should start with Cosmos3-Nano. Cosmos3-Super is heavier and usually needs more GPU resources.

What can the Generator do?

Generator produces non-text outputs such as images, videos, audio and action rollouts. The README lists several generator workflows:³

Example: provide “a small robot moves through a warehouse and stops at a shelf,” and Generator can produce a simulated video.

What can the Reasoner do?

Reasoner returns text or JSON from text, images and videos. The README lists several reasoner workflows:³

Use Reasoner when you want the model to understand visual context rather than generate video.

Supported inputs and outputs

The README lists these key specs:³

Supported resolution tiers are 256p, 480p and 720p. Supported aspect ratios include 16:9, 4:3, 1:1, 3:4 and 9:16. Supported frame rates are 10, 16, 24 and 30 FPS, and supported frame count is 5 to 300 frames.³

Hardware requirements

Cosmos 3 is a heavy model family. The README lists BF16, Linux and NVIDIA Ampere/Hopper/Blackwell GPUs as the tested setup.³

Practical notes:

• Start with Cosmos3-Nano.
• Text-to-video and image-to-video are much heavier than text reasoning.
• 720p and 189 frames are much heavier than 256p or single-image generation.
• Cosmos3-Super often requires multi-GPU setups or tensor parallelism.
• If you only need production Reasoner, NIM is usually easier than assembling vLLM yourself.
• If you are researching Generator behavior in Python, Diffusers is easier to inspect.

The README recommends CUDA 13 or 12.8 and says system CUDA and PyTorch CUDA major versions must match.⁷

Hugging Face login

Before running examples, the README says to create a Hugging Face access token and authenticate locally:⁸

uvx hf@latest auth login

If your default disk is small, set HF_HOME to a larger cache path:

export HF_HOME=/data/huggingface

Cosmos models are large, so disk cache planning matters.

Generator with Diffusers

This path is good for research, Python experimentation and pipeline inspection.

Environment setup:

uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate

uv pip install --torch-backend=auto \
  "diffusers @ git+https://github.com/huggingface/diffusers.git" \
  accelerate \
  av \
  cosmos_guardrail \
  huggingface_hub \
  imageio \
  imageio-ffmpeg \
  torch \
  torchvision \
  transformers

The README explains that --torch-backend=auto lets uv pick a CUDA build matching the NVIDIA driver.⁸

Text-to-video example:

import torch
from diffusers import Cosmos3OmniPipeline
from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepScheduler
from diffusers.utils import export_to_video

pipe = Cosmos3OmniPipeline.from_pretrained(
    "nvidia/Cosmos3-Nano",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)

pipe.scheduler = UniPCMultistepScheduler.from_config(
    pipe.scheduler.config,
    flow_shift=10.0,
)

result = pipe(
    prompt="A mobile robot navigates a warehouse aisle and stops at a shelf.",
    negative_prompt="",
    image=None,
    num_frames=189,
    height=720,
    width=1280,
    fps=24,
    num_inference_steps=35,
    guidance_scale=6.0,
    enable_sound=False,
    add_resolution_template=False,
    add_duration_template=False,
    generator=torch.Generator(device="cuda").manual_seed(1234),
)

export_to_video(result.video, "cosmos3_t2v.mp4", fps=24, macro_block_size=1)

The README warns that text-to-video takes time: the first run downloads Cosmos3-Nano, and diffusion workloads finish all inference steps before output appears.⁸

Generator production with vLLM-Omni

For OpenAI-compatible image/video/sound/action serving, use vLLM-Omni.

Docker server example:

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v "$(pwd):/workspace" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-omni:cosmos3 \
  vllm serve nvidia/Cosmos3-Nano \
  --omni \
  --model-class-name Cosmos3OmniDiffusersPipeline \
  --allowed-local-media-path / \
  --port 8000 \
  --init-timeout 1800

The README says Cosmos3 checkpoints can exceed the default initialization timeout, so --init-timeout 1800 is recommended.⁹

Text-to-video request:

curl -sS -X POST http://localhost:8000/v1/videos/sync \
  --form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \
  --form-string "negative_prompt=blurry, distorted, low quality" \
  --form-string "size=1280x720" \
  --form-string "num_frames=189" \
  --form-string "fps=24" \
  --form-string "num_inference_steps=35" \
  --form-string "guidance_scale=6.0" \
  --form-string "flow_shift=10.0" \
  --form-string "seed=0" \
  --form-string 'extra_params={"use_resolution_template":false,"use_duration_template":false,"guardrails":true}' \
  -o cosmos3_t2v_output.mp4

The README notes that --form-string should be used instead of -F for text fields to avoid curl truncation issues when values contain ;.¹⁰

Reasoner with vLLM

If you only need image/video understanding that returns text, Reasoner with vLLM is lighter than loading the full Generator path.

Install:

uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate

uv pip install --torch-backend=cu130 "vllm==0.21.0" \
  "vllm-cosmos3 @ git+https://github.com/NVIDIA/cosmos-framework.git#subdirectory=packages/vllm-cosmos3"

Start server:

vllm serve nvidia/Cosmos3-Nano \
  --hf-overrides '{"architectures": ["Cosmos3ReasonerForConditionalGeneration"]}' \
  --async-scheduling \
  --allowed-local-media-path / \
  --port 8000

The README describes this path as production inference for Reasoner behind an OpenAI-compatible chat-completions API.¹¹

Reasoner with NIM

For the easiest production Reasoner deployment, use Cosmos 3 Reasoner NIM. The README describes it as a prebuilt optimized container that serves text outputs from text, image and video inputs.¹²

Run:

export CONTAINER_NAME="nvidia-cosmos3-reasoner"
export IMG_NAME="nvcr.io/nim/nvidia/cosmos3-reasoner:1.7.0"
export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"

docker run -it --rm --name=$CONTAINER_NAME \
  --runtime=nvidia \
  --gpus all \
  --shm-size=32GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NIM_MODEL_SIZE=nano \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -u $(id -u) \
  -p 8000:8000 \
  $IMG_NAME

The API is available at:

http://127.0.0.1:8000/v1

Python client example:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-used")

response = client.chat.completions.create(
    model="nvidia/cosmos3-nano-reasoner",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {
            "role": "user",
            "content": [
                {"type": "video_url", "video_url": {"url": "https://download.samplelib.com/mp4/sample-5s.mp4"}},
                {"type": "text", "text": "List the notable events with approximate timestamps."},
            ],
        },
    ],
    max_tokens=256,
    stream=False,
    extra_body={"media_io_kwargs": {"video": {"fps": 4.0}}},
)

print(response.choices[0].message.content)

NIM is usually the easiest path when you want an OpenAI-compatible server without manual vLLM/CUDA setup.

Choosing an integration

The README provides a clear integration guide:¹³

Beginner path:

1. Read the README and cookbooks/cosmos3.
2. Try Cosmos3-Nano + Diffusers if you want generation.
3. Try NIM if you want a Reasoner API.
4. Use vLLM-Omni for Generator production.

Guardrails and safety

The README says Cosmos 3 ships with safety guardrails that screen prompts and blur faces in generated output. Guardrails can be disabled per request using extra_params={"guardrails": false}, or disabled server-wide through deploy configuration.¹⁴

For production, do not disable guardrails unless you have replacement controls. Generated images and videos may be used for simulation or synthetic data, but content involving people, faces, real locations or sensitive environments still needs policy review.

Known limitations

The README says Cosmos 3 can produce artifacts in long, high-resolution or physically complex outputs. Common failure modes include:¹⁵

• temporal inconsistency;
• unstable camera or object motion;
• inaccurate sound-video alignment;
• imperfect action-state consistency;
• object morphing;
• inaccurate 3D structure;
• implausible physical dynamics.

The README also warns that applications requiring physically grounded simulation, safety-critical control or complex multi-agent behavior need additional validation, guardrails and system-level safety analysis before deployment.¹⁵

Do not treat generated output as physically correct ground truth without validation.

Cosmos ecosystem

The README lists three related ecosystem projects:¹⁶

Think of NVIDIA/cosmos as the entry point and examples for Cosmos 3, while Cosmos Framework is where deeper setup, inference, training and evaluation workflows live.

License

The repository uses the OpenMDW-1.1 License for NVIDIA Cosmos source code and models.⁶ The license states that the model materials are provided “as is,” without warranty, and that users are responsible for third-party rights, consents, permissions and due diligence.⁶

Important notes:

• Do not assume it is Apache/MIT.
• Read OpenMDW-1.1 before commercial or redistribution use.
• The license text says it does not impose restrictions on outputs, but users are still responsible for rights and compliance.
• The README also says the project may download and install third-party open-source software and users should review those license terms before use.¹⁷

Personal setup guide

Goal: try Reasoner quickly

1. Create an NGC API key.
2. Log Docker into nvcr.io.
3. Run the NIM container.
4. Call the local OpenAI-compatible endpoint with curl or Python.

This avoids most vLLM/CUDA setup.

Goal: try video generation

1. Log into Hugging Face.
2. Create a Python 3.13 environment with uv.
3. Install Diffusers from GitHub and dependencies.
4. Load nvidia/Cosmos3-Nano.
5. Start with lower resolution/fewer frames if hardware is limited.

Goal: research or post-train

Use:

NVIDIA/cosmos-framework

Then follow framework training and evaluation docs.

Team deployment guide

Phase 1: define the use case

Decide:

• understanding or generation?
• robotics, autonomous driving or infrastructure?
• synthetic data or decision support?
• research prototype or production API?
• is latency or output quality more important?

Phase 2: pick runtime

Phase 3: standardize infrastructure

• NVIDIA Ampere/Hopper/Blackwell GPUs.
• CUDA 13 or 12.8.
• Large Hugging Face/NGC cache disks.
• Docker with NVIDIA Container Toolkit.
• GPU memory/utilization monitoring.
• Queueing for long video jobs.
• Artifact storage for MP4/JPG/JSON output.
• Metadata logging for prompt, seed, model, fps and resolution.

Phase 4: validate safety

• Compare with ground truth where possible.
• Check temporal consistency.
• Check object morphing.
• Check physical plausibility.
• Review face/person content.
• Do not use generated output for safety-critical control without external validation.

Production checklist

• Read OpenMDW-1.1.
• Choose Nano or Super.
• Pick Diffusers, vLLM-Omni, vLLM, NIM or Cosmos Framework.
• Pin package/container versions.
• Use Hugging Face/NGC tokens with appropriate scope.
• Use a dedicated disk cache.
• Run in containers where possible.
• Keep guardrails on unless replaced by other controls.
• Do not expose APIs without authentication.
• Set timeouts for video jobs.
• Record seed, model, prompt, resolution, FPS and steps.
• Store outputs and metadata for debugging.
• Add human review for important workflows.
• Do not treat model output as simulation ground truth without validation.

When should you use NVIDIA Cosmos?

Use it when:

• you work on robotics, autonomous vehicles, embodied AI or simulation;
• you need physical-world video/image understanding;
• you need synthetic data for training;
• you need future rollouts from image/video/action;
• you need a reasoner for physical common sense;
• you have suitable GPU infrastructure;
• you want to experiment with Cosmos 3.

Avoid it when:

• you only need a normal text chatbot;
• you do not have GPU infrastructure or hosted containers;
• real-time constraints are strict and unbenchmarked;
• the workflow is safety-critical and not independently validated;
• you have not reviewed the license;
• you need physically exact simulation.

Cosmos vs previous repositories

Cosmos operates at the Physical AI and world simulation layer, not the developer tooling or document-processing layer.

FAQ

What is NVIDIA Cosmos?

NVIDIA Cosmos is an open platform of world models, datasets and tools for building Physical AI systems such as robots, autonomous vehicles and smart infrastructure.²

What is Cosmos 3?

Cosmos 3 is NVIDIA’s omnimodal world model family that processes and generates language, images, video, audio and action sequences in a unified Mixture-of-Transformers architecture.³⁴

What is the difference between Reasoner and Generator?

Reasoner takes text/vision inputs and returns text or JSON for understanding, grounding, planning and reasoning. Generator takes text/vision/sound/action inputs and generates image, video, sound or action rollouts.³

Can Cosmos run on CPU?

Not realistically for the main workflows. The README lists BF16, Linux and NVIDIA Ampere/Hopper/Blackwell GPUs as the tested setup.³

Where should beginners start?

Start with Cosmos3-Nano, cookbooks/cosmos3, Hugging Face authentication, then Diffusers for generation or NIM for Reasoner API deployment.

Is Cosmos production-ready?

It has production paths through vLLM-Omni, vLLM and NIM, but the README warns that physically grounded simulation, safety-critical control and complex multi-agent behavior require validation, guardrails and system-level safety analysis.¹⁵

Conclusion

NVIDIA/cosmos is important for anyone working on Physical AI. It is not just a video generation repository. Its broader purpose is to help AI understand, simulate and predict the physical world for robotics, autonomous systems and embodied agents. Cosmos 3 exposes two main paths: Reasoner for understanding and planning, and Generator for simulation and multimodal generation.

For a beginner, the simplest mental model is: Reasoner answers “what is happening and what may happen next?”, while Generator creates “a possible world or future rollout.” For real deployment, focus on license, GPU requirements, CUDA compatibility, runtime choice, guardrails, benchmarking and validation.

References

Footnotes

1. GitHub raw asset. NVIDIA/cosmos/cosmos-logo-thumbnail.png. https://github.com/NVIDIA/cosmos/raw/main/cosmos-logo-thumbnail.png ↩

2. GitHub. NVIDIA/cosmos. https://github.com/NVIDIA/cosmos ↩ ↩² ↩³

3. NVIDIA Cosmos README. https://raw.githubusercontent.com/NVIDIA/cosmos/main/README.md ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴

4. “Cosmos 3: Omnimodal World Models for Physical AI.” https://arxiv.org/abs/2606.02800 ↩ ↩²

5. NVIDIA Cosmos README, News section, Cosmos 3 release note. https://raw.githubusercontent.com/NVIDIA/cosmos/main/README.md ↩

6. OpenMDW-1.1 License in NVIDIA/cosmos. https://raw.githubusercontent.com/NVIDIA/cosmos/main/LICENSE ↩ ↩² ↩³

7. NVIDIA Cosmos README, CUDA and troubleshooting notes. https://raw.githubusercontent.com/NVIDIA/cosmos/main/README.md ↩

8. NVIDIA Cosmos README, Quickstart and Hugging Face login. https://raw.githubusercontent.com/NVIDIA/cosmos/main/README.md ↩ ↩² ↩³

9. NVIDIA Cosmos README, Generator with vLLM-Omni. https://raw.githubusercontent.com/NVIDIA/cosmos/main/README.md ↩

10. NVIDIA Cosmos README, vLLM-Omni request fields and curl notes. https://raw.githubusercontent.com/NVIDIA/cosmos/main/README.md ↩

11. NVIDIA Cosmos README, Reasoner with vLLM. https://raw.githubusercontent.com/NVIDIA/cosmos/main/README.md ↩

12. NVIDIA Cosmos README, Reasoner with NIM. https://raw.githubusercontent.com/NVIDIA/cosmos/main/README.md ↩

13. NVIDIA Cosmos README, Choosing an Integration. https://raw.githubusercontent.com/NVIDIA/cosmos/main/README.md ↩

14. NVIDIA Cosmos README, guardrails configuration. https://raw.githubusercontent.com/NVIDIA/cosmos/main/README.md ↩

15. NVIDIA Cosmos README, Limitations. https://raw.githubusercontent.com/NVIDIA/cosmos/main/README.md ↩ ↩² ↩³

16. NVIDIA Cosmos README, Ecosystem. https://raw.githubusercontent.com/NVIDIA/cosmos/main/README.md ↩

17. NVIDIA Cosmos README, License and Contact. https://raw.githubusercontent.com/NVIDIA/cosmos/main/README.md ↩