👁 JoyAI-Echo generated video gallery

Model Summary

JoyAI-Echo is a long-form, multi-shot, audio-video generation framework that breaks the barriers of error accumulation, weak temporal coherence, and prohibitive latency in long video generation. A cross-modal audio-visual memory bank preserves character appearance and voice timbre consistently over five-minute videos, while a post-training pipeline combining memory-based reinforcement learning with distribution matching distillation (DMD) delivers a 7.5× inference speedup without sacrificing quality.

JoyAI-Echo decisively outperforms HappyOyster (directing mode) on long-form generation and even surpasses the short-video specialist Wan 2.6 on human-centric tasks.

This repository hosts the released checkpoint. Inference code is released separately — see the Usage section.

Model Details

• Developed by: Echo Team @ Joy Future Academy, JD
• Model type: Text-to-(Audio+Video) diffusion transformer, DMD 8-step
• Modality: Text → synchronized video + audio
• Backbone: Built on top of LTX-Video
• Text encoder: google/gemma-3-12b-it (downloaded separately)
• Resolution / length (by default): 1280 × 736, 241 frames @ 25 fps per shot
• Max story length: up to 5 minutes (multi-shot)
• License: LTX-2 Community License Agreement

Highlights

• 🎞️ Minute-level multi-shot stories: generate a sequence of coherent shots from one prompt JSON.
• ⚡ DMD-distilled few-step inference: ~7.5× faster than the original pipeline.
• 🔊 Joint audio-video generation: one pipeline produces synchronized video and audio.
• 🧠 Paired cross-modal memory bank: conditions each new shot on prior visual identity and voice context for story-level consistency.

Demo Gallery

Explore long-form and short-form JoyAI-Echo cases on the Project Page. 🍿

Usage

Inference is run with the standalone JoyAI-Echo inference repository.

1. Download the checkpoint

huggingface-cli download jdopensource/JoyAI-Echo \
 --local-dir checkpoints

Also download the Gemma text encoder:

huggingface-cli download google/gemma-3-12b-it \
 --local-dir checkpoints/gemma-3-12b

Expected layout:

checkpoints/
├── echo-longvideo-release.safetensors
└── gemma-3-12b/

2. Get the inference code

git clone https://github.com/jd-opensource/JoyAI-Echo.git
cd JoyAI-Echo

Environment: Python 3.11 + PyTorch 2.8 + CUDA 12.8 (see the inference repo's environment.yml / requirements.txt).

3. Write a story prompt

Enhance your prompt first. We provide prompt enhancers — system prompts that expand a short story or idea into well-formed shot prompts: prompts/long_story_writer_system_prompt.md for long, multi-shot video, and prompts/short_story_writer_system_prompt.md for single-shot short video. We strongly recommend running your input through the matching enhancer before inference; un-enhanced prompts tend to produce noticeably weaker results.

Create a JSON file under prompts/. Each file is a single object with a prompts list, where every string is one complete shot. A single string produces one shot; multiple strings produce a multi-shot story, with each new shot conditioned on the previous ones through the paired audio-video memory bank.

Inside each string, write these parts in order:

4. Run

python inference.py

Outputs land in inference_result/outputs/<prompt-name>/inference_<timestamp>/.

Hardware

Peak GPU memory is ~46–50 GB at the default 1280 × 736 × 241 frame setting — a single H100/A100 (80 GB) or 48 GB GPU is sufficient. For smaller GPUs, lower resolution or frame count:

python inference.py --num-frames 121 --video-height 480 --video-width 832

Results

Reported Scale

Human Evaluation

GSB user study on long- and short-video generation. The numbers denote the percentage of user preferences.

Links

• Project page: https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/
• Inference code: https://github.com/jd-opensource/JoyAI-Echo
• HuggingFace: https://huggingface.co/jdopensource/JoyAI-Echo

Acknowledgements

We gratefully acknowledge the open-source projects this work builds upon — in particular LTX2.3 for the base video generator and Gemma for the text encoder. Thanks to the broader research community whose contributions made this release possible.

Citation

If JoyAI-Echo helps your research or products, please cite:

@techreport{echo2026JoyEcho,
 title = {JoyAI-Echo: Pushing the Frontier of Long Video Generation},
 author = {{Echo Team @ Joy Future Academy, JD}},
 institution = {Joy Future Academy, JD},
 year = {2026},
 month = {May}
}

License

This project is based on LTX-2 by Lightricks Ltd.

Portions of the original LTX-2 codebase have been modified by JD.com for academic and research purposes only. This project is not intended for commercial use. For commercial use of LTX-2 or its derivatives, please contact Lightricks Ltd.

All original copyright, license, patent, trademark, and attribution notices from LTX-2 are retained. This project remains subject to the LTX-2 Community License Agreement.