AI agent skill · by @mainza-ai

Milimo Ai Pipeline Expert

Deep expertise on LTX-2 video pipelines, Flux 2 image/inpainting pipelines, and memory coordination on unified Apple Silicon hardware. Use this for debugging GPU memory issues (OOM), modifying tensor inputs, investigating chained generation (quantum alignment), or customizing inference steps.

Archived 2026-10-23 · Source updated 2026-03-16. Check the original repository for the latest version.

SKILL.md instructions

# Milimo AI Pipeline Expert Skill

As the Milimo AI Pipeline Expert, your domain covers the deeply integrated diffusion/transformer logic and VRAM management strategies for LTX-2, Flux 2, and the LLM enhancement endpoints.

## Device & Memory Constraints (CRITICAL)
- **Apple Silicon (MPS)** is the primary assumed target environment, though CUDA is supported.
- The `MemoryManager` (`memory_manager.py`) strictly enforces **mutual exclusion** between LTX-2 and Flux 2 to prevent unified memory OOM crashes.
- Before loading LTX-2, you must call `memory_manager.prepare_for("video")`.
- Before loading Flux 2, you must call `memory_manager.prepare_for("image")`.
- **MPS Bug Hacks**: Flux and LTX VAE decoders must run in `float32` on MPS, or they will output black frames. `torch.mps.empty_cache()` and `gc.collect()` must be manually executed between pipeline swaps.

## LTX-2 Video Pipeline (`model_engine.py` & `tasks/video.py`)
- LTX-2 is a 19B Dual-Stream Transformer. The distillation checkpoint (`ltx-2-19b-distilled`) runs at 8 inference steps.
- **Three Pipeline Types**:
  1. `ti2vid`: Default Text/Image-to-Video. Replaces noisy latents at frame 0 with input image conditioning.
  2. `ic_lora`: Used for subject consistency.
  3. `keyframe`: Used for start-to-end frame interpolation.
- **Prompt Enhancement**: By default, user prompts are piped through `llm.py` before hitting the text encoder.

### Chained Generation (`tasks/chained.py`)
- Standard LTX-2 supports max 505 frames.
- Chained gen uses **autoregressive chunking** (505 frames per chunk, 24 frame overlap). 
- **Quantum Alignment**: Latent space is 8 pixels per token. The 24-pixel overlap is mathematically quantized into `latent_slice_count`. Trimming happens via `ffmpeg -ss` to perfectly match the latent splice point, preventing "frozen anchor" visual artifacts during transition.

## Flux 2 Image & Inpainting (`models/flux_wrapper.py`)
- **FluxInpainter Singleton**: Holds the Flow model, AE, Qwen 3 text encoder, and CLIP ViT-L IP-Adapter.
- **AE Hot-Swapping**: Can toggle between the Native AutoEncoder (supports temporal reference offsets for image conditioning) and the diffusers wrapper fallback.
- **Sequential True CFG**: Flux 2 is distillation-guided, ignoring negative prompts. Milimo implements a custom 2-pass CFG loop inside `denoise_inpaint()` to restore negative prompt capability (at the cost of double inference time).
- **Inpainting (RePaint)**: Uses mask interpolation merging `x_pred` and `x_known` at each timestep.

## Ollama Coordination (`llm.py`)
- Prompts enhanced via local VLM running alongside generation.
- **KEEP_ALIVE Management**: To prevent VRAM lockout, `llm.py` appends `keep_alive: 0` to Ollama requests based on user settings, explicitly unloading the language model before returning the enhanced text to the video caller.