AI agent skill · by @aiming-lab

Data Loading

Optimize data loading pipeline to prevent GPU starvation. Use when setting up DataLoader or data preprocessing.

Archived 2026-10-23 · Source updated 2026-08-19 · License: MIT. Check the original repository for the latest version.

SKILL.md instructions

## Efficient Data Loading Best Practice
1. Use num_workers = min(8, os.cpu_count()) for DataLoader
2. Enable pin_memory=True when using GPU
3. Use persistent_workers=True to avoid re-spawning
4. Pre-compute and cache transformations when possible
5. For image data: use torchvision.transforms.v2 (faster)
6. For large datasets: consider memory-mapped files or WebDataset
7. Profile with torch.utils.bottleneck to find I/O bottlenecks