vLLM

What it does

The industry-standard engine for high-throughput model serving, used by many AI products behind the scenes.

Summary based on the project's own description. The Japanese page has a fuller beginner guide: setup difficulty, what to prepare, and a step-by-step start.

Screenshots & visuals