# Serve CLI

`transformers serve` CLI 是本地或自托管服务器的轻量级选项。它避免了 vLLM 等专用推理引擎带来的额外运行时和运维开销。适用于评估、实验以及中等负载的部署场景。[连续批处理](../continuous_batching)等功能可提高吞吐量并降低延迟。

> [!TIP]
> 对于大规模生产部署，请使用 vLLM 或 SGLang，并以 Transformer 模型作为后端。有关更多信息，请参阅 [推理后端](../community_integrations/transformers_as_backend) 指南。

`transformers serve` 命令会启动一个与 [OpenAI SDK](https://platform.openai.com/docs/overview) 兼容的本地服务器。该服务器可与许多第三方应用程序配合使用，并支持以下 REST API。

- 用于文本、图像、音频和视频请求的 `/v1/chat/completions`
- `/v1/completions` 用于从自由格式提示词生成传统文本补全
- `/v1/responses` 支持 [Responses API](https://platform.openai.com/docs/api-reference/responses)
- `/v1/audio/transcriptions` 用于音频转录
- `/v1/systemone` 回答类型化问题（选择题、评分题、是/否题），并为每个允许的答案提供概率
- `/v1/models` 列出可用于第三方集成的模型
- `/load_model` 通过 SSE 流式传输模型加载进度

安装服务依赖项。

```bash
pip install transformers[serving]
```

运行 `transformers serve` 以启动服务器。默认服务器地址为 http://localhost:8000。

```shell
transformers serve
```

## v1/chat/completions

`v1/chat/completions` API 基于 [Chat Completions API](https://platform.openai.com/docs/api-reference/chat)。它支持针对 LLM、VLM 和多模态模型的文本、图像、音频和视频请求。您可以使用 `curl`、[InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) 或 [OpenAI](https://platform.openai.com/docs/quickstart) 客户端来调用该 API。

### 基于文本的补全

```python
from huggingface_hub import InferenceClient

messages = [{"role": "user", "content": "What is the Transformers library known for?"}]
client = InferenceClient("http://localhost:8000")

result = client.chat_completion(messages, model="Qwen/Qwen2.5-0.5B-Instruct", max_tokens=256)
print(result.choices[0].message.content)
```

[InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) 会返回一个打印的字符串。

```shell
The Transformers library is primarily known for its ability to create and manipulate large-scale language models [...]
```

```python
from huggingface_hub import InferenceClient

messages = [{"role": "user", "content": "What is the Transformers library known for?"}]
client = InferenceClient("http://localhost:8000")

stream = client.chat_completion(messages, model="Qwen/Qwen2.5-0.5B-Instruct", max_tokens=256, stream=True)
for chunk in stream:
    token = chunk.choices[0].delta.content
    if token:
        print(token, end="")
```

[InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) 会返回一个打印的字符串。

```shell
The Transformers library is primarily known for its ability to create and manipulate large-scale language models [...]
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

completion = client.chat.completions.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "What is the Transformers library known for?"
        }
    ],
)
print(completion.choices[0].message.content)
```

[OpenAI](https://platform.openai.com/docs/quickstart) 客户端会返回一个打印的字符串。

```shell
The Transformers library is primarily known for its ability to create and manipulate large-scale language models [...]
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

completion = client.chat.completions.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "What is the Transformers library known for?"
        }
    ],
    stream=True,
)

for chunk in completion:
    token = chunk.choices[0].delta.content
    if token:
        print(token, end="")
```

[OpenAI](https://platform.openai.com/docs/quickstart) 客户端会返回一个打印的字符串。

```shell
The Transformers library is primarily known for its ability to create and manipulate large-scale language models [...]
```

```shell
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [{"role": "user", "content": "What is the Transformers library known for?"}],
    "temperature": 0.9,
    "max_tokens": 256
  }'
```

该命令返回一个 JSON 字符串。

```json
{
  "id": "ade3578d-e2c3-47ed-b651-c014991a92f6",
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "message": {
        "content": "The Transformers library is primarily known for its ability to create and manipulate large-scale language models [...]",
        "role": "assistant"
      }
    }
  ],
  "created": 1776184004,
  "model": "Qwen/Qwen2.5-0.5B-Instruct@main",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 256,
    "prompt_tokens": 37,
    "total_tokens": 293
  }
}
```

```shell
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [{"role": "user", "content": "What is the Transformers library known for?"}],
    "temperature": 0.9,
    "max_tokens": 256,
    "stream": true
  }'
```

该命令返回一个数据块流。

```shell
data: {"id":"596d3941-754d-43b4-8773-9bf9940049f5","choices":[{"delta":{"role":"assistant"},"index":0}],"created":1776183469,"model":"Qwen/Qwen2.5-0.5B-Instruct@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"596d3941-754d-43b4-8773-9bf9940049f5","choices":[{"delta":{"content":"The"},"index":0}],"created":1776183469,"model":"Qwen/Qwen2.5-0.5B-Instruct@main","object":"chat.completion.chunk","system_fingerprint":""}

(...)
```

### 基于图像的补全

```python
from huggingface_hub import InferenceClient

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
                }
            },
        ],
    }
]
client = InferenceClient("http://localhost:8000")

result = client.chat_completion(messages, model="Qwen/Qwen2.5-VL-7B-Instruct", max_tokens=256)
print(result.choices[0].message.content)
```

[InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) 会返回一个打印的字符串。

```xmp
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.
```

```python
from huggingface_hub import InferenceClient

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
                }
            },
        ],
    }
]
client = InferenceClient("http://localhost:8000")

stream = client.chat_completion(messages, model="Qwen/Qwen2.5-VL-7B-Instruct", max_tokens=256, stream=True)
for chunk in stream:
    token = chunk.choices[0].delta.content
    if token:
        print(token, end="")
```

[InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) 会返回一个打印的字符串。

```xmp
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

completion = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-7B-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
                    }
                },
            ],
        }
    ],
)
print(completion.choices[0].message.content)
```

[OpenAI](https://platform.openai.com/docs/quickstart) 客户端会返回一个打印的字符串。

```xmp
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

completion = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-7B-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
                    }
                },
            ],
        }
    ],
    stream=True,
)

for chunk in completion:
    token = chunk.choices[0].delta.content
    if token:
        print(token, end="")
```

[OpenAI](https://platform.openai.com/docs/quickstart) 客户端会返回一个打印的字符串。

```xmp
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.
```

```shell
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What's in this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"
            }
          }
        ]
      }
    ],
    "max_tokens": 256
  }'
```

该命令返回一个 JSON 字符串。

```json
{
  "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon [...]",
        "role": "assistant"
      }
    }
  ],
  "created": 1753366665,
  "model": "Qwen/Qwen2.5-VL-7B-Instruct@main",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 50,
    "prompt_tokens": 120,
    "total_tokens": 170
  }
}
```

```shell
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
    "stream": true,
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What's in this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"
            }
          }
        ]
      }
    ],
    "max_tokens": 256
  }'
```

该命令返回一个数据块流。

```shell
data: {"id":"f47ac10b-58cc-4372-a567-0e02b2c3d479","choices":[{"delta":{"role":"assistant"},"index":0}],"created":1753366665,"model":"Qwen/Qwen2.5-VL-7B-Instruct@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"f47ac10b-58cc-4372-a567-0e02b2c3d479","choices":[{"delta":{"content":"The "},"index":0}],"created":1753366701,"model":"Qwen/Qwen2.5-VL-7B-Instruct@main","object":"chat.completion.chunk","system_fingerprint":""}

(...)
```

### 基于音频的补全

像 [Gemma 4](https://huggingface.co/google/gemma-4-E2B-it) 和 [Qwen2.5-Omni](https://huggingface.co/Qwen/Qwen2.5-Omni-3B) 这样的多模态模型，可以通过 OpenAI 的 `input_audio` 内容类型接收音频输入。请将音频进行 Base64 编码，并指定格式（`mp3` 或 `wav`）。

```python
import base64
from huggingface_hub.utils import httpx
from huggingface_hub import InferenceClient

audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Transcribe this audio."},
            {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
        ],
    }
]
client = InferenceClient("http://localhost:8000")

result = client.chat_completion(messages, model="google/gemma-4-E2B-it", max_tokens=256)
print(result.choices[0].message.content)
```

[InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) 会返回一个打印的字符串。

```
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.
```

```python
import base64
from huggingface_hub.utils import httpx
from huggingface_hub import InferenceClient

audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Transcribe this audio."},
            {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
        ],
    }
]
client = InferenceClient("http://localhost:8000")

stream = client.chat_completion(messages, model="google/gemma-4-E2B-it", max_tokens=256, stream=True)
for chunk in stream:
    token = chunk.choices[0].delta.content
    if token:
        print(token, end="")
```

[InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) 会返回一个打印的字符串。

```
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.
```

```python
import base64
from huggingface_hub.utils import httpx
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()

completion = client.chat.completions.create(
    model="google/gemma-4-E2B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Transcribe this audio."},
                {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
            ],
        }
    ],
)
print(completion.choices[0].message.content)
```

[OpenAI](https://platform.openai.com/docs/quickstart) 客户端会返回一个打印的字符串。

```
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.
```

```python
import base64
from huggingface_hub.utils import httpx
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()

completion = client.chat.completions.create(
    model="google/gemma-4-E2B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Transcribe this audio."},
                {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
            ],
        }
    ],
    stream=True,
)

for chunk in completion:
    token = chunk.choices[0].delta.content
    if token:
        print(token, end="")
```

[OpenAI](https://platform.openai.com/docs/quickstart) 客户端会返回一个打印的字符串。

```
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.
```

```shell
# First, base64-encode the audio file and build the JSON payload
AUDIO_B64=$(curl -sL https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3 | base64 -w 0)

cat <<EOF > /tmp/audio_request.json
{
  "model": "google/gemma-4-E2B-it",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "Transcribe this audio."},
        {"type": "input_audio", "input_audio": {"data": "$AUDIO_B64", "format": "mp3"}}
      ]
    }
  ]
}
EOF

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d @/tmp/audio_request.json
```

该命令返回一个 JSON 字符串。

```json
{
  "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "This week, I traveled to Chicago to deliver my final farewell address to the nation [...]",
        "role": "assistant"
      }
    }
  ],
  "created": 1753366665,
  "model": "google/gemma-4-E2B-it@main",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 85,
    "prompt_tokens": 200,
    "total_tokens": 285
  }
}
```

```shell
# First, base64-encode the audio file and build the JSON payload
AUDIO_B64=$(curl -sL https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3 | base64 -w 0)

cat <<EOF > /tmp/audio_request.json
{
  "model": "google/gemma-4-E2B-it",
  "stream": true,
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "Transcribe this audio."},
        {"type": "input_audio", "input_audio": {"data": "$AUDIO_B64", "format": "mp3"}}
      ]
    }
  ]
}
EOF

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d @/tmp/audio_request.json
```

该命令返回一个数据块流。

```shell
data: {"id":"cb997e1d-98b9-414a-be89-1880288610ef","choices":[{"delta":{"role":"assistant"},"index":0}],"created":1776254856,"model":"google/gemma-4-E2B-it@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"cb997e1d-98b9-414a-be89-1880288610ef","choices":[{"delta":{"content":"This"},"index":0}],"created":1776254856,"model":"google/gemma-4-E2B-it@main","object":"chat.completion.chunk","system_fingerprint":""}

(...)
```

> [!WARNING]
> `audio_url` 内容类型是一种扩展，不属于 OpenAI 标准，未来版本中可能会发生变化。

你还可以通过 `audio_url` 内容类型以 URL 形式传入音频，以跳过 base64 编码。

```python
completion = client.chat.completions.create(
    model="google/gemma-4-E2B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Transcribe this audio."},
                {"type": "audio_url", "audio_url": {"url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"}},
            ],
        }
    ],
)
```

### 基于视频的补全

> [!WARNING]
> `video_url` 内容类型是 OpenAI 标准之外的扩展，未来版本中可能会发生变化。

使用 `video_url` 内容类型进行视频输入。如果模型支持音频（例如 Gemma 4、Qwen2.5-Omni），服务器会从视频中提取音轨，并将其与视觉帧一起处理。

> [!TIP]
> 视频处理需要 [torchcodec](https://github.com/pytorch/torchcodec)。请使用 `pip install torchcodec` 进行安装。

```python
from huggingface_hub import InferenceClient

messages = [
    {
        "role": "user",
        "content": [
            {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
            {"type": "text", "text": "What is happening in the video and what is the song about?"},
        ],
    }
]
client = InferenceClient("http://localhost:8000")

result = client.chat_completion(messages, model="google/gemma-4-E2B-it", max_tokens=256)
print(result.choices[0].message.content)
```

[InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) 会返回一个打印的字符串。

```
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.

The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."
```

```python
from huggingface_hub import InferenceClient

messages = [
    {
        "role": "user",
        "content": [
            {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
            {"type": "text", "text": "What is happening in the video and what is the song about?"},
        ],
    }
]
client = InferenceClient("http://localhost:8000")

stream = client.chat_completion(messages, model="google/gemma-4-E2B-it", max_tokens=256, stream=True)
for chunk in stream:
    token = chunk.choices[0].delta.content
    if token:
        print(token, end="")
```

[InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) 会返回一个打印的字符串。

```
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.

The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

completion = client.chat.completions.create(
    model="google/gemma-4-E2B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
                {"type": "text", "text": "What is happening in the video and what is the song about?"},
            ],
        }
    ],
    max_tokens=256,
)
print(completion.choices[0].message.content)
```

[OpenAI](https://platform.openai.com/docs/quickstart) 客户端会返回一个打印的字符串。

```
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.

The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

completion = client.chat.completions.create(
    model="google/gemma-4-E2B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
                {"type": "text", "text": "What is happening in the video and what is the song about?"},
            ],
        }
    ],
    max_tokens=256,
    stream=True,
)

for chunk in completion:
    token = chunk.choices[0].delta.content
    if token:
        print(token, end="")
```

[OpenAI](https://platform.openai.com/docs/quickstart) 客户端会返回一个打印的字符串。

```
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.

The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."
```

```shell
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-E2B-it",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
          {"type": "text", "text": "What is happening in the video and what is the song about?"}
        ]
      }
    ],
    "max_tokens": 256
  }'
```

该命令返回一个 JSON 字符串。

```json
{
  "id": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "message": {
        "content": "The video captures a live music performance at a music festival or a large concert [...]",
        "role": "assistant"
      }
    }
  ],
  "created": 1753366665,
  "model": "google/gemma-4-E2B-it@main",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 256,
    "prompt_tokens": 350,
    "total_tokens": 606
  }
}
```

```shell
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-E2B-it",
    "stream": true,
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
          {"type": "text", "text": "What is happening in the video and what is the song about?"}
        ]
      }
    ],
    "max_tokens": 256
  }'
```

该命令返回一个数据块流。

```shell
data: {"id":"cb997e1d-98b9-414a-be89-1880288610ef","choices":[{"delta":{"role":"assistant"},"index":0}],"created":1776254856,"model":"google/gemma-4-E2B-it@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"cb997e1d-98b9-414a-be89-1880288610ef","choices":[{"delta":{"content":"Based"},"index":0}],"created":1776254856,"model":"google/gemma-4-E2B-it@main","object":"chat.completion.chunk","system_fingerprint":""}

(...)
```

### 多轮对话[[completions]]

要进行多轮对话，请在 `messages` 列表中提供完整的对话历史，并按 `user` 和 `assistant` 角色交替排列。与所有 OpenAI 兼容的服务器一样，该 API 是无状态的，因此每个请求都必须包含完整的对话历史。

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

completion = client.chat.completions.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    messages=[
        {"role": "user", "content": "What is the capital of France?"},
        {"role": "assistant", "content": "The capital of France is Paris."},
        {"role": "user", "content": "How many people live there?"},
    ],
)
print(completion.choices[0].message.content)
```

后续问题“那里有多少人居住？”依赖于先前的上下文，因此模型会针对巴黎进行回答。

```
As of 2021, the population of Paris is approximately 2.2 million people.
```

## v1/completions

`v1/completions` API 基于[旧版 Completions API](https://platform.openai.com/docs/api-reference/completions)。与 `/v1/chat/completions` 不同，它接受自由格式的文本 `prompt` 而非聊天消息，并在 `choices[].text` 中返回生成的文本。这对于基础（非指令）模型以及无需聊天模板的文本补全任务非常有用。它还支持 `suffix` 以实现中间填充式的文本插入。

```shell
curl -X POST http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B",
    "prompt": "The capital of France is",
    "max_tokens": 20
  }'
```

该命令返回以下响应。

```json
{
  "id": "chatcmpl-abc123",
  "object": "text_completion",
  "created": 1234567890,
  "model": "Qwen/Qwen2.5-0.5B@main",
  "choices": [
    {
      "text": " Paris, and the capital of the United States is Washington, D.C.",
      "index": 0,
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 5,
    "completion_tokens": 16,
    "total_tokens": 21
  }
}
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

# Non-streaming
completion = client.completions.create(
    model="Qwen/Qwen2.5-0.5B",
    prompt="The capital of France is",
    max_tokens=20,
)
print(completion.choices[0].text)
```

[OpenAI](https://platform.openai.com/docs/quickstart) 客户端返回以下内容。

```shell
 Paris, and the capital of the United States is Washington, D.C.
```

同样支持流式处理。

```python
stream = client.completions.create(
    model="Qwen/Qwen2.5-0.5B",
    prompt="The capital of France is",
    max_tokens=20,
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].text, end="")
```

## v1/responses

[Responses API](https://platform.openai.com/docs/api-reference/responses) 是 OpenAI 最新的生成 API 端点。它支持有状态交互，并集成了内置工具以扩展模型的功能。对于新项目，OpenAI [建议](https://platform.openai.com/docs/guides/migrate-to-responses)使用 Responses API 而非 Chat Completions API。

`v1/responses` API 支持通过 `curl` 命令和 [OpenAI](https://platform.openai.com/docs/quickstart) 客户端发送文本、图像、音频和视频请求。

### 基于文本的响应

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

response = client.responses.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    input="Tell me a three sentence bedtime story about a unicorn.",
    max_output_tokens=256,
    stream=False,
)
print(response.output[0].content[0].text)
```

[OpenAI](https://platform.openai.com/docs/quickstart) 客户端会返回一个打印的字符串。

```shell
Once upon a time, in a faraway land, there lived a beautiful unicorn named Luna [...]
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

response = client.responses.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    input="Tell me a three sentence bedtime story about a unicorn.",
    max_output_tokens=256,
    stream=True,
)

for event in response:
    if hasattr(event, "delta") and event.delta:
        print(event.delta, end="")
```

[OpenAI](https://platform.openai.com/docs/quickstart) 客户端会返回一个打印的字符串。

```shell
Once upon a time, in a faraway land, there lived a beautiful unicorn named Luna [...]
```

```shell
curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "input": "Tell me a three sentence bedtime story about a unicorn.",
    "max_output_tokens": 256,
    "stream": false
  }'
```

该命令返回一个 JSON 字符串。

```json
{
  "id": "resp_8693af9b-b561-4267-bd1a-22d1b43284cb",
  "created_at": 1776258580.7114341,
  "model": "Qwen/Qwen2.5-0.5B-Instruct@main",
  "object": "response",
  "output": [
    {
      "id": "msg_8693af9b-b561-4267-bd1a-22d1b43284cb",
      "content": [
        {
          "annotations": [],
          "text": "Once upon a time, in a faraway land, there lived a beautiful unicorn named Luna [...]",
          "type": "output_text"
        }
      ],
      "role": "assistant",
      "status": "completed",
      "type": "message"
    }
  ],
  "status": "completed"
}
```

```shell
curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "stream": true,
    "input": "Tell me a three sentence bedtime story about a unicorn.",
    "max_output_tokens": 256
  }'
```

该命令返回一个数据块流。

```shell
data: {"response":{"id":"resp_req_0","created_at":1754059817.783648,"model":"Qwen/Qwen2.5-0.5B-Instruct@main","object":"response","output":[],"status":"queued"},"sequence_number":0,"type":"response.created"}

data: {"content_index":0,"delta":"Once ","item_id":"msg_req_0","output_index":0,"sequence_number":5,"type":"response.output_text.delta"}

data: {"content_index":0,"delta":"upon ","item_id":"msg_req_0","output_index":0,"sequence_number":6,"type":"response.output_text.delta"}

(...)
```

### 基于图像的响应

Responses API 还支持图像、音频和视频输入。

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

response = client.responses.create(
    model="Qwen/Qwen2.5-VL-7B-Instruct",
    input=[
        {"type": "input_text", "text": "What's in this image?"},
        {
            "type": "input_image",
            "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
        },
    ],
    max_output_tokens=256,
    stream=False,
)
print(response.output[0].content[0].text)
```

[OpenAI](https://platform.openai.com/docs/quickstart) 客户端会返回一个打印的字符串。

```xmp
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

response = client.responses.create(
    model="Qwen/Qwen2.5-VL-7B-Instruct",
    input=[
        {"type": "input_text", "text": "What's in this image?"},
        {
            "type": "input_image",
            "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
        },
    ],
    max_output_tokens=256,
    stream=True,
)

for event in response:
    if hasattr(event, "delta") and event.delta:
        print(event.delta, end="")
```

[OpenAI](https://platform.openai.com/docs/quickstart) 客户端会返回一个打印的字符串。

```xmp
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.
```

```shell
curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
    "input": [
      {"type": "input_text", "text": "What'\''s in this image?"},
      {"type": "input_image", "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"}
    ],
    "max_output_tokens": 256,
    "stream": false
  }'
```

该命令返回一个 JSON 字符串。

```json
{
  "id": "resp_f47ac10b-58cc-4372-a567-0e02b2c3d479",
  "created_at": 1776258580.7114341,
  "model": "Qwen/Qwen2.5-VL-7B-Instruct@main",
  "object": "response",
  "output": [
    {
      "id": "msg_f47ac10b-58cc-4372-a567-0e02b2c3d479",
      "content": [
        {
          "annotations": [],
          "text": "The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon [...]",
          "type": "output_text"
        }
      ],
      "role": "assistant",
      "status": "completed",
      "type": "message"
    }
  ],
  "status": "completed"
}
```

```shell
curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
    "stream": true,
    "input": [
      {"type": "input_text", "text": "What'\''s in this image?"},
      {"type": "input_image", "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"}
    ],
    "max_output_tokens": 256
  }'
```

该命令返回一个数据块流。

```shell
data: {"response":{"id":"resp_f47ac10b-58cc-4372-a567-0e02b2c3d479","created_at":1776258580.7114341,"model":"Qwen/Qwen2.5-VL-7B-Instruct@main","object":"response","output":[],"status":"queued"},"sequence_number":0,"type":"response.created"}

data: {"content_index":0,"delta":"The ","item_id":"msg_f47ac10b","output_index":0,"sequence_number":5,"type":"response.output_text.delta"}

(...)
```

### 基于音频的响应

```python
import base64
from huggingface_hub.utils import httpx
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()

response = client.responses.create(
    model="google/gemma-4-E2B-it",
    input=[
        {"type": "input_text", "text": "Transcribe this audio."},
        {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
    ],
    max_output_tokens=256,
    stream=False,
)
print(response.output[0].content[0].text)
```

[OpenAI](https://platform.openai.com/docs/quickstart) 客户端会返回一个打印的字符串。

```
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.
```

```python
import base64
from huggingface_hub.utils import httpx
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()

response = client.responses.create(
    model="google/gemma-4-E2B-it",
    input=[
        {"type": "input_text", "text": "Transcribe this audio."},
        {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
            ],
        }
    ],
    max_output_tokens=256,
    stream=True,
)

for event in response:
    if hasattr(event, "delta") and event.delta:
        print(event.delta, end="")
```

[OpenAI](https://platform.openai.com/docs/quickstart) 客户端会返回一个打印的字符串。

```
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.
```

```shell
# First, base64-encode the audio file and build the JSON payload
AUDIO_B64=$(curl -sL https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3 | base64 -w 0)

cat <<EOF > /tmp/audio_request.json
{
  "model": "google/gemma-4-E2B-it",
  "input": [
    {"type": "input_text", "text": "Transcribe this audio."},
    {"type": "input_audio", "input_audio": {"data": "$AUDIO_B64", "format": "mp3"}}
  ],
  "max_output_tokens": 256,
  "stream": false
}
EOF

curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d @/tmp/audio_request.json
```

该命令返回一个 JSON 字符串。

```json
{
  "id": "resp_a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "created_at": 1776258580.7114341,
  "model": "google/gemma-4-E2B-it@main",
  "object": "response",
  "output": [
    {
      "id": "msg_a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "content": [
        {
          "annotations": [],
          "text": "This week, I traveled to Chicago to deliver my final farewell address to the nation [...]",
          "type": "output_text"
        }
      ],
      "role": "assistant",
      "status": "completed",
      "type": "message"
    }
  ],
  "status": "completed"
}
```

```shell
# First, base64-encode the audio file and build the JSON payload
AUDIO_B64=$(curl -sL https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3 | base64 -w 0)

cat <<EOF > /tmp/audio_request.json
{
  "model": "google/gemma-4-E2B-it",
  "stream": true,
  "input": [
    {"type": "input_text", "text": "Transcribe this audio."},
    {"type": "input_audio", "input_audio": {"data": "$AUDIO_B64", "format": "mp3"}}
  ],
  "max_output_tokens": 256
}
EOF

curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d @/tmp/audio_request.json
```

该命令返回一个数据块流。

```shell
data: {"response":{"id":"resp_a1b2c3d4-e5f6-7890-abcd-ef1234567890","created_at":1776258580.7114341,"model":"google/gemma-4-E2B-it@main","object":"response","output":[],"status":"queued"},"sequence_number":0,"type":"response.created"}

data: {"content_index":0,"delta":"This ","item_id":"msg_a1b2c3d4","output_index":0,"sequence_number":5,"type":"response.output_text.delta"}

(...)
```

> [!WARNING]
> `audio_url` 内容类型是一种扩展，不属于 OpenAI 标准，未来版本中可能会发生变化。

你还可以通过 `audio_url` 内容类型以 URL 形式传入音频，以跳过 base64 编码。

```python
response = client.responses.create(
    model="google/gemma-4-E2B-it",
    input=[
        {"type": "input_text", "text": "Transcribe this audio."},
        {"type": "audio_url", "audio_url": {"url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"}},
    ],
    max_output_tokens=256,
    stream=False,
)
```

### 基于视频的响应

> [!WARNING]
> `video_url` 内容类型是 OpenAI 标准之外的扩展，未来版本中可能会发生变化。

> [!TIP]
> 视频处理需要 [torchcodec](https://github.com/pytorch/torchcodec)。请使用 `pip install torchcodec` 进行安装。

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

response = client.responses.create(
    model="google/gemma-4-E2B-it",
    input=[
        {"type": "input_text", "text": "What is happening in the video and what is the song about?"},
        {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
    ],
    max_output_tokens=256,
    stream=False,
)
print(response.output[0].content[0].text)
```

[OpenAI](https://platform.openai.com/docs/quickstart) 客户端会返回一个打印的字符串。

```
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.

The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

response = client.responses.create(
    model="google/gemma-4-E2B-it",
    input=[
        {"type": "input_text", "text": "What is happening in the video and what is the song about?"},
        {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
    ],
    max_output_tokens=256,
    stream=True,
)

for event in response:
    if hasattr(event, "delta") and event.delta:
        print(event.delta, end="")
```

[OpenAI](https://platform.openai.com/docs/quickstart) 客户端会返回一个打印的字符串。

```
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.

The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."
```

```shell
curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-E2B-it",
    "input": [
      {"type": "input_text", "text": "What is happening in the video and what is the song about?"},
      {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}}
    ],
    "max_output_tokens": 256,
    "stream": false
  }'
```

该命令返回一个 JSON 字符串。

```json
{
  "id": "resp_b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "created_at": 1776258580.7114341,
  "model": "google/gemma-4-E2B-it@main",
  "object": "response",
  "output": [
    {
      "id": "msg_b2c3d4e5-f6a7-8901-bcde-f12345678901",
      "content": [
        {
          "annotations": [],
          "text": "The video captures a live music performance at a music festival or a large concert [...]",
          "type": "output_text"
        }
      ],
      "role": "assistant",
      "status": "completed",
      "type": "message"
    }
  ],
  "status": "completed"
}
```

```shell
curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-E2B-it",
    "stream": true,
    "input": [
      {"type": "input_text", "text": "What is happening in the video and what is the song about?"},
      {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}}
    ],
    "max_output_tokens": 256
  }'
```

该命令返回一个数据块流。

```shell
data: {"response":{"id":"resp_b2c3d4e5-f6a7-8901-bcde-f12345678901","created_at":1776258580.7114341,"model":"google/gemma-4-E2B-it@main","object":"response","output":[],"status":"queued"},"sequence_number":0,"type":"response.created"}

data: {"content_index":0,"delta":"Based ","item_id":"msg_b2c3d4e5","output_index":0,"sequence_number":5,"type":"response.output_text.delta"}

(...)
```

### 多轮对话[[responses]]

对于多轮对话，请在 `input` 字段中传入一个消息列表，每条消息都包含 `role` 键。与所有 OpenAI 兼容的服务器一样，该 API 是无状态的，因此每个请求都必须包含完整的对话历史。

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

response = client.responses.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    input=[
        {"role": "user", "content": "What is the capital of France?"},
        {"role": "assistant", "content": "The capital of France is Paris."},
        {"role": "user", "content": "How many people live there?"},
    ],
    max_output_tokens=256,
    stream=False,
)
print(response.output[0].content[0].text)
```

后续问题“那里有多少人居住？”依赖于先前的上下文，因此模型会针对巴黎进行回答。

```
As of 2021, Paris has a population of approximately 2.8 million people.
```

## v1/systemone

`v1/systemone` 端点用于回答有关共享 `state` 的带类型问题，遵循 [TypeSafe 的 System One API](https://api.typesafe.ai/openapi.json)。可用于支持工单路由、紧急程度评级或操作选择等决策。

模型在一次批量前向传播中对所有允许的候选答案进行评分，而无需生成文本。该端点支持带有聊天模板的自回归模型；

发送模型、要评估的状态以及按名称索引的问题：

```shell
curl http://localhost:8000/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "state": "Our integration has been failing for 3 days and we are losing sales!",
    "questions": {
      "team": {
        "type": "choice",
        "instructions": "Which team handles this?",
        "criteria": {"billing": null, "technical": "Bugs or integration problems"}
      },
      "urgency": {
        "type": "score",
        "instructions": "How urgent is this ticket?",
        "criteria": ["low", "medium", "high"]
      },
      "upset": {"type": "noul", "instructions": "Is the customer upset?"}
    }
  }'
```

安装适用于 Node.js 20 或更高版本的 [TypeSafe SDK](https://github.com/typesafe-ai/typesafe-sdk-js)：

```shell
npm install @typesafe-ai/sdk
```

将 `baseURL` 设置为服务器根路径；SDK 会自动添加 `/v1/systemone`。SDK 需要一个 API 密钥，但在本地服务器上可以使用占位符。

```typescript
import { choice, noul, score, TypeSafeClient } from "@typesafe-ai/sdk";

const client = new TypeSafeClient({
  baseURL: "http://localhost:8000",
  apiKey: "<random_string>",
});

const response = await client.systemOne({
  model: "Qwen/Qwen2.5-1.5B-Instruct",
  state: "Our integration has been failing for 3 days and we are losing sales!",
  questions: {
    team: choice("Which team handles this?", {
      billing: null,
      technical: "Bugs or integration problems",
    }),
    urgency: score("How urgent is this ticket?", ["low", "medium", "high"]),
    upset: noul("Is the customer upset?"),
  },
});

console.log(response.answers.team.choice);
console.log(response.answers.urgency.score);
console.log(response.answers.upset.noul);
```

客户端会打印所选团队、紧急程度分数以及客户不满的概率。例如：

```shell
technical
1.6739138662815094
0.851952850818634
```

答案使用与问题相同的名称。例如：

```json
{
  "model": "Qwen/Qwen2.5-1.5B-Instruct",
  "answers": {
    "team": {"type": "choice", "choice": "technical", "probabilities": {"billing": 0.000006962256520637311, "technical": 0.999993085861206}, "confidence": 0.9999861717224121},
    "urgency": {"type": "score", "score": 1.6739138662815094, "legend": {"0": "low", "1": "medium", "2": "high"}, "probabilities": {"0": 0.0031353600788861513, "1": 0.31981542706489563, "2": 0.6770492196083069}, "confidence": 0.5108707791659981},
    "upset": {"type": "noul", "noul": 0.851952850818634}
  },
  "usage": {"input_tokens": 213, "output_tokens": 0}
}
```

| 问题类型 | `criteria` | 答案 |
| :--- | :--- | :--- |
| `choice` | 映射到描述的选项名称，可以为 `null`。 | 最可能的选项、选项概率以及置信度。 |
| `score` | 按从低到高的顺序排列的级别描述。 | 概率加权层级索引、概率、图例以及置信度。 |
| `noul` | `true` 和 `false` 的可选描述。 | `true` 的概率。 |

`state`、可选的 `instructions` 以及描述可以包含文本或 JSON 对象和数组。每个请求最多接受 64 个问题，每个问题包含 1 到 26 个选项或评分等级。

对于支持视觉的模型，请添加一个 `images` 数组，其中最多包含四个 `data:image/...;base64,...` URL。每个问题都会接收相同的图像。模型的处理器和聊天模板必须支持图像；不接受远程图像 URL。

### 决策配置

决策模型目前尚未就提示词、答案标签和校准设置的存储方式达成统一标准。由于不同模型实现之间存在差异，`--decision-config` 允许你显式地提供这些配置。具有自定义决策头的模型仍然需要兼容的实现；该文件仅用于配置提示词格式和下一个 token 的评分。

默认情况下，问题使用回退提示词以及禁用思考功能的模型聊天模板。对于选项和分数，答案标签为 `A`–`Z`；对于布尔值，则为 `Yes`/`No`。概率是这些标签在温度为 1 时的 softmax 值。

传入 `--decision-config` 以覆盖模型的这些默认值：

```shell
transformers serve openjev/openjev --decision-config decision.json
```

JSON 文件支持三个可选字段：

| 字段 | 含义 |
| :--- | :--- |
| `labels` | 按问题类型和选项顺序标注答案（对于 `noul`，先 `true`，后 `false`）。每个标签必须是一个不同的、已知的单个 token。 |
| `temperature` | 按问题类型划分的正温度值，应用方式为 `softmax(label_logits / temperature)`。 |
| `chat_template` | 一个内联 Jinja 模板，用于替换完整的提示词，包括聊天格式和助手前缀。 |

例如，要在保留默认提示词的同时更改标签和校准：

```json
{
  "labels": {"noul": ["A", "B"]},
  "temperature": {"choice": 0.85, "score": 0.85, "noul": 1.5547129}
}
```

自定义模板接收常规的 `messages`，以及原始的 `state`、`images` 和 `question` 字段。question 字段暴露了 `name`、`type`、`instructions`、`options`、`labels`、`descriptions` 和 `legend`；options、labels 和 descriptions 共享相同的顺序。使用这些字段来替换回退措辞，并使用 `tojson` 来渲染结构化值。用于图像输入的模板必须包含模型的图像占位符。

该文件在启动时加载，仅适用于 `/v1/systemone`。省略的设置将保留其默认值；不会自动从检查点读取决策配置。

## v1/audio/transcriptions

`v1/audio/transcriptions` 端点使用语音转文本模型对音频进行转录。它遵循 [Audio transcription API](https://platform.openai.com/docs/api-reference/audio/createTranscription) 格式。

```shell
curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -H "Content-Type: multipart/form-data" \
  -F "file=@/path/to/audio.wav" \
  -F "model=openai/whisper-large-v3"
```

该命令返回以下响应。

```shell
{
  "text": "Transcribed text from the audio file",
}
```

## v1/models

`v1/models` 端点会扫描您的本地 Hugging Face 缓存，并以 OpenAI 兼容格式返回已下载模型的列表。第三方工具使用该端点来发现可用的模型。

在运行 `transformers serve` 之前，请先下载一个模型。

```bash
transformers download Qwen/Qwen2.5-0.5B-Instruct
```

下载完成后，该模型会出现在 `/v1/models` 的响应中。

```shell
curl http://localhost:8000/v1/models
```

该端点返回一个包含可用模型的 JSON 对象。

## 加载模型

`/load_model` 端点会预先加载模型，并通过 [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events)（SSE）流式传输进度。`transformers chat` CLI 会自动使用该端点，使用户能够看到下载和加载进度，而不是面对一个挂起的提示符。请在发送推理请求之前使用它来预热模型。

### 请求

```shell
curl -N -X POST http://localhost:8000/load_model \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct"}'
```

`model` 字段是一个 Hugging Face 模型标识符，可选地带有 `@revision` 后缀（`Qwen/Qwen2.5-0.5B-Instruct@main`）。省略 revision 时，默认使用 `main`。

### 响应

响应是一个 SSE 流（`Content-Type: text/event-stream`）。每个帧都是 `data:` 行上的一个 JSON 对象。

```
data: {"status": "loading", "model": "Qwen/Qwen2.5-0.5B-Instruct@main", "stage": "processor"}
```

每个事件至少包含 `status` 和 `model` 字段。其他字段取决于状态。

| 字段 | 存在时 | 描述 |
|-------|-------------|-------------|
| `status` | 始终 | `loading`、`ready` 或 `error` |
| `model` | 始终 | 标准 `model_id@revision` |
| `stage` | `status == "loading"` | `processor`、`config`、`download`、`weights` 之一（参见下文各阶段） |
| `progress` | `download` 和 `weights` 阶段 | 包含 `current` 和 `total`（整数或 null）的对象 |
| `cached` | `status == "ready"` | 如果模型已在内存中，则为 `true` |
| `message` | `status == "error"` | 错误描述 |

### 阶段

加载按顺序经历以下阶段。某些阶段可能会被跳过（当文件已在本地缓存时，会跳过 `download`）。

| 阶段 | 有进展吗？ | 描述 |
|-------|---------------|-------------|
| `processor` | No | 加载分词器/处理器 |
| `config` | No | 加载模型配置 |
| `download` | 是（字节） | 下载模型文件 |
| `weights` | 是（条目） | 将权重张量加载到内存 |

流以且仅以一个终止事件结束：`ready`（成功）或 `error`（失败）。

## 超时

`transformers serve` 处理任意模型的请求。每个模型按需加载，并驻留在 GPU 内存中。模型在 300 秒无活动后会自动卸载，以释放 GPU 内存。将 `--model-timeout` 设置为不同的秒数，或设置为 `-1` 以禁用自动卸载。

```shell
transformers serve --model-timeout 400
```

### 加载示例

以下示例分别展示了全新下载的模型、从本地缓存加载的模型（跳过下载阶段）以及已在内存中的模型的响应。

```
data: {"status": "loading", "model": "org/model@main", "stage": "processor"}
data: {"status": "loading", "model": "org/model@main", "stage": "config"}
data: {"status": "loading", "model": "org/model@main", "stage": "download", "progress": {"current": 0, "total": 269100000}}
data: {"status": "loading", "model": "org/model@main", "stage": "download", "progress": {"current": 134600000, "total": 269100000}}
data: {"status": "loading", "model": "org/model@main", "stage": "download", "progress": {"current": 269100000, "total": 269100000}}
data: {"status": "loading", "model": "org/model@main", "stage": "weights", "progress": {"current": 1, "total": 272}}
data: {"status": "loading", "model": "org/model@main", "stage": "weights", "progress": {"current": 272, "total": 272}}
data: {"status": "ready", "model": "org/model@main", "cached": false}
```

```
data: {"status": "loading", "model": "org/model@main", "stage": "processor"}
data: {"status": "loading", "model": "org/model@main", "stage": "config"}
data: {"status": "loading", "model": "org/model@main", "stage": "weights", "progress": {"current": 1, "total": 272}}
data: {"status": "loading", "model": "org/model@main", "stage": "weights", "progress": {"current": 272, "total": 272}}
data: {"status": "ready", "model": "org/model@main", "cached": false}
```

```
data: {"status": "ready", "model": "org/model@main", "cached": true}
```

## 工具调用

`transformers serve` 服务器支持 OpenAI 风格的函数调用。经过工具使用训练的模型会生成结构化的函数调用，由您的应用程序执行。

> [!NOTE]
> 工具调用适用于任何其分词器声明了工具调用令牌的模型。Qwen 和 Gemma 4 开箱即用。如需支持特定模型，请提交一个 [issue](https://github.com/huggingface/transformers/issues/new/choose)。

按照 OpenAI 格式，将工具定义为函数规范列表。

```py
import json
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<KEY>")

tools = [
  {
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Get the current weather in a location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {
            "type": "string",
            "description": "The city name, e.g. San Francisco"
          },
          "unit": {
            "type": "string",
            "enum": ["celsius", "fahrenheit"],
            "description": "temperature unit"
          }
        },
        "required": ["location"]
      }
    }
  }
]
```

通过向 [create](https://platform.openai.com/docs/api-reference/responses/create) 中的 `extra_body` 参数传递 [GenerationConfig](/docs/transformers/main/zh/main_classes/text_generation#transformers.GenerationConfig) 参数来自定义生成。

```py
generation_config = {
  "max_new_tokens": 512,
  "temperature": 0.7,
  "top_p": 0.9,
  "top_k": 50,
  "do_sample": True,
  "repetition_penalty": 1.1,
  "no_repeat_ngram_size": 3,
}

response = client.responses.create(
  model="Qwen/Qwen2.5-7B-Instruct",
  instructions="You are a helpful weather assistant. Use the get_weather tool to answer questions.",
  input="What's the weather like in San Francisco?",
  tools=tools,
  stream=True,
  extra_body={"generation_config": json.dumps(generation_config)}
)

for event in response:
  print(event)
```

### 多轮工具调用

模型返回工具调用后，在本地执行该函数，然后通过后续请求将结果发送回去，以获取模型的最终答案。这两种 API 之间的模式略有不同。有关完整规范，请参阅 [OpenAI 函数调用指南](https://developers.openai.com/api/docs/guides/function-calling?api-mode=chat)。

以下示例复用了上面定义的 `tools` 列表。

将工具结果作为带有匹配 `tool_call_id` 的 `role: "tool"` 消息传递。

```py
# Model returns a tool call
messages = [{"role": "user", "content": "What's the weather like in San Francisco?"}]
response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=messages,
    tools=tools,
)
assistant_message = response.choices[0].message

# Execute the tool locally
tool_call = assistant_message.tool_calls[0]
result = {"temperature": 22, "condition": "sunny"}  # your actual function call here

# Send the tool result back
messages.append(assistant_message)
messages.append({
    "role": "tool",
    "tool_call_id": tool_call.id,
    "content": json.dumps(result),
})
final_response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=messages,
    tools=tools,
)
print(final_response.choices[0].message.content)
```

在后续请求的 `input` 列表中，将工具结果作为 `function_call_output` 项传入。

```py
user_message = {"role": "user", "content": "What's the weather like in San Francisco?"}
response = client.responses.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    input=[user_message],
    tools=tools,
    stream=False,
)
tool_call = next(item for item in response.output if item.type == "function_call")

result = {"temperature": 22, "condition": "sunny"}

final_response = client.responses.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    input=[
        user_message,
        tool_call.model_dump(exclude_none=True),
        {"type": "function_call_output", "call_id": tool_call.call_id, "output": json.dumps(result)},
    ],
    tools=tools,
    stream=False,
)
print(final_response.output_text)
```

## 推理

推理模型在输出最终答案之前会生成一段隐藏的思维链。服务器会检测这些思考片段，从可见答案中去除分隔符，并将推理文本放入专用字段，以便客户端将其单独渲染。

- Chat Completions 会在助手消息中通过 `reasoning_content` 返回推理内容。
- Responses API 将推理内容作为 `reasoning` 输出项返回，该输出项位于 `message` 项之前。

推理检测依赖于[响应模板](../chat_response_parsing)，该模板告知服务器思考片段的起始和结束位置。服务器会按顺序在两个位置中查找该模板。

1. 如果模型附带了该属性，则使用分词器的 `response_template` 属性。
2. 针对尚未提供模板的分词器，根据模型架构匹配的内建回退方案。

对于既没有这两种情况的模型，服务将不进行响应解析。原始解码文本会作为 `content` 返回，并且不会显示任何推理字段，即使模型输出了 `<think>...</think>` 标签也是如此。

### 在服务器上启用推理

使用 `--reasoning` 来控制聊天模板是否输出思考 token。

| 值 | 行为 |
|---|---|
| `auto`（默认值） | 回退到聊天模板的默认值。 |
| `on` | 通过在聊天模板的 kwargs 中设置 `enable_thinking=True` 来强制启用思考。 |
| `off` | 通过设置 `enable_thinking=False` 来强制关闭思考功能。 |

```sh
transformers serve Qwen/Qwen3-1.7B --reasoning on
```

> [!WARNING]
> 并非所有模型都支持推理。`--reasoning` 仅在模型的聊天模板处理 `enable_thinking` 变量（或其分词器声明了思考分隔符）时生效。对于其他模型，该标志会被静默忽略。

`--reasoning on` 和 `--reasoning off` 通过设置聊天模板变量 `enable_thinking` 来工作。某些聊天模板使用不同的变量名来切换思考模式，或者接受额外的变量以更改提示的渲染方式。可以直接通过 `--chat-template-kwargs` 以 JSON 对象的形式传入任何聊天模板变量。

```sh
transformers serve Qwen/Qwen3-1.7B \
  --chat-template-kwargs '{"enable_thinking": true}'
```

客户端还可以在请求体中发送 `chat_template_kwargs`，以在单次请求中覆盖服务器默认值，而无需重启服务器。

### Chat Completions

非流式响应会在助手消息中同时包含 `reasoning_content` 和 `content`。

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<KEY>")

completion = client.chat.completions.create(
    model="Qwen/Qwen3-1.7B",
    messages=[{"role": "user", "content": "What is 17 * 23?"}],
)
message = completion.choices[0].message
print("reasoning:", message.reasoning_content)
print("answer:", message.content)
```

流式输出会先发出 `reasoning_content` 增量，再发出 `content` 增量。请分别拼接每个字段。

```python
chunks = list(
    client.chat.completions.create(
        model="Qwen/Qwen3-1.7B",
        messages=[{"role": "user", "content": "What is 17 * 23?"}],
        stream=True,
    )
)
reasoning = "".join(getattr(c.choices[0].delta, "reasoning_content", None) or "" for c in chunks)
content = "".join(c.choices[0].delta.content or "" for c in chunks)
```

```shell
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-1.7B",
    "messages": [{"role": "user", "content": "What is 17 * 23?"}],
    "chat_template_kwargs": {"enable_thinking": true}
  }'
```

响应中的助手消息包含 `reasoning_content`。

```json
{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "17 * 23 = 391.",
        "reasoning_content": "Let me compute 17 * 23. 17 * 20 = 340, 17 * 3 = 51, so 340 + 51 = 391."
      },
      "finish_reason": "stop",
      "index": 0
    }
  ]
}
```

### Responses API

Responses API 将推理内容作为独立的 `reasoning` 输出项返回，该输出项位于 `message` 项之前。

```python
response = client.responses.create(
    model="Qwen/Qwen3-1.7B",
    input="What is 17 * 23?",
    stream=False,
)
reasoning_item = next(i for i in response.output if i.type == "reasoning")
message_item = next(i for i in response.output if i.type == "message")
print("reasoning:", reasoning_item.content[0].text)
print("answer:", message_item.content[0].text)
```

流式处理会先打开推理项并发出 `response.reasoning_text.delta` 事件，然后关闭该项并打开消息项，以发出 `response.output_text.delta` 事件。

```python
stream = client.responses.create(
    model="Qwen/Qwen3-1.7B",
    input="What is 17 * 23?",
    stream=True,
)

for event in stream:
    if event.type == "response.reasoning_text.delta":
        print(event.delta, end="")
    elif event.type == "response.output_text.delta":
        print(event.delta, end="")
```

### 多轮往返

对于多轮对话，请在下一次请求中包含上一次响应中的推理文本。服务器会将其传递给聊天模板，该模板可以将其渲染为先前助手轮次的一部分，从而使模型能够看到自己之前的思考过程。

将 `reasoning_content` 附加到上一条助手消息。

```python
first = client.chat.completions.create(
    model="Qwen/Qwen3-1.7B",
    messages=[{"role": "user", "content": "What is 17 * 23?"}],
).choices[0].message

second = client.chat.completions.create(
    model="Qwen/Qwen3-1.7B",
    messages=[
        {"role": "user", "content": "What is 17 * 23?"},
        {
            "role": "assistant",
            "content": first.content,
            "reasoning_content": first.reasoning_content,
        },
        {"role": "user", "content": "Now multiply that result by 2."},
    ],
)
print(second.choices[0].message.content)
```

在助手消息之前，将先前的 `reasoning` 项作为输入项传回。

```python
first = client.responses.create(
    model="Qwen/Qwen3-1.7B",
    input="What is 17 * 23?",
    stream=False,
)
reasoning_item = next(i for i in first.output if i.type == "reasoning")
message_item = next(i for i in first.output if i.type == "message")

second = client.responses.create(
    model="Qwen/Qwen3-1.7B",
    input=[
        {"role": "user", "content": "What is 17 * 23?"},
        reasoning_item.model_dump(exclude_none=True),
        {"role": "assistant", "content": message_item.content[0].text},
        {"role": "user", "content": "Now multiply that result by 2."},
    ],
    stream=False,
)
print(second.output_text)
```

## 端口转发

端口转发允许您从远程服务器提供模型服务。请确保您拥有该服务器的 SSH 访问权限，然后在您的本地机器上运行此命令。

```bash
ssh -N -f -L 8000:localhost:8000 your_server_account@your_server_IP -p port_to_ssh_into_your_server
```

## 可复现性

将模型作为位置参数传入，以避免每次请求时的模型提示，从而产生稳定且可重复的运行结果。

```sh
transformers serve Qwen/Qwen2.5-0.5B-Instruct \
  --continuous-batching \
  --dtype "bfloat16"
```