mirror of
https://github.com/alibaba/higress.git
synced 2026-03-06 09:30:49 +08:00
add model router plugin (#1414)
This commit is contained in:
63
plugins/wasm-cpp/extensions/model_router/README_EN.md
Normal file
63
plugins/wasm-cpp/extensions/model_router/README_EN.md
Normal file
@@ -0,0 +1,63 @@
|
||||
## Function Description
|
||||
The `model-router` plugin implements the functionality of routing based on the `model` parameter in the LLM protocol.
|
||||
|
||||
## Runtime Properties
|
||||
|
||||
Plugin Execution Phase: `Default Phase`
|
||||
Plugin Execution Priority: `260`
|
||||
|
||||
## Configuration Fields
|
||||
|
||||
| Name | Data Type | Filling Requirement | Default Value | Description |
|
||||
| -------------------- | ------------- | --------------------- | ---------------------- | ----------------------------------------------------- |
|
||||
| `enable` | bool | Optional | false | Whether to enable routing based on the `model` parameter |
|
||||
| `model_key` | string | Optional | model | The location of the `model` parameter in the request body |
|
||||
| `add_header_key` | string | Optional | x-higress-llm-provider | The header where the parsed provider name from the `model` parameter will be placed |
|
||||
|
||||
## Effect Description
|
||||
|
||||
To enable routing based on the `model` parameter, use the following configuration:
|
||||
|
||||
```yaml
|
||||
enable: true
|
||||
```
|
||||
|
||||
After enabling, the plugin extracts the provider part (if any) from the `model` parameter in the request, and sets it in the `x-higress-llm-provider` request header for subsequent routing. It also rewrites the `model` parameter to the model name part. For example, the original LLM request body is:
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "openai/gpt-4o",
|
||||
"frequency_penalty": 0,
|
||||
"max_tokens": 800,
|
||||
"stream": false,
|
||||
"messages": [{
|
||||
"role": "user",
|
||||
"content": "What is the GitHub address for the main repository of the Higress project?"
|
||||
}],
|
||||
"presence_penalty": 0,
|
||||
"temperature": 0.7,
|
||||
"top_p": 0.95
|
||||
}
|
||||
```
|
||||
|
||||
After processing by the plugin, the following request header (which can be used for routing matching) will be added:
|
||||
|
||||
`x-higress-llm-provider: openai`
|
||||
|
||||
The original LLM request body will be modified to:
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "gpt-4o",
|
||||
"frequency_penalty": 0,
|
||||
"max_tokens": 800,
|
||||
"stream": false,
|
||||
"messages": [{
|
||||
"role": "user",
|
||||
"content": "What is the GitHub address for the main repository of the Higress project?"
|
||||
}],
|
||||
"presence_penalty": 0,
|
||||
"temperature": 0.7,
|
||||
"top_p": 0.95
|
||||
}
|
||||
```
|
||||
Reference in New Issue
Block a user