jiazhizhong/higress
1
0
Fork 0
mirror of https://github.com/alibaba/higress.git synced 2026-03-20 02:07:27 +08:00
Code Issues Packages Projects Releases Wiki Activity

add model-mapper plugin & optimize model-router plugin (#1538)

Browse Source
This commit is contained in:
澄潭
2024-11-22 22:24:42 +08:00
committed by GitHub
parent 96575b982e
commit e68a8ac25f
11 changed files with 1067 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files

76
plugins/wasm-cpp/extensions/model_router/README_EN.md
View File

@@ -1,38 +1,41 @@
## Function Description
The `model-router` plugin implements the functionality of routing based on the `model` parameter in the LLM protocol.
## Runtime Properties
Plugin Execution Phase: `Default Phase`
Plugin Execution Priority: `260`
The `model-router` plugin implements the function of routing based on the model parameter in the LLM protocol.
## Configuration Fields
| Name | Data Type | Filling Requirement | Default Value | Description |
| -------------------- | ------------- | --------------------- | ---------------------- | ----------------------------------------------------- |
| `enable` | bool | Optional | false | Whether to enable routing based on the `model` parameter |
| `model_key` | string | Optional | model | The location of the `model` parameter in the request body |
| `add_header_key` | string | Optional | x-higress-llm-provider | The header where the parsed provider name from the `model` parameter will be placed |
| Name | Data Type | Filling Requirement | Default Value | Description |
| ----------- | --------------- | ----------------------- | ------ | ------------------------------------------- |
| `modelKey` | string | Optional | model | The location of the model parameter in the request body |
| `addProviderHeader` | string | Optional | - | Which request header to place the provider name parsed from the model parameter |
| `modelToHeader` | string | Optional | - | Which request header to directly place the model parameter |
| `enableOnPathSuffix` | array of string | Optional | ["/v1/chat/completions"] | Only effective for requests with these specific path suffixes |
## Runtime Attributes
Plugin execution phase: Authentication phase
Plugin execution priority: 900
## Effect Description
To enable routing based on the `model` parameter, use the following configuration:
### Routing Based on the model Parameter
The following configuration is required:
```yaml
enable: true
modelToHeader: x-higress-llm-model
```
After enabling, the plugin extracts the provider part (if any) from the `model` parameter in the request, and sets it in the `x-higress-llm-provider` request header for subsequent routing. It also rewrites the `model` parameter to the model name part. For example, the original LLM request body is:
The plugin will extract the model parameter from the request and set it in the x-higress-llm-model request header, which can be used for subsequent routing. For example, the original LLM request body:
```json
{
"model": "openai/gpt-4o",
"model": "qwen-long",
"frequency_penalty": 0,
"max_tokens": 800,
"stream": false,
"messages": [{
"role": "user",
"content": "What is the GitHub address for the main repository of the Higress project?"
"content": "What is the GitHub address of the main repository for the higress project"
}],
"presence_penalty": 0,
"temperature": 0.7,
@@ -40,24 +43,55 @@ After enabling, the plugin extracts the provider part (if any) from the `model`
}
```
After processing by the plugin, the following request header (which can be used for routing matching) will be added:
After processing by this plugin, the following request header (which can be used for route matching) will be added:
`x-higress-llm-provider: openai`
x-higress-llm-model: qwen-long
The original LLM request body will be modified to:
### Extracting the provider Field from the model Parameter for Routing
> Note that this mode requires the client to specify the provider using a `/` separator in the model parameter.
The following configuration is required:
```yaml
addProviderHeader: x-higress-llm-provider
```
The plugin will extract the provider part (if present) from the model parameter in the request and set it in the x-higress-llm-provider request header, which can be used for subsequent routing, and rewrite the model parameter to the model name part. For example, the original LLM request body:
```json
{
"model": "gpt-4o",
"model": "dashscope/qwen-long",
"frequency_penalty": 0,
"max_tokens": 800,
"stream": false,
"messages": [{
"role": "user",
"content": "What is the GitHub address for the main repository of the Higress project?"
"content": "What is the GitHub address of the main repository for the higress project"
}],
"presence_penalty": 0,
"temperature": 0.7,
"top_p": 0.95
}
```
After processing by this plugin, the following request header (which can be used for route matching) will be added:
x-higress-llm-provider: dashscope
The original LLM request body will be changed to:
```json
{
"model": "qwen-long",
"frequency_penalty": 0,
"max_tokens": 800,
"stream": false,
"messages": [{
"role": "user",
"content": "What is the GitHub address of the main repository for the higress project"
}],
"presence_penalty": 0,
"temperature": 0.7,
"top_p": 0.95
}