From 0471249e7f9b69f781df6a4b309a7b1f2188e9f8 Mon Sep 17 00:00:00 2001 From: xingyunyang01 <94745901+xingyunyang01@users.noreply.github.com> Date: Wed, 18 Sep 2024 10:52:23 +0800 Subject: [PATCH] =?UTF-8?q?ai-agent=E6=8F=92=E4=BB=B6=E6=96=B0=E7=89=88?= =?UTF-8?q?=E6=9C=AC=20(#1311)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Kent Dong --- plugins/wasm-go/extensions/ai-agent/README.md | 43 ++- .../wasm-go/extensions/ai-agent/README_EN.md | 31 +- plugins/wasm-go/extensions/ai-agent/config.go | 98 +++--- plugins/wasm-go/extensions/ai-agent/main.go | 309 ++++++++++-------- .../extensions/ai-agent/promptTpl/prompt.go | 164 +++++++--- 5 files changed, 385 insertions(+), 260 deletions(-) diff --git a/plugins/wasm-go/extensions/ai-agent/README.md b/plugins/wasm-go/extensions/ai-agent/README.md index 38915a106..1057d35d3 100644 --- a/plugins/wasm-go/extensions/ai-agent/README.md +++ b/plugins/wasm-go/extensions/ai-agent/README.md @@ -5,14 +5,10 @@ description: AI Agent插件配置参考 --- ## 功能说明 -一个可定制化的 API AI Agent，支持配置 http method 类型为 GET 与 POST 的 API，目前只支持非流式模式。 +一个可定制化的 API AI Agent，支持配置 http method 类型为 GET 与 POST 的 API，支持多轮对话，支持流式与非流式模式。 agent流程图如下： ![ai-agent](https://github.com/user-attachments/assets/b0761a0c-1afa-496c-a98e-bb9f38b340f8) -## 运行属性 - -插件执行阶段：`默认阶段` -插件执行优先级：`20` ## 配置字段 @@ -46,18 +42,19 @@ agent流程图如下： `apiProvider`的配置字段说明如下： -| 名称 | 数据类型 | 填写要求 | 默认值 | 描述 | -|-----------------|-----------|---------|--------|------------------------------------------| -| `apiKey` | object | 非必填 | - | 用于在访问外部 API 服务时进行认证的令牌。 | -| `serviceName` | string | 必填 | - | 访问外部 API 服务名 | -| `servicePort` | int | 必填 | - | 访问外部 API 服务端口 | -| `domain` | string | 必填 | - | 访访问外部 API 时域名 | +| 名称 | 数据类型 | 填写要求 | 默认值 | 描述 | +|-------------------|-----------|---------|--------|------------------------------------------| +| `apiKey` | object | 非必填 | - | 用于在访问外部 API 服务时进行认证的令牌。 | +| `maxExecutionTime`| int | 非必填 | 50000 | 每一次请求API的超时时间，单位毫秒。 | +| `serviceName` | string | 必填 | - | 访问外部 API 服务名 | +| `servicePort` | int | 必填 | - | 访问外部 API 服务端口 | +| `domain` | string | 必填 | - | 访访问外部 API 时域名 | `apiKey`的配置字段说明如下： | 名称 | 数据类型 | 填写要求 | 默认值 | 描述 | -|-------------------|---------|------------|--------|-------------------------------------------------------------------------------| -| `in` | string | 非必填 | header | 在访问外部 API 服务时进行认证的令牌是放在 header 中还是放在 query 中，默认是 header。 +|-------------------|---------|------------|--------|-----------------------------------------------------------------------------------------| +| `in` | string | 非必填 | none | 在访问外部 API 服务时进行认证的令牌是放在 header 中还是放在 query 中，如果API没有令牌，填none。 | `name` | string | 非必填 | - | 用于在访问外部 API 服务时进行认证的令牌的名称。 | | `value` | string | 非必填 | - | 用于在访问外部 API 服务时进行认证的令牌的值。 | @@ -75,11 +72,8 @@ agent流程图如下： |-----------------|-----------|-----------|--------|---------------------------------------------| | `question` | string | 非必填 | - | Agent ReAct 模板的 question 部分 | | `thought1` | string | 非必填 | - | Agent ReAct 模板的 thought1 部分 | -| `actionInput` | string | 非必填 | - | Agent ReAct 模板的 actionInput 部分 | | `observation` | string | 非必填 | - | Agent ReAct 模板的 observation 部分 | | `thought2` | string | 非必填 | - | Agent ReAct 模板的 thought2 部分 | -| `finalAnswer` | string | 非必填 | - | Agent ReAct 模板的 finalAnswer 部分 | -| `begin` | string | 非必填 | - | Agent ReAct 模板的 begin 部分 | ## 用法示例 @@ -325,6 +319,21 @@ curl 'http://<这里换成网关公网IP>/api/openai/v1/chat/completions' \ **请求示例** +```shell +curl 'http://<这里换成网关公网IP>/api/openai/v1/chat/completions' \ +-H 'Accept: application/json, text/event-stream' \ +-H 'Content-Type: application/json' \ +--data-raw '{"model":"qwen","frequency_penalty":0,"max_tokens":800,"stream":false,"messages":[{"role": "user","content": "济南的天气如何？"},{ "role": "assistant","content": "目前，济南市的天气为多云，气温为24℃，数据更新时间为2024年9月12日21时50分14秒。"},{"role": "user","content": "北京呢？"}],"presence_penalty":0,"temperature":0,"top_p":0}' +``` + +**响应示例** + +```json +{"id":"ebd6ea91-8e38-9e14-9a5b-90178d2edea4","choices":[{"index":0,"message":{"role":"assistant","content":"目前，北京市的天气为晴朗，气温为19℃，数据更新时间为2024年9月12日22时17分40秒。"},"finish_reason":"stop"}],"created":1723187991,"model":"qwen-max-0403","object":"chat.completion","usage":{"prompt_tokens":999,"completion_tokens":76,"total_tokens":1075}} +``` + +**请求示例** + ```shell curl 'http://<这里换成网关公网IP>/api/openai/v1/chat/completions' \ -H 'Accept: application/json, text/event-stream' \ @@ -351,4 +360,4 @@ curl 'http://<这里换成网关公网IP>/api/openai/v1/chat/completions' \ ```json {"id":"65dcf12c-61ff-9e68-bffa-44fc9e6070d5","choices":[{"index":0,"message":{"role":"assistant","content":" “九头蛇万岁!”的德语翻译为“Hoch lebe Hydra!”。"},"finish_reason":"stop"}],"created":1724043865,"model":"qwen-max-0403","object":"chat.completion","usage":{"prompt_tokens":908,"completion_tokens":52,"total_tokens":960}} -``` +``` \ No newline at end of file diff --git a/plugins/wasm-go/extensions/ai-agent/README_EN.md b/plugins/wasm-go/extensions/ai-agent/README_EN.md index 4d818f9a6..b594b102a 100644 --- a/plugins/wasm-go/extensions/ai-agent/README_EN.md +++ b/plugins/wasm-go/extensions/ai-agent/README_EN.md @@ -4,7 +4,7 @@ keywords: [ AI Gateway, AI Agent ] description: AI Agent plugin configuration reference --- ## Functional Description -A customizable API AI Agent that supports configuring HTTP method types as GET and POST APIs. Currently, it only supports non-streaming mode. +A customizable API AI Agent that supports configuring HTTP method types as GET and POST APIs. Supports multiple dialogue rounds, streaming and non-streaming modes. The agent flow chart is as follows: ![ai-agent](https://github.com/user-attachments/assets/b0761a0c-1afa-496c-a98e-bb9f38b340f8) @@ -41,17 +41,18 @@ The configuration fields for `apis` are as follows: | `api` | string | Required | - | OpenAPI documentation of the tool | The configuration fields for `apiProvider` are as follows: -| Name | Data Type | Requirement | Default Value | Description | -|-----------------|-----------|-------------|---------------|--------------------------------------------------| -| `apiKey` | object | Optional | - | Token for authentication when accessing external API services. | -| `serviceName` | string | Required | - | Name of the external API service | -| `servicePort` | int | Required | - | Port of the external API service | -| `domain` | string | Required | - | Domain for accessing the external API | +| Name | Data Type | Requirement | Default Value | Description | +|-------------------|-----------|-------------|---------------|--------------------------------------------------| +| `apiKey` | object | Optional | - | Token for authentication when accessing external API services. | +| `maxExecutionTime`| int | Optional | 50000 | Timeout for each request to the API, in milliseconds| +| `serviceName` | string | Required | - | Name of the external API service | +| `servicePort` | int | Required | - | Port of the external API service | +| `domain` | string | Required | - | Domain for accessing the external API | The configuration fields for `apiKey` are as follows: | Name | Data Type | Requirement | Default Value | Description | |-------------------|-----------|-------------|---------------|-------------------------------------------------------------------------------------| -| `in` | string | Optional | header | Whether the authentication token for accessing the external API service is in the header or in the query; default is header. | +| `in` | string | Optional | none | Whether the authentication token for accessing the external API service is in the header or in the query; If the API does not have a token, fill in none. | | `name` | string | Optional | - | The name of the token for authentication when accessing the external API service. | | `value` | string | Optional | - | The value of the token for authentication when accessing the external API service. | @@ -67,11 +68,8 @@ The configuration fields for `chTemplate` and `enTemplate` are as follows: |-----------------|-----------|-------------|---------------|---------------------------------------------------| | `question` | string | Optional | - | The question part of the Agent ReAct template | | `thought1` | string | Optional | - | The thought1 part of the Agent ReAct template | -| `actionInput` | string | Optional | - | The actionInput part of the Agent ReAct template | | `observation` | string | Optional | - | The observation part of the Agent ReAct template | | `thought2` | string | Optional | - | The thought2 part of the Agent ReAct template | -| `finalAnswer` | string | Optional | - | The finalAnswer part of the Agent ReAct template | -| `begin` | string | Optional | - | The begin part of the Agent ReAct template | ## Usage Example **Configuration Information** @@ -308,6 +306,17 @@ curl 'http:///api/openai/v1/chat/completions' \ curl 'http:///api/openai/v1/chat/completions' \ -H 'Accept: application/json, text/event-stream' \ -H 'Content-Type: application/json' \ +--data-raw '{"model":"qwen","frequency_penalty":0,"max_tokens":800,"stream":false,"messages":[{"role":"user","content":"What is the current weather in Jinan?"},{"role":"assistant","content":" The current weather condition in Jinan is overcast, with a temperature of 31°C. This information was last updated on August 9, 2024, at 15:12 (Beijing time)."},{"role":"user","content":"BeiJing?"}],"presence_penalty":0,"temperature":0,"top_p":0}' +``` +**Response Example** +```json +{"id":"ebd6ea91-8e38-9e14-9a5b-90178d2edea4","choices":[{"index":0,"message":{"role":"assistant","content":" The current weather condition in Beijing is overcast, with a temperature of 19°C. This information was last updated on Sep 12, 2024, at 22:17 (Beijing time)."},"finish_reason":"stop"}],"created":1723187991,"model":"qwen-max-0403","object":"chat.completion","usage":{"prompt_tokens":999,"completion_tokens":76,"total_tokens":1075}} +``` +**Request Example** +```shell +curl 'http:///api/openai/v1/chat/completions' \ +-H 'Accept: application/json, text/event-stream' \ +-H 'Content-Type: application/json' \ --data-raw '{"model":"qwen","frequency_penalty":0,"max_tokens":800,"stream":false,"messages":[{"role":"user","content":"What is the current weather in Jinan? Please indicate in Fahrenheit and respond in Japanese."}],"presence_penalty":0,"temperature":0,"top_p":0}' ``` **Response Example** diff --git a/plugins/wasm-go/extensions/ai-agent/config.go b/plugins/wasm-go/extensions/ai-agent/config.go index 31445c17f..ae694a42e 100644 --- a/plugins/wasm-go/extensions/ai-agent/config.go +++ b/plugins/wasm-go/extensions/ai-agent/config.go @@ -46,7 +46,7 @@ type Response struct { } // 用于存放拆解出来的工具相关信息 -type Tool_Param struct { +type ToolsParam struct { ToolName string `yaml:"toolName"` Path string `yaml:"path"` Method string `yaml:"method"` @@ -56,10 +56,11 @@ type Tool_Param struct { } // 用于存放拆解出来的api相关信息 -type APIParam struct { - APIKey APIKey `yaml:"apiKey"` - URL string `yaml:"url"` - Tool_Param []Tool_Param `yaml:"tool_Param"` +type APIsParam struct { + APIKey APIKey `yaml:"apiKey"` + URL string `yaml:"url"` + MaxExecutionTime int64 `yaml:"maxExecutionTime"` + ToolsParam []ToolsParam `yaml:"toolsParam"` } type Info struct { @@ -153,7 +154,10 @@ type APIProvider struct { ServicePort int64 `required:"true" yaml:"servicePort" json:"servicePort"` // @Title zh-CN 服务域名 // @Description zh-CN 服务域名，例如 restapi.amap.com - Domin string `required:"true" yaml:"domain" json:"domain"` + Domain string `required:"true" yaml:"domain" json:"domain"` + // @Title zh-CN 每一次请求api的超时时间 + // @Description zh-CN 每一次请求api的超时时间，单位毫秒，默认50000 + MaxExecutionTime int64 `yaml:"maxExecutionTime" json:"maxExecutionTime"` // @Title zh-CN 通义千问大模型服务的key // @Description zh-CN 通义千问大模型服务的key APIKey APIKey `required:"true" yaml:"apiKey" json:"apiKey"` @@ -167,11 +171,8 @@ type APIs struct { type Template struct { Question string `yaml:"question" json:"question"` Thought1 string `yaml:"thought1" json:"thought1"` - ActionInput string `yaml:"actionInput" json:"actionInput"` Observation string `yaml:"observation" json:"observation"` Thought2 string `yaml:"thought2" json:"thought2"` - FinalAnswer string `yaml:"finalAnswer" json:"finalAnswer"` - Begin string `yaml:"begin" json:"begin"` } type PromptTemplate struct { @@ -189,7 +190,7 @@ type LLMInfo struct { ServicePort int64 `required:"true" yaml:"servicePort" json:"servicePort"` // @Title zh-CN 大模型服务域名 // @Description zh-CN 大模型服务域名，例如 dashscope.aliyuncs.com - Domin string `required:"true" yaml:"domin" json:"domin"` + Domain string `required:"true" yaml:"domain" json:"domain"` // @Title zh-CN 大模型服务的key // @Description zh-CN 大模型服务的key APIKey string `required:"true" yaml:"apiKey" json:"apiKey"` @@ -222,7 +223,7 @@ type PluginConfig struct { // @Description zh-CN 用于存储llm使用信息 LLMInfo LLMInfo `required:"true" yaml:"llm" json:"llm"` LLMClient wrapper.HttpClient `yaml:"-" json:"-"` - APIParam []APIParam `yaml:"-" json:"-"` + APIsParam []APIsParam `yaml:"-" json:"-"` PromptTemplate PromptTemplate `yaml:"promptTemplate" json:"promptTemplate"` } @@ -260,11 +261,13 @@ func initAPIs(gjson gjson.Result, c *PluginConfig) error { return errors.New("apiProvider domain is required") } - apiKeyIn := item.Get("apiProvider.apiKey.in").String() - if apiKeyIn != "query" { - apiKeyIn = "header" + maxExecutionTime := item.Get("apiProvider.maxExecutionTime").Int() + if maxExecutionTime == 0 { + maxExecutionTime = 50000 } + apiKeyIn := item.Get("apiProvider.apiKey.in").String() + apiKeyName := item.Get("apiProvider.apiKey.name") apiKeyValue := item.Get("apiProvider.apiKey.value") @@ -289,13 +292,13 @@ func initAPIs(gjson gjson.Result, c *PluginConfig) error { return err } - var allTool_param []Tool_Param + var allTool_param []ToolsParam //拆除服务下面的每个api的path for path, pathmap := range apiStruct.Paths { //拆解出每个api对应的参数 for method, submap := range pathmap { //把参数列表存起来 - var param Tool_Param + var param ToolsParam param.Path = path param.ToolName = submap.OperationID if method == "get" { @@ -319,13 +322,14 @@ func initAPIs(gjson gjson.Result, c *PluginConfig) error { allTool_param = append(allTool_param, param) } } - apiParam := APIParam{ - APIKey: APIKey{In: apiKeyIn, Name: apiKeyName.String(), Value: apiKeyValue.String()}, - URL: apiStruct.Servers[0].URL, - Tool_Param: allTool_param, + apiParam := APIsParam{ + APIKey: APIKey{In: apiKeyIn, Name: apiKeyName.String(), Value: apiKeyValue.String()}, + URL: apiStruct.Servers[0].URL, + MaxExecutionTime: maxExecutionTime, + ToolsParam: allTool_param, } - c.APIParam = append(c.APIParam, apiParam) + c.APIsParam = append(c.APIsParam, apiParam) } return nil } @@ -338,60 +342,36 @@ func initReActPromptTpl(gjson gjson.Result, c *PluginConfig) { if c.PromptTemplate.Language == "EN" { c.PromptTemplate.ENTemplate.Question = gjson.Get("promptTemplate.enTemplate.question").String() if c.PromptTemplate.ENTemplate.Question == "" { - c.PromptTemplate.ENTemplate.Question = "the input question you must answer" + c.PromptTemplate.ENTemplate.Question = "input question to answer" } c.PromptTemplate.ENTemplate.Thought1 = gjson.Get("promptTemplate.enTemplate.thought1").String() if c.PromptTemplate.ENTemplate.Thought1 == "" { - c.PromptTemplate.ENTemplate.Thought1 = "you should always think about what to do" - } - c.PromptTemplate.ENTemplate.ActionInput = gjson.Get("promptTemplate.enTemplate.actionInput").String() - if c.PromptTemplate.ENTemplate.ActionInput == "" { - c.PromptTemplate.ENTemplate.ActionInput = "the input to the action" + c.PromptTemplate.ENTemplate.Thought1 = "consider previous and subsequent steps" } c.PromptTemplate.ENTemplate.Observation = gjson.Get("promptTemplate.enTemplate.observation").String() if c.PromptTemplate.ENTemplate.Observation == "" { - c.PromptTemplate.ENTemplate.Observation = "the result of the action" + c.PromptTemplate.ENTemplate.Observation = "action result" } - c.PromptTemplate.ENTemplate.Thought1 = gjson.Get("promptTemplate.enTemplate.thought2").String() - if c.PromptTemplate.ENTemplate.Thought1 == "" { - c.PromptTemplate.ENTemplate.Thought1 = "I now know the final answer" - } - c.PromptTemplate.ENTemplate.FinalAnswer = gjson.Get("promptTemplate.enTemplate.finalAnswer").String() - if c.PromptTemplate.ENTemplate.FinalAnswer == "" { - c.PromptTemplate.ENTemplate.FinalAnswer = "the final answer to the original input question, please give the most direct answer directly in Chinese, not English, and do not add extra content." - } - c.PromptTemplate.ENTemplate.Begin = gjson.Get("promptTemplate.enTemplate.begin").String() - if c.PromptTemplate.ENTemplate.Begin == "" { - c.PromptTemplate.ENTemplate.Begin = "Begin! Remember to speak as a pirate when giving your final answer. Use lots of \"Arg\"s" + c.PromptTemplate.ENTemplate.Thought2 = gjson.Get("promptTemplate.enTemplate.thought2").String() + if c.PromptTemplate.ENTemplate.Thought2 == "" { + c.PromptTemplate.ENTemplate.Thought2 = "I know what to respond" } } else if c.PromptTemplate.Language == "CH" { c.PromptTemplate.CHTemplate.Question = gjson.Get("promptTemplate.chTemplate.question").String() if c.PromptTemplate.CHTemplate.Question == "" { - c.PromptTemplate.CHTemplate.Question = "你需要回答的输入问题" + c.PromptTemplate.CHTemplate.Question = "输入要回答的问题" } c.PromptTemplate.CHTemplate.Thought1 = gjson.Get("promptTemplate.chTemplate.thought1").String() if c.PromptTemplate.CHTemplate.Thought1 == "" { - c.PromptTemplate.CHTemplate.Thought1 = "你应该总是思考该做什么" - } - c.PromptTemplate.CHTemplate.ActionInput = gjson.Get("promptTemplate.chTemplate.actionInput").String() - if c.PromptTemplate.CHTemplate.ActionInput == "" { - c.PromptTemplate.CHTemplate.ActionInput = "行动的输入，必须出现在Action后" + c.PromptTemplate.CHTemplate.Thought1 = "考虑之前和之后的步骤" } c.PromptTemplate.CHTemplate.Observation = gjson.Get("promptTemplate.chTemplate.observation").String() if c.PromptTemplate.CHTemplate.Observation == "" { - c.PromptTemplate.CHTemplate.Observation = "行动的结果" + c.PromptTemplate.CHTemplate.Observation = "行动结果" } - c.PromptTemplate.CHTemplate.Thought1 = gjson.Get("promptTemplate.chTemplate.thought2").String() - if c.PromptTemplate.CHTemplate.Thought1 == "" { - c.PromptTemplate.CHTemplate.Thought1 = "我现在知道最终答案" - } - c.PromptTemplate.CHTemplate.FinalAnswer = gjson.Get("promptTemplate.chTemplate.finalAnswer").String() - if c.PromptTemplate.CHTemplate.FinalAnswer == "" { - c.PromptTemplate.CHTemplate.FinalAnswer = "对原始输入问题的最终答案" - } - c.PromptTemplate.CHTemplate.Begin = gjson.Get("promptTemplate.chTemplate.begin").String() - if c.PromptTemplate.CHTemplate.Begin == "" { - c.PromptTemplate.CHTemplate.Begin = "再次重申，不要修改以上模板的字段名称，开始吧！" + c.PromptTemplate.CHTemplate.Thought2 = gjson.Get("promptTemplate.chTemplate.thought2").String() + if c.PromptTemplate.CHTemplate.Thought2 == "" { + c.PromptTemplate.CHTemplate.Thought2 = "我知道该回应什么" } } } @@ -400,7 +380,7 @@ func initLLMClient(gjson gjson.Result, c *PluginConfig) { c.LLMInfo.APIKey = gjson.Get("llm.apiKey").String() c.LLMInfo.ServiceName = gjson.Get("llm.serviceName").String() c.LLMInfo.ServicePort = gjson.Get("llm.servicePort").Int() - c.LLMInfo.Domin = gjson.Get("llm.domain").String() + c.LLMInfo.Domain = gjson.Get("llm.domain").String() c.LLMInfo.Path = gjson.Get("llm.path").String() c.LLMInfo.Model = gjson.Get("llm.model").String() c.LLMInfo.MaxIterations = gjson.Get("llm.maxIterations").Int() @@ -419,6 +399,6 @@ func initLLMClient(gjson gjson.Result, c *PluginConfig) { c.LLMClient = wrapper.NewClusterClient(wrapper.FQDNCluster{ FQDN: c.LLMInfo.ServiceName, Port: c.LLMInfo.ServicePort, - Host: c.LLMInfo.Domin, + Host: c.LLMInfo.Domain, }) } diff --git a/plugins/wasm-go/extensions/ai-agent/main.go b/plugins/wasm-go/extensions/ai-agent/main.go index 4359e63fb..97acb8e18 100644 --- a/plugins/wasm-go/extensions/ai-agent/main.go +++ b/plugins/wasm-go/extensions/ai-agent/main.go @@ -17,6 +17,7 @@ import ( // 用于统计函数的递归调用次数 const ToolCallsCount = "ToolCallsCount" +const StreamContextKey = "Stream" // react的正则规则 const ActionPattern = `Action:\s*(.*?)[.\n]` @@ -53,7 +54,7 @@ func onHttpRequestHeaders(ctx wrapper.HttpContext, config PluginConfig, log wrap return types.ActionContinue } -func firstReq(config PluginConfig, prompt string, rawRequest Request, log wrapper.Log) types.Action { +func firstReq(ctx wrapper.HttpContext, config PluginConfig, prompt string, rawRequest Request, log wrapper.Log) types.Action { log.Debugf("[onHttpRequestBody] firstreq:%s", prompt) var userMessage Message @@ -62,13 +63,17 @@ func firstReq(config PluginConfig, prompt string, rawRequest Request, log wrappe newMessages := []Message{userMessage} rawRequest.Messages = newMessages + if rawRequest.Stream { + ctx.SetContext(StreamContextKey, struct{}{}) + rawRequest.Stream = false + } //replace old message and resume request qwen newbody, err := json.Marshal(rawRequest) if err != nil { return types.ActionContinue } else { - log.Debugf("[onHttpRequestBody] newRequestBody: ", string(newbody)) + log.Debugf("[onHttpRequestBody] newRequestBody: %s", string(newbody)) err := proxywasm.ReplaceHttpRequestBody(newbody) if err != nil { log.Debug("替换失败") @@ -87,18 +92,26 @@ func onHttpRequestBody(ctx wrapper.HttpContext, config PluginConfig, body []byte var rawRequest Request err := json.Unmarshal(body, &rawRequest) if err != nil { - log.Debugf("[onHttpRequestBody] body json umarshal err: ", err.Error()) + log.Debugf("[onHttpRequestBody] body json umarshal err: %s", err.Error()) return types.ActionContinue } log.Debugf("onHttpRequestBody rawRequest: %v", rawRequest) //获取用户query var query string + var history string messageLength := len(rawRequest.Messages) - log.Debugf("[onHttpRequestBody] messageLength: %s\n", messageLength) + log.Debugf("[onHttpRequestBody] messageLength: %s", messageLength) if messageLength > 0 { query = rawRequest.Messages[messageLength-1].Content - log.Debugf("[onHttpRequestBody] query: %s\n", query) + log.Debugf("[onHttpRequestBody] query: %s", query) + if messageLength >= 3 { + for i := 0; i < messageLength-1; i += 2 { + history += "human: " + rawRequest.Messages[i].Content + "\nAI: " + rawRequest.Messages[i+1].Content + } + } else { + history = "" + } } else { return types.ActionContinue } @@ -111,8 +124,8 @@ func onHttpRequestBody(ctx wrapper.HttpContext, config PluginConfig, body []byte //拼装agent prompt模板 tool_desc := make([]string, 0) tool_names := make([]string, 0) - for _, apiParam := range config.APIParam { - for _, tool_param := range apiParam.Tool_Param { + for _, apisParam := range config.APIsParam { + for _, tool_param := range apisParam.ToolsParam { tool_desc = append(tool_desc, fmt.Sprintf(prompttpl.TOOL_DESC, tool_param.ToolName, tool_param.Description, tool_param.Description, tool_param.Description, tool_param.Parameter), "\n") tool_names = append(tool_names, tool_param.ToolName) } @@ -122,26 +135,22 @@ func onHttpRequestBody(ctx wrapper.HttpContext, config PluginConfig, body []byte if config.PromptTemplate.Language == "CH" { prompt = fmt.Sprintf(prompttpl.CH_Template, tool_desc, + tool_names, config.PromptTemplate.CHTemplate.Question, config.PromptTemplate.CHTemplate.Thought1, - tool_names, - config.PromptTemplate.CHTemplate.ActionInput, config.PromptTemplate.CHTemplate.Observation, config.PromptTemplate.CHTemplate.Thought2, - config.PromptTemplate.CHTemplate.FinalAnswer, - config.PromptTemplate.CHTemplate.Begin, + history, query) } else { prompt = fmt.Sprintf(prompttpl.EN_Template, tool_desc, + tool_names, config.PromptTemplate.ENTemplate.Question, config.PromptTemplate.ENTemplate.Thought1, - tool_names, - config.PromptTemplate.ENTemplate.ActionInput, config.PromptTemplate.ENTemplate.Observation, config.PromptTemplate.ENTemplate.Thought2, - config.PromptTemplate.ENTemplate.FinalAnswer, - config.PromptTemplate.ENTemplate.Begin, + history, query) } @@ -154,7 +163,7 @@ func onHttpRequestBody(ctx wrapper.HttpContext, config PluginConfig, body []byte dashscope.MessageStore.AddForUser(prompt) //开始第一次请求 - ret := firstReq(config, prompt, rawRequest, log) + ret := firstReq(ctx, config, prompt, rawRequest, log) return ret } @@ -168,7 +177,7 @@ func onHttpResponseHeaders(ctx wrapper.HttpContext, config PluginConfig, log wra func toolsCallResult(ctx wrapper.HttpContext, config PluginConfig, content string, rawResponse Response, log wrapper.Log, statusCode int, responseBody []byte) { if statusCode != http.StatusOK { - log.Debugf("statusCode: %d\n", statusCode) + log.Debugf("statusCode: %d", statusCode) } log.Info("========函数返回结果========") log.Infof(string(responseBody)) @@ -193,30 +202,36 @@ func toolsCallResult(ctx wrapper.HttpContext, config PluginConfig, content strin //得到gpt的返回结果 var responseCompletion dashscope.CompletionResponse _ = json.Unmarshal(responseBody, &responseCompletion) - log.Infof("[toolsCall] content: %s\n", responseCompletion.Choices[0].Message.Content) + log.Infof("[toolsCall] content: %s", responseCompletion.Choices[0].Message.Content) if responseCompletion.Choices[0].Message.Content != "" { - retType := toolsCall(ctx, config, responseCompletion.Choices[0].Message.Content, rawResponse, log) + retType, actionInput := toolsCall(ctx, config, responseCompletion.Choices[0].Message.Content, rawResponse, log) if retType == types.ActionContinue { //得到了Final Answer var assistantMessage Message - assistantMessage.Role = "assistant" - startIndex := strings.Index(responseCompletion.Choices[0].Message.Content, "Final Answer:") - if startIndex != -1 { - startIndex += len("Final Answer:") // 移动到"Final Answer:"之后的位置 - extractedText := responseCompletion.Choices[0].Message.Content[startIndex:] - assistantMessage.Content = extractedText - } + if ctx.GetContext(StreamContextKey) == nil { + assistantMessage.Role = "assistant" + assistantMessage.Content = actionInput + rawResponse.Choices[0].Message = assistantMessage + newbody, err := json.Marshal(rawResponse) + if err != nil { + proxywasm.ResumeHttpResponse() + return + } else { + proxywasm.ReplaceHttpResponseBody(newbody) - rawResponse.Choices[0].Message = assistantMessage - - newbody, err := json.Marshal(rawResponse) - if err != nil { - proxywasm.ResumeHttpResponse() - return + log.Debug("[onHttpResponseBody] response替换成功") + proxywasm.ResumeHttpResponse() + } } else { - log.Infof("[onHttpResponseBody] newResponseBody: ", string(newbody)) - proxywasm.ReplaceHttpResponseBody(newbody) + headers := [][2]string{{"content-type", "text/event-stream; charset=utf-8"}} + proxywasm.ReplaceHttpResponseHeaders(headers) + // Remove quotes from actionInput + actionInput = strings.Trim(actionInput, "\"") + returnStreamResponseTemplate := `data:{"id":"%s","choices":[{"index":0,"delta":{"role":"assistant","content":"%s"},"finish_reason":"stop"}],"model":"%s","object":"chat.completion","usage":{"prompt_tokens":%d,"completion_tokens":%d,"total_tokens":%d}}` + "\n\ndata:[DONE]\n\n" + newbody := fmt.Sprintf(returnStreamResponseTemplate, rawResponse.ID, actionInput, rawResponse.Model, rawResponse.Usage.PromptTokens, rawResponse.Usage.CompletionTokens, rawResponse.Usage.TotalTokens) + log.Infof("[onHttpResponseBody] newResponseBody: ", newbody) + proxywasm.ReplaceHttpResponseBody([]byte(newbody)) log.Debug("[onHttpResponseBody] response替换成功") proxywasm.ResumeHttpResponse() @@ -232,121 +247,156 @@ func toolsCallResult(ctx wrapper.HttpContext, config PluginConfig, content strin } } -func toolsCall(ctx wrapper.HttpContext, config PluginConfig, content string, rawResponse Response, log wrapper.Log) types.Action { +func outputParser(response string, log wrapper.Log) (string, string) { + log.Debugf("Raw response:%s", response) + + start := strings.Index(response, "```") + end := strings.LastIndex(response, "```") + + var jsonStr string + if start != -1 && end != -1 { + jsonStr = strings.TrimSpace(response[start+3 : end]) + } else { + jsonStr = response + } + + log.Debugf("Extracted JSON string:%s", jsonStr) + + var action map[string]interface{} + err := json.Unmarshal([]byte(jsonStr), &action) + if err == nil { + var actionName, actionInput string + for key, value := range action { + if strings.Contains(strings.ToLower(key), "input") { + actionInput = fmt.Sprintf("%v", value) + } else { + actionName = fmt.Sprintf("%v", value) + } + } + if actionName != "" && actionInput != "" { + return actionName, actionInput + } + } + log.Debugf("json parse err: %s", err.Error()) + // Fallback to regex parsing if JSON unmarshaling fails + pattern := `\{\s*"action":\s*"([^"]+)",\s*"action_input":\s*((?:\{[^}]+\})|"[^"]+")\s*\}` + re := regexp.MustCompile(pattern) + match := re.FindStringSubmatch(jsonStr) + + if len(match) == 3 { + action := match[1] + actionInput := match[2] + log.Debugf("Parsed action:%s, action_input:%s", action, actionInput) + return action, actionInput + } + + log.Debug("No valid action and action_input found in the response") + return "", "" +} + +func toolsCall(ctx wrapper.HttpContext, config PluginConfig, content string, rawResponse Response, log wrapper.Log) (types.Action, string) { dashscope.MessageStore.AddForAssistant(content) + action, actionInput := outputParser(content, log) + //得到最终答案 - regexPattern := regexp.MustCompile(FinalAnswerPattern) - finalAnswer := regexPattern.FindStringSubmatch(content) - if len(finalAnswer) > 1 { - return types.ActionContinue + if action == "Final Answer" { + return types.ActionContinue, actionInput } count := ctx.GetContext(ToolCallsCount).(int) count++ - log.Debugf("toolCallsCount:%d, config.LLMInfo.MaxIterations=%d\n", count, config.LLMInfo.MaxIterations) + log.Debugf("toolCallsCount:%d, config.LLMInfo.MaxIterations=%d", count, config.LLMInfo.MaxIterations) //函数递归调用次数，达到了预设的循环次数，强制结束 if int64(count) > config.LLMInfo.MaxIterations { ctx.SetContext(ToolCallsCount, 0) - return types.ActionContinue + return types.ActionContinue, "" } else { ctx.SetContext(ToolCallsCount, count) } //没得到最终答案 - regexAction := regexp.MustCompile(ActionPattern) - regexActionInput := regexp.MustCompile(ActionInputPattern) - action := regexAction.FindStringSubmatch(content) - actionInput := regexActionInput.FindStringSubmatch(content) + var url string + var headers [][2]string + var apiClient wrapper.HttpClient + var method string + var reqBody []byte + var key string + var maxExecutionTime int64 - if len(action) > 1 && len(actionInput) > 1 { - var url string - var headers [][2]string - var apiClient wrapper.HttpClient - var method string - var reqBody []byte - var key string + for i, apisParam := range config.APIsParam { + maxExecutionTime = apisParam.MaxExecutionTime + for _, tools_param := range apisParam.ToolsParam { + if action == tools_param.ToolName { + log.Infof("calls %s", tools_param.ToolName) + log.Infof("actionInput: %s", actionInput) - for i, apiParam := range config.APIParam { - for _, tool_param := range apiParam.Tool_Param { - if action[1] == tool_param.ToolName { - log.Infof("calls %s\n", tool_param.ToolName) - log.Infof("actionInput[1]: %s", actionInput[1]) - - //将大模型需要的参数反序列化 - var data map[string]interface{} - if err := json.Unmarshal([]byte(actionInput[1]), &data); err != nil { - log.Debugf("Error: %s\n", err.Error()) - return types.ActionContinue - } - - method = tool_param.Method - - //key or header组装 - if apiParam.APIKey.Name != "" { - if apiParam.APIKey.In == "query" { //query类型的key要放到url中 - headers = nil - key = "?" + apiParam.APIKey.Name + "=" + apiParam.APIKey.Value - } else if apiParam.APIKey.In == "header" { //header类型的key放在header中 - headers = [][2]string{{"Content-Type", "application/json"}, {"Authorization", apiParam.APIKey.Name + " " + apiParam.APIKey.Value}} - } - } - - if method == "GET" { - //query组装 - var args string - for i, param := range tool_param.ParamName { //从参数列表中取出参数 - if i == 0 && apiParam.APIKey.In != "query" { - args = "?" + param + "=%s" - args = fmt.Sprintf(args, data[param]) - } else { - args = args + "&" + param + "=%s" - args = fmt.Sprintf(args, data[param]) - } - } - - //url组装 - url = apiParam.URL + tool_param.Path + key + args - } else if method == "POST" { - reqBody = nil - //json参数组装 - jsonData, err := json.Marshal(data) - if err != nil { - log.Debugf("Error: %s\n", err.Error()) - return types.ActionContinue - } - reqBody = jsonData - - //url组装 - url = apiParam.URL + tool_param.Path + key - } - - log.Infof("url: %s\n", url) - - apiClient = config.APIClient[i] - break + //将大模型需要的参数反序列化 + var data map[string]interface{} + if err := json.Unmarshal([]byte(actionInput), &data); err != nil { + log.Debugf("Error: %s", err.Error()) + return types.ActionContinue, "" } - } - } - if apiClient != nil { - err := apiClient.Call( - method, - url, - headers, - reqBody, - func(statusCode int, responseHeaders http.Header, responseBody []byte) { - toolsCallResult(ctx, config, content, rawResponse, log, statusCode, responseBody) - }, 50000) - if err != nil { - log.Debugf("tool calls error: %s\n", err.Error()) - proxywasm.ResumeHttpRequest() + method = tools_param.Method + + // 组装 headers 和 key + headers = [][2]string{{"Content-Type", "application/json"}} + if apisParam.APIKey.Name != "" { + if apisParam.APIKey.In == "query" { + key = "?" + apisParam.APIKey.Name + "=" + apisParam.APIKey.Value + } else if apisParam.APIKey.In == "header" { + headers = append(headers, [2]string{"Authorization", apisParam.APIKey.Name + " " + apisParam.APIKey.Value}) + } + } + + // 组装 URL 和请求体 + url = apisParam.URL + tools_param.Path + key + if method == "GET" { + queryParams := make([]string, 0, len(tools_param.ParamName)) + for _, param := range tools_param.ParamName { + if value, ok := data[param]; ok { + queryParams = append(queryParams, fmt.Sprintf("%s=%v", param, value)) + } + } + if len(queryParams) > 0 { + url += "&" + strings.Join(queryParams, "&") + } + } else if method == "POST" { + var err error + reqBody, err = json.Marshal(data) + if err != nil { + log.Debugf("Error marshaling JSON: %s", err.Error()) + return types.ActionContinue, "" + } + } + + log.Infof("url: %s", url) + + apiClient = config.APIClient[i] + break } - } else { - return types.ActionContinue } } - return types.ActionPause + + if apiClient != nil { + err := apiClient.Call( + method, + url, + headers, + reqBody, + func(statusCode int, responseHeaders http.Header, responseBody []byte) { + toolsCallResult(ctx, config, content, rawResponse, log, statusCode, responseBody) + }, uint32(maxExecutionTime)) + if err != nil { + log.Debugf("tool calls error: %s", err.Error()) + proxywasm.ResumeHttpRequest() + } + } else { + return types.ActionContinue, "" + } + + return types.ActionPause, "" } // 从response接收到firstreq的大模型返回 @@ -361,11 +411,12 @@ func onHttpResponseBody(ctx wrapper.HttpContext, config PluginConfig, body []byt log.Debugf("[onHttpResponseBody] body to json err: %s", err.Error()) return types.ActionContinue } - log.Infof("first content: %s\n", rawResponse.Choices[0].Message.Content) + log.Infof("first content: %s", rawResponse.Choices[0].Message.Content) //如果gpt返回的内容不是空的 if rawResponse.Choices[0].Message.Content != "" { //进入agent的循环思考，工具调用的过程中 - return toolsCall(ctx, config, rawResponse.Choices[0].Message.Content, rawResponse, log) + retType, _ := toolsCall(ctx, config, rawResponse.Choices[0].Message.Content, rawResponse, log) + return retType } else { return types.ActionContinue } diff --git a/plugins/wasm-go/extensions/ai-agent/promptTpl/prompt.go b/plugins/wasm-go/extensions/ai-agent/promptTpl/prompt.go index ab8bffc57..47ae6f71f 100644 --- a/plugins/wasm-go/extensions/ai-agent/promptTpl/prompt.go +++ b/plugins/wasm-go/extensions/ai-agent/promptTpl/prompt.go @@ -13,81 +13,157 @@ Parameters: Format the arguments as a JSON object.` /* -Answer the following questions as best you can, but speaking as a pirate might speak. You have access to the following tools: +Respond to the human as helpfully and accurately as possible. You have access to the following tools: -%s +{{tools_desc}} -Use the following format: +Use a json blob to specify a tool by providing an action key (tool name) and an action_input key (tool input). +Valid "action" values: "Final Answer" or {{tool_names}} -Question: the input question you must answer -Thought: you should always think about what to do -Action: the action to take, should be one of %s -Action Input: the input to the action -Observation: the result of the action -... (this Thought/Action/Action Input/Observation can repeat N times) -Thought: I now know the final answer -Final Answer: the final answer to the original input question, please give the most direct answer directly in Chinese, not English, and do not add extra content. +Provide only ONE action per $JSON_BLOB, as shown: -Begin! Remember to speak as a pirate when giving your final answer. Use lots of "Arg"s +``` -Question: %s + { + "action": $TOOL_NAME, + "action_input": $ACTION_INPUT + } + +``` + +Follow this format: + +Question: input question to answer +Thought: consider previous and subsequent steps +Action: +``` +$JSON_BLOB +``` +Observation: action result +... (repeat Thought/Action/Observation N times) +Thought: I know what to respond +Action: +``` + + { + "action": "Final Answer", + "action_input": "Final response to human" + } + +``` + +Begin! Reminder to ALWAYS respond with a valid json blob of a single action. Use tools if necessary. Respond directly if appropriate. Format is Action:```$JSON_BLOB```then Observation:. +{{historic_messages}} +Question: {{query}} */ const EN_Template = ` -Answer the following questions as best you can, but speaking as a pirate might speak. You have access to the following tools: +Respond to the human as helpfully and accurately as possible.You have access to the following tools: %s -Use the following format: +Use a json blob to specify a tool by providing an action key (tool name) and an action_input key (tool input). +Valid "action" values: "Final Answer" or %s +Provide only ONE action per $JSON_BLOB, as shown: +` + "```" + ` +{ + "action": $TOOL_NAME, + "action_input": $ACTION_INPUT +} +` + "```" + ` +Follow this format: Question: %s -Thought: %s -Action: the action to take, should be one of %s -Action Input: %s -Observation: %s -... (this Thought/Action/Action Input/Observation can repeat N times) -Thought: %s -Final Answer: %s +Thought: %s +Action: ` + "```" + `$JSON_BLOB` + "```" + ` +Observation: %s +... (repeat Thought/Action/Observation N times) +Thought: %s +Action:` + "```" + ` +{ + "action": "Final Answer", + "action_input": "Final response to human" +} +` + "```" + ` +Begin! Reminder to ALWAYS respond with a valid json blob of a single action. Use tools if necessary. Respond directly if appropriate.Format is Action:` + "```" + `$JSON_BLOB` + "```" + `then Observation:. %s - Question: %s ` /* -尽你所能回答以下问题。你可以使用以下工具： +尽可能帮助和准确地回答人的问题。您可以使用以下工具： -%s +{tool_descs} -请使用以下格式，其中Action字段后必须跟着Action Input字段，并且不要将Action Input替换成Input或者tool等字段，不能出现格式以外的字段名，每个字段在每个轮次只出现一次： -Question: 你需要回答的输入问题 -Thought: 你应该总是思考该做什么 -Action: 要采取的动作，动作只能是%s中的一个 ，一定不要加入其它内容 -Action Input: 行动的输入，必须出现在Action后。 -Observation: 行动的结果 -...（这个Thought/Action/Action Input/Observation可以重复N次） -Thought: 我现在知道最终答案 -Final Answer: 对原始输入问题的最终答案 +使用 json blob，通过提供 action key（工具名称）和 action_input key（工具输入）来指定工具。 +有效的 "action"值为 "Final Answer"或 {tool_names} -再次重申，不要修改以上模板的字段名称，开始吧！ +每个 $JSON_BLOB 只能提供一个操作，如图所示： -Question: %s +``` + + {{ + "action": $TOOL_NAME, + "action_input": $ACTION_INPUT + }} + +``` + +按照以下格式: +Question: 输入要回答的问题 +Thought: 考虑之前和之后的步骤 +Action: +``` +$JSON_BLOB +``` + +Observation: 行动结果 +...（这个Thought/Action//Observation可以重复N次） +Thought: 我知道该回应什么 +Action: +``` + + {{ + "action": "Final Answer", + "action_input": "Final response to human" + }} + +``` + +开始！提醒您始终使用单个操作的有效 json blob 进行响应。必要时使用工具。如果合适，可直接响应。格式为 Action:```$JSON_BLOB```then Observation:. +{historic_messages} +Question: {input} */ const CH_Template = ` -尽你所能回答以下问题。你可以使用以下工具： +尽可能帮助和准确地回答人的问题。您可以使用以下工具： %s -请使用以下格式，其中Action字段后必须跟着Action Input字段，并且不要将Action Input替换成Input或者tool等字段，不能出现格式以外的字段名，每个字段在每个轮次只出现一次： +使用 json blob，通过提供 action key（工具名称）和 action_input key（工具输入）来指定工具。 +有效的 "action"值为 "Final Answer"或 %s + +每个 $JSON_BLOB 只能提供一个操作，如图所示： +` + "```" + ` +{ + "action": $TOOL_NAME, + "action_input": $ACTION_INPUT +} +` + "```" + ` +按照以下格式: Question: %s Thought: %s -Action: 要采取的动作，动作只能是%s中的一个 ，一定不要加入其它内容 -Action Input: %s +Action: ` + "```" + `$JSON_BLOB` + "```" + ` + Observation: %s -...（这个Thought/Action/Action Input/Observation可以重复N次） +...（这个Thought/Action//Observation可以重复N次） Thought: %s -Final Answer: %s - +Action:` + "```" + ` +{ + "action": "Final Answer", + "action_input": "Final response to human" +} +` + "```" + ` +开始！提醒您始终使用单个操作的有效 json blob 进行响应。必要时使用工具。如果合适，可直接响应。格式为 Action:` + "```" + `$JSON_BLOB` + "```" + `then Observation:. %s - Question: %s `