jiazhizhong/higress
1
0
Fork 0
mirror of https://github.com/alibaba/higress.git synced 2026-03-01 07:00:50 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat: Add request and response transformer wasm plugin (#549)

Browse Source
This commit is contained in:
WeixinX
2023-10-07 15:47:20 +08:00
committed by GitHub
parent be8563765e
commit 1ded5322a5
12 changed files with 2402 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
plugins/wasm-go/Dockerfile
View File

@@ -14,7 +14,7 @@ COPY . .
WORKDIR /workspace/extensions/$PLUGIN_NAME
RUN go mod tidy
RUN tinygo build -o /main.wasm -scheduler=none -gc=custom -tags='custommalloc nottinygc_finalizer' -target=wasi ./main.go
RUN tinygo build -o /main.wasm -scheduler=none -gc=custom -tags='custommalloc nottinygc_finalizer' -target=wasi ./
FROM scratch as output

500
plugins/wasm-go/extensions/transformer/README.md Normal file
View File

@@ -0,0 +1,500 @@
# 功能说明
`transformer` 插件可以对请求/响应头、请求查询参数、请求/响应体参数进行转换，支持的转换操作类型包括删除、重命名、更新、添加、追加、映射、去重。
# 配置字段
| 名称 | 数据类型 | 填写要求 | 默认值 | 描述 |
| -------- | -------- | -------- | -------- | -------- |
| type | string | 必填，可选值为 `request`, `response` | - | 指定转换器类型 |
| rules | array of object | 选填 | - | 指定转换操作类型以及请求/响应头、请求查询参数、请求/响应体参数的转换规则 |
`rules`中每一项的配置字段说明如下：
| 名称 | 数据类型 | 填写要求 | 默认值 | 描述 |
| -------- | -------- | -------- | -------- | -------- |
| operate | string | 必填，可选值为 `remove`, `rename`, `replace`, `add`, `append`, `map`, `dedupe` | - | 指定转换操作类型，支持的操作类型有删除 (remove)、重命名 (rename)、更新 (replace)、添加 (add)、追加 (append)、映射 (map)、去重 (dedupe)，当存在多项不同类型的转换规则时，按照上述操作类型顺序依次执行 |
| headers | array of object | 选填 | - | 指定请求/响应头转换规则 |
| querys | array of object | 选填 | - | 指定请求查询参数转换规则 |
| body | array of object | 选填 | - | 指定请求/响应体参数转换规则，请求体转换允许 content-type 为 `application/json`, `application/x-www-form-urlencoded`, `multipart/form-data`；响应体转换仅允许 content-type 为 `application/json` |
`headers`, `querys`, `body`中每一项的配置字段说明如下：
| 名称 | 数据类型 | 填写要求 | 默认值 | 描述 |
| -------- | -------- | -------- | -------- |---------------------------------------------------|
| key | string | 选填 | - | 指定键，详见[转换操作类型](#转换操作类型) |
| value | string | 选填 | - | 指定值，详见[转换操作类型](#转换操作类型) |
| value_type | string | 选填，可选值为 `object`, `boolean`, `number`, `string` | string | 当`content-type: application/json`时，该字段指定请求/响应体参数的值类型 |
| host_pattern | string | 选填 | - | 指定请求主机名匹配规则，当转换操作类型为 `replace`, `add`, `append` 时有效 |
| path_pattern | string | 选填 | - | 指定请求路径匹配规则，当转换操作类型为 `replace`, `add`, `append` 时有效 |
注意：
* `request transformer` 支持以下转换对象：请求头部、请求查询参数、请求体（application/json, application/x-www-form-urlencoded, multipart/form-data）
* `response transformer` 支持以下转换对象：响应头部、响应体（application/json）
* 转换操作类型的执行顺序：remove → rename → replace → add → append → map → dedupe
* 当转换对象为 headers 时，` key` 不区分大小写；当为 headers 且为 `rename`, `map` 操作时，`value` 也不区分大小写（因为此时该字段具有 key 含义）；而 querys 和 body 的 `key`, `value` 字段均区分大小写
* `value_type` 仅对 content-type 为 application/json 的请求/响应体有效
* `host_pattern` 和 `path_pathern` 支持 [RE2 语法](https://pkg.go.dev/regexp/syntax)，仅对 `replace`, `add`, `append` 操作有效，且在一项转换规则中两者只能选填其一，若均填写，则 `host_pattern` 生效，而 `path_pattern` 失效
# 转换操作类型
| 操作类型 | key 字段含义 | value 字段含义 | 描述 |
| ------------- | ----------------- |-----| ------------------------------------------------------------ |
| 删除 remove | 目标 key |无需设置| 若存在指定的 `key`，则删除；否则无操作 |
| 重命名 rename | 目标 oldKey |新的 key 名称 newKey| 若存在指定的 `oldKey:value`，则将其键名重命名为 `newKey`，得到 `newKey:value`；否则无操作 |
| 更新 replace | 目标 key |新的 value 值 newValue| 若存在指定的 `key:value`，则将其 value 更新为 `newValue`，得到 `key:newValue`；否则无操作 |
| 添加 add | 添加的 key | 添加的 value |若不存在指定的 `key:value`，则添加；否则无操作 |
| 追加 append | 目标 key |追加的 value值 appendValue| 若存在指定的 `key:value`，则追加 appendValue 得到 `key:[value..., appendValue]`；否则相当于执行 add 操作，得到 `key:appendValue` |
| 映射 map | 映射来源 fromKey |映射目标 toKey| 若存在指定的 `fromKey:fromValue`，则将其值 fromValue 映射给 toKey 的值，得到 `toKey:fromValue`，同时保留 `fromKey:fromValue`（注：若 toKey 已存在则其值会被覆盖）；否则无操作 |
| 去重 dedupe | 目标 key |指定去重策略 strategy| `strategy` 可选值为：<br>`RETAIN_UNIQUE`: 按顺序保留所有唯一值，如 `k1:[v1,v2,v3,v3,v2,v1]`，去重后得到 `k1:[v1,v2,v3]` <br>`RETAIN_LAST`: 保留最后一个值，如 `k1:[v1,v2,v3]`，去重后得到 `k1:v3` <br>`RETAIN_FIRST` (default): 保留第一个值，如 `k1:[v1,v2,v3]`，去重后得到 `k1:v1`<br>（注：若去重后只剩下一个元素 v1 时，键值对变为 `k1:v1`, 而不是 `k1:[v1]`） |
# 配置示例
## Request Transformer
### 转换请求头部
```yaml
type: request
rules:
- operate: remove
headers:
- key: X-remove
- operate: rename
headers:
- key: X-not-renamed
value: X-renamed
- operate: replace
headers:
- key: X-replace
value: replaced
- operate: add
headers:
- key: X-add-append
value: host-$1
host_pattern: ^(.*)\.com$
- operate: append
headers:
- key: X-add-append
value: path-$1
path_pattern: ^.*?\/(\w+)[\?]{0,1}.*$
- operate: map
headers:
- key: X-add-append
value: X-map
- operate: dedupe
headers:
- key: X-dedupe-first
value: RETAIN_FIRST
- key: X-dedupe-last
value: RETAIN_LAST
- key: X-dedupe-unique
value: RETAIN_UNIQUE
```
发送请求
```bash
$ curl -v console.higress.io/get -H 'host: foo.bar.com' \
-H 'X-remove: exist' -H 'X-not-renamed:test' -H 'X-replace:not-replaced' \
-H 'X-dedupe-first:1' -H 'X-dedupe-first:2' -H 'X-dedupe-first:3' \
-H 'X-dedupe-last:a' -H 'X-dedupe-last:b' -H 'X-dedupe-last:c' \
-H 'X-dedupe-unique:1' -H 'X-dedupe-unique:2' -H 'X-dedupe-unique:3' \
-H 'X-dedupe-unique:3' -H 'X-dedupe-unique:2' -H 'X-dedupe-unique:1'
# httpbin 响应结果
{
"args": {},
"headers": {
...
"X-Add-Append": "host-foo.bar,path-get",
...
"X-Dedupe-First": "1",
"X-Dedupe-Last": "c",
"X-Dedupe-Unique": "1,2,3",
...
"X-Map": "host-foo.bar,path-get",
"X-Renamed": "test",
"X-Replace": "replaced"
},
...
}
```
### 转换请求查询参数
```yaml
type: request
rules:
- operate: remove
querys:
- key: k1
- operate: rename
querys:
- key: k2
value: k2-new
- operate: replace
querys:
- key: k2-new
value: v2-new
- operate: add
querys:
- key: k3
value: v31-$1
path_pattern: ^.*?\/(\w+)[\?]{0,1}.*$
- operate: append
querys:
- key: k3
value: v32
- operate: map
querys:
- key: k3
value: k4
- operate: dedupe
querys:
- key: k4
value: RETAIN_FIRST
```
发送请求
```bash
$ curl -v "console.higress.io/get?k1=v11&k1=v12&k2=v2"
# httpbin 响应结果
{
"args": {
"k2-new": "v2-new",
"k3": [
"v31-get",
"v32"
],
"k4": "v31-get"
},
...
"url": "http://foo.bar.com/get?k2-new=v2-new&k3=v31-get&k3=v32&k4=v31-get"
}
```
### 转换请求体
```yaml
type: request
rules:
- operate: remove
body:
- key: a1
- operate: rename
body:
- key: a2
value: a2-new
- operate: replace
body:
- key: a3
value: t3-new
value_type: string
- operate: add
body:
- key: a1-new
value: t1-new
value_type: string
- operate: append
body:
- key: a1-new
value: t1-$1-append
value_type: string
host_pattern: ^(.*)\.com$
- operate: map
body:
- key: a1-new
value: a4
- operate: dedupe
body:
- key: a4
value: RETAIN_FIRST
```
发送请求：
**1. Content-Type: application/json**
```bash
$ curl -v -x POST console.higress.io/post -H 'host: foo.bar.com' \
-H 'Content-Type: application/json' -d '{"a1":"t1","a2":"t2","a3":"t3"}'
# httpbin 响应结果
{
...
"headers": {
...
"Content-Type": "application/json",
...
},
"json": {
"a1-new": [
"t1-new",
"t1-foo.bar-append"
],
"a2-new": "t2",
"a3": "t3-new",
"a4": "t1-new"
},
...
}
```
**2. Content-Type: application/x-www-form-urlencoded**
```bash
$ curl -v -X POST console.higress.io/post -H 'host: foo.bar.com' \
-d 'a1=t1&a2=t2&a3=t3'
# httpbin 响应结果
{
...
"form": {
"a1-new": [
"t1-new",
"t1-foo.bar-append"
],
"a2-new": "t2",
"a3": "t3-new",
"a4": "t1-new"
},
"headers": {
...
"Content-Type": "application/x-www-form-urlencoded",
...
},
...
}
```
**3. Content-Type: multipart/form-data**
```bash
$ curl -v -X POST console.higress.io/post -H 'host: foo.bar.com' \
-F a1=t1 -F a2=t2 -F a3=t3
# httpbin 响应结果
{
...
"form": {
"a1-new": [
"t1-new",
"t1-foo.bar-append"
],
"a2-new": "t2",
"a3": "t3-new",
"a4": "t1-new"
},
"headers": {
...
"Content-Type": "multipart/form-data; boundary=------------------------1118b3fab5afbc4e",
...
},
...
}
```
## Response Transformer
与 Request Transformer 类似，在此仅说明转换 JSON 形式的请求/响应体时的注意事项：
### key 嵌套 `.`
1.通常情况下，指定的 key 中含有 `.` 表示嵌套含义，如下：
```yaml
type: response
rules:
- operate: add
body:
- key: foo.bar
value: value
```
```bash
$ curl -v console.higress.io/get
# httpbin 响应结果
{
...
"foo": {
"bar": "value"
},
...
}
```
2.当使用 `\.` 对 key 中的 `.` 进行转义后，表示非嵌套含义，如下：
> 当使用双引号括住字符串时使用 `\\.` 进行转义
```yaml
type: response
rules:
- operate: add
body:
- key: foo\.bar
value: value
```
```bash
$ curl -v console.higress.io/get
# httpbin 响应结果
{
...
"foo.bar": "value",
...
}
```
### 访问数组元素 `.index`
可以通过数组下标 `array.index 访问数组元素，下标从 0 开始：
```json
{
"users": [
{
"123": { "name": "zhangsan", "age": 18 }
},
{
"456": { "name": "lisi", "age": 19 }
}
]
}
```
1.移除 `user` 第一个元素：
```yaml
type: request
rules:
- operate: remove
body:
- key: users.0
```
```bash
$ curl -v -X POST console.higress.io/post \
-H 'Content-Type: application/json' \
-d '{"users":[{"123":{"name":"zhangsan"}},{"456":{"name":"lisi"}}]}'
# httpbin 响应结果
{
...
"json": {
"users": [
{
"456": {
"name": "lisi"
}
}
]
},
...
}
```
2.将 `users` 第一个元素的 key 为 `123` 重命名为 `msg`:
```yaml
type: request
rules:
- operate: rename
body:
- key: users.0.123
value: users.0.first
```
```bash
$ curl -v -X POST console.higress.io/post \
-H 'Content-Type: application/json' \
-d '{"users":[{"123":{"name":"zhangsan"}},{"456":{"name":"lisi"}}]}'
# httpbin 响应结果
{
...
"json": {
"users": [
{
"msg": {
"name": "zhangsan"
}
},
{
"456": {
"name": "lisi"
}
}
]
},
...
}
```
### 遍历数组元素 `.#`
可以使用 `array.#` 对数组进行遍历操作：
> ❗️该操作目前只能用在 replace 上，请勿在其他转换中尝试该操作，以免造成无法预知的结果
```json
{
"users": [
{
"name": "zhangsan",
"age": 18
},
{
"name": "lisi",
"age": 19
}
]
}
```
```yaml
type: request
rules:
- operate: replace
body:
- key: users.#.age
value: 20
```
```bash
$ curl -v -X POST console.higress.io/post \
-H 'Content-Type: application/json' \
-d '{"users":[{"name":"zhangsan","age":18},{"name":"lisi","age":19}]}'
# httpbin 响应结果
{
...
"json": {
"users": [
{
"age": "20",
"name": "zhangsan"
},
{
"age": "20",
"name": "lisi"
}
]
},
...
}
```

1
plugins/wasm-go/extensions/transformer/VERSION Normal file
View File

@@ -0,0 +1 @@
1.0.0

25
plugins/wasm-go/extensions/transformer/go.mod Normal file
View File

@@ -0,0 +1,25 @@
module github.com/alibaba/higress/plugins/wasm-go/extensions/transformer
go 1.19
replace github.com/alibaba/higress/plugins/wasm-go => ../..
require (
github.com/alibaba/higress/plugins/wasm-go v0.0.0-20230829022308-8747e1ddadf0
github.com/pkg/errors v0.9.1
github.com/stretchr/testify v1.8.0
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0
github.com/tidwall/gjson v1.17.0
)
require (
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/magefile/mage v1.14.0 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.1 // indirect
github.com/tidwall/sjson v1.2.5 // indirect
github.com/wasilibs/nottinygc v0.3.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

40
plugins/wasm-go/extensions/transformer/go.sum Normal file
View File

@@ -0,0 +1,40 @@
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/jinzhu/copier v0.4.0 h1:w3ciUoD19shMCRargcpm0cm91ytaBhDvuRpz1ODO/U8=
github.com/jinzhu/copier v0.4.0/go.mod h1:DfbEm0FYsaqBcKcFuvmOZb218JkPGtvSHsKg8S8hyyg=
github.com/magefile/mage v1.14.0 h1:6QDX3g6z1YvJ4olPhT1wksUcSa/V0a1B+pJb73fBjyo=
github.com/magefile/mage v1.14.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0 h1:kS7BvMKN+FiptV4pfwiNX8e3q14evxAWkhYbxt8EI1M=
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0/go.mod h1:qkW5MBz2jch2u8bS59wws65WC+Gtx3x0aPUX5JL7CXI=
github.com/tidwall/gjson v1.14.2/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/gjson v1.14.3 h1:9jvXn7olKEHU1S9vwoMGliaT8jq1vJ7IH/n9zD9Dnlw=
github.com/tidwall/gjson v1.14.3/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/gjson v1.17.0 h1:/Jocvlh98kcTfpN2+JzGQWQcqrPQwDrVEMApx/M5ZwM=
github.com/tidwall/gjson v1.17.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
github.com/tidwall/pretty v1.2.0 h1:RWIZEg2iJ8/g6fDDYzMpobmaoGh5OLl4AXtGUGPcqCs=
github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
github.com/tidwall/pretty v1.2.1 h1:qjsOFOWWQl+N3RsoF5/ssm1pHmJJwhjlSbZ51I6wMl4=
github.com/tidwall/pretty v1.2.1/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
github.com/tidwall/sjson v1.2.5 h1:kLy8mja+1c9jlljvWTlSazM7cKDRfJuR/bOJhcY5NcY=
github.com/tidwall/sjson v1.2.5/go.mod h1:Fvgq9kS/6ociJEDnK0Fk1cpYF4FIW6ZF7LAe+6jwd28=
github.com/wasilibs/nottinygc v0.3.0 h1:0L1jsJ1MsyN5tdinmFbLfuEA0TnHRcqaBM9pDTJVJmU=
github.com/wasilibs/nottinygc v0.3.0/go.mod h1:oDcIotskuYNMpqMF23l7Z8uzD4TC0WXHK8jetlB3HIo=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

965
plugins/wasm-go/extensions/transformer/main.go Normal file
View File

@@ -0,0 +1,965 @@
// Copyright (c) 2022 Alibaba Group Holding Ltd.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"regexp"
"strings"
"github.com/alibaba/higress/plugins/wasm-go/pkg/wrapper"
"github.com/pkg/errors"
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
"github.com/tidwall/gjson"
"github.com/tidwall/sjson"
)
func main() {
wrapper.SetCtx(
"transformer",
wrapper.ParseConfigBy(parseConfig),
wrapper.ProcessRequestHeadersBy(onHttpRequestHeaders),
wrapper.ProcessRequestBodyBy(onHttpRequestBody),
wrapper.ProcessResponseHeadersBy(onHttpResponseHeaders),
wrapper.ProcessResponseBodyBy(onHttpResponseBody),
)
}
// @Name transformer
// @Category custom
// @Phase UNSPECIFIED_PHASE
// @Priority 100
// @Title zh-CN 请求/响应转换器
// @Title en-US Request/Response Transformer
// @Description zh-CN transformer 插件可以对请求/响应头、请求查询参数、请求/响应体参数进行转换，支持的转换操作类型包括删除、重命名、更新、添加、追加、映射、去重。
// @Description en-US The transformer plugin can transform request/response headers, request query parameters, and request/response body parameters. Supported transform operations include remove, rename, replace, add, append, map, and dedupe.
// @IconUrl https://img.alicdn.com/imgextra/i1/O1CN018iKKih1iVx287RltL_!!6000000004419-2-tps-42-42.png
// @Version 1.0.0
//
// @Contact.name Higress Team
// @Contact.url http://higress.io/
// @Contact.email admin@higress.io
//
// @Example
// type: request
// rules:
// - operate: remove
// headers:
// - key: X-remove
// querys:
// - key: k1
// body:
// - key: a1
// - operate: rename
// headers:
// - key: X-not-renamed
// value: X-renamed
// - operate: replace
// headers:
// - key: X-replace
// value: replaced
// - operate: add
// headers:
// - key: X-add-append
// value: host-$1
// host_pattern: ^(.*)\.com$
// - operate: append
// headers:
// - key: X-add-append
// value: path-$1
// path_pattern: ^.*?\/(\w+)[\?]{0,1}.*$
// body:
// - key: a1-new
// value: t1-$1-append
// value_type: string
// host_pattern: ^(.*)\.com$
// - operate: map
// headers:
// - key: X-add-append
// value: X-map
// - operate: dedupe
// headers:
// - key: X-dedupe-first
// value: RETAIN_FIRST
// @End
type TransformerConfig struct {
// @Title 转换器类型
// @Description 指定转换器类型，可选值为 request, response。
typ string `yaml:"type"`
// @Title 转换规则
// @Description 指定转换操作类型以及请求/响应头、请求查询参数、请求/响应体参数的转换规则
rules []TransformRule `yaml:"rules"`
// this field is not exposed to the user and is used to store the request/response transformer instance
trans Transformer `yaml:"-"`
}
type TransformRule struct {
// @Title 转换操作类型
// @Description 指定转换操作类型，可选值为 remove, rename, replace, add, append, map, dedupe
operate string `yaml:"operate"`
// @Title 请求/响应头转换规则
// @Description 指定请求/响应头转换规则
headers []Param `yaml:"headers"`
// @Title 请求查询参数转换规则
// @Description 指定请求查询参数转换规则
querys []Param `yaml:"querys"`
// @Title 请求/响应体参数转换规则
// @Description 指定请求/响应体参数转换规则，请求体转换允许 content-type 为 application/json, application/x-www-form-urlencoded, multipart/form-data；响应体转换仅允许 content-type 为 application/json
body []Param `yaml:"body"`
}
type Param struct {
// @Title 参数的键
// @Description 指定键值对的键
key string `yaml:"key"`
// @Title 参数的值
// @Description 指定键值对的值，可能的含义有：空 (remove)，key (rename, map), value (replace, add, append), strategy (dedupe)
value string `yaml:"value"`
// @Title 值类型
// @Description 当 content-type=application/json 时，为请求/响应体参数指定值类型，可选值为 object, boolean, number, string(default)
valueType string `yaml:"value_type"`
// @Title 请求主机名匹配规则
// @Description 指定主机名匹配规则，当转换操作类型为 replace, add, append 时有效
hostPattern string `yaml:"host_pattern"`
// @Title 请求路径匹配规则
// @Description 指定路径匹配规则，当转换操作类型为 replace, add, append 时有效
pathPattern string `yaml:"path_pattern"`
}
func parseConfig(json gjson.Result, config *TransformerConfig, log wrapper.Log) (err error) {
config.typ = strings.ToLower(json.Get("type").String())
if config.typ != "request" && config.typ != "response" {
return errors.Errorf("invalid transformer type %q", config.typ)
}
config.rules = make([]TransformRule, 0)
rules := json.Get("rules").Array()
for _, r := range rules {
var tRule TransformRule
tRule.operate = strings.ToLower(r.Get("operate").String())
if !isValidOperation(tRule.operate) {
return errors.Errorf("invalid operate type %q", tRule.operate)
}
for _, h := range r.Get("headers").Array() {
tRule.headers = append(tRule.headers, constructParam(&h, tRule.operate, ""))
}
for _, q := range r.Get("querys").Array() {
tRule.querys = append(tRule.querys, constructParam(&q, tRule.operate, ""))
}
for _, b := range r.Get("body").Array() {
valueType := strings.ToLower(b.Get("value_type").String())
if valueType == "" { // default
valueType = "string"
}
if !isValidJsonType(valueType) {
return errors.Errorf("invalid body params type %q", valueType)
}
tRule.body = append(tRule.body, constructParam(&b, tRule.operate, valueType))
}
config.rules = append(config.rules, tRule)
}
switch config.typ {
case "request":
config.trans, err = newRequestTransformer(config)
case "response":
config.trans, err = newResponseTransformer(config)
}
if err != nil {
return errors.Wrapf(err, "failed to new %s transformer", config.typ)
}
return nil
}
func constructParam(item *gjson.Result, op, valueType string) Param {
p := Param{
key: item.Get("key").String(),
value: item.Get("value").String(),
valueType: valueType,
}
if op == "replace" || op == "add" || op == "append" {
p.hostPattern = item.Get("host_pattern").String()
p.pathPattern = item.Get("path_pattern").String()
}
return p
}
func onHttpRequestHeaders(ctx wrapper.HttpContext, config TransformerConfig, log wrapper.Log) types.Action {
// because it may be a response transformer, so the setting of host and path have to advance
host, path := ctx.Host(), ctx.Path()
ctx.SetContext("host", host)
ctx.SetContext("path", path)
if config.typ == "response" {
return types.ActionContinue
}
log.Debug("on http request headers ...")
headers, err := proxywasm.GetHttpRequestHeaders()
if err != nil {
log.Warn("failed to get request headers")
return types.ActionContinue
}
hs := convertHeaders(headers)
if hs[":authority"] == nil {
log.Warn(errGetRequestHost.Error())
return types.ActionContinue
}
if hs[":path"] == nil {
log.Warn(errGetRequestPath.Error())
return types.ActionContinue
}
contentType := ""
if hs["content-type"] != nil {
contentType = hs["content-type"][0]
}
if config.trans.IsBodyChange() && isValidRequestContentType(contentType) {
delete(hs, "content-length")
ctx.SetContext("content-type", contentType)
} else {
ctx.DontReadRequestBody()
}
if config.trans.IsHeaderChange() {
if err = config.trans.TransformHeaders(host, path, hs); err != nil {
log.Warnf("failed to transform request headers: %v", err)
return types.ActionContinue
}
}
if config.trans.IsQueryChange() {
qs, err := parseQueryByPath(path)
if err != nil {
log.Warnf("failed to parse query params by path: %v", err)
return types.ActionContinue
}
if err = config.trans.TransformQuerys(host, path, qs); err != nil {
log.Warnf("failed to transform request query params: %v", err)
return types.ActionContinue
}
path, err = constructPath(path, qs)
if err != nil {
log.Warnf("failed to construct path: %v", err)
return types.ActionContinue
}
hs[":path"] = []string{path}
}
headers = reconvertHeaders(hs)
if err = proxywasm.ReplaceHttpRequestHeaders(headers); err != nil {
log.Warnf("failed to replace request headers: %v", err)
return types.ActionContinue
}
return types.ActionContinue
}
func onHttpRequestBody(ctx wrapper.HttpContext, config TransformerConfig, body []byte, log wrapper.Log) types.Action {
if config.typ == "response" || !config.trans.IsBodyChange() {
return types.ActionContinue
}
log.Debug("on http request body ...")
host, path, err := getHostAndPathFromHttpCtx(ctx)
if err != nil {
log.Warn(err.Error())
return types.ActionContinue
}
contentType, ok := ctx.GetContext("content-type").(string)
if !ok {
log.Warn(errGetContentType.Error())
return types.ActionContinue
}
structuredBody, err := parseBody(contentType, body)
if err != nil {
if !errors.Is(err, errEmptyBody) {
log.Warnf("failed to parse request body: %v", err)
}
log.Debug("request body is empty")
return types.ActionContinue
}
if err = config.trans.TransformBody(host, path, structuredBody); err != nil {
log.Warnf("failed to transform request body: %v", err)
return types.ActionContinue
}
body, err = constructBody(contentType, structuredBody)
if err != nil {
log.Warnf("failed to construct request body: %v", err)
return types.ActionContinue
}
if err = proxywasm.ReplaceHttpRequestBody(body); err != nil {
log.Warnf("failed to replace request body: %v", err)
return types.ActionContinue
}
return types.ActionContinue
}
func onHttpResponseHeaders(ctx wrapper.HttpContext, config TransformerConfig, log wrapper.Log) types.Action {
if config.typ == "request" {
return types.ActionContinue
}
log.Debug("on http response headers ...")
host, path, err := getHostAndPathFromHttpCtx(ctx)
if err != nil {
log.Warn(err.Error())
return types.ActionContinue
}
headers, err := proxywasm.GetHttpResponseHeaders()
if err != nil {
log.Warnf("failed to get response headers: %v", err)
return types.ActionContinue
}
hs := convertHeaders(headers)
contentType := ""
if hs["content-type"] != nil {
contentType = hs["content-type"][0]
}
if config.trans.IsBodyChange() && isValidResponseContentType(contentType) {
delete(hs, "content-length")
ctx.SetContext("content-type", contentType)
} else {
ctx.DontReadResponseBody()
}
if config.trans.IsHeaderChange() {
if err = config.trans.TransformHeaders(host, path, hs); err != nil {
log.Warnf("failed to transform response headers: %v", err)
return types.ActionContinue
}
}
headers = reconvertHeaders(hs)
if err = proxywasm.ReplaceHttpResponseHeaders(headers); err != nil {
log.Warnf("failed to replace response headers: %v", err)
return types.ActionContinue
}
return types.ActionContinue
}
func onHttpResponseBody(ctx wrapper.HttpContext, config TransformerConfig, body []byte, log wrapper.Log) types.Action {
if config.typ == "request" || !config.trans.IsBodyChange() {
return types.ActionContinue
}
log.Debug("on http response body ...")
host, path, err := getHostAndPathFromHttpCtx(ctx)
if err != nil {
log.Warn(err.Error())
return types.ActionContinue
}
contentType, ok := ctx.GetContext("content-type").(string)
if !ok {
log.Warn(errGetContentType.Error())
return types.ActionContinue
}
structuredBody, err := parseBody(contentType, body)
if err != nil {
if !errors.Is(err, errEmptyBody) {
log.Warnf("failed to parse response body: %v", err)
}
log.Debug("response body is empty")
return types.ActionContinue
}
if err = config.trans.TransformBody(host, path, structuredBody); err != nil {
log.Warnf("failed to transform response body: %v", err)
return types.ActionContinue
}
body, err = constructBody(contentType, structuredBody)
if err != nil {
log.Warnf("failed to construct response body: %v", err)
return types.ActionContinue
}
if err = proxywasm.ReplaceHttpResponseBody(body); err != nil {
log.Warnf("failed to replace response body: %v", err)
return types.ActionContinue
}
return types.ActionContinue
}
func getHostAndPathFromHttpCtx(ctx wrapper.HttpContext) (host, path string, err error) {
host, ok := ctx.GetContext("host").(string)
if !ok {
return "", "", errGetRequestHost
}
path, ok = ctx.GetContext("path").(string)
if !ok {
return "", "", errGetRequestPath
}
return host, path, nil
}
type Transformer interface {
TransformHeaders(host, path string, hs map[string][]string) error
TransformQuerys(host, path string, qs map[string][]string) error
TransformBody(host, path string, body interface{}) error
IsHeaderChange() bool
IsQueryChange() bool
IsBodyChange() bool
}
var _ Transformer = (*requestTransformer)(nil)
var _ Transformer = (*responseTransformer)(nil)
type requestTransformer struct {
headerHandler *kvHandler
queryHandler *kvHandler
bodyHandler *requestBodyHandler
isHeaderChange bool
isQueryChange bool
isBodyChange bool
}
func newRequestTransformer(config *TransformerConfig) (Transformer, error) {
headerKvtGroup, isHeaderChange, err := newKvtGroup(config.rules, "headers")
if err != nil {
return nil, errors.Wrap(err, "failed to new kvt group for headers")
}
queryKvtGroup, isQueryChange, err := newKvtGroup(config.rules, "querys")
if err != nil {
return nil, errors.Wrap(err, "failed to new kvt group for querys")
}
bodyKvtGroup, isBodyChange, err := newKvtGroup(config.rules, "body")
if err != nil {
return nil, errors.Wrap(err, "failed to new kvt group for body")
}
return &requestTransformer{
headerHandler: &kvHandler{headerKvtGroup},
queryHandler: &kvHandler{queryKvtGroup},
bodyHandler: &requestBodyHandler{
formDataHandler: &kvHandler{bodyKvtGroup},
jsonHandler: &jsonHandler{bodyKvtGroup},
},
isHeaderChange: isHeaderChange,
isQueryChange: isQueryChange,
isBodyChange: isBodyChange,
}, nil
}
func (t requestTransformer) TransformHeaders(host, path string, hs map[string][]string) error {
return t.headerHandler.handle(host, path, hs)
}
func (t requestTransformer) TransformQuerys(host, path string, qs map[string][]string) error {
return t.queryHandler.handle(host, path, qs)
}
func (t requestTransformer) TransformBody(host, path string, body interface{}) error {
switch body.(type) {
case map[string][]string:
return t.bodyHandler.formDataHandler.handle(host, path, body.(map[string][]string))
case map[string]interface{}:
m := body.(map[string]interface{})
newBody, err := t.bodyHandler.handle(host, path, m["body"].([]byte))
if err != nil {
return err
}
m["body"] = newBody
default:
return errBodyType
}
return nil
}
func (t requestTransformer) IsHeaderChange() bool { return t.isHeaderChange }
func (t requestTransformer) IsQueryChange() bool { return t.isQueryChange }
func (t requestTransformer) IsBodyChange() bool { return t.isBodyChange }
type responseTransformer struct {
headerHandler *kvHandler
bodyHandler *responseBodyHandler
isHeaderChange bool
isBodyChange bool
}
func newResponseTransformer(config *TransformerConfig) (Transformer, error) {
headerKvtGroup, isHeaderChange, err := newKvtGroup(config.rules, "headers")
if err != nil {
return nil, errors.Wrap(err, "failed to new kvt group for headers")
}
bodyKvtGroup, isBodyChange, err := newKvtGroup(config.rules, "body")
if err != nil {
return nil, errors.Wrap(err, "failed to new kvt group for body")
}
return &responseTransformer{
headerHandler: &kvHandler{headerKvtGroup},
bodyHandler: &responseBodyHandler{&jsonHandler{bodyKvtGroup}},
isHeaderChange: isHeaderChange,
isBodyChange: isBodyChange,
}, nil
}
func (t responseTransformer) TransformHeaders(host, path string, hs map[string][]string) error {
return t.headerHandler.handle(host, path, hs)
}
func (t responseTransformer) TransformQuerys(host, path string, qs map[string][]string) error {
// the response does not need to transform the query params, always returns nil
return nil
}
func (t responseTransformer) TransformBody(host, path string, body interface{}) error {
switch body.(type) {
case map[string]interface{}:
m := body.(map[string]interface{})
newBody, err := t.bodyHandler.handle(host, path, m["body"].([]byte))
if err != nil {
return err
}
m["body"] = newBody
default:
return errBodyType
}
return nil
}
func (t responseTransformer) IsHeaderChange() bool { return t.isHeaderChange }
func (t responseTransformer) IsQueryChange() bool { return false } // the response does not need to transform the query params, always returns false
func (t responseTransformer) IsBodyChange() bool { return t.isBodyChange }
type requestBodyHandler struct {
formDataHandler *kvHandler
*jsonHandler
}
type responseBodyHandler struct {
*jsonHandler
}
type kvHandler struct {
*kvtGroup
}
type jsonHandler struct {
*kvtGroup
}
func (h kvHandler) handle(host, path string, kvs map[string][]string) error {
// order: remove → rename → replace → add → append → map → dedupe
// remove
for _, key := range h.remove {
delete(kvs, key)
}
// rename: 若指定 oldKey 不存在则无操作；否则将 oldKey 的值追加给 newKey，并删除 oldKey:value
for _, item := range h.rename {
oldKey, newKey := item.key, item.value
if ovs, ok := kvs[oldKey]; ok {
kvs[newKey] = append(kvs[newKey], ovs...)
delete(kvs, oldKey)
}
}
// replace: 若指定 key 不存在，则无操作；否则替换 value 为 newValue
for _, item := range h.replace {
key, newValue := item.key, item.value
if _, ok := kvs[key]; !ok {
continue
}
if item.reg != nil {
newValue = item.reg.matchAndReplace(newValue, host, path)
}
kvs[item.key] = []string{newValue}
}
// add: 若指定 key 存在则无操作；否则添加 key:value
for _, item := range h.add {
key, value := item.key, item.value
if _, ok := kvs[key]; ok {
continue
}
if item.reg != nil {
value = item.reg.matchAndReplace(value, host, path)
}
kvs[key] = []string{value}
}
// append: 若指定 key 存在，则追加同名 kv；否则相当于添加操作
for _, item := range h.append {
key, appendValue := item.key, item.value
if item.reg != nil {
appendValue = item.reg.matchAndReplace(appendValue, host, path)
}
kvs[key] = append(kvs[key], appendValue)
}
// map: 若指定 fromKey 不存在则无操作；否则将 fromKey 的值映射给 toKey 的值
for _, item := range h.map_ {
fromKey, toKey := item.key, item.value
if vs, ok := kvs[fromKey]; ok {
kvs[toKey] = vs
}
}
// dedupe: 根据 strategy 去重：RETAIN_UNIQUE 保留所有唯一值，RETAIN_LAST 保留最后一个值，RETAIN_FIRST 保留第一个值 (default)
for _, item := range h.dedupe {
key, strategy := item.key, item.value
switch strings.ToUpper(strategy) {
case "RETAIN_UNIQUE":
uniSet, uniques := make(map[string]struct{}), make([]string, 0)
for _, v := range kvs[key] {
if _, ok := uniSet[v]; !ok {
uniSet[v] = struct{}{}
uniques = append(uniques, v)
}
}
kvs[key] = uniques
case "RETAIN_LAST":
if vs, ok := kvs[key]; ok && len(vs) >= 1 {
kvs[key] = vs[len(vs)-1:]
}
case "RETAIN_FIRST":
fallthrough
default:
if vs, ok := kvs[key]; ok && len(vs) >= 1 {
kvs[key] = vs[:1]
}
}
}
return nil
}
func (h jsonHandler) handle(host, path string, oriData []byte) (data []byte, err error) {
// order: remove → rename → replace → add → append → map → dedupe
if !gjson.ValidBytes(oriData) {
return nil, errors.New("invalid json body")
}
data = oriData
// remove
for _, key := range h.remove {
if data, err = sjson.DeleteBytes(data, key); err != nil {
return nil, errors.Wrap(err, errRemove.Error())
}
}
// rename: 若指定 oldKey 不存在则无操作；否则将 oldKey 的值追加给 newKey，并删除 oldKey:value
for _, item := range h.rename {
oldKey, newKey := item.key, item.value
value := gjson.GetBytes(data, oldKey)
if !value.Exists() {
continue
}
if data, err = sjson.SetBytes(data, newKey, value.Value()); err != nil {
return nil, errors.Wrap(err, errRename.Error())
}
if data, err = sjson.DeleteBytes(data, oldKey); err != nil {
return nil, errors.Wrap(err, errRename.Error())
}
}
// replace: 若指定 key 不存在，则无操作；否则替换 value 为 newValue
for _, item := range h.replace {
key, value, valueType := item.key, item.value, item.typ
if !gjson.GetBytes(data, key).Exists() {
continue
}
if valueType == "string" && item.reg != nil {
value = item.reg.matchAndReplace(value, host, path)
}
newValue, err := convertByJsonType(valueType, value)
if err != nil {
return nil, errors.Wrap(err, errReplace.Error())
}
if data, err = sjson.SetBytes(data, key, newValue); err != nil {
return nil, errors.Wrap(err, errReplace.Error())
}
}
// add: 若指定 key 存在则无操作；否则添加 key:value
for _, item := range h.add {
key, value, valueType := item.key, item.value, item.typ
if gjson.GetBytes(data, key).Exists() {
continue
}
if valueType == "string" && item.reg != nil {
value = item.reg.matchAndReplace(value, host, path)
}
newValue, err := convertByJsonType(valueType, value)
if err != nil {
return nil, errors.Wrap(err, errAdd.Error())
}
if data, err = sjson.SetBytes(data, key, newValue); err != nil {
return nil, errors.Wrap(err, errAdd.Error())
}
}
// append: 若指定 key 存在，则追加同名 kv；否则相当于添加操作
// 当原本的 value 为数组时，追加；当原本的 value 不为数组时，将原本的 value 和 appendValue 组成数组
for _, item := range h.append {
key, value, valueType := item.key, item.value, item.typ
if valueType == "string" && item.reg != nil {
value = item.reg.matchAndReplace(value, host, path)
}
appendValue, err := convertByJsonType(valueType, value)
if err != nil {
return nil, errors.Wrapf(err, errAppend.Error())
}
oldValue := gjson.GetBytes(data, key)
if !oldValue.Exists() {
if data, err = sjson.SetBytes(data, key, appendValue); err != nil { // key: appendValue
return nil, errors.Wrap(err, errAppend.Error())
}
continue
}
// oldValue exists
if oldValue.IsArray() {
if len(oldValue.Array()) == 0 {
if data, err = sjson.SetBytes(data, key, []interface{}{appendValue}); err != nil { // key: [appendValue]
return nil, errors.Wrap(err, errAppend.Error())
}
continue
}
// len(oldValue.Array()) != 0
oldValues := make([]interface{}, 0, len(oldValue.Array())+1)
for _, val := range oldValue.Array() {
oldValues = append(oldValues, val.Value())
}
if data, err = sjson.SetBytes(data, key, append(oldValues, appendValue)); err != nil { // key: [oldValue..., appendValue]
return nil, errors.Wrap(err, errAppend.Error())
}
continue
}
// oldValue is not array
if data, err = sjson.SetBytes(data, key, []interface{}{oldValue.Value(), appendValue}); err != nil { // key: [oldValue, appendValue]
return nil, errors.Wrap(err, errAppend.Error())
}
}
// map: 若指定 fromKey 不存在则无操作；否则将 fromKey 的值映射给 toKey 的值
for _, item := range h.map_ {
fromKey, toKey := item.key, item.value
fromValue := gjson.GetBytes(data, fromKey)
if !fromValue.Exists() {
continue
}
if data, err = sjson.SetBytes(data, toKey, fromValue.Value()); err != nil {
return nil, errors.Wrap(err, errMap.Error())
}
}
// dedupe: 根据 strategy 去重：RETAIN_UNIQUE 保留所有唯一值，RETAIN_LAST 保留最后一个值，RETAIN_FIRST 保留第一个值 (default)
for _, item := range h.dedupe {
key, strategy := item.key, item.value
value := gjson.GetBytes(data, key)
if !value.Exists() || !value.IsArray() {
continue
}
// value is array
values := value.Array()
if len(values) == 0 {
continue
}
var dedupedVal interface{}
switch strings.ToUpper(strategy) {
case "RETAIN_UNIQUE":
uniSet, uniques := make(map[string]struct{}), make([]interface{}, 0)
for _, v := range values {
vstr := v.String()
if _, ok := uniSet[vstr]; !ok {
uniSet[vstr] = struct{}{}
uniques = append(uniques, v.Value())
}
}
if len(uniques) == 1 {
dedupedVal = uniques[0] // key: uniques[0]
} else if len(uniques) > 1 {
dedupedVal = uniques // key: [uniques...]
}
case "RETAIN_LAST":
dedupedVal = values[len(values)-1].Value() // key: last
case "RETAIN_FIRST":
fallthrough
default:
dedupedVal = values[0].Value() // key: first
}
if dedupedVal == nil {
continue
}
if data, err = sjson.SetBytes(data, key, dedupedVal); err != nil {
return nil, errors.Wrap(err, errDedupe.Error())
}
}
return data, nil
}
type kvtGroup struct {
remove []string // key
rename []kvt // oldKey:newKey
replace []kvtReg // key:newValue
add []kvtReg // newKey:newValue
append []kvtReg // key:appendValue
map_ []kvt // fromKey:toKey
dedupe []kvt // key:strategy
}
func newKvtGroup(rules []TransformRule, typ string) (g *kvtGroup, isChange bool, err error) {
g = &kvtGroup{}
for _, r := range rules {
var prams []Param
switch typ {
case "headers":
prams = r.headers
case "querys":
prams = r.querys
case "body":
prams = r.body
}
for _, p := range prams {
switch r.operate {
case "remove":
key := p.key
if typ == "headers" {
key = strings.ToLower(key)
}
g.remove = append(g.remove, key)
case "rename", "map", "dedupe":
var kt kvt
kt.key, kt.value = p.key, p.value
if typ == "headers" {
kt.key = strings.ToLower(kt.key)
if r.operate == "rename" || r.operate == "map" {
kt.value = strings.ToLower(kt.value)
}
}
if typ == "body" {
kt.typ = p.valueType
}
switch r.operate {
case "rename":
g.rename = append(g.rename, kt)
case "map":
g.map_ = append(g.map_, kt)
case "dedupe":
g.dedupe = append(g.dedupe, kt)
}
case "replace", "add", "append":
var kr kvtReg
kr.key, kr.value = p.key, p.value
if typ == "headers" {
kr.key = strings.ToLower(kr.key)
}
if p.hostPattern != "" || p.pathPattern != "" {
kr.reg, err = newReg(p.hostPattern, p.pathPattern)
if err != nil {
return nil, false, errors.Wrap(err, "failed to new reg")
}
}
if typ == "body" {
kr.typ = p.valueType
}
switch r.operate {
case "replace":
g.replace = append(g.replace, kr)
case "add":
g.add = append(g.add, kr)
case "append":
g.append = append(g.append, kr)
}
}
}
}
isChange = len(g.remove) != 0 ||
len(g.rename) != 0 || len(g.replace) != 0 ||
len(g.add) != 0 || len(g.append) != 0 ||
len(g.map_) != 0 || len(g.dedupe) != 0
return g, isChange, nil
}
type kvtReg struct {
kvt
*reg
}
type kvt struct {
key string
value string
typ string
}
type reg struct {
hostReg *regexp.Regexp
pathReg *regexp.Regexp
}
// you can only choose one between host and path
func newReg(hostPatten, pathPatten string) (r *reg, err error) {
r = &reg{}
if hostPatten != "" {
r.hostReg, err = regexp.Compile(hostPatten)
return
}
if pathPatten != "" {
r.pathReg, err = regexp.Compile(pathPatten)
return
}
return
}
func (r reg) matchAndReplace(value, host, path string) string {
if r.hostReg != nil && r.hostReg.MatchString(host) {
return r.hostReg.ReplaceAllString(host, value)
}
if r.pathReg != nil && r.pathReg.MatchString(path) {
return r.pathReg.ReplaceAllString(path, value)
}
return value
}

258
plugins/wasm-go/extensions/transformer/utils.go Normal file
View File

@@ -0,0 +1,258 @@
// Copyright (c) 2022 Alibaba Group Holding Ltd.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"bytes"
"encoding/json"
"io"
"mime"
"mime/multipart"
"net/url"
"strconv"
"strings"
"github.com/pkg/errors"
"github.com/tidwall/pretty"
)
const (
ContentTypeApplicationJson = "application/json"
ContentTypeFormUrlencoded = "application/x-www-form-urlencoded"
ContentTypeMultipartForm = "multipart/form-data"
)
var (
errGetRequestHost = errors.New("failed to get request host")
errGetRequestPath = errors.New("failed to get request path")
errEmptyBody = errors.New("body is empty")
errBodyType = errors.New("unsupported body type")
errGetContentType = errors.New("failed to get content-type from http context")
errRemove = errors.New("failed to remove")
errRename = errors.New("failed to rename")
errReplace = errors.New("failed to replace")
errAdd = errors.New("failed to add")
errAppend = errors.New("failed to append")
errMap = errors.New("failed to map")
errDedupe = errors.New("failed to dedupe")
errContentTypeFmt = "unsupported content-type: %s"
)
func isValidOperation(op string) bool {
switch op {
case "remove", "rename", "replace", "add", "append", "map", "dedupe":
return true
default:
return false
}
}
func parseQueryByPath(path string) (map[string][]string, error) {
u, err := url.Parse(path)
if err != nil {
return nil, err
}
qs := make(map[string][]string)
for k, vs := range u.Query() {
qs[k] = vs
}
return qs, nil
}
func constructPath(path string, qs map[string][]string) (string, error) {
u, err := url.Parse(path)
if err != nil {
return path, err
}
query := url.Values{}
for k, vs := range qs {
for _, v := range vs {
query.Add(k, v)
}
}
u.RawQuery = query.Encode()
return u.String(), nil
}
// 返回值为 map[string]interface{} 或 map[string][]string，使用时断言即可
func parseBody(contentType string, body []byte) (interface{}, error) {
if len(body) == 0 {
return nil, errEmptyBody
}
typ, params, err := mime.ParseMediaType(contentType)
if err != nil {
return nil, err
}
switch typ {
case ContentTypeApplicationJson:
return map[string]interface{}{"body": body}, nil
case ContentTypeFormUrlencoded:
ret := make(map[string][]string)
kvs, err := url.ParseQuery(string(body))
if err != nil {
return nil, err
}
for k, vs := range kvs {
ret[k] = vs
}
return ret, nil
case ContentTypeMultipartForm:
ret := make(map[string][]string)
mr := multipart.NewReader(bytes.NewReader(body), params["boundary"])
for {
p, err := mr.NextPart()
if err != nil {
if err == io.EOF {
break
}
return nil, err
}
formName := p.FormName()
fileName := p.FileName()
if formName == "" || fileName != "" {
continue
}
formValue, err := io.ReadAll(p)
if err != nil {
return nil, err
}
ret[formName] = append(ret[formName], string(formValue))
}
return ret, nil
default:
return nil, errors.Errorf(errContentTypeFmt, contentType)
}
}
func constructBody(contentType string, body interface{}) ([]byte, error) {
typ, params, err := mime.ParseMediaType(contentType)
if err != nil {
return nil, err
}
switch typ {
case ContentTypeApplicationJson:
bd, ok := body.(map[string]interface{})["body"].([]byte)
if !ok {
return nil, errBodyType
}
return pretty.Pretty(bd), nil
case ContentTypeFormUrlencoded:
bd, ok := body.(map[string][]string)
if !ok {
return nil, errBodyType
}
query := url.Values{}
for k, vs := range bd {
for _, v := range vs {
query.Add(k, v)
}
}
return []byte(query.Encode()), nil
case ContentTypeMultipartForm:
bd, ok := body.(map[string][]string)
if !ok {
return nil, errBodyType
}
buf := new(bytes.Buffer)
w := multipart.NewWriter(buf)
if err = w.SetBoundary(params["boundary"]); err != nil {
return nil, err
}
for k, vs := range bd {
for _, v := range vs {
if err = w.WriteField(k, v); err != nil {
return nil, err
}
}
}
if err = w.Close(); err != nil {
return nil, err
}
return buf.Bytes(), nil
default:
return nil, errors.Errorf(errContentTypeFmt, contentType)
}
}
func isValidRequestContentType(contentType string) bool {
typ, _, err := mime.ParseMediaType(contentType)
if err != nil {
return false
}
return typ == ContentTypeApplicationJson || typ == ContentTypeFormUrlencoded || typ == ContentTypeMultipartForm
}
func isValidResponseContentType(contentType string) bool {
typ, _, err := mime.ParseMediaType(contentType)
if err != nil {
return false
}
return typ == ContentTypeApplicationJson
}
func convertByJsonType(typ string, value string) (ret interface{}, err error) {
switch strings.ToLower(typ) {
case "object":
err = json.Unmarshal([]byte(value), &ret)
case "boolean":
ret, err = strconv.ParseBool(value)
case "number":
ret, err = strconv.ParseFloat(value, 64)
case "string":
fallthrough
default:
ret = value
}
return
}
func isValidJsonType(typ string) bool {
switch typ {
case "object", "boolean", "number", "string":
return true
default:
return false
}
}
// headers: [][2]string -> map[string][]string
func convertHeaders(hs [][2]string) map[string][]string {
ret := make(map[string][]string)
for _, h := range hs {
k, v := strings.ToLower(h[0]), h[1]
ret[k] = append(ret[k], v)
}
return ret
}
// headers: map[string][]string -> [][2]string
func reconvertHeaders(hs map[string][]string) [][2]string {
var ret [][2]string
for k, vs := range hs {
for _, v := range vs {
ret = append(ret, [2]string{k, v})
}
}
return ret
}

328
plugins/wasm-go/extensions/transformer/utils_test.go Normal file
View File

@@ -0,0 +1,328 @@
// Copyright (c) 2022 Alibaba Group Holding Ltd.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"fmt"
"testing"
"github.com/stretchr/testify/require"
)
func TestParseQueryByPath(t *testing.T) {
cases := []struct {
name string
path string
expected map[string][]string
errMsg string
}{
{
name: "common",
path: "/get?k1=v1&k2=v2&k3=v3",
expected: map[string][]string{
"k1": {"v1"},
"k2": {"v2"},
"k3": {"v3"},
},
},
{
name: "empty query",
path: "www.example.com/get",
expected: map[string][]string{},
},
{
name: "multiple values",
path: "www.example.com/get?k1=v11&k1=v12&k2=v2&k1=v13",
expected: map[string][]string{
"k1": {"v11", "v12", "v13"},
"k2": {"v2"},
},
},
{
name: "encoded url",
path: "/get%20with%3Freserved%20characters?key=Hello+World",
expected: map[string][]string{
"key": {"Hello World"},
},
},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
actual, err := parseQueryByPath(c.path)
if c.errMsg != "" {
require.EqualError(t, err, c.errMsg)
return
}
require.NoError(t, err)
require.Equal(t, c.expected, actual)
})
}
}
func TestConstructPath(t *testing.T) {
cases := []struct {
name string
path string
qs map[string][]string
expected string
errMsg string
}{
{
name: "common",
path: "/get",
qs: map[string][]string{
"k1": {"v1"},
"k2": {"v2"},
"k3": {"v3"},
},
expected: "/get?k1=v1&k2=v2&k3=v3",
},
{
name: "empty query",
path: "www.example.com/get",
qs: map[string][]string{},
expected: "www.example.com/get",
},
{
name: "encoded url",
path: "/get with?",
qs: map[string][]string{
"key": {"Hello World"},
},
expected: "/get%20with?key=Hello+World",
},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
actual, err := constructPath(c.path, c.qs)
if c.errMsg != "" {
require.EqualError(t, err, c.errMsg)
return
}
require.NoError(t, err)
require.Equal(t, c.expected, actual)
})
}
}
func TestParseBody(t *testing.T) {
cases := []struct {
name string
mediaType string
body []byte
expected interface{}
errMsg string
}{
{
name: "application/json",
mediaType: "application/json",
body: []byte(`{
"k1": "v2",
"k2": 20,
"k3": true,
"k4": [1, 2, 3],
"k5": {
"k6": "v6"
}
}`),
expected: map[string]interface{}{"body": []byte(`{
"k1": "v2",
"k2": 20,
"k3": true,
"k4": [1, 2, 3],
"k5": {
"k6": "v6"
}
}`)},
},
{
name: "application/x-www-form-urlencoded",
mediaType: "application/x-www-form-urlencoded",
body: []byte("k1=v11&k1=v12&k2=v2&k3=v3"),
expected: map[string][]string{
"k1": {"v11", "v12"},
"k2": {"v2"},
"k3": {"v3"},
},
},
{
name: "multipart/form-data",
mediaType: "multipart/form-data; boundary=--------------------------962785348548682888818907",
body: []byte("----------------------------962785348548682888818907\r\nContent-Disposition: form-data; name=\"k1\"\r\n\r\nv11\r\n----------------------------962785348548682888818907\r\nContent-Disposition: form-data; name=\"k1\"\r\n\r\nv12\r\n----------------------------962785348548682888818907\r\nContent-Disposition: form-data; name=\"k2\"\r\n\r\nv2\r\n----------------------------962785348548682888818907--\r\n"),
expected: map[string][]string{
"k1": {"v11", "v12"},
"k2": {"v2"},
},
},
{
name: "unsupported content type",
mediaType: "plain/text",
body: []byte(`qwe`),
errMsg: fmt.Sprintf(errContentTypeFmt, "plain/text"),
},
{
name: "empty body",
mediaType: "application/json",
body: []byte(``),
errMsg: errEmptyBody.Error(),
},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
actual, err := parseBody(c.mediaType, c.body)
if c.errMsg != "" {
require.EqualError(t, err, c.errMsg)
return
}
require.NoError(t, err)
require.Equal(t, c.expected, actual)
})
}
}
func TestConstructBody(t *testing.T) {
cases := []struct {
name string
mediaType string
body interface{}
expected []byte
errMsg string
}{
{
name: "application/json",
mediaType: "application/json",
body: map[string]interface{}{"body": []byte(`{
"k1": {
"k2": [1, 2, 3]
}
}
`)},
expected: []byte(`{
"k1": {
"k2": [1, 2, 3]
}
}
`),
},
{
name: "application/x-www-form-urlencoded",
mediaType: "application/x-www-form-urlencoded",
body: map[string][]string{
"k1": {"v11", "v12"},
},
expected: []byte("k1=v11&k1=v12"),
},
{
name: "multipart/form-data",
mediaType: "multipart/form-data; boundary=--------------------------962785348548682888818907",
body: map[string][]string{
"k1": {"v11", "v12"},
},
expected: []byte("----------------------------962785348548682888818907\r\nContent-Disposition: form-data; name=\"k1\"\r\n\r\nv11\r\n----------------------------962785348548682888818907\r\nContent-Disposition: form-data; name=\"k1\"\r\n\r\nv12\r\n----------------------------962785348548682888818907--\r\n"),
},
{
name: "unsupported media type",
mediaType: "plain/text",
body: []byte(`qwe`),
errMsg: fmt.Sprintf(errContentTypeFmt, "plain/text"),
},
{
name: "empty body",
mediaType: "application/json",
body: map[string]interface{}{"body": []byte{}},
expected: []byte{},
},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
actual, err := constructBody(c.mediaType, c.body)
if c.errMsg != "" {
require.EqualError(t, err, c.errMsg)
return
}
require.NoError(t, err)
require.Equal(t, c.expected, actual)
})
}
}
func TestConvertByJsonType(t *testing.T) {
cases := []struct {
name string
valueTyp string
value string
expected interface{}
errMsg string
}{
{
name: "object",
valueTyp: "object",
value: "{\"array\": [1, 2, 3], \"object\": { \"first\": \"hello\", \"second\": \"world\" } }",
expected: map[string]interface{}{
"array": []interface{}{float64(1), float64(2), float64(3)},
"object": map[string]interface{}{
"first": "hello",
"second": "world",
},
},
},
{
name: "boolean",
valueTyp: "boolean",
value: "true",
expected: true,
},
{
name: "boolean: failed",
valueTyp: "boolean",
value: "null",
errMsg: "strconv.ParseBool: parsing \"null\": invalid syntax",
},
{
name: "number",
valueTyp: "number",
value: "10",
expected: float64(10),
},
{
name: "string",
valueTyp: "string",
value: "hello world",
expected: "hello world",
},
{
name: "unsupported type",
valueTyp: "integer",
value: "10",
expected: "10", // default string
},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
actual, err := convertByJsonType(c.valueTyp, c.value)
if c.errMsg != "" {
require.EqualError(t, err, c.errMsg)
return
}
require.NoError(t, err)
require.Equal(t, c.expected, actual)
})
}
}

121
test/e2e/conformance/tests/go-wasm-transformer.go Normal file
View File

@@ -0,0 +1,121 @@
// Copyright (c) 2022 Alibaba Group Holding Ltd.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package tests
import (
"testing"
"github.com/alibaba/higress/test/e2e/conformance/utils/http"
"github.com/alibaba/higress/test/e2e/conformance/utils/suite"
)
func init() {
Register(WasmPluginsTransformer)
}
// TODO(WeixinX): Request and response body conformance check is not supported now
var WasmPluginsTransformer = suite.ConformanceTest{
ShortName: "WasmPluginTransformer",
Description: "The Ingress in the higress-conformance-infra namespace test the transformer WASM plugin.",
Features: []suite.SupportedFeature{suite.WASMGoConformanceFeature},
Manifests: []string{"tests/go-wasm-transformer.yaml"},
Test: func(t *testing.T, suite *suite.ConformanceTestSuite) {
testcases := []http.Assertion{
{
Meta: http.AssertionMeta{
TestCaseName: "case 1: request transformer",
TargetBackend: "infra-backend-v1",
TargetNamespace: "higress-conformance-infra",
},
Request: http.AssertionRequest{
ActualRequest: http.Request{
Host: "foo1.com",
Path: "/get?k1=v11&k1=v12&k2=v2",
Headers: map[string]string{
"X-remove": "exist",
"X-not-renamed": "test",
"X-replace": "not-replaced",
"X-dedupe-first": "1,2,3",
"X-dedupe-last": "a,b,c",
"X-dedupe-unique": "1,2,3,3,2,1",
},
},
ExpectedRequest: &http.ExpectedRequest{
Request: http.Request{
Host: "foo1.com",
Path: "/get?k2-new=v2-new&k3=v31&k3=v32&k4=v31", // url.Value.Encode() is ordered by key
Headers: map[string]string{
"X-renamed": "test",
"X-replace": "replaced",
"X-add-append": "add,append", // header with same name
"X-map": "add,append",
"X-dedupe-first": "1",
"X-dedupe-last": "c",
"X-dedupe-unique": "1,2,3",
},
},
AbsentHeaders: []string{"X-remove"},
},
},
Response: http.AssertionResponse{
ExpectedResponse: http.Response{
StatusCode: 200,
},
},
},
{
Meta: http.AssertionMeta{
TestCaseName: "case 2: response transformer",
TargetBackend: "infra-backend-v1",
TargetNamespace: "higress-conformance-infra",
},
Request: http.AssertionRequest{
ActualRequest: http.Request{
Host: "foo2.com",
Path: "/get/index.html",
},
ExpectedRequest: &http.ExpectedRequest{
Request: http.Request{
Host: "foo2.com",
Path: "/get/index.html",
},
},
},
Response: http.AssertionResponse{
AdditionalResponseHeaders: map[string]string{
"X-remove": "exist",
"X-not-renamed": "test",
"X-replace": "not-replaced",
},
ExpectedResponse: http.Response{
StatusCode: 200,
Headers: map[string]string{
"X-renamed": "test",
"X-replace": "replace-get", // regexp matches path and replace "replace-$1"
"X-add-append": "add-foo2,append-index", // regexp matches host and replace "add-$1"
"X-map": "add-foo2,append-index",
},
AbsentHeaders: []string{"X-remove"},
},
},
},
}
t.Run("WasmPlugin transformer", func(t *testing.T) {
for _, testcase := range testcases {
http.MakeRequestAndExpectEventuallyConsistentResponse(t, suite.RoundTripper, suite.TimeoutConfig, suite.GatewayAddress, testcase)
}
})
},
}

155
test/e2e/conformance/tests/go-wasm-transformer.yaml Normal file
View File

@@ -0,0 +1,155 @@
# Copyright (c) 2022 Alibaba Group Holding Ltd.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
name: wasmplugin-transform-request
namespace: higress-conformance-infra
spec:
ingressClassName: higress
rules:
- host: "foo1.com"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: infra-backend-v1
port:
number: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
name: wasmplugin-transform-response
namespace: higress-conformance-infra
spec:
ingressClassName: higress
rules:
- host: "foo2.com"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: infra-backend-v1
port:
number: 8080
---
apiVersion: extensions.higress.io/v1alpha1
kind: WasmPlugin
metadata:
name: transformer
namespace: higress-system
spec:
matchRules:
# request transformer
- ingress:
- higress-conformance-infra/wasmplugin-transform-request
configDisable: false
config:
type: request
rules:
- operate: remove
headers:
- key: X-remove
querys:
- key: k1
- operate: rename
headers:
- key: X-not-renamed
value: X-renamed
querys:
- key: k2
value: k2-new
- operate: replace
headers:
- key: X-replace
value: replaced
querys:
- key: k2-new
value: v2-new
- operate: add
headers:
- key: X-add-append
value: add
querys:
- key: k3
value: v31
- operate: append
headers:
- key: X-add-append
value: append
querys:
- key: k3
value: v32
- operate: map
headers:
- key: X-add-append
value: X-map
querys:
- key: k3
value: k4
- operate: dedupe
headers:
- key: X-dedupe-first
value: RETAIN_FIRST
- key: X-dedupe-last
value: RETAIN_LAST
- key: X-dedupe-unique
value: RETAIN_UNIQUE
querys:
- key: k4
value: RETAIN_FIRST
# response transformer
- ingress:
- higress-conformance-infra/wasmplugin-transform-response
configDisable: false
config:
type: response
rules:
- operate: remove
headers:
- key: X-remove
- operate: rename
headers:
- key: X-not-renamed
value: X-renamed
- operate: replace
headers:
- key: X-replace
value: replace-$1
path_pattern: ^.*?\/(\w+)[\?]{0,1}.*$
- operate: add
headers:
- key: X-add-append
value: add-$1
host_pattern: ^(.*)\.com$
- operate: append
headers:
- key: X-add-append
value: append-$1
path_pattern: ^\/get\/(.*)\.html$
- operate: map
headers:
- key: X-add-append
value: X-map
url: file:///opt/plugins/wasm-go/extensions/transformer/plugin.wasm

5
test/e2e/conformance/utils/http/http.go
View File

@@ -189,7 +189,10 @@ func MakeRequestAndExpectEventuallyConsistentResponse(t *testing.T, r roundtripp
if expected.Request.ActualRequest.Headers != nil {
for name, value := range expected.Request.ActualRequest.Headers {
req.Headers[name] = []string{value}
vals := strings.Split(value, ",")
for _, val := range vals {
req.Headers[name] = append(req.Headers[name], strings.TrimSpace(val))
}
}
}

6
test/e2e/conformance/utils/roundtripper/roundtripper.go
View File

@@ -164,8 +164,10 @@ func (d *DefaultRoundTripper) CaptureRoundTrip(request Request) (*CapturedReques
}
if request.Headers != nil {
for name, value := range request.Headers {
req.Header.Set(name, value[0])
for name, values := range request.Headers {
for _, value := range values {
req.Header.Add(name, value)
}
}
}