Compare commits

..

42 Commits

Author SHA1 Message Date
澄潭
021387c9d3 Update Makefile.core.mk 2024-02-05 00:02:34 +08:00
澄潭
ea0a694d81 Update build-image-and-push.yaml 2024-02-04 23:57:49 +08:00
澄潭
8028fe03ca Update build-image-and-push.yaml 2024-02-04 23:57:00 +08:00
澄潭
a68dde0b91 Update build-image-and-push.yaml 2024-02-04 23:44:31 +08:00
澄潭
48c3db85c4 Update build-image-and-push.yaml 2024-02-04 23:41:25 +08:00
澄潭
7e85065832 increase health check timeout (#820) 2024-02-04 18:52:14 +08:00
SJC
7967f5db70 feat: add new param for global option (#813)
Signed-off-by: sjcsjc123 <1401189096@qq.com>
2024-02-04 13:58:46 +08:00
Jun
5026973d59 fix: hgctl latest version bug (#816) 2024-02-03 12:17:22 +08:00
澄潭
7097eef6ba Update Makefile.core.mk 2024-02-03 11:54:58 +08:00
onlypiglet
fae222806b Implement the Go Wasm plugin: bot-detect (#747) 2024-02-02 16:57:41 +08:00
SJC
c63cdb62ea feat: add request-validation plugin (#700)
Signed-off-by: sjcsjc123 <1401189096@qq.com>
2024-02-02 16:52:43 +08:00
澄潭
e43f5d106f fix rds cache (#815) 2024-02-02 16:42:25 +08:00
澄潭
29c95ea557 Update Makefile.core.mk 2024-02-01 19:59:03 +08:00
澄潭
73d5cc3f1d rel: Release verion 1.3.4-rc.2 (#812) 2024-02-01 19:41:24 +08:00
澄潭
c1ddbcef7c Enable srds by default (#811) 2024-02-01 18:48:02 +08:00
rinfx
dd39c87311 bugfix: "path=/" will cause panic (#809) 2024-02-01 11:50:26 +08:00
baerwang
612c94dd8a feat: wasm support opa (Open Policy Agent) (#760) 2024-01-30 15:29:51 +08:00
llahu
e67ed481cf feat: Use new goproxy in makefile (#805) 2024-01-29 10:03:05 +08:00
rinfx
ccea33655f add devcontainer for wasm-go plugin developing (#749)
Co-authored-by: Xunzhuo <bitliu@tencent.com>
2024-01-26 16:59:06 +08:00
Uncle-Justice
ad4cfdbd40 Unify wasm go sdk version (#797) 2024-01-26 16:48:57 +08:00
Uncle-Justice
3598c21da0 test: add codecov target for patch (#792) 2024-01-26 16:47:34 +08:00
SJC
a624351f84 feat: e2e test opt (#746)
Signed-off-by: sjcsjc123 <1401189096@qq.com>
2024-01-26 16:45:33 +08:00
Kent Dong
c41264816e feat: Support matching pseudo headers (#803) 2024-01-26 16:44:12 +08:00
SJC
acd80d2528 feat: add global option for upstream (#798)
Signed-off-by: sjcsjc123 <1401189096@qq.com>
2024-01-26 11:25:59 +08:00
澄潭
073c10df77 optimize rewrite regex (#801) 2024-01-25 10:20:49 +08:00
SJC
90f89cf588 fix: bug with close setting of idle timeout (#775)
Signed-off-by: sjcsjc123 <1401189096@qq.com>
2024-01-22 13:44:51 +08:00
Uncle-Justice
879192cf99 feat: optimize transformer go wasm plugin (#712) 2024-01-22 13:40:55 +08:00
澄潭
d3d000753d optimize destinationRule generate (#782) 2024-01-19 18:06:19 +08:00
hansedong
b8a01113e3 improve log output format (#789)
Signed-off-by: hansedong <skipper1314@gmail.com>
2024-01-19 18:02:19 +08:00
baerwang
0bb9e6dd89 docs: Pre-development preparation (#762) 2024-01-18 10:40:29 +08:00
澄潭
ecdd077c72 Optimize rds cache (#779) 2024-01-18 10:39:58 +08:00
澄潭
e971faeb0b remove useless kube client in nacos v1 watcher (#788) 2024-01-18 10:30:56 +08:00
澄潭
77013e28b6 fix CVE-2023-44487 (#784) 2024-01-17 15:01:08 +08:00
澄潭
9faa5f37d1 Update CODEOWNERS 2024-01-16 20:28:46 +08:00
Uncle-Justice
665d9fa943 test: add testcases for CompareRequest&Response (#778) 2024-01-16 11:43:30 +08:00
Zihan Li
a71ecf41d1 bugfix(main): fix test coverage panic permission denied in darwin/arm64 (#741) 2024-01-12 14:09:22 +08:00
Uncle-Justice
b825f9176f feat: e2e test support http body check (#733) 2024-01-12 14:08:46 +08:00
澄潭
d35d23e2d5 Update Makefile.core.mk 2024-01-12 11:33:38 +08:00
澄潭
6d1e09c146 Update Makefile.core.mk 2024-01-11 18:04:40 +08:00
澄潭
87c39d393f rel: Release verion 1.3.4-rc.1 (#773) 2024-01-11 17:42:15 +08:00
澄潭
c97260c4a9 fix memleak of srds compute (#772) 2024-01-11 17:21:03 +08:00
澄潭
5e509e7032 fix: strip port from host when match scope rds (#770) 2024-01-10 20:16:36 +08:00
124 changed files with 13085 additions and 702 deletions

View File

@@ -52,11 +52,9 @@ jobs:
uses: actions/cache@v3
with:
path: |-
envoy
istio
.git/modules
key: ${{ runner.os }}-submodules-new-${{ github.run_id }}
restore-keys: ${{ runner.os }}-submodules-new
key: ${{ runner.os }}-submodules-cache-${{ github.run_id }}
restore-keys: ${{ runner.os }}-submodules-cache
- run: git stash # restore patch

View File

@@ -36,11 +36,9 @@ jobs:
uses: actions/cache@v3
with:
path: |-
envoy
istio
.git/modules
key: ${{ runner.os }}-submodules-new-${{ github.run_id }}
restore-keys: ${{ runner.os }}-submodules-new
key: ${{ runner.os }}-submodules-cache-${{ github.run_id }}
restore-keys: ${{ runner.os }}-submodules-cache
- run: git stash # restore patch
@@ -82,11 +80,9 @@ jobs:
uses: actions/cache@v3
with:
path: |-
envoy
istio
.git/modules
key: ${{ runner.os }}-submodules-new-${{ github.run_id }}
restore-keys: ${{ runner.os }}-submodules-new
key: ${{ runner.os }}-submodules-cache-${{ github.run_id }}
restore-keys: ${{ runner.os }}-submodules-cache
- run: git stash # restore patch
@@ -130,11 +126,9 @@ jobs:
uses: actions/cache@v3
with:
path: |-
envoy
istio
.git/modules
key: ${{ runner.os }}-submodules-new-${{ github.run_id }}
restore-keys: ${{ runner.os }}-submodules-new
key: ${{ runner.os }}-submodules-cache-${{ github.run_id }}
restore-keys: ${{ runner.os }}-submodules-cache
- run: git stash # restore patch

View File

@@ -20,6 +20,16 @@ jobs:
with:
fetch-depth: 1
- name: Free Up GitHub Actions Ubuntu Runner Disk Space 🔧
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
android: true
dotnet: true
haskell: true
large-packages: true
swap-storage: true
- name: "Setup Go"
uses: actions/setup-go@v3
with:
@@ -86,6 +96,16 @@ jobs:
with:
fetch-depth: 1
- name: Free Up GitHub Actions Ubuntu Runner Disk Space 🔧
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
android: true
dotnet: true
haskell: true
large-packages: true
swap-storage: true
- name: "Setup Go"
uses: actions/setup-go@v3
with:
@@ -153,6 +173,16 @@ jobs:
with:
fetch-depth: 1
- name: Free Up GitHub Actions Ubuntu Runner Disk Space 🔧
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
android: true
dotnet: true
haskell: true
large-packages: true
swap-storage: true
- name: "Setup Go"
uses: actions/setup-go@v3
with:

View File

@@ -26,6 +26,7 @@ header:
- 'VERSION'
- 'tools/'
- 'test/README.md'
- 'test/README_CN.md'
- 'cmd/hgctl/config/testdata/config'
- 'pkg/cmd/hgctl/manifests'

View File

@@ -1,7 +1,7 @@
/api @johnlanni @CH3CHO
/envoy @gengleilei @johnlanni
/istio @SpecialYang @johnlanni
/pkg @SpecialYang @johnlanni @CH3CHO @Xunzhuo
/pkg @SpecialYang @johnlanni @CH3CHO
/plugins @johnlanni @WeixinX
/registry @NameHaibinZhang @2456868764 @johnlanni
/test @Xunzhuo @2456868764 @CH3CHO

View File

@@ -73,6 +73,7 @@
* [分支定义](#分支定义)
* [提交规则](#提交规则)
* [PR说明](#PR说明)
* [开发前准备](#开发前准备)
### 工作区准备
@@ -168,6 +169,12 @@ git config --get user.email
PR 是更改 Higress 项目文件的唯一方法。为了帮助审查人更好地理解你的目的PR 描述不能太详细。我们鼓励贡献者遵循 [PR 模板](./.github/PULL_REQUEST_TEMPLATE.md) 来完成拉取请求。
### 开发前准备
```shell
make prebuild && go mod tidy
```
## 测试用例贡献
任何测试用例都会受到欢迎。目前Higress 功能测试用例是高优先级的。

View File

@@ -169,6 +169,12 @@ No matter commit message, or commit content, we do take more emphasis on code re
PR is the only way to make change to Higress project files. To help reviewers better get your purpose, PR description could not be too detailed. We encourage contributors to follow the [PR template](./.github/PULL_REQUEST_TEMPLATE.md) to finish the pull request.
### Pre-development preparation
```shell
make prebuild && go mod tidy
```
## Test case contribution
Any test case would be welcomed. Currently, Higress function test cases are high priority.

View File

@@ -15,7 +15,7 @@ GO_LDFLAGS += -X $(VERSION_PACKAGE).higressVersion=$(shell cat VERSION) \
GO ?= go
export GOPROXY ?= https://proxy.golang.com.cn,direct
export GOPROXY ?= https://proxy.golang.org,direct
TARGET_ARCH ?= amd64
@@ -138,11 +138,11 @@ export ENVOY_TAR_PATH:=/home/package/envoy.tar.gz
external/package/envoy-amd64.tar.gz:
# cd external/proxy; BUILD_WITH_CONTAINER=1 make test_release
cd external/package; wget "https://github.com/alibaba/higress/releases/download/v1.3.2/envoy-amd64.tar.gz"
cd external/package; wget -O envoy-amd64.tar.gz "https://github.com/alibaba/higress/releases/download/v1.3.4-rc.1/envoy-symbol-amd64.tar.gz"
external/package/envoy-arm64.tar.gz:
# cd external/proxy; BUILD_WITH_CONTAINER=1 make test_release
cd external/package; wget "https://github.com/alibaba/higress/releases/download/v1.3.2/envoy-arm64.tar.gz"
cd external/package; wget -O envoy-arm64.tar.gz "https://github.com/alibaba/higress/releases/download/v1.3.4-rc.1/envoy-symbol-arm64.tar.gz"
build-pilot:
cd external/istio; rm -rf out/linux_amd64; GOOS_LOCAL=linux TARGET_OS=linux TARGET_ARCH=amd64 BUILD_WITH_CONTAINER=1 make build-linux
@@ -154,13 +154,13 @@ build-pilot-local:
build-gateway: prebuild external/package/envoy-amd64.tar.gz external/package/envoy-arm64.tar.gz build-pilot
cd external/istio; BUILD_WITH_CONTAINER=1 BUILDX_PLATFORM=true DOCKER_BUILD_VARIANTS=default DOCKER_TARGETS="docker.proxyv2" make docker
build-gateway-local: prebuild external/package/envoy-amd64.tar.gz external/package/envoy-arm64.tar.gz build-pilot
build-gateway-local: prebuild external/package/envoy-amd64.tar.gz external/package/envoy-arm64.tar.gz
cd external/istio; rm -rf out/linux_${GOARCH_LOCAL}; GOOS_LOCAL=linux TARGET_OS=linux BUILD_WITH_CONTAINER=1 BUILDX_PLATFORM=false DOCKER_BUILD_VARIANTS=default DOCKER_TARGETS="docker.proxyv2" make docker
build-istio: prebuild build-pilot
cd external/istio; BUILD_WITH_CONTAINER=1 BUILDX_PLATFORM=true DOCKER_BUILD_VARIANTS=default DOCKER_TARGETS="docker.pilot" make docker
build-istio-local: prebuild build-pilot-local
build-istio-local: prebuild
cd external/istio; rm -rf out/linux_${GOARCH_LOCAL}; GOOS_LOCAL=linux TARGET_OS=linux BUILD_WITH_CONTAINER=1 BUILDX_PLATFORM=false DOCKER_BUILD_VARIANTS=default DOCKER_TARGETS="docker.pilot" make docker
build-wasmplugins:
@@ -177,13 +177,13 @@ install: pre-install
cd helm/higress; helm dependency build
helm install higress helm/higress -n higress-system --create-namespace --set 'global.local=true'
ENVOY_LATEST_IMAGE_TAG ?= sha-53ff28c
ISTIO_LATEST_IMAGE_TAG ?= sha-53ff28c
ENVOY_LATEST_IMAGE_TAG ?= sha-a68dde0
ISTIO_LATEST_IMAGE_TAG ?= sha-a68dde0
install-dev: pre-install
helm install higress helm/core -n higress-system --create-namespace --set 'controller.tag=$(TAG)' --set 'gateway.replicas=1' --set 'pilot.tag=$(ISTIO_LATEST_IMAGE_TAG)' --set 'gateway.tag=$(ENVOY_LATEST_IMAGE_TAG)' --set 'global.local=true'
install-dev-wasmplugin: build-wasmplugins pre-install
helm install higress helm/core -n higress-system --create-namespace --set 'controller.tag=$(TAG)' --set 'gateway.replicas=1' --set 'pilot.tag=$(ISTIO_LATEST_IMAGE_TAG)' --set 'gateway.tag=$(ENVOY_LATEST_IMAGE_TAG)' --set 'global.local=true' --set 'global.volumeWasmPlugins=true'
helm install higress helm/core -n higress-system --create-namespace --set 'controller.tag=$(TAG)' --set 'gateway.replicas=1' --set 'pilot.tag=$(ISTIO_LATEST_IMAGE_TAG)' --set 'gateway.tag=$(ENVOY_LATEST_IMAGE_TAG)' --set 'global.local=true' --set 'global.volumeWasmPlugins=true' --set 'global.onlyPushRouteCluster=false'
uninstall:
helm uninstall higress -n higress-system
@@ -233,14 +233,30 @@ include tools/lint.mk
.PHONY: gateway-conformance-test
gateway-conformance-test:
# higress-conformance-test-prepare prepares the environment for higress conformance tests.
.PHONY: higress-conformance-test-prepare
higress-conformance-test-prepare: $(tools/kind) delete-cluster create-cluster docker-build kube-load-image install-dev
# higress-conformance-test runs ingress api conformance tests.
.PHONY: higress-conformance-test
higress-conformance-test: $(tools/kind) delete-cluster create-cluster docker-build kube-load-image install-dev run-higress-e2e-test delete-cluster
# higress-conformance-test-clean cleans the environment for higress conformance tests.
.PHONY: higress-conformance-test-clean
higress-conformance-test-clean: $(tools/kind) delete-cluster
# higress-wasmplugin-test-prepare prepares the environment for higress wasmplugin tests.
.PHONY: higress-wasmplugin-test-prepare
higress-wasmplugin-test-prepare: $(tools/kind) delete-cluster create-cluster docker-build kube-load-image install-dev-wasmplugin
# higress-wasmplugin-test runs ingress wasmplugin tests.
.PHONY: higress-wasmplugin-test
higress-wasmplugin-test: $(tools/kind) delete-cluster create-cluster docker-build kube-load-image install-dev-wasmplugin run-higress-e2e-test-wasmplugin delete-cluster
# higress-wasmplugin-test-clean cleans the environment for higress wasmplugin tests.
.PHONY: higress-wasmplugin-test-clean
higress-wasmplugin-test-clean: $(tools/kind) delete-cluster
# create-cluster creates a kube cluster with kind.
.PHONY: create-cluster
create-cluster: $(tools/kind)
@@ -270,6 +286,17 @@ kube-load-image: $(tools/kind) ## Install the Higress image to a kind cluster us
tools/hack/kind-load-image.sh docker.io/alihigress/httpbin 1.0.2
tools/hack/kind-load-image.sh docker.io/charlie1380/eureka-registry-provider v0.3.0
tools/hack/kind-load-image.sh docker.io/bitinit/eureka latest
# run-higress-e2e-test-setup starts to setup ingress e2e tests.
.PHONT: run-higress-e2e-test-setup
run-higress-e2e-test-setup:
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
@echo -e "\n\033[36mWaiting higress-controller to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-controller --for=condition=Available
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
go test -v -tags conformance ./test/e2e/e2e_test.go --ingress-class=higress --debug=true --test-area=setup
# run-higress-e2e-test starts to run ingress e2e tests.
.PHONY: run-higress-e2e-test
run-higress-e2e-test:
@@ -278,9 +305,39 @@ run-higress-e2e-test:
kubectl wait --timeout=10m -n higress-system deployment/higress-controller --for=condition=Available
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
go test -v -tags conformance ./test/e2e/e2e_test.go --ingress-class=higress --debug=true
go test -v -tags conformance ./test/e2e/e2e_test.go --ingress-class=higress --debug=true --test-area=all
# run-higress-e2e-test starts to run ingress e2e tests.
# run-higress-e2e-test-run starts to run ingress e2e conformance tests.
.PHONY: run-higress-e2e-test-run
run-higress-e2e-test-run:
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
@echo -e "\n\033[36mWaiting higress-controller to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-controller --for=condition=Available
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
go test -v -tags conformance ./test/e2e/e2e_test.go --ingress-class=higress --debug=true --test-area=run
# run-higress-e2e-test-clean starts to clean ingress e2e tests.
.PHONY: run-higress-e2e-test-clean
run-higress-e2e-test-clean:
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
@echo -e "\n\033[36mWaiting higress-controller to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-controller --for=condition=Available
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
go test -v -tags conformance ./test/e2e/e2e_test.go --ingress-class=higress --debug=true --test-area=clean
# run-higress-e2e-test-wasmplugin-setup starts to prepare ingress e2e tests.
.PHONY: run-higress-e2e-test-wasmplugin-setup
run-higress-e2e-test-wasmplugin-setup:
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
@echo -e "\n\033[36mWaiting higress-controller to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-controller --for=condition=Available
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
go test -v -tags conformance ./test/e2e/e2e_test.go -isWasmPluginTest=true -wasmPluginType=$(PLUGIN_TYPE) -wasmPluginName=$(PLUGIN_NAME) --ingress-class=higress --debug=true --test-area=setup
# run-higress-e2e-test-wasmplugin starts to run ingress e2e tests.
.PHONY: run-higress-e2e-test-wasmplugin
run-higress-e2e-test-wasmplugin:
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
@@ -288,4 +345,24 @@ run-higress-e2e-test-wasmplugin:
kubectl wait --timeout=10m -n higress-system deployment/higress-controller --for=condition=Available
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
go test -v -tags conformance ./test/e2e/e2e_test.go -isWasmPluginTest=true -wasmPluginType=$(PLUGIN_TYPE) -wasmPluginName=$(PLUGIN_NAME) --ingress-class=higress --debug=true
go test -v -tags conformance ./test/e2e/e2e_test.go -isWasmPluginTest=true -wasmPluginType=$(PLUGIN_TYPE) -wasmPluginName=$(PLUGIN_NAME) --ingress-class=higress --debug=true --test-area=all
# run-higress-e2e-test-wasmplugin-run starts to run ingress e2e conformance tests.
.PHONY: run-higress-e2e-test-wasmplugin-run
run-higress-e2e-test-wasmplugin-run:
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
@echo -e "\n\033[36mWaiting higress-controller to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-controller --for=condition=Available
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
go test -v -tags conformance ./test/e2e/e2e_test.go -isWasmPluginTest=true -wasmPluginType=$(PLUGIN_TYPE) -wasmPluginName=$(PLUGIN_NAME) --ingress-class=higress --debug=true --test-area=run
# run-higress-e2e-test-wasmplugin-clean starts to clean ingress e2e tests.
.PHONY: run-higress-e2e-test-wasmplugin-clean
run-higress-e2e-test-wasmplugin-clean:
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
@echo -e "\n\033[36mWaiting higress-controller to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-controller --for=condition=Available
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
go test -v -tags conformance ./test/e2e/e2e_test.go -isWasmPluginTest=true -wasmPluginType=$(PLUGIN_TYPE) -wasmPluginName=$(PLUGIN_NAME) --ingress-class=higress --debug=true --test-area=clean

View File

@@ -1 +1 @@
v1.3.3
v1.3.4-rc.2

View File

@@ -2,7 +2,11 @@ codecov:
require_ci_to_pass: yes
coverage:
status:
patch: no
patch:
default:
target: 50%
threshold: 0%
if_ci_failed: error # success, failure, error, ignore
project:
default:
target: auto
@@ -17,4 +21,4 @@ ignore:
comment:
layout: "reach,diff,flags,tree"
behavior: default
require_changes: no
require_changes: no

View File

@@ -0,0 +1,49 @@
diff -Naur envoy/source/common/router/BUILD envoy-new/source/common/router/BUILD
--- envoy/source/common/router/BUILD 2024-01-10 20:10:14.505600746 +0800
+++ envoy-new/source/common/router/BUILD 2024-01-10 20:07:25.960379955 +0800
@@ -212,6 +212,7 @@
"//envoy/router:rds_interface",
"//envoy/router:scopes_interface",
"//envoy/thread_local:thread_local_interface",
+ "//source/common/http:header_utility_lib",
"@envoy_api//envoy/config/route/v3:pkg_cc_proto",
"@envoy_api//envoy/extensions/filters/network/http_connection_manager/v3:pkg_cc_proto",
],
diff -Naur envoy/source/common/router/scoped_config_impl.cc envoy-new/source/common/router/scoped_config_impl.cc
--- envoy/source/common/router/scoped_config_impl.cc 2024-01-10 20:10:14.529600924 +0800
+++ envoy-new/source/common/router/scoped_config_impl.cc 2024-01-10 20:09:50.161422411 +0800
@@ -3,6 +3,8 @@
#include "envoy/config/route/v3/scoped_route.pb.h"
#include "envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.pb.h"
+#include "source/common/http/header_utility.h"
+
namespace Envoy {
namespace Router {
@@ -74,18 +76,20 @@
HostValueExtractorImpl::computeFragment(const Http::HeaderMap& headers,
const StreamInfo::StreamInfo&,
ReComputeCbPtr& recompute) const {
- auto fragment = computeFragment(headers);
auto host = static_cast<const Http::RequestHeaderMap&>(headers).getHostValue();
+ auto port_start = Http::HeaderUtility::getPortStart(host);
+ if (port_start != absl::string_view::npos) {
+ host = host.substr(0, port_start);
+ }
*recompute = [this, host, recompute]() mutable -> std::unique_ptr<ScopeKeyFragmentBase> {
return reComputeHelper(std::string(host), recompute, 0);
};
- return fragment;
+ return std::make_unique<StringKeyFragment>(host);
}
std::unique_ptr<ScopeKeyFragmentBase>
-HostValueExtractorImpl::computeFragment(const Http::HeaderMap& headers) const {
- return std::make_unique<StringKeyFragment>(
- static_cast<const Http::RequestHeaderMap&>(headers).getHostValue());
+HostValueExtractorImpl::computeFragment(const Http::HeaderMap&) const {
+ return nullptr;
}
std::unique_ptr<ScopeKeyFragmentBase>

View File

@@ -0,0 +1,65 @@
diff -Naur envoy/source/common/router/scoped_config_impl.cc envoy-new/source/common/router/scoped_config_impl.cc
--- envoy/source/common/router/scoped_config_impl.cc 2024-01-11 16:23:55.407881263 +0800
+++ envoy-new/source/common/router/scoped_config_impl.cc 2024-01-11 16:23:42.311786814 +0800
@@ -53,21 +53,26 @@
}
std::unique_ptr<ScopeKeyFragmentBase>
-HostValueExtractorImpl::reComputeHelper(const std::string& host, ReComputeCbPtr& next_recompute,
+HostValueExtractorImpl::reComputeHelper(const std::string& host,
+ ReComputeCbWeakPtr& weak_next_recompute,
uint32_t recompute_seq) const {
if (recompute_seq == max_recompute_num_) {
ENVOY_LOG_MISC(warn,
"recompute host fragment failed, maximum number of recalculations exceeded");
return nullptr;
}
+ auto next_recompute = weak_next_recompute.lock();
+ if (!next_recompute) {
+ return nullptr;
+ }
if (host == "*") {
*next_recompute = nullptr;
return nullptr;
}
auto masked_host = maskFirstDNSLabel(host);
*next_recompute = [this, masked_host, recompute_seq,
- next_recompute]() mutable -> std::unique_ptr<ScopeKeyFragmentBase> {
- return reComputeHelper(masked_host, next_recompute, recompute_seq + 1);
+ weak_next_recompute]() mutable -> std::unique_ptr<ScopeKeyFragmentBase> {
+ return reComputeHelper(masked_host, weak_next_recompute, recompute_seq + 1);
};
return std::make_unique<StringKeyFragment>(masked_host);
}
@@ -81,8 +86,9 @@
if (port_start != absl::string_view::npos) {
host = host.substr(0, port_start);
}
- *recompute = [this, host, recompute]() mutable -> std::unique_ptr<ScopeKeyFragmentBase> {
- return reComputeHelper(std::string(host), recompute, 0);
+ *recompute = [this, host, weak_recompute = ReComputeCbWeakPtr(recompute)]() mutable
+ -> std::unique_ptr<ScopeKeyFragmentBase> {
+ return reComputeHelper(std::string(host), weak_recompute, 0);
};
return std::make_unique<StringKeyFragment>(host);
}
diff -Naur envoy/source/common/router/scoped_config_impl.h envoy-new/source/common/router/scoped_config_impl.h
--- envoy/source/common/router/scoped_config_impl.h 2024-01-11 16:23:55.407881263 +0800
+++ envoy-new/source/common/router/scoped_config_impl.h 2024-01-11 16:23:42.311786814 +0800
@@ -25,6 +25,7 @@
#if defined(ALIMESH)
using ReComputeCb = std::function<std::unique_ptr<ScopeKeyFragmentBase>()>;
using ReComputeCbPtr = std::shared_ptr<ReComputeCb>;
+using ReComputeCbWeakPtr = std::weak_ptr<ReComputeCb>;
#endif
/**
@@ -83,7 +84,7 @@
private:
std::unique_ptr<ScopeKeyFragmentBase> reComputeHelper(const std::string& host,
- ReComputeCbPtr& next_recompute,
+ ReComputeCbWeakPtr& weak_next_recompute,
uint32_t recompute_seq) const;
static constexpr uint32_t DefaultMaxRecomputeNum = 100;

View File

File diff suppressed because it is too large Load Diff

View File

File diff suppressed because it is too large Load Diff

View File

File diff suppressed because it is too large Load Diff

2
go.mod
View File

@@ -15,7 +15,7 @@ replace github.com/imdario/mergo => github.com/imdario/mergo v0.3.5
require (
github.com/AlecAivazis/survey/v2 v2.3.6
github.com/agiledragon/gomonkey/v2 v2.9.0
github.com/agiledragon/gomonkey/v2 v2.11.0
github.com/avast/retry-go/v4 v4.3.4
github.com/compose-spec/compose-go v1.8.2
github.com/docker/cli v20.10.20+incompatible

4
go.sum
View File

@@ -160,8 +160,8 @@ github.com/VividCortex/ewma v1.1.1/go.mod h1:2Tkkvm3sRDVXaiyucHiACn4cqf7DpdyLvmx
github.com/VividCortex/gohistogram v1.0.0/go.mod h1:Pf5mBqqDxYaXu3hDrrU+w6nw50o/4+TcAqDqk/vUH7g=
github.com/afex/hystrix-go v0.0.0-20180502004556-fa1af6a1f4f5/go.mod h1:SkGFH1ia65gfNATL8TAiHDNxPzPdmEL5uirI2Uyuz6c=
github.com/agext/levenshtein v1.2.1/go.mod h1:JEDfjyjHDjOF/1e4FlBE/PkbqA9OfWu2ki2W0IB5558=
github.com/agiledragon/gomonkey/v2 v2.9.0 h1:PDiKKybR596O6FHW+RVSG0Z7uGCBNbmbUXh3uCNQ7Hc=
github.com/agiledragon/gomonkey/v2 v2.9.0/go.mod h1:ap1AmDzcVOAz1YpeJ3TCzIgstoaWLA6jbbgxfB4w2iY=
github.com/agiledragon/gomonkey/v2 v2.11.0 h1:5oxSgA+tC1xuGsrIorR+sYiziYltmJyEZ9qA25b6l5U=
github.com/agiledragon/gomonkey/v2 v2.11.0/go.mod h1:ap1AmDzcVOAz1YpeJ3TCzIgstoaWLA6jbbgxfB4w2iY=
github.com/agl/ed25519 v0.0.0-20170116200512-5312a6153412/go.mod h1:WPjqKcmVOxf0XSf3YxCJs6N6AOSrOx3obionmG7T0y0=
github.com/ahmetb/gen-crd-api-reference-docs v0.3.0/go.mod h1:TdjdkYhlOifCQWPs1UdTma97kQQMozf5h26hTuG70u8=
github.com/alecthomas/jsonschema v0.0.0-20180308105923-f2c93856175a/go.mod h1:qpebaTNSsyUn5rPSJMsfqEtDw71TTggXM6stUDI16HA=

View File

@@ -1,5 +1,5 @@
apiVersion: v2
appVersion: 1.3.3
appVersion: 1.3.4-rc.2
description: Helm chart for deploying higress gateways
icon: https://higress.io/img/higress_logo_small.png
home: http://higress.io/
@@ -10,4 +10,4 @@ name: higress-core
sources:
- http://github.com/alibaba/higress
type: application
version: 1.3.3
version: 1.3.4-rc.2

View File

@@ -154,6 +154,11 @@ spec:
type: array
httpPath:
type: string
paramFromEntireBody:
properties:
paramType:
type: string
type: object
params:
items:
properties:

View File

@@ -70,6 +70,10 @@ spec:
periodSeconds: 3
timeoutSeconds: 5
env:
- name: DEFAULT_UPSTREAM_CONCURRENCY_THRESHOLD
value: "{{ .Values.global.defaultUpstreamConcurrencyThreshold }}"
- name: ISTIO_GPRC_MAXRECVMSGSIZE
value: "{{ .Values.global.xdsMaxRecvMsgSize }}"
- name: ENBALE_SCOPED_RDS
value: "{{ .Values.global.enableSRDS }}"
- name: ON_DEMAND_RDS

View File

@@ -175,15 +175,15 @@ spec:
protocol: TCP
{{- end }}
readinessProbe:
failureThreshold: 30
failureThreshold: {{ .Values.gateway.readinessFailureThreshold }}
httpGet:
path: /healthz/ready
port: 15021
scheme: HTTP
initialDelaySeconds: 1
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 3
initialDelaySeconds: {{ .Values.gateway.readinessInitialDelaySeconds }}
periodSeconds: {{ .Values.gateway.readinessPeriodSeconds }}
successThreshold: {{ .Values.gateway.readinessSuccessThreshold }}
timeoutSeconds: {{ .Values.gateway.readinessTimeoutSeconds }}
{{- if not (or .Values.global.local .Values.global.kind) }}
resources:
{{- toYaml .Values.gateway.resources | nindent 12 }}

View File

@@ -1,8 +1,10 @@
revision: ""
global:
enableSRDS: false
onDemandRDS: true
hostRDSMergeSubset: true
xdsMaxRecvMsgSize: 104857600
defaultUpstreamConcurrencyThreshold: 10000
enableSRDS: true
onDemandRDS: false
hostRDSMergeSubset: false
onlyPushRouteCluster: true
# IngressClass filters which ingress resources the higress controller watches.
# The default ingress class is higress.
@@ -13,7 +15,7 @@ global:
# resources in the k8s cluster.
ingressClass: "higress"
watchNamespace: ""
disableAlpnH2: true
disableAlpnH2: false
enableStatus: true
# whether to use autoscaling/v2 template for HPA settings
# for internal usage only, not to be configured by users.
@@ -151,12 +153,18 @@ global:
# The number of successive failed probes before indicating readiness failure.
readinessFailureThreshold: 30
# The number of successive successed probes before indicating readiness success.
readinessSuccessThreshold: 30
# The initial delay for readiness probes in seconds.
readinessInitialDelaySeconds: 1
# The period between readiness probes.
readinessPeriodSeconds: 2
# The readiness timeout seconds
readinessTimeoutSeconds: 3
# Resources for the sidecar.
resources:
requests:
@@ -373,6 +381,21 @@ gateway:
replicas: 2
image: gateway
# The number of successive failed probes before indicating readiness failure.
readinessFailureThreshold: 30
# The number of successive successed probes before indicating readiness success.
readinessSuccessThreshold: 1
# The initial delay for readiness probes in seconds.
readinessInitialDelaySeconds: 1
# The period between readiness probes.
readinessPeriodSeconds: 2
# The readiness timeout seconds
readinessTimeoutSeconds: 3
hub: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress
tag: ""
# revision declares which revision this gateway is a part of

View File

@@ -1,9 +1,9 @@
dependencies:
- name: higress-core
repository: file://../core
version: 1.3.3
version: 1.3.4-rc.2
- name: higress-console
repository: https://higress.io/helm-charts/
version: 1.3.1
digest: sha256:585666df5da403450c5e586a71388bc0d029354b1100b20a50616f56711fa171
generated: "2024-01-08T21:40:10.446936+08:00"
version: 1.3.2
digest: sha256:aa10fa4ae01c7e7f8297db5fe6f4752aa6526e2411d46c93abddbb4e7e84d36d
generated: "2024-02-01T19:33:51.268079+08:00"

View File

@@ -1,5 +1,5 @@
apiVersion: v2
appVersion: 1.3.3
appVersion: 1.3.4-rc.2
description: Helm chart for deploying Higress gateways
icon: https://higress.io/img/higress_logo_small.png
home: http://higress.io/
@@ -12,9 +12,9 @@ sources:
dependencies:
- name: higress-core
repository: "file://../core"
version: 1.3.3
version: 1.3.4-rc.2
- name: higress-console
repository: "https://higress.io/helm-charts/"
version: 1.3.1
version: 1.3.2
type: application
version: 1.3.3
version: 1.3.4-rc.2

View File

@@ -0,0 +1,373 @@
diff -Naur istio/pilot/pkg/model/push_context.go istio-new/pilot/pkg/model/push_context.go
--- istio/pilot/pkg/model/push_context.go 2024-01-15 20:46:45.000000000 +0800
+++ istio-new/pilot/pkg/model/push_context.go 2024-01-15 19:20:45.000000000 +0800
@@ -96,6 +96,9 @@
publicByGateway map[string][]config.Config
// root vs namespace/name ->delegate vs virtualservice gvk/namespace/name
delegates map[ConfigKey][]ConfigKey
+ // Added by ingress
+ byHost map[string][]config.Config
+ // End added by ingress
}
func newVirtualServiceIndex() virtualServiceIndex {
@@ -104,6 +107,9 @@
privateByNamespaceAndGateway: map[string]map[string][]config.Config{},
exportedToNamespaceByGateway: map[string]map[string][]config.Config{},
delegates: map[ConfigKey][]ConfigKey{},
+ // Added by ingress
+ byHost: map[string][]config.Config{},
+ // End added by ingress
}
}
@@ -857,6 +863,13 @@
return res
}
+// Added by ingress
+func (ps *PushContext) VirtualServicesForHost(proxy *Proxy, host string) []config.Config {
+ return ps.virtualServiceIndex.byHost[host]
+}
+
+// End added by ingress
+
// DelegateVirtualServicesConfigKey lists all the delegate virtual services configkeys associated with the provided virtual services
func (ps *PushContext) DelegateVirtualServicesConfigKey(vses []config.Config) []ConfigKey {
var out []ConfigKey
@@ -1468,6 +1481,11 @@
for _, virtualService := range vservices {
ns := virtualService.Namespace
rule := virtualService.Spec.(*networking.VirtualService)
+ // Added by ingress
+ for _, host := range rule.Hosts {
+ ps.virtualServiceIndex.byHost[host] = append(ps.virtualServiceIndex.byHost[host], virtualService)
+ }
+ // End added by ingress
gwNames := getGatewayNames(rule)
if len(rule.ExportTo) == 0 {
// No exportTo in virtualService. Use the global default
diff -Naur istio/pilot/pkg/networking/core/v1alpha3/gateway.go istio-new/pilot/pkg/networking/core/v1alpha3/gateway.go
--- istio/pilot/pkg/networking/core/v1alpha3/gateway.go 2024-01-15 20:46:45.000000000 +0800
+++ istio-new/pilot/pkg/networking/core/v1alpha3/gateway.go 2024-01-15 20:04:05.000000000 +0800
@@ -28,6 +28,7 @@
route "github.com/envoyproxy/go-control-plane/envoy/config/route/v3"
hcm "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/network/http_connection_manager/v3"
tls "github.com/envoyproxy/go-control-plane/envoy/extensions/transport_sockets/tls/v3"
+ discovery "github.com/envoyproxy/go-control-plane/envoy/service/discovery/v3"
"github.com/hashicorp/go-multierror"
meshconfig "istio.io/api/mesh/v1alpha1"
@@ -35,6 +36,7 @@
"istio.io/istio/pilot/pkg/features"
"istio.io/istio/pilot/pkg/model"
istionetworking "istio.io/istio/pilot/pkg/networking"
+ "istio.io/istio/pilot/pkg/networking/core/v1alpha3/envoyfilter"
"istio.io/istio/pilot/pkg/networking/core/v1alpha3/extension"
"istio.io/istio/pilot/pkg/networking/core/v1alpha3/mseingress"
istio_route "istio.io/istio/pilot/pkg/networking/core/v1alpha3/route"
@@ -423,8 +425,15 @@
gatewayName string
}
-func (configgen *ConfigGeneratorImpl) buildHostRDSConfig(node *model.Proxy, push *model.PushContext,
- routeName string) *route.RouteConfiguration {
+func (configgen *ConfigGeneratorImpl) buildHostRDSConfig(
+ node *model.Proxy,
+ req *model.PushRequest,
+ routeName string,
+ vsCache map[int][]virtualServiceContext,
+ efw *model.EnvoyFilterWrapper,
+ efKeys []string,
+) (*discovery.Resource, bool) {
+ push := req.Push
var (
hostRDSPort string
hostRDSHost string
@@ -432,7 +441,7 @@
portAndHost := strings.SplitN(strings.TrimPrefix(routeName, constants.HigressHostRDSNamePrefix), ".", 2)
if len(portAndHost) != 2 {
log.Errorf("Invalid route %s when using Higress hostRDS", routeName)
- return nil
+ return nil, false
}
hostRDSPort = portAndHost[0]
hostRDSHost = portAndHost[1]
@@ -441,10 +450,24 @@
rdsPort, err := strconv.Atoi(hostRDSPort)
if err != nil {
log.Errorf("Invalid port %s of route %s when using Higress hostRDS", hostRDSPort, routeName)
- return nil
+ return nil, false
+ }
+
+ routeCache := &istio_route.Cache{
+ RouteName: routeName,
+ ProxyVersion: node.Metadata.IstioVersion,
+ ListenerPort: rdsPort,
+ // Use same host vs to cache, although the cache can be cleared when the port is different, this can be accepted
+ VirtualServices: push.VirtualServicesForHost(node, hostRDSHost),
+ EnvoyFilterKeys: efKeys,
+ }
+
+ resource, exist := configgen.Cache.Get(routeCache)
+ if exist {
+ return resource, true
}
+
listenerPort := uint32(rdsPort)
- globalHTTPFilters := mseingress.ExtractGlobalHTTPFilters(node, push)
isH3DiscoveryNeeded := false
@@ -457,9 +480,9 @@
break
}
}
-
gatewayRoutes := make(map[string]map[string][]*route.Route)
gatewayVirtualServices := make(map[string][]config.Config)
+ var listenerVirtualServices []virtualServiceContext
var selectedVirtualServices []virtualServiceContext
var vHost *route.VirtualHost
serverIterator := func(mergedServers map[model.ServerPort]*model.MergedServers) {
@@ -478,31 +501,8 @@
gatewayVirtualServices[gatewayName] = virtualServices
}
for _, virtualService := range virtualServices {
- hostMatch := false
- var selectHost string
- virtualServiceHosts := host.NewNames(virtualService.Spec.(*networking.VirtualService).Hosts)
- for _, hostname := range virtualServiceHosts {
- // exact match
- if hostname == host.Name(hostRDSHost) {
- hostMatch = true
- selectHost = hostRDSHost
- break
- }
- if features.HostRDSMergeSubset {
- // subset match
- if host.Name(hostRDSHost).SubsetOf(hostname) {
- hostMatch = true
- selectHost = string(hostname)
- }
- }
- }
- if !hostMatch {
- continue
- }
- copiedVS := virtualService.DeepCopy()
- copiedVS.Spec.(*networking.VirtualService).Hosts = []string{selectHost}
- selectedVirtualServices = append(selectedVirtualServices, virtualServiceContext{
- virtualService: copiedVS,
+ listenerVirtualServices = append(listenerVirtualServices, virtualServiceContext{
+ virtualService: virtualService,
server: server,
gatewayName: gatewayName,
})
@@ -510,15 +510,63 @@
}
}
}
- serverIterator(merged.MergedServers)
- serverIterator(merged.MergedQUICTransportServers)
- // Sort by subset
- // before: ["*.abc.com", "*.com", "www.abc.com"]
- // after: ["www.abc.com", "*.abc.com", "*.com"]
- sort.SliceStable(selectedVirtualServices, func(i, j int) bool {
- return host.Name(selectedVirtualServices[i].virtualService.Spec.(*networking.VirtualService).Hosts[0]).SubsetOf(
- host.Name(selectedVirtualServices[j].virtualService.Spec.(*networking.VirtualService).Hosts[0]))
- })
+ var vsExists bool
+ if listenerVirtualServices, vsExists = vsCache[rdsPort]; !vsExists {
+ serverIterator(merged.MergedServers)
+ serverIterator(merged.MergedQUICTransportServers)
+ vsCache[rdsPort] = listenerVirtualServices
+ }
+ for _, vsCtx := range listenerVirtualServices {
+ virtualService := vsCtx.virtualService
+ hostMatch := false
+ var selectHost string
+ for _, hostname := range virtualService.Spec.(*networking.VirtualService).Hosts {
+ // exact match
+ if hostname == hostRDSHost {
+ hostMatch = true
+ selectHost = hostRDSHost
+ break
+ }
+ if features.HostRDSMergeSubset {
+ // subset match
+ if host.Name(hostRDSHost).SubsetOf(host.Name(hostname)) {
+ hostMatch = true
+ selectHost = hostname
+ }
+ }
+ }
+ if !hostMatch {
+ continue
+ }
+ if len(virtualService.Spec.(*networking.VirtualService).Hosts) > 1 {
+ copiedVS := &networking.VirtualService{}
+ copiedVS = virtualService.Spec.(*networking.VirtualService)
+ copiedVS.Hosts = []string{selectHost}
+ selectedVirtualServices = append(selectedVirtualServices, virtualServiceContext{
+ virtualService: config.Config{
+ Meta: virtualService.Meta,
+ Spec: copiedVS,
+ Status: virtualService.Status,
+ },
+ server: vsCtx.server,
+ gatewayName: vsCtx.gatewayName,
+ })
+ } else {
+ selectedVirtualServices = append(selectedVirtualServices, vsCtx)
+ }
+ }
+ if features.HostRDSMergeSubset {
+ // Sort by subset
+ // before: ["*.abc.com", "*.com", "www.abc.com"]
+ // after: ["www.abc.com", "*.abc.com", "*.com"]
+ sort.SliceStable(selectedVirtualServices, func(i, j int) bool {
+ return host.Name(selectedVirtualServices[i].virtualService.Spec.(*networking.VirtualService).Hosts[0]).SubsetOf(
+ host.Name(selectedVirtualServices[j].virtualService.Spec.(*networking.VirtualService).Hosts[0]))
+ })
+ }
+
+ globalHTTPFilters := mseingress.ExtractGlobalHTTPFilters(node, push)
+
port := int(listenerPort)
for _, ctx := range selectedVirtualServices {
virtualService := ctx.virtualService
@@ -605,25 +653,42 @@
ValidateClusters: proto.BoolFalse,
}
- return routeCfg
+ routeCfg = envoyfilter.ApplyRouteConfigurationPatches(networking.EnvoyFilter_GATEWAY, node, efw, routeCfg)
+ resource = &discovery.Resource{
+ Name: routeName,
+ Resource: util.MessageToAny(routeCfg),
+ }
+
+ if features.EnableRDSCaching {
+ configgen.Cache.Add(routeCache, req, resource)
+ }
+
+ return resource, false
}
// End added by ingress
-func (configgen *ConfigGeneratorImpl) buildGatewayHTTPRouteConfig(node *model.Proxy, push *model.PushContext,
- routeName string) *route.RouteConfiguration {
+// Modifed by ingress
+func (configgen *ConfigGeneratorImpl) buildGatewayHTTPRouteConfig(
+ node *model.Proxy,
+ req *model.PushRequest,
+ routeName string,
+ vsCache map[int][]virtualServiceContext,
+ efw *model.EnvoyFilterWrapper,
+ efKeys []string,
+) (*discovery.Resource, bool) {
if node.MergedGateway == nil {
log.Warnf("buildGatewayRoutes: no gateways for router %v", node.ID)
- return &route.RouteConfiguration{
- Name: routeName,
- VirtualHosts: []*route.VirtualHost{},
- ValidateClusters: proto.BoolFalse,
- }
+ return nil, false
}
-
// Added by ingress
+ push := req.Push
if strings.HasPrefix(routeName, constants.HigressHostRDSNamePrefix) {
- return configgen.buildHostRDSConfig(node, push, routeName)
+ resource, cacheHit := configgen.buildHostRDSConfig(node, req, routeName, vsCache, efw, efKeys)
+ if resource == nil {
+ return nil, false
+ }
+ return resource, cacheHit
}
// End added by ingress
@@ -636,7 +701,7 @@
// This can happen when a gateway has recently been deleted. Envoy will still request route
// information due to the draining of listeners, so we should not return an error.
- return nil
+ return nil, false
}
servers := merged.ServersByRouteName[routeName]
@@ -768,9 +833,16 @@
ValidateClusters: proto.BoolFalse,
}
- return routeCfg
+ routeCfg = envoyfilter.ApplyRouteConfigurationPatches(networking.EnvoyFilter_GATEWAY, node, efw, routeCfg)
+ resource := &discovery.Resource{
+ Name: routeName,
+ Resource: util.MessageToAny(routeCfg),
+ }
+ return resource, false
}
+// End modified by ingress
+
// hashRouteList returns a hash of a list of pointers
func hashRouteList(r []*route.Route) uint64 {
hash := md5.New()
diff -Naur istio/pilot/pkg/networking/core/v1alpha3/httproute.go istio-new/pilot/pkg/networking/core/v1alpha3/httproute.go
--- istio/pilot/pkg/networking/core/v1alpha3/httproute.go 2024-01-15 20:46:41.000000000 +0800
+++ istio-new/pilot/pkg/networking/core/v1alpha3/httproute.go 2024-01-15 10:29:09.000000000 +0800
@@ -78,17 +78,30 @@
routeConfigurations = append(routeConfigurations, rc)
}
case model.Router:
+ // Modified by ingress
+ vsCache := make(map[int][]virtualServiceContext)
+ envoyfilterKeys := efw.Keys()
for _, routeName := range routeNames {
- rc := configgen.buildGatewayHTTPRouteConfig(node, req.Push, routeName)
- if rc != nil {
- rc = envoyfilter.ApplyRouteConfigurationPatches(networking.EnvoyFilter_GATEWAY, node, efw, rc)
- resource := &discovery.Resource{
+ rc, cached := configgen.buildGatewayHTTPRouteConfig(node, req, routeName, vsCache, efw, envoyfilterKeys)
+ if cached && !features.EnableUnsafeAssertions {
+ hit++
+ } else {
+ miss++
+ }
+ if rc == nil {
+ emptyRoute := &route.RouteConfiguration{
+ Name: routeName,
+ VirtualHosts: []*route.VirtualHost{},
+ ValidateClusters: proto.BoolFalse,
+ }
+ rc = &discovery.Resource{
Name: routeName,
- Resource: util.MessageToAny(rc),
+ Resource: util.MessageToAny(emptyRoute),
}
- routeConfigurations = append(routeConfigurations, resource)
}
+ routeConfigurations = append(routeConfigurations, rc)
}
+ // End modified by ingress
}
if !features.EnableRDSCaching {
return routeConfigurations, model.DefaultXdsLogDetails
diff -Naur istio/pilot/pkg/xds/discovery.go istio-new/pilot/pkg/xds/discovery.go
--- istio/pilot/pkg/xds/discovery.go 2024-01-15 20:46:45.000000000 +0800
+++ istio-new/pilot/pkg/xds/discovery.go 2024-01-12 19:56:02.000000000 +0800
@@ -392,6 +392,9 @@
// ConfigUpdate implements ConfigUpdater interface, used to request pushes.
// It replaces the 'clear cache' from v1.
func (s *DiscoveryServer) ConfigUpdate(req *model.PushRequest) {
+ if req.Full {
+ log.Infof("full push happen, reason:%v", req.Reason)
+ }
inboundConfigUpdates.Increment()
s.InboundUpdates.Inc()
s.pushChannel <- req

View File

@@ -0,0 +1,60 @@
diff -Naur istio/pilot/cmd/pilot-agent/status/util/stats.go istio-new/pilot/cmd/pilot-agent/status/util/stats.go
--- istio/pilot/cmd/pilot-agent/status/util/stats.go 2024-02-01 10:20:13.000000000 +0800
+++ istio-new/pilot/cmd/pilot-agent/status/util/stats.go 2024-01-31 22:44:53.000000000 +0800
@@ -73,7 +73,7 @@
localHostAddr = "localhost"
}
- readinessURL := fmt.Sprintf("http://%s:%d/stats?usedonly&filter=%s", localHostAddr, adminPort, readyStatsRegex)
+ readinessURL := fmt.Sprintf("http://%s:%d/stats?usedonly", localHostAddr, adminPort)
stats, err := http.DoHTTPGetWithTimeout(readinessURL, readinessTimeout)
if err != nil {
return nil, false, err
@@ -105,7 +105,7 @@
localHostAddr = "localhost"
}
- stats, err := http.DoHTTPGet(fmt.Sprintf("http://%s:%d/stats?usedonly&filter=%s", localHostAddr, adminPort, updateStatsRegex))
+ stats, err := http.DoHTTPGet(fmt.Sprintf("http://%s:%d/stats?usedonly", localHostAddr, adminPort))
if err != nil {
return nil, err
}
diff -Naur istio/pilot/pkg/features/pilot.go istio-new/pilot/pkg/features/pilot.go
--- istio/pilot/pkg/features/pilot.go 2024-02-01 10:20:17.000000000 +0800
+++ istio-new/pilot/pkg/features/pilot.go 2024-02-01 10:16:18.000000000 +0800
@@ -575,6 +575,8 @@
"If enabled, each host in virtualservice will have an independent RDS, which is used with SRDS").Get()
OnDemandRDS = env.RegisterBoolVar("ON_DEMAND_RDS", false,
"If enabled, the on demand filter will be added to the HCM filters").Get()
+ DefaultUpstreamConcurrencyThreshold = env.RegisterIntVar("DEFAULT_UPSTREAM_CONCURRENCY_THRESHOLD", 1000000,
+ "The default threshold of max_requests/max_pending_requests/max_connections of circuit breaker").Get()
// End added by ingress
)
diff -Naur istio/pilot/pkg/networking/core/v1alpha3/cluster.go istio-new/pilot/pkg/networking/core/v1alpha3/cluster.go
--- istio/pilot/pkg/networking/core/v1alpha3/cluster.go 2024-02-01 10:20:17.000000000 +0800
+++ istio-new/pilot/pkg/networking/core/v1alpha3/cluster.go 2024-02-01 10:16:05.000000000 +0800
@@ -61,6 +61,7 @@
// getDefaultCircuitBreakerThresholds returns a copy of the default circuit breaker thresholds for the given traffic direction.
func getDefaultCircuitBreakerThresholds() *cluster.CircuitBreakers_Thresholds {
+ // Modified by ingress
return &cluster.CircuitBreakers_Thresholds{
// DefaultMaxRetries specifies the default for the Envoy circuit breaker parameter max_retries. This
// defines the maximum number of parallel retries a given Envoy will allow to the upstream cluster. Envoy defaults
@@ -68,11 +69,12 @@
// where multiple endpoints in a cluster are terminated. In these scenarios the circuit breaker can kick
// in before Pilot is able to deliver an updated endpoint list to Envoy, leading to client-facing 503s.
MaxRetries: &wrappers.UInt32Value{Value: math.MaxUint32},
- MaxRequests: &wrappers.UInt32Value{Value: math.MaxUint32},
- MaxConnections: &wrappers.UInt32Value{Value: math.MaxUint32},
- MaxPendingRequests: &wrappers.UInt32Value{Value: math.MaxUint32},
+ MaxRequests: &wrappers.UInt32Value{Value: uint32(features.DefaultUpstreamConcurrencyThreshold)},
+ MaxConnections: &wrappers.UInt32Value{Value: uint32(features.DefaultUpstreamConcurrencyThreshold)},
+ MaxPendingRequests: &wrappers.UInt32Value{Value: uint32(features.DefaultUpstreamConcurrencyThreshold)},
TrackRemaining: true,
}
+ // End modified by ingress
}
// BuildClusters returns the list of clusters for the given proxy. This is the CDS output

View File

@@ -0,0 +1,88 @@
diff -Naur istio/pilot/pkg/networking/core/v1alpha3/gateway.go istio-new/pilot/pkg/networking/core/v1alpha3/gateway.go
--- istio/pilot/pkg/networking/core/v1alpha3/gateway.go 2024-02-01 13:53:17.000000000 +0800
+++ istio-new/pilot/pkg/networking/core/v1alpha3/gateway.go 2024-02-01 13:52:11.000000000 +0800
@@ -501,6 +501,16 @@
gatewayVirtualServices[gatewayName] = virtualServices
}
for _, virtualService := range virtualServices {
+ virtualServiceHosts := host.NewNames(virtualService.Spec.(*networking.VirtualService).Hosts)
+ serverHosts := host.NamesForNamespace(server.Hosts, virtualService.Namespace)
+
+ // We have two cases here:
+ // 1. virtualService hosts are 1.foo.com, 2.foo.com, 3.foo.com and server hosts are ns/*.foo.com
+ // 2. virtualService hosts are *.foo.com, and server hosts are ns/1.foo.com, ns/2.foo.com, ns/3.foo.com
+ intersectingHosts := serverHosts.Intersection(virtualServiceHosts)
+ if len(intersectingHosts) == 0 {
+ continue
+ }
listenerVirtualServices = append(listenerVirtualServices, virtualServiceContext{
virtualService: virtualService,
server: server,
@@ -615,22 +625,24 @@
// check all hostname if is not exist with HttpsRedirect set to true
// create VirtualHost to redirect
- for _, hostname := range server.Hosts {
- if !server.GetTls().GetHttpsRedirect() {
- continue
- }
- if vHost != nil && host.Name(hostname) == host.Name(hostRDSHost) {
+ if server.GetTls().GetHttpsRedirect() {
+ if vHost != nil {
vHost.RequireTls = route.VirtualHost_ALL
- continue
+ } else {
+ vHost = &route.VirtualHost{
+ Name: util.DomainName(hostRDSHost, port),
+ Domains: buildGatewayVirtualHostDomains(hostRDSHost, port),
+ IncludeRequestAttemptCount: true,
+ RequireTls: route.VirtualHost_ALL,
+ }
}
- vHost = &route.VirtualHost{
- Name: util.DomainName(hostname, port),
- Domains: buildGatewayVirtualHostDomains(hostname, port),
- IncludeRequestAttemptCount: true,
- RequireTls: route.VirtualHost_ALL,
+ } else if vHost != nil {
+ mode := server.GetTls().GetMode()
+ if mode == networking.ServerTLSSettings_MUTUAL ||
+ mode == networking.ServerTLSSettings_ISTIO_MUTUAL {
+ vHost.AllowServerNames = append(vHost.AllowServerNames, server.Hosts...)
}
}
-
}
var virtualHosts []*route.VirtualHost
if vHost == nil {
@@ -642,6 +654,30 @@
Routes: []*route.Route{},
}}
} else {
+ sort.SliceStable(vHost.AllowServerNames, func(i, j int) bool {
+ hostI := vHost.AllowServerNames[i]
+ hostJ := vHost.AllowServerNames[j]
+ if host.Name(hostI).SubsetOf(host.Name(hostJ)) {
+ return true
+ }
+ return hostI < hostJ
+ })
+ var uniqueServerNames []string
+ hasAllCatch := false
+ for i, name := range vHost.AllowServerNames {
+ if name == "*" {
+ hasAllCatch = true
+ break
+ }
+ if i == 0 || vHost.AllowServerNames[i-1] != name {
+ uniqueServerNames = append(uniqueServerNames, name)
+ }
+ }
+ if hasAllCatch {
+ vHost.AllowServerNames = nil
+ } else {
+ vHost.AllowServerNames = uniqueServerNames
+ }
vHost.Routes = istio_route.CombineVHostRoutes(vHost.Routes)
virtualHosts = append(virtualHosts, vHost)
}

View File

@@ -0,0 +1,41 @@
diff -Naur istio/pilot/pkg/xds/discovery.go istio-new/pilot/pkg/xds/discovery.go
--- istio/pilot/pkg/xds/discovery.go 2024-02-02 16:26:49.000000000 +0800
+++ istio-new/pilot/pkg/xds/discovery.go 2024-02-02 15:38:53.000000000 +0800
@@ -18,6 +18,7 @@
"context"
"fmt"
"strconv"
+ "strings"
"sync"
"time"
@@ -41,6 +42,7 @@
"istio.io/istio/pilot/pkg/util/sets"
v3 "istio.io/istio/pilot/pkg/xds/v3"
"istio.io/istio/pkg/cluster"
+ "istio.io/istio/pkg/config/constants"
"istio.io/istio/pkg/security"
)
@@ -332,6 +334,21 @@
} else {
// Otherwise, just clear the updated configs
s.Cache.Clear(req.ConfigsUpdated)
+ //Added by ingress
+ trimKeyMap := make(map[model.ConfigKey]struct{})
+ for configKey := range req.ConfigsUpdated {
+ if strings.HasPrefix(configKey.Name, constants.IstioIngressGatewayName+"-") {
+ trimKeyMap[model.ConfigKey{
+ Kind: configKey.Kind,
+ Name: strings.TrimPrefix(configKey.Name, constants.IstioIngressGatewayName+"-"),
+ Namespace: configKey.Namespace,
+ }] = struct{}{}
+ }
+ }
+ if len(trimKeyMap) > 0 {
+ s.Cache.Clear(trimKeyMap)
+ }
+ //End added by ingress
}
}

View File

@@ -0,0 +1,21 @@
diff -Naur istio/pilot/cmd/pilot-agent/status/util/stats.go istio-new/pilot/cmd/pilot-agent/status/util/stats.go
--- istio/pilot/cmd/pilot-agent/status/util/stats.go 2024-02-04 18:48:18.000000000 +0800
+++ istio-new/pilot/cmd/pilot-agent/status/util/stats.go 2024-02-04 09:35:42.000000000 +0800
@@ -37,7 +37,7 @@
updateStatsRegex = "^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$"
)
-var readinessTimeout = time.Second * 3 // Default Readiness timeout. It is set the same in helm charts.
+var readinessTimeout = time.Second * 60 // Default Readiness timeout. It is set the same in helm charts.
type stat struct {
name string
@@ -105,7 +105,7 @@
localHostAddr = "localhost"
}
- stats, err := http.DoHTTPGet(fmt.Sprintf("http://%s:%d/stats?usedonly", localHostAddr, adminPort))
+ stats, err := http.DoHTTPGetWithTimeout(fmt.Sprintf("http://%s:%d/stats?usedonly", localHostAddr, adminPort), readinessTimeout)
if err != nil {
return nil, err
}

View File

@@ -584,7 +584,7 @@ func locateChart(cpOpts *action.ChartPathOptions, name string, settings *cli.Env
return fileAbsPath, nil
}
func ParseLatestVersion(repoUrl string, version string) (string, error) {
func ParseLatestVersion(repoUrl string, version string, devel bool) (string, error) {
cpOpts := &action.ChartPathOptions{
RepoURL: repoUrl,
@@ -632,7 +632,16 @@ func ParseLatestVersion(repoUrl string, version string) (string, error) {
// get higress helm chart latest version
if entries, ok := indexFile.Entries[RepoChartIndexYamlHigressIndex]; ok {
return entries[0].AppVersion, nil
if devel {
return entries[0].AppVersion, nil
}
if chatVersion, err := indexFile.Get(RepoChartIndexYamlHigressIndex, ""); err != nil {
return "", errors.New("can't find higress latest version")
} else {
return chatVersion.Version, nil
}
}
return "", errors.New("can't find higress latest version")

View File

@@ -52,6 +52,8 @@ type InstallArgs struct {
Set []string
// ManifestsPath is a path to a ManifestsPath and profiles directory in the local filesystem with a release tgz.
ManifestsPath string
// Devel if set true when version is latest, it will get latest version, otherwise it will get latest stable version
Devel bool
}
func (a *InstallArgs) String() string {
@@ -67,6 +69,7 @@ func addInstallFlags(cmd *cobra.Command, args *InstallArgs) {
cmd.PersistentFlags().StringSliceVarP(&args.InFilenames, "filename", "f", nil, filenameFlagHelpStr)
cmd.PersistentFlags().StringArrayVarP(&args.Set, "set", "s", nil, setFlagHelpStr)
cmd.PersistentFlags().StringVarP(&args.ManifestsPath, "manifests", "d", "", manifestsFlagHelpStr)
cmd.PersistentFlags().BoolVar(&args.Devel, "devel", false, "use development versions (alpha, beta, and release candidate releases), If version is set, this is ignored")
}
// --manifests is an alias for --set installPackagePath=
@@ -141,7 +144,7 @@ func install(writer io.Writer, iArgs *InstallArgs) error {
return err
}
err = installManifests(profile, writer)
err = installManifests(profile, writer, iArgs.Devel)
if err != nil {
return fmt.Errorf("failed to install manifests: %v", err)
}
@@ -192,8 +195,8 @@ func promptProfileName(writer io.Writer) string {
}
func installManifests(profile *helm.Profile, writer io.Writer) error {
installer, err := installer.NewInstaller(profile, writer, false)
func installManifests(profile *helm.Profile, writer io.Writer, devel bool) error {
installer, err := installer.NewInstaller(profile, writer, false, devel, installer.InstallInstallerMode)
if err != nil {
return err
}

View File

@@ -52,6 +52,8 @@ type ComponentOptions struct {
Quiet bool
// Capabilities
Capabilities *chartutil.Capabilities
// devel
Devel bool
}
type ComponentOption func(*ComponentOptions)
@@ -98,6 +100,12 @@ func WithQuiet() ComponentOption {
}
}
func WithDevel(devel bool) ComponentOption {
return func(opts *ComponentOptions) {
opts.Devel = devel
}
}
func renderComponentManifest(spec any, renderer helm.Renderer, addOn bool, name ComponentName, namespace string) (string, error) {
var valsBytes []byte
var valsYaml string

View File

@@ -52,7 +52,7 @@ func (h *HigressComponent) Run() error {
// Parse latest version
if h.opts.Version == helm.RepoLatestVersion {
latestVersion, err := helm.ParseLatestVersion(h.opts.RepoURL, h.opts.Version)
latestVersion, err := helm.ParseLatestVersion(h.opts.RepoURL, h.opts.Version, h.opts.Devel)
if err != nil {
return err
}

View File

@@ -28,6 +28,8 @@ import (
"k8s.io/client-go/util/homedir"
)
type InstallerMode int32
const (
HgctlHomeDirPath = ".hgctl"
StandaloneInstalledPath = "higress-standalone"
@@ -37,20 +39,26 @@ const (
DefaultIstioNamespace = "istio-system"
)
const (
InstallInstallerMode InstallerMode = iota
UpgradeInstallerMode
UninstallInstallerMode
)
type Installer interface {
Install() error
UnInstall() error
Upgrade() error
}
func NewInstaller(profile *helm.Profile, writer io.Writer, quiet bool) (Installer, error) {
func NewInstaller(profile *helm.Profile, writer io.Writer, quiet bool, devel bool, installerMode InstallerMode) (Installer, error) {
switch profile.Global.Install {
case helm.InstallK8s, helm.InstallLocalK8s:
cliClient, err := kubernetes.NewCLIClient(options.DefaultConfigFlags.ToRawKubeConfigLoader())
if err != nil {
return nil, fmt.Errorf("failed to build kubernetes client: %w", err)
}
installer, err := NewK8sInstaller(profile, cliClient, writer, quiet)
installer, err := NewK8sInstaller(profile, cliClient, writer, quiet, devel, installerMode)
return installer, err
case helm.InstallLocalDocker:
installer, err := NewDockerInstaller(profile, writer, quiet)

View File

@@ -254,7 +254,7 @@ func (o *K8sInstaller) isNamespacedObject(obj *object.K8sObject) bool {
return false
}
func NewK8sInstaller(profile *helm.Profile, cli kubernetes.CLIClient, writer io.Writer, quiet bool) (*K8sInstaller, error) {
func NewK8sInstaller(profile *helm.Profile, cli kubernetes.CLIClient, writer io.Writer, quiet bool, devel bool, installerMode InstallerMode) (*K8sInstaller, error) {
if profile == nil {
return nil, errors.New("install profile is empty")
}
@@ -267,14 +267,20 @@ func NewK8sInstaller(profile *helm.Profile, cli kubernetes.CLIClient, writer io.
}
fmt.Fprintf(writer, "%s\n", capabilities.KubeVersion.Version)
// initialize components
higressVersion := profile.Charts.Higress.Version
if installerMode == UninstallInstallerMode {
// uninstall
higressVersion = profile.HigressVersion
}
components := make(map[ComponentName]Component)
opts := []ComponentOption{
WithComponentNamespace(profile.Global.Namespace),
WithComponentChartPath(profile.InstallPackagePath),
WithComponentVersion(profile.Charts.Higress.Version),
WithComponentVersion(higressVersion),
WithComponentRepoURL(profile.Charts.Higress.Url),
WithComponentChartName(profile.Charts.Higress.Name),
WithComponentCapabilities(capabilities),
WithDevel(devel),
}
if quiet {
opts = append(opts, WithQuiet())

View File

@@ -37,6 +37,8 @@ type ManifestArgs struct {
Set []string
// ManifestsPath is a path to a ManifestsPath and profiles directory in the local filesystem with a release tgz.
ManifestsPath string
// Devel if set true when version is latest, it will get latest version, otherwise it will get latest stable version
Devel bool
}
func (a *ManifestArgs) String() string {
@@ -70,6 +72,7 @@ func addManifestFlags(cmd *cobra.Command, args *ManifestArgs) {
cmd.PersistentFlags().StringSliceVarP(&args.InFilenames, "filename", "f", nil, filenameFlagHelpStr)
cmd.PersistentFlags().StringArrayVarP(&args.Set, "set", "s", nil, setFlagHelpStr)
cmd.PersistentFlags().StringVarP(&args.ManifestsPath, "manifests", "d", "", manifestsFlagHelpStr)
cmd.PersistentFlags().BoolVar(&args.Devel, "devel", false, "use development versions (alpha, beta, and release candidate releases), If version is set, this is ignored")
}
// newManifestGenerateCmd generates a higress install manifest and applies it to a cluster
@@ -113,20 +116,20 @@ func generate(writer io.Writer, iArgs *ManifestArgs) error {
return err
}
err = genManifests(profile, writer)
err = genManifests(profile, writer, iArgs.Devel)
if err != nil {
return fmt.Errorf("failed to install manifests: %v", err)
}
return nil
}
func genManifests(profile *helm.Profile, writer io.Writer) error {
func genManifests(profile *helm.Profile, writer io.Writer, devel bool) error {
cliClient, err := kubernetes.NewCLIClient(options.DefaultConfigFlags.ToRawKubeConfigLoader())
if err != nil {
return fmt.Errorf("failed to build kubernetes client: %w", err)
}
op, err := installer.NewK8sInstaller(profile, cliClient, writer, true)
op, err := installer.NewK8sInstaller(profile, cliClient, writer, true, devel, installer.InstallInstallerMode)
if err != nil {
return err
}

View File

@@ -126,7 +126,7 @@ func promptUninstall(writer io.Writer) bool {
}
func uninstallManifests(profile *helm.Profile, writer io.Writer, uiArgs *uninstallArgs) error {
installer, err := installer.NewInstaller(profile, writer, false)
installer, err := installer.NewInstaller(profile, writer, false, false, installer.UninstallInstallerMode)
if err != nil {
return err
}

View File

@@ -37,6 +37,7 @@ func addUpgradeFlags(cmd *cobra.Command, args *upgradeArgs) {
cmd.PersistentFlags().StringSliceVarP(&args.InFilenames, "filename", "f", nil, filenameFlagHelpStr)
cmd.PersistentFlags().StringArrayVarP(&args.Set, "set", "s", nil, setFlagHelpStr)
cmd.PersistentFlags().StringVarP(&args.ManifestsPath, "manifests", "d", "", manifestsFlagHelpStr)
cmd.PersistentFlags().BoolVar(&args.Devel, "devel", false, "use development versions (alpha, beta, and release candidate releases), If version is set, this is ignored")
}
// newUpgradeCmd upgrades Istio control plane in-place with eligibility checks.
@@ -91,7 +92,7 @@ func upgrade(writer io.Writer, iArgs *InstallArgs) error {
return nil
}
err = upgradeManifests(profile, writer)
err = upgradeManifests(profile, writer, iArgs.Devel)
if err != nil {
return err
}
@@ -120,8 +121,8 @@ func promptUpgrade(writer io.Writer) bool {
}
}
func upgradeManifests(profile *helm.Profile, writer io.Writer) error {
installer, err := installer.NewInstaller(profile, writer, false)
func upgradeManifests(profile *helm.Profile, writer io.Writer, devel bool) error {
installer, err := installer.NewInstaller(profile, writer, false, devel, installer.UpgradeInstallerMode)
if err != nil {
return err
}

View File

@@ -19,6 +19,7 @@ import (
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"sync"
@@ -474,9 +475,6 @@ func (m *IngressConfig) convertVirtualService(configs []common.WrapperConfig) []
gateways := []string{m.namespace + "/" +
common.CreateConvertedName(m.clusterId, cleanHost),
common.CreateConvertedName(constants.IstioIngressGatewayName, cleanHost)}
if host != "*" {
gateways = append(gateways, m.namespace+"/"+common.CreateConvertedName(m.clusterId, common.CleanHost("*")))
}
wrapperVS, exist := convertOptions.VirtualServices[host]
if !exist {
@@ -673,6 +671,18 @@ func (m *IngressConfig) convertDestinationRule(configs []common.WrapperConfig) [
out := make([]config.Config, 0, len(destinationRules))
for _, dr := range destinationRules {
sort.SliceStable(dr.DestinationRule.TrafficPolicy.PortLevelSettings, func(i, j int) bool {
portI := dr.DestinationRule.TrafficPolicy.PortLevelSettings[i].Port
portJ := dr.DestinationRule.TrafficPolicy.PortLevelSettings[j].Port
if portI == nil && portJ == nil {
return true
} else if portI == nil {
return true
} else if portJ == nil {
return false
}
return portI.Number < portJ.Number
})
drName := util.CreateDestinationRuleName(m.clusterId, dr.ServiceKey.Namespace, dr.ServiceKey.Name)
out = append(out, config.Config{
Meta: config.Meta{

View File

@@ -314,9 +314,6 @@ func (m *KIngressConfig) convertVirtualService(configs []common.WrapperConfig) [
gateways := []string{m.namespace + "/" +
common.CreateConvertedName(m.clusterId, cleanHost),
common.CreateConvertedName(constants.IstioIngressGatewayName, cleanHost)}
if host != "*" {
gateways = append(gateways, m.namespace+"/"+common.CreateConvertedName(m.clusterId, common.CleanHost("*")))
}
wrapperVS, exist := convertOptions.VirtualServices[host]
if !exist {

View File

@@ -83,7 +83,7 @@ func (i *Ingress) NeedRegexMatch(path string) bool {
if i.Rewrite == nil {
return false
}
if strings.ContainsAny(path, `\.+*?()|[]{}^$`) {
if i.Rewrite.RewriteTarget != "" && strings.ContainsAny(path, `\.+*?()|[]{}^$`) {
return true
}
if strings.ContainsAny(i.Rewrite.RewriteTarget, `$\`) {

View File

@@ -67,6 +67,15 @@ func TestNeedRegexMatch(t *testing.T) {
inputPath: "/.*",
expect: true,
},
{
input: &Ingress{
Rewrite: &RewriteConfig{
UseRegex: false,
},
},
inputPath: "/.",
expect: false,
},
{
input: &Ingress{
Rewrite: &RewriteConfig{

View File

@@ -23,13 +23,14 @@ import (
)
const (
exact = "exact"
regex = "regex"
prefix = "prefix"
MatchMethod = "match-method"
MatchQuery = "match-query"
MatchHeader = "match-header"
sep = " "
exact = "exact"
regex = "regex"
prefix = "prefix"
MatchMethod = "match-method"
MatchQuery = "match-query"
MatchHeader = "match-header"
MatchPseudoHeader = "match-pseudo-header"
sep = " "
)
var (
@@ -56,6 +57,24 @@ func (m match) Parse(annotations Annotations, config *Ingress, _ *GlobalContext)
IngressLog.Errorf("parse headers error %v within ingress %s/%s", err, config.Namespace, config.Name)
}
var pseudoHeaderMatches map[string]map[string]string
if pseudoHeaderMatches, err = m.matchByHeaderOrQueryParma(annotations, MatchPseudoHeader, pseudoHeaderMatches); err != nil {
IngressLog.Errorf("parse headers error %v within ingress %s/%s", err, config.Namespace, config.Name)
}
if pseudoHeaderMatches != nil && len(pseudoHeaderMatches) > 0 {
if config.Match.Headers == nil {
config.Match.Headers = make(map[string]map[string]string)
}
for typ, mmap := range pseudoHeaderMatches {
if config.Match.Headers[typ] == nil {
config.Match.Headers[typ] = make(map[string]string)
}
for k, v := range mmap {
config.Match.Headers[typ][":"+k] = v
}
}
}
if config.Match.QueryParams, err = m.matchByHeaderOrQueryParma(annotations, MatchQuery, config.Match.QueryParams); err != nil {
IngressLog.Errorf("parse query params error %v within ingress %s/%s", err, config.Namespace, config.Name)
}

View File

@@ -15,6 +15,7 @@
package annotations
import (
"strings"
"testing"
"github.com/google/go-cmp/cmp"
@@ -112,11 +113,47 @@ func TestMatch_ParseHeaders(t *testing.T) {
},
},
},
{
typ: "exact",
key: ":method",
value: "GET",
expect: map[string]map[string]string{
exact: {
":method": "GET",
},
},
},
{
typ: "prefix",
key: ":path",
value: "/foo",
expect: map[string]map[string]string{
prefix: {
":path": "/foo",
},
},
},
{
typ: "regex",
key: ":authority",
value: "test\\d+\\.com",
expect: map[string]map[string]string{
regex: {
":authority": "test\\d+\\.com",
},
},
},
}
for _, tt := range testCases {
t.Run("", func(t *testing.T) {
key := buildHigressAnnotationKey(tt.typ + "-" + MatchHeader + "-" + tt.key)
matchKeyword := MatchHeader
headerKey := tt.key
if strings.HasPrefix(headerKey, ":") {
headerKey = strings.TrimPrefix(headerKey, ":")
matchKeyword = MatchPseudoHeader
}
key := buildHigressAnnotationKey(tt.typ + "-" + matchKeyword + "-" + headerKey)
input := Annotations{key: tt.value}
config := &Ingress{}
_ = parser.Parse(input, config, nil)

View File

@@ -65,7 +65,12 @@ func (u upstreamTLS) Parse(annotations Annotations, config *Ingress, _ *GlobalCo
}
defer func() {
config.UpstreamTLS = upstreamTLSConfig
if upstreamTLSConfig.BackendProtocol == defaultBackendProtocol {
// no need destination rule when use HTTP protocol
config.UpstreamTLS = nil
} else {
config.UpstreamTLS = upstreamTLSConfig
}
}()
if proto, err := annotations.ParseStringASAP(backendProtocol); err == nil {

View File

@@ -33,6 +33,12 @@ func TestUpstreamTLSParse(t *testing.T) {
input: Annotations{},
expect: nil,
},
{
input: Annotations{
buildNginxAnnotationKey(backendProtocol): "HTTP",
},
expect: nil,
},
{
input: Annotations{
buildNginxAnnotationKey(proxySSLSecret): "",

View File

@@ -36,7 +36,6 @@ func ValidateBackendResource(resource *v1.TypedLocalObjectReference) bool {
if resource == nil || resource.APIGroup == nil ||
*resource.APIGroup != netv1.SchemeGroupVersion.Group ||
resource.Kind != "McpBridge" || resource.Name != "default" {
IngressLog.Warnf("invalid mcpbridge resource: %v", resource)
return false
}
return true

View File

@@ -37,6 +37,7 @@ type HigressConfig struct {
Tracing *Tracing `json:"tracing,omitempty"`
Gzip *Gzip `json:"gzip,omitempty"`
Downstream *Downstream `json:"downstream,omitempty"`
Upstream *Upstream `json:"upstream,omitempty"`
DisableXEnvoyHeaders bool `json:"disableXEnvoyHeaders,omitempty"`
AddXRealIpHeader bool `json:"addXRealIpHeader,omitempty"`
}
@@ -47,6 +48,7 @@ func NewDefaultHigressConfig() *HigressConfig {
Tracing: NewDefaultTracing(),
Gzip: NewDefaultGzip(),
Downstream: globalOption.Downstream,
Upstream: globalOption.Upstream,
DisableXEnvoyHeaders: globalOption.DisableXEnvoyHeaders,
AddXRealIpHeader: globalOption.AddXRealIpHeader,
}

View File

@@ -15,7 +15,6 @@
package configmap
import (
"encoding/json"
"fmt"
"reflect"
"sync/atomic"
@@ -38,19 +37,22 @@ const (
minInitialConnectionWindowSize = 65535
maxInitialConnectionWindowSize = 2147483647
defaultIdleTimeout = 180
defaultMaxRequestHeadersKb = 60
defaultConnectionBufferLimits = 32768
defaultMaxConcurrentStreams = 100
defaultInitialStreamWindowSize = 65535
defaultInitialConnectionWindowSize = 1048576
defaultAddXRealIpHeader = false
defaultDisableXEnvoyHeaders = false
defaultIdleTimeout = 180
defaultUpStreamIdleTimeout = 10
defaultUpStreamConnectionBufferLimits = 10485760
defaultMaxRequestHeadersKb = 60
defaultConnectionBufferLimits = 32768
defaultMaxConcurrentStreams = 100
defaultInitialStreamWindowSize = 65535
defaultInitialConnectionWindowSize = 1048576
defaultAddXRealIpHeader = false
defaultDisableXEnvoyHeaders = false
)
// Global configures the behavior of the downstream connection, x-real-ip header and x-envoy headers.
type Global struct {
Downstream *Downstream `json:"downstream,omitempty"`
Upstream *Upstream `json:"upstream,omitempty"`
AddXRealIpHeader bool `json:"addXRealIpHeader,omitempty"`
DisableXEnvoyHeaders bool `json:"disableXEnvoyHeaders,omitempty"`
}
@@ -58,7 +60,7 @@ type Global struct {
// Downstream configures the behavior of the downstream connection.
type Downstream struct {
// IdleTimeout limits the time that a connection may be idle and stream idle.
IdleTimeout uint32 `json:"idleTimeout,omitempty"`
IdleTimeout uint32 `json:"idleTimeout"`
// MaxRequestHeadersKb limits the size of request headers allowed.
MaxRequestHeadersKb uint32 `json:"maxRequestHeadersKb,omitempty"`
// ConnectionBufferLimits configures the buffer size limits for connections.
@@ -67,6 +69,14 @@ type Downstream struct {
Http2 *Http2 `json:"http2,omitempty"`
}
// Upstream configures the behavior of the upstream connection.
type Upstream struct {
// IdleTimeout limits the time that a connection may be idle on the upstream.
IdleTimeout uint32 `json:"idleTimeout"`
// ConnectionBufferLimits configures the buffer size limits for connections.
ConnectionBufferLimits uint32 `json:"connectionBufferLimits,omitempty"`
}
// Http2 configures HTTP/2 specific options.
type Http2 struct {
// MaxConcurrentStreams limits the number of concurrent streams allowed.
@@ -125,7 +135,7 @@ func compareGlobal(old *Global, new *Global) (Result, error) {
return ResultDelete, nil
}
if new.Downstream == nil && !new.AddXRealIpHeader && !new.DisableXEnvoyHeaders {
if new.Downstream == nil && new.Upstream == nil && !new.AddXRealIpHeader && !new.DisableXEnvoyHeaders {
return ResultDelete, nil
}
@@ -139,18 +149,30 @@ func compareGlobal(old *Global, new *Global) (Result, error) {
// deepCopyGlobal deep copies the global option.
func deepCopyGlobal(global *Global) (*Global, error) {
newGlobal := NewDefaultGlobalOption()
bytes, err := json.Marshal(global)
if err != nil {
return nil, err
if global.Downstream != nil {
newGlobal.Downstream.IdleTimeout = global.Downstream.IdleTimeout
newGlobal.Downstream.MaxRequestHeadersKb = global.Downstream.MaxRequestHeadersKb
newGlobal.Downstream.ConnectionBufferLimits = global.Downstream.ConnectionBufferLimits
if global.Downstream.Http2 != nil {
newGlobal.Downstream.Http2.MaxConcurrentStreams = global.Downstream.Http2.MaxConcurrentStreams
newGlobal.Downstream.Http2.InitialStreamWindowSize = global.Downstream.Http2.InitialStreamWindowSize
newGlobal.Downstream.Http2.InitialConnectionWindowSize = global.Downstream.Http2.InitialConnectionWindowSize
}
}
err = json.Unmarshal(bytes, newGlobal)
return newGlobal, err
if global.Upstream != nil {
newGlobal.Upstream.IdleTimeout = global.Upstream.IdleTimeout
newGlobal.Upstream.ConnectionBufferLimits = global.Upstream.ConnectionBufferLimits
}
newGlobal.AddXRealIpHeader = global.AddXRealIpHeader
newGlobal.DisableXEnvoyHeaders = global.DisableXEnvoyHeaders
return newGlobal, nil
}
// NewDefaultGlobalOption returns a default global config.
func NewDefaultGlobalOption() *Global {
return &Global{
Downstream: NewDefaultDownstream(),
Upstream: NewDefaultUpStream(),
AddXRealIpHeader: defaultAddXRealIpHeader,
DisableXEnvoyHeaders: defaultDisableXEnvoyHeaders,
}
@@ -166,6 +188,14 @@ func NewDefaultDownstream() *Downstream {
}
}
// NewDefaultUpStream returns a default upstream config.
func NewDefaultUpStream() *Upstream {
return &Upstream{
IdleTimeout: defaultUpStreamIdleTimeout,
ConnectionBufferLimits: defaultUpStreamConnectionBufferLimits,
}
}
// NewDefaultHttp2 returns a default http2 config.
func NewDefaultHttp2() *Http2 {
return &Http2{
@@ -215,12 +245,14 @@ func (g *GlobalOptionController) GetName() string {
func (g *GlobalOptionController) AddOrUpdateHigressConfig(name util.ClusterNamespacedName, old *HigressConfig, new *HigressConfig) error {
newGlobal := &Global{
Downstream: new.Downstream,
Upstream: new.Upstream,
AddXRealIpHeader: new.AddXRealIpHeader,
DisableXEnvoyHeaders: new.DisableXEnvoyHeaders,
}
oldGlobal := &Global{
Downstream: old.Downstream,
Upstream: old.Upstream,
AddXRealIpHeader: old.AddXRealIpHeader,
DisableXEnvoyHeaders: old.DisableXEnvoyHeaders,
}
@@ -264,6 +296,7 @@ func (g *GlobalOptionController) ValidHigressConfig(higressConfig *HigressConfig
global := &Global{
Downstream: higressConfig.Downstream,
Upstream: higressConfig.Upstream,
AddXRealIpHeader: higressConfig.AddXRealIpHeader,
DisableXEnvoyHeaders: higressConfig.DisableXEnvoyHeaders,
}
@@ -306,6 +339,18 @@ func (g *GlobalOptionController) ConstructEnvoyFilters() ([]*config.Config, erro
downstreamConfig := g.generateDownstreamEnvoyFilter(downstreamStruct, bufferLimitStruct, namespace)
configPatch = append(configPatch, downstreamConfig...)
if global.Upstream == nil {
return generateEnvoyFilter(namespace, configPatch), nil
}
upstreamStruct := g.constructUpstream(global.Upstream)
bufferLimitStruct = g.constructUpstreamBufferLimit(global.Upstream)
if len(upstreamStruct) == 0 {
return generateEnvoyFilter(namespace, configPatch), nil
}
upstreamConfig := g.generateUpstreamEnvoyFilter(upstreamStruct, bufferLimitStruct, namespace)
configPatch = append(configPatch, upstreamConfig...)
return generateEnvoyFilter(namespace, configPatch), nil
}
@@ -365,6 +410,32 @@ func (g *GlobalOptionController) generateDownstreamEnvoyFilter(downstreamValueSt
return downstreamConfig
}
func (g *GlobalOptionController) generateUpstreamEnvoyFilter(upstreamValueStruct string, bufferLimit string, namespace string) []*networking.EnvoyFilter_EnvoyConfigObjectPatch {
upstreamConfig := []*networking.EnvoyFilter_EnvoyConfigObjectPatch{
{
ApplyTo: networking.EnvoyFilter_CLUSTER,
Match: &networking.EnvoyFilter_EnvoyConfigObjectMatch{
Context: networking.EnvoyFilter_GATEWAY,
},
Patch: &networking.EnvoyFilter_Patch{
Operation: networking.EnvoyFilter_Patch_MERGE,
Value: util.BuildPatchStruct(upstreamValueStruct),
},
},
{
ApplyTo: networking.EnvoyFilter_CLUSTER,
Match: &networking.EnvoyFilter_EnvoyConfigObjectMatch{
Context: networking.EnvoyFilter_GATEWAY,
},
Patch: &networking.EnvoyFilter_Patch{
Operation: networking.EnvoyFilter_Patch_MERGE,
Value: util.BuildPatchStruct(bufferLimit),
},
},
}
return upstreamConfig
}
// generateAddXRealIpHeaderEnvoyFilter generates the add x-real-ip header envoy filter.
func (g *GlobalOptionController) generateAddXRealIpHeaderEnvoyFilter(addXRealIpHeaderStruct string, namespace string) []*networking.EnvoyFilter_EnvoyConfigObjectPatch {
addXRealIpHeaderConfig := []*networking.EnvoyFilter_EnvoyConfigObjectPatch{
@@ -460,6 +531,32 @@ func (g *GlobalOptionController) constructDownstream(downstream *Downstream) str
return downstreamConfig
}
// constructUpstream constructs the upstream config.
func (g *GlobalOptionController) constructUpstream(upstream *Upstream) string {
upstreamConfig := ""
idleTimeout := upstream.IdleTimeout
upstreamConfig = fmt.Sprintf(`
{
"common_http_protocol_options": {
"idleTimeout": "%ds"
}
}
`, idleTimeout)
return upstreamConfig
}
// constructUpstreamBufferLimit constructs the upstream buffer limit config.
func (g *GlobalOptionController) constructUpstreamBufferLimit(upstream *Upstream) string {
upstreamBufferLimitStruct := fmt.Sprintf(`
{
"per_connection_buffer_limit_bytes": %d
}
`, upstream.ConnectionBufferLimits)
return upstreamBufferLimitStruct
}
// constructAddXRealIpHeader constructs the add x-real-ip header config.
func (g *GlobalOptionController) constructAddXRealIpHeader() string {
addXRealIpHeaderStruct := fmt.Sprintf(`

View File

@@ -41,6 +41,7 @@ func Test_validGlobal(t *testing.T) {
name: "downstream nil",
global: &Global{
Downstream: nil,
Upstream: NewDefaultUpStream(),
AddXRealIpHeader: true,
DisableXEnvoyHeaders: true,
},
@@ -131,35 +132,27 @@ func Test_deepCopyGlobal(t *testing.T) {
name: "deep copy 2",
global: &Global{
Downstream: &Downstream{
IdleTimeout: 1,
IdleTimeout: 0,
MaxRequestHeadersKb: 9600,
ConnectionBufferLimits: 4096,
Http2: NewDefaultHttp2(),
},
Upstream: &Upstream{
IdleTimeout: 10,
},
AddXRealIpHeader: true,
DisableXEnvoyHeaders: true,
},
want: &Global{
Downstream: &Downstream{
IdleTimeout: 1,
IdleTimeout: 0,
MaxRequestHeadersKb: 9600,
ConnectionBufferLimits: 4096,
Http2: NewDefaultHttp2(),
},
AddXRealIpHeader: true,
DisableXEnvoyHeaders: true,
},
wantErr: nil,
},
{
name: "deep copy 3",
global: &Global{
Downstream: &Downstream{},
AddXRealIpHeader: true,
DisableXEnvoyHeaders: true,
},
want: &Global{
Downstream: NewDefaultDownstream(),
Upstream: &Upstream{
IdleTimeout: 10,
},
AddXRealIpHeader: true,
DisableXEnvoyHeaders: true,
},
@@ -203,7 +196,13 @@ func Test_AddOrUpdateHigressConfig(t *testing.T) {
old: NewDefaultHigressConfig(),
new: &HigressConfig{
Downstream: &Downstream{
IdleTimeout: 1,
IdleTimeout: 1,
MaxRequestHeadersKb: defaultMaxRequestHeadersKb,
ConnectionBufferLimits: defaultConnectionBufferLimits,
Http2: NewDefaultHttp2(),
},
Upstream: &Upstream{
IdleTimeout: 10,
},
AddXRealIpHeader: true,
DisableXEnvoyHeaders: true,
@@ -217,6 +216,9 @@ func Test_AddOrUpdateHigressConfig(t *testing.T) {
ConnectionBufferLimits: defaultConnectionBufferLimits,
Http2: NewDefaultHttp2(),
},
Upstream: &Upstream{
IdleTimeout: 10,
},
AddXRealIpHeader: true,
DisableXEnvoyHeaders: true,
},
@@ -225,6 +227,7 @@ func Test_AddOrUpdateHigressConfig(t *testing.T) {
name: "delete and push",
old: &HigressConfig{
Downstream: NewDefaultDownstream(),
Upstream: NewDefaultUpStream(),
AddXRealIpHeader: defaultAddXRealIpHeader,
DisableXEnvoyHeaders: defaultDisableXEnvoyHeaders,
},
@@ -233,6 +236,7 @@ func Test_AddOrUpdateHigressConfig(t *testing.T) {
wantEventPush: "push",
wantGlobal: &Global{
Downstream: NewDefaultDownstream(),
Upstream: NewDefaultUpStream(),
AddXRealIpHeader: defaultAddXRealIpHeader,
DisableXEnvoyHeaders: defaultDisableXEnvoyHeaders,
},

View File

@@ -1279,8 +1279,10 @@ func createRuleKey(annots map[string]string, hostAndPath string) string {
if idx := strings.Index(k, annotations.MatchHeader); idx != -1 {
key := k[start:idx] + k[idx+len(annotations.MatchHeader)+1:]
headers = append(headers, [2]string{key, val})
}
if idx := strings.Index(k, annotations.MatchQuery); idx != -1 {
} else if idx := strings.Index(k, annotations.MatchPseudoHeader); idx != -1 {
key := k[start:idx] + ":" + k[idx+len(annotations.MatchPseudoHeader)+1:]
headers = append(headers, [2]string{key, val})
} else if idx := strings.Index(k, annotations.MatchQuery); idx != -1 {
key := k[start:idx] + k[idx+len(annotations.MatchQuery)+1:]
params = append(params, [2]string{key, val})
}

View File

@@ -1302,15 +1302,18 @@ func TestCreateRuleKey(t *testing.T) {
}
annots := annotations.Annotations{
buildHigressAnnotationKey(annotations.MatchMethod): "GET PUT",
buildHigressAnnotationKey("exact-" + annotations.MatchHeader + "-abc"): "123",
buildHigressAnnotationKey("prefix-" + annotations.MatchHeader + "-def"): "456",
buildHigressAnnotationKey("exact-" + annotations.MatchQuery + "-region"): "beijing",
buildHigressAnnotationKey("prefix-" + annotations.MatchQuery + "-user-id"): "user-",
buildHigressAnnotationKey(annotations.MatchMethod): "GET PUT",
buildHigressAnnotationKey("exact-" + annotations.MatchHeader + "-abc"): "123",
buildHigressAnnotationKey("prefix-" + annotations.MatchHeader + "-def"): "456",
buildHigressAnnotationKey("exact-" + annotations.MatchPseudoHeader + "-authority"): "foo.bar.com",
buildHigressAnnotationKey("prefix-" + annotations.MatchPseudoHeader + "-scheme"): "htt",
buildHigressAnnotationKey("exact-" + annotations.MatchQuery + "-region"): "beijing",
buildHigressAnnotationKey("prefix-" + annotations.MatchQuery + "-user-id"): "user-",
}
expect := "higress.com-prefix-/foo" + sep + //host-pathType-path
"GET PUT" + sep + // method
"exact-abc\t123" + "\n" + "prefix-def\t456" + sep + // header
"exact-:authority\tfoo.bar.com" + "\n" + "exact-abc\t123" + "\n" +
"prefix-:scheme\thtt" + "\n" + "prefix-def\t456" + sep + // header
"exact-region\tbeijing" + "\n" + "prefix-user-id\tuser-" + sep // params
key := createRuleKey(annots, wrapperHttpRoute.PathFormat())

View File

@@ -1226,8 +1226,10 @@ func createRuleKey(annots map[string]string, hostAndPath string) string {
if idx := strings.Index(k, annotations.MatchHeader); idx != -1 {
key := k[start:idx] + k[idx+len(annotations.MatchHeader)+1:]
headers = append(headers, [2]string{key, val})
}
if idx := strings.Index(k, annotations.MatchQuery); idx != -1 {
} else if idx := strings.Index(k, annotations.MatchPseudoHeader); idx != -1 {
key := k[start:idx] + ":" + k[idx+len(annotations.MatchPseudoHeader)+1:]
headers = append(headers, [2]string{key, val})
} else if idx := strings.Index(k, annotations.MatchQuery); idx != -1 {
key := k[start:idx] + k[idx+len(annotations.MatchQuery)+1:]
params = append(params, [2]string{key, val})
}

View File

@@ -699,8 +699,10 @@ func createRuleKey(annots map[string]string, hostAndPath string) string {
if idx := strings.Index(k, annotations.MatchHeader); idx != -1 {
key := k[start:idx] + k[idx+len(annotations.MatchHeader)+1:]
headers = append(headers, [2]string{key, val})
}
if idx := strings.Index(k, annotations.MatchQuery); idx != -1 {
} else if idx := strings.Index(k, annotations.MatchPseudoHeader); idx != -1 {
key := k[start:idx] + ":" + k[idx+len(annotations.MatchPseudoHeader)+1:]
headers = append(headers, [2]string{key, val})
} else if idx := strings.Index(k, annotations.MatchQuery); idx != -1 {
key := k[start:idx] + k[idx+len(annotations.MatchQuery)+1:]
params = append(params, [2]string{key, val})
}

View File

@@ -581,15 +581,18 @@ func TestCreateRuleKey(t *testing.T) {
}
annots := annotations.Annotations{
buildHigressAnnotationKey(annotations.MatchMethod): "GET PUT",
buildHigressAnnotationKey("exact-" + annotations.MatchHeader + "-abc"): "123",
buildHigressAnnotationKey("prefix-" + annotations.MatchHeader + "-def"): "456",
buildHigressAnnotationKey("exact-" + annotations.MatchQuery + "-region"): "beijing",
buildHigressAnnotationKey("prefix-" + annotations.MatchQuery + "-user-id"): "user-",
buildHigressAnnotationKey(annotations.MatchMethod): "GET PUT",
buildHigressAnnotationKey("exact-" + annotations.MatchHeader + "-abc"): "123",
buildHigressAnnotationKey("prefix-" + annotations.MatchHeader + "-def"): "456",
buildHigressAnnotationKey("exact-" + annotations.MatchPseudoHeader + "-authority"): "foo.bar.com",
buildHigressAnnotationKey("prefix-" + annotations.MatchPseudoHeader + "-scheme"): "htt",
buildHigressAnnotationKey("exact-" + annotations.MatchQuery + "-region"): "beijing",
buildHigressAnnotationKey("prefix-" + annotations.MatchQuery + "-user-id"): "user-",
}
expect := "higress.com-prefix-/foo" + sep + //host-pathType-path
"GET PUT" + sep + // method
"exact-abc\t123" + "\n" + "prefix-def\t456" + sep + // header
"exact-:authority\tfoo.bar.com" + "\n" + "exact-abc\t123" + "\n" +
"prefix-:scheme\thtt" + "\n" + "prefix-def\t456" + sep + // header
"exact-region\tbeijing" + "\n" + "prefix-user-id\tuser-" + sep // params
key := createRuleKey(annots, wrapperHttpRoute.PathFormat())

View File

@@ -0,0 +1,23 @@
FROM higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/gateway:1.3.1
FROM ubuntu:20.04
RUN apt -y update \
&& apt install -y --no-install-recommends python3-pip net-tools vim wget make curl git 2>&1 \
&& apt install -y --reinstall ca-certificates \
&& apt-get autoremove -y && apt-get clean \
&& rm -rf /tmp/* /var/tmp/* \
&& rm -rf /var/lib/apt/lists/*
ENV PATH=/opt/tinygo/bin:/opt/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
RUN wget --no-check-certificate https://github.com/tinygo-org/tinygo/releases/download/v0.29.0/tinygo0.29.0.linux-amd64.tar.gz \
&& tar -zvxf tinygo0.29.0.linux-amd64.tar.gz -C /opt \
&& rm tinygo0.29.0.linux-amd64.tar.gz
RUN wget --no-check-certificate https://go.dev/dl/go1.19.linux-amd64.tar.gz \
&& tar -zvxf go1.19.linux-amd64.tar.gz -C /opt \
&& rm go1.19.linux-amd64.tar.gz \
&& go install -v golang.org/x/tools/gopls@latest
COPY --from=0 /usr/local/bin/envoy /usr/local/bin/envoy

View File

@@ -0,0 +1,21 @@
{
"name": "Wasm Go Dev",
// "dockerFile": "Dockerfile",
"image": "liuxr25/wasm-go:tinygo-0.29.0",
"runArgs": [
"--user=root"
],
"customizations": {
"vscode": {
"settings": {
"terminal.integrated.shell.linux": "/bin/bash",
"python.pythonPath": "/usr/bin/python3"
},
"extensions": [
"ms-python.python",
"golang.go"
]
}
}
}

View File

@@ -0,0 +1,76 @@
import json
import sys
plugin_name = sys.argv[1]
with open("extensions/"+plugin_name+"/config.json", "r") as f:
plugin_config = json.load(f)
config = f'''static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 8080
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: AUTO
stat_prefix: ingress_http
route_config:
name: test
virtual_hosts:
- name: direct_response_service
domains:
- "*"
routes:
- match:
prefix: "/"
direct_response:
status: 200
body:
inline_string: "hello world\\n"
# - match:
# prefix: "/"
# route:
# cluster: service-backend
http_filters:
- name: {plugin_name}
typed_config:
"@type": type.googleapis.com/udpa.type.v1.TypedStruct
type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
value:
config:
name: wasmdemo
vm_config:
runtime: envoy.wasm.runtime.v8
code:
local:
filename: ./extensions/{plugin_name}/main.wasm
configuration:
"@type": "type.googleapis.com/google.protobuf.StringValue"
value: '{json.dumps(plugin_config)}'
- name: envoy.filters.http.router
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
# clusters:
# - name: service-backend
# connect_timeout: 600s
# type: STATIC
# lb_policy: ROUND_ROBIN
# load_assignment:
# cluster_name: service-backend
# endpoints:
# - lb_endpoints:
# - endpoint:
# address:
# socket_address:
# address: 127.0.0.1
# port_value: 8000
'''
with open("extensions/"+plugin_name+"/config.yaml", "w") as f:
f.write(config)

View File

@@ -60,3 +60,16 @@ builder:
.
@echo ""
@echo "image: ${BUILDER}"
local-build:
tinygo build -scheduler=none -target=wasi -gc=custom -tags='custommalloc nottinygc_finalizer' \
-o extensions/${PLUGIN_NAME}/main.wasm \
extensions/${PLUGIN_NAME}/main.go
@echo ""
@echo "wasm: extensions/${PLUGIN_NAME}/main.wasm"
local-run:
python3 .devcontainer/gen_config.py ${PLUGIN_NAME}
envoy -c extensions/${PLUGIN_NAME}/config.yaml --concurrency 0 --log-level info --component-log-level wasm:debug
local-all: local-build local-run

View File

@@ -0,0 +1,58 @@
<p>
<a href="README_EN.md"> English </a> | 中文
</p>
# 功能说明
`bot-detect`插件可以用于识别并阻止互联网爬虫对站点资源的爬取
# 配置字段
| 名称 | 数据类型 | 填写要求 | 默认值 | 描述 |
| -------- | -------- | -------- | -------- | -------- |
| allow | array of string | 选填 | - | 配置匹配 User-Agent 请求头的正则表达式,匹配命中时将允许其访问 |
| deny | array of string | 选填 | - | 配置匹配 User-Agent 请求头的正则表达式,匹配命中时将屏蔽请求 |
| blocked_code | number | 选填 | 403 | 配置请求被屏蔽时返回的 HTTP 状态码 |
| blocked_message | string | 选填 | - | 配置请求被屏蔽时返回的 HTTP 应答 Body |
`allow``deny` 字段可以均不配置,则执行默认的爬虫判断逻辑,通过配置 `allow` 字段可以将原本命中默认爬虫判断逻辑的请求放行,通过配置 `deny` 字段可以增加额外的爬虫判断逻辑。
默认的爬虫判断正则表达式集合如下:
```bash
# Bots General matcher 'name/0.0'
(?:\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}(?:[Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50}))[/ ](\d+)(?:\.(\d+)(?:\.(\d+)|)|)
# Bots General matcher 'name 0.0'
(?:\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}(?:[Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50})) (\d+)(?:\.(\d+)(?:\.(\d+)|)|)
# Bots containing spider|scrape|bot(but not CUBOT)|Crawl
((?:[A-z0-9]{1,50}|[A-z\-]{1,50} ?|)(?: the |)(?:[Ss][Pp][Ii][Dd][Ee][Rr]|[Ss]crape|[Cc][Rr][Aa][Ww][Ll])[A-z0-9]{0,50})(?:(?:[ /]| v)(\d+)(?:\.(\d+)|)(?:\.(\d+)|)|)
# Bots Pattern '/name-0.0'
/((?:Ant-)?Nutch|[A-z]+[Bb]ot|[A-z]+[Ss]pider|Axtaris|fetchurl|Isara|ShopSalad|Tailsweep)[ \-](\d+)(?:\.(\d+)(?:\.(\d+))?)?
# Bots Pattern 'name/0.0'
\b(008|Altresium|Argus|BaiduMobaider|BoardReader|DNSGroup|DataparkSearch|EDI|Goodzer|Grub|INGRID|Infohelfer|LinkedInBot|LOOQ|Nutch|OgScrper|PathDefender|Peew|PostPost|Steeler|Twitterbot|VSE|WebCrunch|WebZIP|Y!J-BR[A-Z]|YahooSeeker|envolk|sproose|wminer)/(\d+)(?:\.(\d+)|)(?:\.(\d+)|)
# More bots
(CSimpleSpider|Cityreview Robot|CrawlDaddy|CrawlFire|Finderbots|Index crawler|Job Roboter|KiwiStatus Spider|Lijit Crawler|QuerySeekerSpider|ScollSpider|Trends Crawler|USyd-NLP-Spider|SiteCat Webbot|BotName\/\$BotVersion|123metaspider-Bot|1470\.net crawler|50\.nu|8bo Crawler Bot|Aboundex|Accoona-[A-z]{1,30}-Agent|AdsBot-Google(?:-[a-z]{1,30}|)|altavista|AppEngine-Google|archive.{0,30}\.org_bot|archiver|Ask Jeeves|[Bb]ai[Dd]u[Ss]pider(?:-[A-Za-z]{1,30})(?:-[A-Za-z]{1,30}|)|bingbot|BingPreview|blitzbot|BlogBridge|Bloglovin|BoardReader Blog Indexer|BoardReader Favicon Fetcher|boitho.com-dc|BotSeer|BUbiNG|\b\w{0,30}favicon\w{0,30}\b|\bYeti(?:-[a-z]{1,30}|)|Catchpoint(?: bot|)|[Cc]harlotte|Checklinks|clumboot|Comodo HTTP\(S\) Crawler|Comodo-Webinspector-Crawler|ConveraCrawler|CRAWL-E|CrawlConvera|Daumoa(?:-feedfetcher|)|Feed Seeker Bot|Feedbin|findlinks|Flamingo_SearchEngine|FollowSite Bot|furlbot|Genieo|gigabot|GomezAgent|gonzo1|(?:[a-zA-Z]{1,30}-|)Googlebot(?:-[a-zA-Z]{1,30}|)|Google SketchUp|grub-client|gsa-crawler|heritrix|HiddenMarket|holmes|HooWWWer|htdig|ia_archiver|ICC-Crawler|Icarus6j|ichiro(?:/mobile|)|IconSurf|IlTrovatore(?:-Setaccio|)|InfuzApp|Innovazion Crawler|InternetArchive|IP2[a-z]{1,30}Bot|jbot\b|KaloogaBot|Kraken|Kurzor|larbin|LEIA|LesnikBot|Linguee Bot|LinkAider|LinkedInBot|Lite Bot|Llaut|lycos|Mail\.RU_Bot|masscan|masidani_bot|Mediapartners-Google|Microsoft .{0,30} Bot|mogimogi|mozDex|MJ12bot|msnbot(?:-media {0,2}|)|msrbot|Mtps Feed Aggregation System|netresearch|Netvibes|NewsGator[^/]{0,30}|^NING|Nutch[^/]{0,30}|Nymesis|ObjectsSearch|OgScrper|Orbiter|OOZBOT|PagePeeker|PagesInventory|PaxleFramework|Peeplo Screenshot Bot|PlantyNet_WebRobot|Pompos|Qwantify|Read%20Later|Reaper|RedCarpet|Retreiver|Riddler|Rival IQ|scooter|Scrapy|Scrubby|searchsight|seekbot|semanticdiscovery|SemrushBot|Simpy|SimplePie|SEOstats|SimpleRSS|SiteCon|Slackbot-LinkExpanding|Slack-ImgProxy|Slurp|snappy|Speedy Spider|Squrl Java|Stringer|TheUsefulbot|ThumbShotsBot|Thumbshots\.ru|Tiny Tiny RSS|Twitterbot|WhatsApp|URL2PNG|Vagabondo|VoilaBot|^vortex|Votay bot|^voyager|WASALive.Bot|Web-sniffer|WebThumb|WeSEE:[A-z]{1,30}|WhatWeb|WIRE|WordPress|Wotbox|www\.almaden\.ibm\.com|Xenu(?:.s|) Link Sleuth|Xerka [A-z]{1,30}Bot|yacy(?:bot|)|YahooSeeker|Yahoo! Slurp|Yandex\w{1,30}|YodaoBot(?:-[A-z]{1,30}|)|YottaaMonitor|Yowedo|^Zao|^Zao-Crawler|ZeBot_www\.ze\.bz|ZooShot|ZyBorg)(?:[ /]v?(\d+)(?:\.(\d+)(?:\.(\d+)|)|)|)
```
# 配置示例
## 放行原本命中爬虫规则的请求
```yaml
allow:
- ".*Go-http-client.*"
```
若不作该配置,默认的 Golang 网络库请求会被视做爬虫,被禁止访问
## 增加爬虫判断
```yaml
deny:
- "spd-tools.*"
```
根据该配置,下列请求将被禁止访问:
```bash
curl http://example.com -H 'User-Agent: spd-tools/1.1'
curl http://exmaple.com -H 'User-Agent: spd-tools'
```

View File

@@ -0,0 +1,58 @@
<p>
English | <a href="README.md">中文</a>
</p>
# Description
`bot-detect` plugin can be used to identify and prevent web crawlers from crawling websites.
# Configuration Fields
| Name | Type | Requirement | Default Value | Description |
| -------- | -------- | -------- | -------- | -------- |
| allow | array of string | Optional | - | A regular expression to match the User-Agent request header and will allow access if the match hits |
| deny | array of string | Optional | - | A regular expression to match the User-Agent request header and will block the request if the match hits |
| blocked_code | number | Optional | 403 | The HTTP status code returned when a request is blocked |
| blocked_message | string | Optional | - | The HTTP response Body returned when a request is blocked |
If field `allow` and field `deny` are not configured at the same time, the default logic to identify crawlers will be executed. By configuring the `allow` field, requests that would otherwise hit the default logic can be allowed. The judgement can be extended by configuring the `deny` field
The default set of crawler judgment regular expressions is as follows
```bash
# Bots General matcher 'name/0.0'
(?:\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}(?:[Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50}))[/ ](\d+)(?:\.(\d+)(?:\.(\d+)|)|)
# Bots General matcher 'name 0.0'
(?:\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}(?:[Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50})) (\d+)(?:\.(\d+)(?:\.(\d+)|)|)
# Bots containing spider|scrape|bot(but not CUBOT)|Crawl
((?:[A-z0-9]{1,50}|[A-z\-]{1,50} ?|)(?: the |)(?:[Ss][Pp][Ii][Dd][Ee][Rr]|[Ss]crape|[Cc][Rr][Aa][Ww][Ll])[A-z0-9]{0,50})(?:(?:[ /]| v)(\d+)(?:\.(\d+)|)(?:\.(\d+)|)|)
# Bots Pattern '/name-0.0'
/((?:Ant-)?Nutch|[A-z]+[Bb]ot|[A-z]+[Ss]pider|Axtaris|fetchurl|Isara|ShopSalad|Tailsweep)[ \-](\d+)(?:\.(\d+)(?:\.(\d+))?)?
# Bots Pattern 'name/0.0'
\b(008|Altresium|Argus|BaiduMobaider|BoardReader|DNSGroup|DataparkSearch|EDI|Goodzer|Grub|INGRID|Infohelfer|LinkedInBot|LOOQ|Nutch|OgScrper|PathDefender|Peew|PostPost|Steeler|Twitterbot|VSE|WebCrunch|WebZIP|Y!J-BR[A-Z]|YahooSeeker|envolk|sproose|wminer)/(\d+)(?:\.(\d+)|)(?:\.(\d+)|)
# More bots
(CSimpleSpider|Cityreview Robot|CrawlDaddy|CrawlFire|Finderbots|Index crawler|Job Roboter|KiwiStatus Spider|Lijit Crawler|QuerySeekerSpider|ScollSpider|Trends Crawler|USyd-NLP-Spider|SiteCat Webbot|BotName\/\$BotVersion|123metaspider-Bot|1470\.net crawler|50\.nu|8bo Crawler Bot|Aboundex|Accoona-[A-z]{1,30}-Agent|AdsBot-Google(?:-[a-z]{1,30}|)|altavista|AppEngine-Google|archive.{0,30}\.org_bot|archiver|Ask Jeeves|[Bb]ai[Dd]u[Ss]pider(?:-[A-Za-z]{1,30})(?:-[A-Za-z]{1,30}|)|bingbot|BingPreview|blitzbot|BlogBridge|Bloglovin|BoardReader Blog Indexer|BoardReader Favicon Fetcher|boitho.com-dc|BotSeer|BUbiNG|\b\w{0,30}favicon\w{0,30}\b|\bYeti(?:-[a-z]{1,30}|)|Catchpoint(?: bot|)|[Cc]harlotte|Checklinks|clumboot|Comodo HTTP\(S\) Crawler|Comodo-Webinspector-Crawler|ConveraCrawler|CRAWL-E|CrawlConvera|Daumoa(?:-feedfetcher|)|Feed Seeker Bot|Feedbin|findlinks|Flamingo_SearchEngine|FollowSite Bot|furlbot|Genieo|gigabot|GomezAgent|gonzo1|(?:[a-zA-Z]{1,30}-|)Googlebot(?:-[a-zA-Z]{1,30}|)|Google SketchUp|grub-client|gsa-crawler|heritrix|HiddenMarket|holmes|HooWWWer|htdig|ia_archiver|ICC-Crawler|Icarus6j|ichiro(?:/mobile|)|IconSurf|IlTrovatore(?:-Setaccio|)|InfuzApp|Innovazion Crawler|InternetArchive|IP2[a-z]{1,30}Bot|jbot\b|KaloogaBot|Kraken|Kurzor|larbin|LEIA|LesnikBot|Linguee Bot|LinkAider|LinkedInBot|Lite Bot|Llaut|lycos|Mail\.RU_Bot|masscan|masidani_bot|Mediapartners-Google|Microsoft .{0,30} Bot|mogimogi|mozDex|MJ12bot|msnbot(?:-media {0,2}|)|msrbot|Mtps Feed Aggregation System|netresearch|Netvibes|NewsGator[^/]{0,30}|^NING|Nutch[^/]{0,30}|Nymesis|ObjectsSearch|OgScrper|Orbiter|OOZBOT|PagePeeker|PagesInventory|PaxleFramework|Peeplo Screenshot Bot|PlantyNet_WebRobot|Pompos|Qwantify|Read%20Later|Reaper|RedCarpet|Retreiver|Riddler|Rival IQ|scooter|Scrapy|Scrubby|searchsight|seekbot|semanticdiscovery|SemrushBot|Simpy|SimplePie|SEOstats|SimpleRSS|SiteCon|Slackbot-LinkExpanding|Slack-ImgProxy|Slurp|snappy|Speedy Spider|Squrl Java|Stringer|TheUsefulbot|ThumbShotsBot|Thumbshots\.ru|Tiny Tiny RSS|Twitterbot|WhatsApp|URL2PNG|Vagabondo|VoilaBot|^vortex|Votay bot|^voyager|WASALive.Bot|Web-sniffer|WebThumb|WeSEE:[A-z]{1,30}|WhatWeb|WIRE|WordPress|Wotbox|www\.almaden\.ibm\.com|Xenu(?:.s|) Link Sleuth|Xerka [A-z]{1,30}Bot|yacy(?:bot|)|YahooSeeker|Yahoo! Slurp|Yandex\w{1,30}|YodaoBot(?:-[A-z]{1,30}|)|YottaaMonitor|Yowedo|^Zao|^Zao-Crawler|ZeBot_www\.ze\.bz|ZooShot|ZyBorg)(?:[ /]v?(\d+)(?:\.(\d+)(?:\.(\d+)|)|)|)
```
# Configuration Samples
## Release Requests that would otherwise Hit the Crawler Rules
```yaml
allow:
- ".*Go-http-client.*"
```
Without this configuration, the default Golang web library request will be treated as a crawler and access will be denied.
## Add Crawler Judgement
```yaml
deny:
- "spd-tools.*"
```
According to this configuration, the following requests will be denied:
```bash
curl http://example.com -H 'User-Agent: spd-tools/1.1'
curl http://exmaple.com -H 'User-Agent: spd-tools'
```

View File

@@ -0,0 +1 @@
1.0.0

View File

@@ -0,0 +1,30 @@
apiVersion: extensions.higress.io/v1alpha1
kind: WasmPlugin
metadata:
annotations:
higress.io/wasm-plugin-description: 用于识别并阻止互联网爬虫对站点资源的爬取
higress.io/wasm-plugin-title: Bot Detect
creationTimestamp: '2024-01-03T10:34:36Z'
generation: 2
labels:
higress.io/resource-definer: higress
higress.io/wasm-plugin-built-in: 'true'
higress.io/wasm-plugin-category: custom
higress.io/wasm-plugin-name: bot-detect
higress.io/wasm-plugin-version: 1.0.0
name: bot-detect
namespace: higress-system
spec:
defaultConfigDisable: true
matchRules:
- config:
blocked_code: 401
blocked_message: a bot
deny:
- Chrome
configDisable: false
ingress:
- test
phase: AUTHN
priority: 310
url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/20240103/bot-detect:1.0.0

View File

@@ -0,0 +1,68 @@
/*
* Copyright (c) 2022 Alibaba Group Holding Ltd.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package config
import (
regexp "github.com/wasilibs/go-re2"
)
var DefaultBotRegex = []*regexp.Regexp{
regexp.MustCompile(`(\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}([Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50}))[/ ](\d+)(\.(\d+)(\.(\d+)|)|)`),
regexp.MustCompile(`((\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}([Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50})) (\d+)(\.(\d+)(\.(\d+)|)|))`),
regexp.MustCompile(`((([A-z0-9]{1,50}|[A-z\-]{1,} ?|)( the |)(?:[Ss][Pp][Ii][Dd][Ee][Rr]|[Ss]crape|[Cc][Rr][Aa][Ww][Ll])[A-z0-9]{0,50})(([ /]| v)(\d+)(\.(\d+)|)(\.(\d+)|)|))`),
regexp.MustCompile(`((Ant-)?Nutch|[A-z]+[Bb]ot|[A-z]+[Ss]pider|Axtaris|fetchurl|Isara|ShopSalad|Tailsweep)[ \-](\d+)(\.(\d+)(\.(\d+))?)?`),
regexp.MustCompile(`\b(008|Altresium|Argus|BaiduMobaider|BoardReader|DNSGroup|DataparkSearch|EDI|Goodzer|Grub|INGRID|Infohelfer|LinkedInBot|LOOQ|Nutch|OgScrper|PathDefender|Peew|PostPost|Steeler|Twitterbot|VSE|WebCrunch|WebZIP|Y!J-BR[A-Z]|YahooSeeker|envolk|sproose|wminer)/(\d+)(?:\.(\d+)|)(?:\.(\d+)|)`),
regexp.MustCompile(`((CSimpleSpider|Cityreview Robot|CrawlDaddy|CrawlFire|Finderbots|Index crawler|Job Roboter|KiwiStatus Spider|Lijit Crawler|QuerySeekerSpider|ScollSpider|Trends Crawler|USyd-NLP-Spider|SiteCat Webbot|BotName\/\$BotVersion|123metaspider-Bot|1470\.net crawler|50\.nu|8bo Crawler Bot|Aboundex|Accoona-[A-z]{1,30}-Agent|AdsBot-Google(?:-[a-z]{1,30}|)|altavista|AppEngine-Google|archive.{0,30}\.org_bot|archiver|Ask Jeeves|[Bb]ai[Dd]u[Ss]pider(?:-[A-Za-z]{1,30})(?:-[A-Za-z]{1,30}|)|bingbot|BingPreview|blitzbot|BlogBridge|Bloglovin|BoardReader Blog Indexer|BoardReader Favicon Fetcher|boitho.com-dc|BotSeer|BUbiNG|\b\w{0,30}favicon\w{0,30}\b|\bYeti(?:-[a-z]{1,30}|)|Catchpoint(?: bot|)|[Cc]harlotte|Checklinks|clumboot|Comodo HTTP\(S\) Crawler|Comodo-Webinspector-Crawler|ConveraCrawler|CRAWL-E|CrawlConvera|Daumoa(?:-feedfetcher|)|Feed Seeker Bot|Feedbin|findlinks|Flamingo_SearchEngine|FollowSite Bot|furlbot|Genieo|gigabot|GomezAgent|gonzo1|(?:[a-zA-Z]{1,30}-|)Googlebot(?:-[a-zA-Z]{1,30}|)|Google SketchUp|grub-client|gsa-crawler|heritrix|HiddenMarket|holmes|HooWWWer|htdig|ia_archiver|ICC-Crawler|Icarus6j|ichiro(?:/mobile|)|IconSurf|IlTrovatore(?:-Setaccio|)|InfuzApp|Innovazion Crawler|InternetArchive|IP2[a-z]{1,30}Bot|jbot\b|KaloogaBot|Kraken|Kurzor|larbin|LEIA|LesnikBot|Linguee Bot|LinkAider|LinkedInBot|Lite Bot|Llaut|lycos|Mail\.RU_Bot|masscan|masidani_bot|Mediapartners-Google|Microsoft .{0,30} Bot|mogimogi|mozDex|MJ12bot|msnbot(?:-media {0,2}|)|msrbot|Mtps Feed Aggregation System|netresearch|Netvibes|NewsGator[^/]{0,30}|^NING|Nutch[^/]{0,30}|Nymesis|ObjectsSearch|OgScrper|Orbiter|OOZBOT|PagePeeker|PagesInventory|PaxleFramework|Peeplo Screenshot Bot|PlantyNet_WebRobot|Pompos|Qwantify|Read%20Later|Reaper|RedCarpet|Retreiver|Riddler|Rival IQ|scooter|Scrapy|Scrubby|searchsight|seekbot|semanticdiscovery|SemrushBot|Simpy|SimplePie|SEOstats|SimpleRSS|SiteCon|Slackbot-LinkExpanding|Slack-ImgProxy|Slurp|snappy|Speedy Spider|Squrl Java|Stringer|TheUsefulbot|ThumbShotsBot|Thumbshots\.ru|Tiny Tiny RSS|Twitterbot|WhatsApp|URL2PNG|Vagabondo|VoilaBot|^vortex|Votay bot|^voyager|WASALive.Bot|Web-sniffer|WebThumb|WeSEE:[A-z]{1,30}|WhatWeb|WIRE|WordPress|Wotbox|www\.almaden\.ibm\.com|Xenu(?:.s|) Link Sleuth|Xerka [A-z]{1,30}Bot|yacy(?:bot|)|YahooSeeker|Yahoo! Slurp|Yandex\w{1,30}|YodaoBot(?:-[A-z]{1,30}|)|YottaaMonitor|Yowedo|^Zao|^Zao-Crawler|ZeBot_www\.ze\.bz|ZooShot|ZyBorg)(?:[ /]v?(\d+)(?:\.(\d+)(?:\.(\d+)|)|)|))`),
}
type BotDetectConfig struct {
BlockedCode uint32 `json:"blocked_code"`
BlockedMessage string `json:"blocked_message"`
Allow []*regexp.Regexp `json:"allow"`
Deny []*regexp.Regexp `json:"deny"`
}
func (bdc *BotDetectConfig) FillDefaultValue() {
if bdc.BlockedCode == 0 {
bdc.BlockedCode = 403
}
if bdc.BlockedMessage == "" {
bdc.BlockedMessage = "Invalid User-Agent"
}
}
func (bdc *BotDetectConfig) Process(ua string) (bool, string) {
if ua == "" {
return false, "can not be empty"
}
for _, allowRule := range bdc.Allow {
if allowRule.MatchString(ua) {
return true, ""
}
}
for _, denyRule := range bdc.Deny {
if denyRule.MatchString(ua) {
return false, denyRule.String()
}
}
for _, defaultRule := range DefaultBotRegex {
if defaultRule.MatchString(ua) {
return false, defaultRule.String()
}
}
return true, ""
}

View File

@@ -0,0 +1,138 @@
/*
* Copyright (c) 2022 Alibaba Group Holding Ltd.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package config
import (
"github.com/stretchr/testify/assert"
regexp "github.com/wasilibs/go-re2"
"log"
"testing"
)
func toRegexMatch(regexs []string) []*regexp.Regexp {
re := make([]*regexp.Regexp, 0)
for _, regex := range regexs {
c, err := regexp.Compile(regex)
if err != nil {
log.Default().Fatal(err.Error())
}
re = append(re, c)
}
return re
}
func TestBotDetectConfig_ProcessTest(t *testing.T) {
tests := []struct {
name string
ua string
allow []string
deny []string
blockCode uint32
blockMessage string
want bool
}{
{
"test empty bot detect",
"",
[]string{},
[]string{},
401,
"bot has been blocked",
false,
},
{
"test default bot detect",
"Ant-Tailsweep-1",
[]string{},
[]string{},
401,
"bot has been blocked",
false,
},
{
"test default bot detect",
"indexer/1.2",
[]string{},
[]string{},
401,
"bot has been blocked",
false,
},
{
"test default bot detect",
"indexer/1.1.0",
[]string{},
[]string{},
401,
"bot has been blocked",
false,
},
{
"test default bot detect",
"YottaaMonitor",
[]string{},
[]string{},
401,
"bot has been blocked",
false,
},
{
"test allow bot detect",
"BaiduMobaider",
[]string{"BaiduMobaider"},
[]string{},
401,
"bot has been blocked",
true,
},
{
"test deny bot detect",
"Chrome",
[]string{},
[]string{"Chrome"},
401,
"bot has been blocked",
false,
},
{
"test allow and deny bot detect",
"SameBotDetect",
[]string{"SameBotDetect"},
[]string{"SameBotDetect"},
401,
"bot has been blocked",
true,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
bdc := BotDetectConfig{
BlockedCode: test.blockCode,
BlockedMessage: test.blockMessage,
Allow: toRegexMatch(test.allow),
Deny: toRegexMatch(test.deny),
}
actual, _ := bdc.Process(test.ua)
assert.Equal(t, test.want, actual, "")
})
}
}

View File

@@ -0,0 +1,23 @@
module bot-detect
go 1.19
require (
github.com/alibaba/higress/plugins/wasm-go v1.3.2
github.com/stretchr/testify v1.8.0
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c
github.com/tidwall/gjson v1.14.3
github.com/wasilibs/go-re2 v1.4.1
)
require (
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520 // indirect
github.com/magefile/mage v1.15.0 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/tetratelabs/wazero v1.6.0 // indirect
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

View File

@@ -0,0 +1,36 @@
github.com/alibaba/higress/plugins/wasm-go v1.3.2 h1:OKFo9zK7PFxvtSq9TmT8TwI6xqmNq5LZXfDBqPLOgkw=
github.com/alibaba/higress/plugins/wasm-go v1.3.2/go.mod h1:WZ/68vwe8qWhusa6C4/gMwUqas0jvHWSOa1bp8iK8F4=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520 h1:IHDghbGQ2DTIXHBHxWfqCYQW1fKjyJ/I7W1pMyUDeEA=
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520/go.mod h1:Nz8ORLaFiLWotg6GeKlJMhv8cci8mM43uEnLA5t8iew=
github.com/magefile/mage v1.15.0 h1:BvGheCMAsG3bWUDbZ8AyXXpCNwU9u5CB6sM+HNb9HYg=
github.com/magefile/mage v1.15.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c h1:OCUFXVGixHLfNjg6/QYEhv+jHJ5mRGhpEUVFv9eWPJE=
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c/go.mod h1:5t/pWFNJ9eMyu/K/Z+OeGhDJ9sN9eCo8fc2pyM/Qjg4=
github.com/tetratelabs/wazero v1.6.0 h1:z0H1iikCdP8t+q341xqepY4EWvHEw8Es7tlqiVzlP3g=
github.com/tetratelabs/wazero v1.6.0/go.mod h1:0U0G41+ochRKoPKCJlh0jMg1CHkyfK8kDqiirMmKY8A=
github.com/tidwall/gjson v1.14.3 h1:9jvXn7olKEHU1S9vwoMGliaT8jq1vJ7IH/n9zD9Dnlw=
github.com/tidwall/gjson v1.14.3/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
github.com/tidwall/pretty v1.2.0 h1:RWIZEg2iJ8/g6fDDYzMpobmaoGh5OLl4AXtGUGPcqCs=
github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
github.com/wasilibs/go-re2 v1.4.1 h1:E5+9O1M8UoGeqLB2A9omeoaWImqpuYDs9cKwvTJq/Oo=
github.com/wasilibs/go-re2 v1.4.1/go.mod h1:ynB8eCwd9JsqUnsk8WlPDk6cEeme8BguZmnqOSURE4Y=
github.com/wasilibs/nottinygc v0.4.0 h1:h1TJMihMC4neN6Zq+WKpLxgd9xCFMw7O9ETLwY2exJQ=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

View File

@@ -0,0 +1,104 @@
/*
* Copyright (c) 2022 Alibaba Group Holding Ltd.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package main
import (
"bot-detect/config"
"github.com/alibaba/higress/plugins/wasm-go/pkg/wrapper"
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
"github.com/tidwall/gjson"
regexp "github.com/wasilibs/go-re2"
)
func main() {
wrapper.SetCtx(
"bot-detect",
wrapper.ParseConfigBy(parseConfig),
wrapper.ProcessRequestHeadersBy(onHttpRequestHeaders),
)
}
func parseConfig(json gjson.Result, botDetectConfig *config.BotDetectConfig, log wrapper.Log) error {
log.Debug("parseConfig()")
if json.Get("blocked_code").Exists() {
botDetectConfig.BlockedCode = uint32(int(json.Get("blocked_code").Int()))
}
if json.Get("blocked_message").Exists() {
botDetectConfig.BlockedMessage = json.Get("blocked_message").String()
}
allowRules := make([]gjson.Result, 0)
denyRules := make([]gjson.Result, 0)
allowRulesValue := json.Get("allow")
if allowRulesValue.Exists() && allowRulesValue.IsArray() {
allowRules = json.Get("allow").Array()
}
denyRulesValue := json.Get("deny")
if denyRulesValue.Exists() && denyRulesValue.IsArray() {
denyRules = json.Get("deny").Array()
}
for _, allowRule := range allowRules {
c, err := regexp.Compile(allowRule.String())
if err != nil {
return err
}
botDetectConfig.Allow = append(botDetectConfig.Allow, c)
}
for _, denyRule := range denyRules {
c, err := regexp.Compile(denyRule.String())
if err != nil {
return err
}
botDetectConfig.Deny = append(botDetectConfig.Deny, c)
}
// Fill default values
botDetectConfig.FillDefaultValue()
log.Debugf("botDetectConfig:%+v", botDetectConfig)
return nil
}
func onHttpRequestHeaders(ctx wrapper.HttpContext, botDetectConfig config.BotDetectConfig, log wrapper.Log) types.Action {
log.Debug("onHttpRequestHeaders()")
//// Get user-agent header
ua, err := proxywasm.GetHttpRequestHeader("user-agent")
if err != nil {
log.Warnf("failed to get user-agent: %v", err)
return types.ActionPause
}
host := ctx.Host()
scheme := ctx.Scheme()
path := ctx.Path()
method := ctx.Method()
if ok, rule := botDetectConfig.Process(ua); !ok {
proxywasm.SendHttpResponse(botDetectConfig.BlockedCode, nil, []byte(botDetectConfig.BlockedMessage), -1)
log.Debugf("scheme:%s, host:%s, method:%s, path:%s user-agent:%s has been blocked by rule:%s", scheme, host, method, path, ua, rule)
return types.ActionPause
}
log.Debugf("scheme:%s, host:%s, method:%s, path:%s user-agent:%s has been passed", scheme, host, method, path, ua)
return types.ActionContinue
}

View File

@@ -5,7 +5,7 @@ go 1.18
replace github.com/alibaba/higress/plugins/wasm-go => ../..
require (
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0
github.com/tidwall/gjson v1.14.3
)

View File

@@ -7,8 +7,8 @@ github.com/magefile/mage v1.14.0 h1:6QDX3g6z1YvJ4olPhT1wksUcSa/V0a1B+pJb73fBjyo=
github.com/magefile/mage v1.14.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c h1:OCUFXVGixHLfNjg6/QYEhv+jHJ5mRGhpEUVFv9eWPJE=
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c/go.mod h1:5t/pWFNJ9eMyu/K/Z+OeGhDJ9sN9eCo8fc2pyM/Qjg4=
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0 h1:kS7BvMKN+FiptV4pfwiNX8e3q14evxAWkhYbxt8EI1M=
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0/go.mod h1:qkW5MBz2jch2u8bS59wws65WC+Gtx3x0aPUX5JL7CXI=
github.com/tidwall/gjson v1.14.3 h1:9jvXn7olKEHU1S9vwoMGliaT8jq1vJ7IH/n9zD9Dnlw=
github.com/tidwall/gjson v1.14.3/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=

View File

@@ -6,7 +6,7 @@ replace github.com/alibaba/higress/plugins/wasm-go => ../..
require (
github.com/alibaba/higress/plugins/wasm-go v0.0.0
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0
)
require (

View File

@@ -7,8 +7,8 @@ github.com/magefile/mage v1.14.0 h1:6QDX3g6z1YvJ4olPhT1wksUcSa/V0a1B+pJb73fBjyo=
github.com/magefile/mage v1.14.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c h1:OCUFXVGixHLfNjg6/QYEhv+jHJ5mRGhpEUVFv9eWPJE=
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c/go.mod h1:5t/pWFNJ9eMyu/K/Z+OeGhDJ9sN9eCo8fc2pyM/Qjg4=
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0 h1:kS7BvMKN+FiptV4pfwiNX8e3q14evxAWkhYbxt8EI1M=
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0/go.mod h1:qkW5MBz2jch2u8bS59wws65WC+Gtx3x0aPUX5JL7CXI=
github.com/tidwall/gjson v1.14.3 h1:9jvXn7olKEHU1S9vwoMGliaT8jq1vJ7IH/n9zD9Dnlw=
github.com/tidwall/gjson v1.14.3/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=

View File

@@ -6,7 +6,7 @@ replace github.com/alibaba/higress/plugins/wasm-go => ../..
require (
github.com/alibaba/higress/plugins/wasm-go v0.0.0
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0
github.com/tidwall/gjson v1.14.3
)

View File

@@ -7,8 +7,8 @@ github.com/magefile/mage v1.14.0 h1:6QDX3g6z1YvJ4olPhT1wksUcSa/V0a1B+pJb73fBjyo=
github.com/magefile/mage v1.14.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c h1:OCUFXVGixHLfNjg6/QYEhv+jHJ5mRGhpEUVFv9eWPJE=
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c/go.mod h1:5t/pWFNJ9eMyu/K/Z+OeGhDJ9sN9eCo8fc2pyM/Qjg4=
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0 h1:kS7BvMKN+FiptV4pfwiNX8e3q14evxAWkhYbxt8EI1M=
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0/go.mod h1:qkW5MBz2jch2u8bS59wws65WC+Gtx3x0aPUX5JL7CXI=
github.com/tidwall/gjson v1.14.3 h1:9jvXn7olKEHU1S9vwoMGliaT8jq1vJ7IH/n9zD9Dnlw=
github.com/tidwall/gjson v1.14.3/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=

View File

@@ -5,7 +5,7 @@ go 1.19
require (
github.com/alibaba/higress/plugins/wasm-go v0.0.0-20230807053545-d307d0e755f1
github.com/go-jose/go-jose/v3 v3.0.0
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0
github.com/tidwall/gjson v1.14.3
golang.org/x/oauth2 v0.11.0
)

View File

@@ -20,8 +20,8 @@ github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZN
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c h1:OCUFXVGixHLfNjg6/QYEhv+jHJ5mRGhpEUVFv9eWPJE=
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c/go.mod h1:5t/pWFNJ9eMyu/K/Z+OeGhDJ9sN9eCo8fc2pyM/Qjg4=
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0 h1:kS7BvMKN+FiptV4pfwiNX8e3q14evxAWkhYbxt8EI1M=
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0/go.mod h1:qkW5MBz2jch2u8bS59wws65WC+Gtx3x0aPUX5JL7CXI=
github.com/tidwall/gjson v1.14.3 h1:9jvXn7olKEHU1S9vwoMGliaT8jq1vJ7IH/n9zD9Dnlw=
github.com/tidwall/gjson v1.14.3/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=

View File

@@ -0,0 +1,101 @@
# 功能说明
该插件实现了 `OPA` 策略控制
# 该教程使用k8s[k8s配置文件](../../../../test/e2e/conformance/tests/go-wasm-opa.yaml)
支持client `k8s,nacos,ip,route` 策略去访问
## 配置字段
| 字段 | 数据类型 | 填写要求 | 默认值 | 描述 |
|---------------|--------|------|-----|--------------------------------------|
| policy | string | 必填 | - | opa 策略 |
| timeout | string | 必填 | - | 访问超时时间设置 |
| serviceSource | string | 必填 | - | k8s,nacos,ip,route |
| host | string | 非必填 | - | 服务主机serviceSource为`ip`必填) |
| serviceName | string | 非必填 | - | 服务名称serviceSource为`k8s,nacos,ip`必填) |
| servicePort | string | 非必填 | - | 服务端口serviceSource为`k8s,nacos,ip`必填) |
| namespace | string | 非必填 | - | 服务端口serviceSource为`k8s,nacos`必填) |
这是一个用于OPA认证配置的表格确保在提供所有必要的信息时遵循上述指导。
## 配置示例
```yaml
serviceSource: k8s
serviceName: opa
servicePort: 8181
namespace: higress-backend
policy: example1
timeout: 5s
```
# 在宿主机上执行OPA的流程
## 启动opa服务
```shell
docker run -d --name opa -p 8181:8181 openpolicyagent/opa:0.35.0 run -s
```
## 创建opa策略
```shell
curl -X PUT '127.0.0.1:8181/v1/policies/example1' \
-H 'Content-Type: text/plain' \
-d 'package example1
import input.request
default allow = false
allow {
# HTTP method must GET
request.method == "GET"
}'
```
## 查询策略
```shell
curl -X POST '127.0.0.1:8181/v1/data/example1/allow' \
-H 'Content-Type: application/json' \
-d '{"input":{"request":{"method":"GET"}}}'
```
# 测试插件
## 打包 WASM 插件
> 在 `wasm-go` 目录下把Dockerfile文件改成`PLUGIN_NAME=opa`,然后执行以下命令
```shell
docker build -t build-wasm-opa --build-arg GOPROXY=https://goproxy.cn,direct --platform=linux/amd64 .
```
## 拷贝插件
> 在当前的目录执行以下命令,将插件拷贝当前的目录
```shell
docker cp wasm-opa:/plugin.wasm .
```
## 运行插件
> 运行前修改envoy.yaml 这两个字段 `OPA_SERVER` `OPA_PORT` 替换宿主机上的IP和端口
```shell
docker compose up
```
## 使用curl测试插件
```shell
curl http://127.0.0.1:10000/get -X GET -v
```
```shell
curl http://127.0.0.1:10000/get -X POST -v
```

View File

@@ -0,0 +1 @@
1.0.0

View File

@@ -0,0 +1,82 @@
// Copyright (c) 2022 Alibaba Group Holding Ltd.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"errors"
"strings"
"github.com/alibaba/higress/plugins/wasm-go/pkg/wrapper"
"github.com/tidwall/gjson"
)
type OpaConfig struct {
policy string
timeout uint32
client wrapper.HttpClient
}
func Client(json gjson.Result) (wrapper.HttpClient, error) {
serviceSource := strings.TrimSpace(json.Get("serviceSource").String())
serviceName := strings.TrimSpace(json.Get("serviceName").String())
servicePort := json.Get("servicePort").Int()
host := strings.TrimSpace(json.Get("host").String())
if host == "" {
if serviceName == "" || servicePort == 0 {
return nil, errors.New("invalid service config")
}
}
var namespace string
if serviceSource == "k8s" || serviceSource == "nacos" {
if namespace = strings.TrimSpace(json.Get("namespace").String()); namespace == "" {
return nil, errors.New("namespace not allow empty")
}
}
switch serviceSource {
case "k8s":
return wrapper.NewClusterClient(wrapper.K8sCluster{
ServiceName: serviceName,
Namespace: namespace,
Port: servicePort,
}), nil
case "nacos":
return wrapper.NewClusterClient(wrapper.NacosCluster{
ServiceName: serviceName,
NamespaceID: namespace,
Port: servicePort,
}), nil
case "ip":
return wrapper.NewClusterClient(wrapper.StaticIpCluster{
ServiceName: serviceName,
Host: host,
Port: servicePort,
}), nil
case "dns":
return wrapper.NewClusterClient(wrapper.DnsCluster{
ServiceName: serviceName,
Port: servicePort,
Domain: json.Get("domain").String(),
}), nil
case "route":
return wrapper.NewClusterClient(wrapper.RouteCluster{
Host: host,
}), nil
}
return nil, errors.New("unknown service source: " + serviceSource)
}

View File

@@ -0,0 +1,50 @@
// Copyright (c) 2022 Alibaba Group Holding Ltd.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"testing"
"github.com/alibaba/higress/plugins/wasm-go/pkg/wrapper"
"github.com/stretchr/testify/assert"
"github.com/tidwall/gjson"
)
func TestConfig(t *testing.T) {
json := gjson.Result{Type: gjson.JSON, Raw: `{"serviceSource": "k8s","serviceName": "opa","servicePort": 8181,"namespace": "example1","policy": "example1","timeout": "5s"}`}
config := &OpaConfig{}
assert.NoError(t, parseConfig(json, config, wrapper.Log{}))
assert.Equal(t, config.policy, "example1")
assert.Equal(t, config.timeout, uint32(5000))
assert.NotNil(t, config.client)
type tt struct {
raw string
result bool
}
tests := []tt{
{raw: `{}`, result: false},
{raw: `{"policy": "example1","timeout": "5s"}`, result: false},
{raw: `{"serviceSource": "route","host": "example.com","policy": "example1","timeout": "5s"}`, result: true},
{raw: `{"serviceSource": "nacos","serviceName": "opa","servicePort": 8181,"policy": "example1","timeout": "5s"}`, result: false},
{raw: `{"serviceSource": "nacos","serviceName": "opa","servicePort": 8181,"namespace": "example1","policy": "example1","timeout": "5s"}`, result: true},
}
for _, test := range tests {
json = gjson.Result{Type: gjson.JSON, Raw: test.raw}
assert.Equal(t, parseConfig(json, config, wrapper.Log{}) == nil, test.result)
}
}

View File

@@ -0,0 +1,16 @@
version: '3.7'
services:
envoy:
image: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/gateway:1.3.1
entrypoint: /usr/local/bin/envoy
command: -c /etc/envoy/envoy.yaml --component-log-level wasm:debug
networks:
- wasmtest
ports:
- "10000:10000"
volumes:
- ./envoy.yaml:/etc/envoy/envoy.yaml
- ./plugin.wasm:/etc/envoy/plugin.wasm
networks:
wasmtest: { }

View File

@@ -0,0 +1,69 @@
admin:
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 9901
static_resources:
listeners:
- name: listener_0
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: [ "*" ]
routes:
- match:
prefix: "/"
route:
cluster: opa-server
http_filters:
- name: wasmdemo
typed_config:
"@type": type.googleapis.com/udpa.type.v1.TypedStruct
type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
value:
config:
name: wasmdemo
vm_config:
runtime: envoy.wasm.runtime.v8
code:
local:
filename: /etc/envoy/plugin.wasm
configuration:
"@type": "type.googleapis.com/google.protobuf.StringValue"
value: |
{
"serviceSource": "route",
"host": "OPA_SERVER:OPA_PORT",
"policy": "example1",
"timeout": "5s"
}
- name: envoy.filters.http.router
clusters:
- name: opa-server
connect_timeout: 0.5s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
dns_refresh_rate: 5s
dns_lookup_family: V4_ONLY
load_assignment:
cluster_name: opa-server
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: OPA_SERVER # opa server Host IP
port_value: OPA_PORT # opa server Host PORT

View File

@@ -0,0 +1,23 @@
module github.com/alibaba/higress/plugins/wasm-go/extensions/opa
go 1.19
require (
github.com/alibaba/higress/plugins/wasm-go v0.0.0
github.com/stretchr/testify v1.8.4
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0
github.com/tidwall/gjson v1.14.4
)
require (
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/google/uuid v1.5.0 // indirect
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520 // indirect
github.com/magefile/mage v1.14.0 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
replace github.com/alibaba/higress/plugins/wasm-go => ../..

View File

@@ -0,0 +1,24 @@
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/google/uuid v1.5.0 h1:1p67kYwdtXjb0gL0BPiP1Av9wiZPo5A8z2cWkTZ+eyU=
github.com/google/uuid v1.5.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520 h1:IHDghbGQ2DTIXHBHxWfqCYQW1fKjyJ/I7W1pMyUDeEA=
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520/go.mod h1:Nz8ORLaFiLWotg6GeKlJMhv8cci8mM43uEnLA5t8iew=
github.com/magefile/mage v1.14.0 h1:6QDX3g6z1YvJ4olPhT1wksUcSa/V0a1B+pJb73fBjyo=
github.com/magefile/mage v1.14.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0 h1:kS7BvMKN+FiptV4pfwiNX8e3q14evxAWkhYbxt8EI1M=
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0/go.mod h1:qkW5MBz2jch2u8bS59wws65WC+Gtx3x0aPUX5JL7CXI=
github.com/tidwall/gjson v1.14.4 h1:uo0p8EbA09J7RQaflQ1aBRffTR7xedD2bcIVSYxLnkM=
github.com/tidwall/gjson v1.14.4/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
github.com/tidwall/pretty v1.2.0 h1:RWIZEg2iJ8/g6fDDYzMpobmaoGh5OLl4AXtGUGPcqCs=
github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

View File

@@ -0,0 +1,134 @@
// Copyright (c) 2022 Alibaba Group Holding Ltd.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"encoding/json"
"errors"
"fmt"
"net/http"
"net/url"
"strings"
"time"
"github.com/alibaba/higress/plugins/wasm-go/pkg/wrapper"
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
"github.com/tidwall/gjson"
)
func main() {
wrapper.SetCtx(
"opa",
wrapper.ParseConfigBy(parseConfig),
wrapper.ProcessRequestHeadersBy(onHttpRequestHeaders),
wrapper.ProcessRequestBodyBy(onHttpRequestBody),
)
}
type Metadata struct {
Input map[string]interface{} `json:"input"`
}
func parseConfig(json gjson.Result, config *OpaConfig, log wrapper.Log) error {
policy := json.Get("policy").String()
if strings.TrimSpace(policy) == "" {
return errors.New("policy not allow empty")
}
timeout := json.Get("timeout").String()
if strings.TrimSpace(timeout) == "" {
return errors.New("timeout not allow empty")
}
duration, err := time.ParseDuration(timeout)
if err != nil {
return errors.New("timeout parse fail: " + err.Error())
}
var uint32Duration uint32
if duration.Milliseconds() > int64(^uint32(0)) {
} else {
uint32Duration = uint32(duration.Milliseconds())
}
config.timeout = uint32Duration
client, err := Client(json)
if err != nil {
return err
}
config.client = client
config.policy = policy
return nil
}
func onHttpRequestHeaders(ctx wrapper.HttpContext, config OpaConfig, log wrapper.Log) types.Action {
return opaCall(ctx, config, nil, log)
}
func onHttpRequestBody(ctx wrapper.HttpContext, config OpaConfig, body []byte, log wrapper.Log) types.Action {
return opaCall(ctx, config, body, log)
}
func opaCall(ctx wrapper.HttpContext, config OpaConfig, body []byte, log wrapper.Log) types.Action {
request := make(map[string]interface{}, 6)
headers, _ := proxywasm.GetHttpRequestHeaders()
request["method"] = ctx.Method()
request["scheme"] = ctx.Scheme()
request["path"] = ctx.Path()
request["headers"] = headers
if len(body) != 0 {
request["body"] = body
}
parse, _ := url.Parse(ctx.Path())
query, _ := url.ParseQuery(parse.RawQuery)
request["query"] = query
data, _ := json.Marshal(Metadata{Input: map[string]interface{}{"request": request}})
if err := config.client.Post(fmt.Sprintf("/v1/data/%s/allow", config.policy),
[][2]string{{"Content-Type", "application/json"}},
data, rspCall, config.timeout); err != nil {
log.Errorf("client opa fail %v", err)
return types.ActionPause
}
return types.ActionPause
}
func rspCall(statusCode int, _ http.Header, responseBody []byte) {
if statusCode != http.StatusOK {
proxywasm.SendHttpResponse(uint32(statusCode), nil, []byte("opa state not is 200"), -1)
return
}
var rsp map[string]interface{}
if err := json.Unmarshal(responseBody, &rsp); err != nil {
proxywasm.SendHttpResponse(http.StatusInternalServerError, nil, []byte(fmt.Sprintf("opa parse rsp fail %+v", err)), -1)
return
}
result, ok := rsp["result"].(bool)
if !ok {
proxywasm.SendHttpResponse(http.StatusInternalServerError, nil, []byte("rsp type conversion fail"), -1)
return
}
if !result {
proxywasm.SendHttpResponse(http.StatusUnauthorized, nil, []byte("opa server not allowed"), -1)
return
}
proxywasm.ResumeHttpRequest()
}

View File

@@ -6,7 +6,7 @@ replace github.com/alibaba/higress/plugins/wasm-go => ../..
require (
github.com/alibaba/higress/plugins/wasm-go v0.0.0
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0
github.com/tidwall/gjson v1.14.3
)

View File

@@ -7,8 +7,8 @@ github.com/magefile/mage v1.14.0 h1:6QDX3g6z1YvJ4olPhT1wksUcSa/V0a1B+pJb73fBjyo=
github.com/magefile/mage v1.14.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c h1:OCUFXVGixHLfNjg6/QYEhv+jHJ5mRGhpEUVFv9eWPJE=
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c/go.mod h1:5t/pWFNJ9eMyu/K/Z+OeGhDJ9sN9eCo8fc2pyM/Qjg4=
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0 h1:kS7BvMKN+FiptV4pfwiNX8e3q14evxAWkhYbxt8EI1M=
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0/go.mod h1:qkW5MBz2jch2u8bS59wws65WC+Gtx3x0aPUX5JL7CXI=
github.com/tidwall/gjson v1.14.3 h1:9jvXn7olKEHU1S9vwoMGliaT8jq1vJ7IH/n9zD9Dnlw=
github.com/tidwall/gjson v1.14.3/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=

View File

@@ -0,0 +1,146 @@
# 功能说明
`request-validation`插件用于提前验证向上游服务转发的请求。该插件使用`JSON Schema`机制进行数据验证可以验证请求的body及header数据。
# 配置字段
| 名称 | 数据类型 | 填写要求 | 默认值 | 描述 |
| -------- | -------- | -------- |-----| -------- |
|header_schema|object|选填| - |配置用于验证请求header的JSON Schema|
|body_schema|object|选填| - |配置用于验证请求body的JSON Schema|
|rejected_code|number|选填| 403 |配置请求被拒绝时返回的HTTP状态码|
|rejected_msg|string|选填| - |配置请求被拒绝时返回的HTTP应答Body|
|enable_swagger|bool|选填| false |配置是否开启swagger文档验证|
|enable_oas3|bool|选填| false |配置是否开启OAS3文档验证|
**校验规则对header和body是一样的下面以body为例说明**
# 配置示例
## 枚举Enum验证
```yaml
body_schema:
type: object
required:
- enum_payload
properties:
enum_payload:
type: string
enum:
- "enum_string_1"
- "enum_string_2"
default: "enum_string_1"
```
## 布尔Boolean验证
```yaml
body_schema:
type: object
required:
- boolean_payload
properties:
boolean_payload:
type: boolean
default: true
```
## 数字范围Number or Integer验证
```yaml
body_schema:
type: object
required:
- integer_payload
properties:
integer_payload:
type: integer
minimum: 1
maximum: 10
```
## 字符串长度String验证
```yaml
body_schema:
type: object
required:
- string_payload
properties:
string_payload:
type: string
minLength: 1
maxLength: 10
```
## 正则表达式Regex验证
```yaml
body_schema:
type: object
required:
- regex_payload
properties:
regex_payload:
type: string
minLength: 1
maxLength: 10
pattern: "^[a-zA-Z0-9_]+$"
```
## 数组Array验证
```yaml
body_schema:
type: object
required:
- array_payload
properties:
array_payload:
type: array
minItems: 1
items:
type: integer
minimum: 1
maximum: 10
uniqueItems: true
default: [1, 2, 3]
```
## 多字段组合Combined验证
```yaml
body_schema:
type: object
required:
- boolean_payload
- array_payload
- regex_payload
properties:
boolean_payload:
type: boolean
array_payload:
type: array
minItems: 1
items:
type: integer
minimum: 1
maximum: 10
uniqueItems: true
default: [1, 2, 3]
regex_payload:
type: string
minLength: 1
maxLength: 10
pattern: "^[a-zA-Z0-9_]+$"
```
## 自定义拒绝信息
```yaml
body_schema:
type: object
required:
- boolean_payload
properties:
boolean_payload:
type: boolean
rejected_code: 403
rejected_msg: "请求被拒绝"
```
# 本地调试
参考[使用 GO 语言开发 WASM 插件](https://higress.io/zh-cn/docs/user/wasm-go#%E4%B8%89%E6%9C%AC%E5%9C%B0%E8%B0%83%E8%AF%95)

View File

@@ -0,0 +1 @@
1.0.0

View File

@@ -0,0 +1,18 @@
module github.com/alibaba/higress/plugins/wasm-go/extensions/request-validation
go 1.19
require (
github.com/alibaba/higress/plugins/wasm-go v1.3.1
github.com/santhosh-tekuri/jsonschema v1.2.4
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0
github.com/tidwall/gjson v1.17.0
)
require (
github.com/google/uuid v1.3.0 // indirect
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520 // indirect
github.com/magefile/mage v1.14.0 // indirect
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.0 // indirect
)

View File

@@ -0,0 +1,22 @@
github.com/alibaba/higress/plugins/wasm-go v1.3.1 h1:d+t4W2NyqmqUz6DPZENflODfkLgdVlTfyso+nq0fSkg=
github.com/alibaba/higress/plugins/wasm-go v1.3.1/go.mod h1:WZ/68vwe8qWhusa6C4/gMwUqas0jvHWSOa1bp8iK8F4=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520 h1:IHDghbGQ2DTIXHBHxWfqCYQW1fKjyJ/I7W1pMyUDeEA=
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520/go.mod h1:Nz8ORLaFiLWotg6GeKlJMhv8cci8mM43uEnLA5t8iew=
github.com/magefile/mage v1.14.0 h1:6QDX3g6z1YvJ4olPhT1wksUcSa/V0a1B+pJb73fBjyo=
github.com/magefile/mage v1.14.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/santhosh-tekuri/jsonschema v1.2.4 h1:hNhW8e7t+H1vgY+1QeEQpveR6D4+OwKPXCfD2aieJis=
github.com/santhosh-tekuri/jsonschema v1.2.4/go.mod h1:TEAUOeZSmIxTTuHatJzrvARHiuO9LYd+cIxzgEHCQI4=
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0 h1:kS7BvMKN+FiptV4pfwiNX8e3q14evxAWkhYbxt8EI1M=
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0/go.mod h1:qkW5MBz2jch2u8bS59wws65WC+Gtx3x0aPUX5JL7CXI=
github.com/tidwall/gjson v1.17.0 h1:/Jocvlh98kcTfpN2+JzGQWQcqrPQwDrVEMApx/M5ZwM=
github.com/tidwall/gjson v1.17.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
github.com/tidwall/pretty v1.2.0 h1:RWIZEg2iJ8/g6fDDYzMpobmaoGh5OLl4AXtGUGPcqCs=
github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=

View File

@@ -0,0 +1,199 @@
// Copyright (c) 2022 Alibaba Group Holding Ltd.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"encoding/json"
"fmt"
"strings"
"github.com/alibaba/higress/plugins/wasm-go/pkg/wrapper"
"github.com/santhosh-tekuri/jsonschema"
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
"github.com/tidwall/gjson"
)
const (
defaultHeaderSchema = "header"
defaultBodySchema = "body"
defaultRejectedCode = 403
)
func main() {
wrapper.SetCtx(
"request-validation",
wrapper.ProcessRequestHeadersBy(onHttpRequestHeaders),
wrapper.ProcessRequestBodyBy(onHttpRequestBody),
wrapper.ParseConfigBy(parseConfig),
)
}
// Config is the config for request validation.
type Config struct {
// compiler is the compiler for json schema.
compiler *jsonschema.Compiler
// rejectedCode is the code for rejected request.
rejectedCode uint32
// rejectedMsg is the message for rejected request.
rejectedMsg string
// draft is the draft version of json schema.
draft *jsonschema.Draft
// enableBodySchema is the flag for enable body schema.
enableBodySchema bool
// enableHeaderSchema is the flag for enable header schema.
enableHeaderSchema bool
}
func parseConfig(result gjson.Result, config *Config, log wrapper.Log) error {
headerSchema := result.Get("header_schema").String()
bodySchema := result.Get("body_schema").String()
enableSwagger := result.Get("enable_swagger").Bool()
enableOas3 := result.Get("enable_oas3").Bool()
code := result.Get("rejected_code").Int()
msg := result.Get("rejected_msg").String()
// set config default value
config.enableBodySchema = false
config.enableHeaderSchema = false
// check enable_swagger and enable_oas3
if enableSwagger && enableOas3 {
return fmt.Errorf("enable_swagger and enable_oas3 can not be true at the same time")
}
// set draft version
if enableSwagger {
config.draft = jsonschema.Draft4
}
if enableOas3 {
config.draft = jsonschema.Draft7
}
if !enableSwagger && !enableOas3 {
config.draft = jsonschema.Draft7
}
// create compiler
compiler := jsonschema.NewCompiler()
compiler.Draft = config.draft
config.compiler = compiler
// add header schema to compiler
if headerSchema != "" {
err := config.compiler.AddResource(defaultHeaderSchema, strings.NewReader(headerSchema))
if err != nil {
return err
}
config.enableHeaderSchema = true
}
// add body schema to compiler
if bodySchema != "" {
err := config.compiler.AddResource(defaultBodySchema, strings.NewReader(bodySchema))
if err != nil {
return err
}
config.enableBodySchema = true
}
// check rejected_code is valid
if code != 0 && code > 100 && code < 600 {
config.rejectedCode = uint32(code)
} else {
config.rejectedCode = defaultRejectedCode
}
config.rejectedMsg = msg
return nil
}
func onHttpRequestHeaders(ctx wrapper.HttpContext, config Config, log wrapper.Log) types.Action {
if !config.enableHeaderSchema {
return types.ActionContinue
}
// get headers
headers, err := proxywasm.GetHttpRequestHeaders()
if err != nil {
log.Errorf("get request headers failed: %v", err)
return types.ActionContinue
}
// covert to schema
schema := make(map[string]interface{})
for _, header := range headers {
schema[header[0]] = header[1]
}
// convert to json string
schemaBytes, err := json.Marshal(schema)
if err != nil {
log.Errorf("marshal schema failed: %v", err)
return types.ActionContinue
}
// validate
document := strings.NewReader(string(schemaBytes))
compile, err := config.compiler.Compile(defaultHeaderSchema)
if err != nil {
log.Errorf("compile schema failed: %v", err)
return types.ActionContinue
}
err = compile.Validate(document)
if err != nil {
log.Errorf("validate request headers failed: %v", err)
proxywasm.SendHttpResponse(config.rejectedCode, nil, []byte(config.rejectedMsg), -1)
return types.ActionPause
}
return types.ActionContinue
}
func onHttpRequestBody(ctx wrapper.HttpContext, config Config, body []byte, log wrapper.Log) types.Action {
if !config.enableBodySchema {
return types.ActionContinue
}
// covert to schema
schema := make(map[string]interface{})
err := json.Unmarshal(body, &schema)
if err != nil {
log.Errorf("unmarshal body failed: %v", err)
return types.ActionContinue
}
// convert to json string
schemaBytes, err := json.Marshal(schema)
if err != nil {
log.Errorf("marshal schema failed: %v", err)
return types.ActionContinue
}
// validate
document := strings.NewReader(string(schemaBytes))
compile, err := config.compiler.Compile(defaultBodySchema)
if err != nil {
log.Errorf("compile schema failed: %v", err)
return types.ActionContinue
}
err = compile.Validate(document)
if err != nil {
log.Errorf("validate request body failed: %v", err)
proxywasm.SendHttpResponse(config.rejectedCode, nil, []byte(config.rejectedMsg), -1)
return types.ActionPause
}
return types.ActionContinue
}

View File

@@ -5,15 +5,16 @@
# 配置字段
| 名称 | 数据类型 | 填写要求 | 默认值 | 描述 |
| -------- | -------- | -------- | -------- | -------- |
| type | string | 填,可选值为 `request`, `response` | - | 指定转换器类型 |
| rules | array of object | 选填 | - | 指定转换操作类型以及请求/响应头、请求查询参数、请求/响应体参数的转换规则 |
| :----: | :----: | :----: | :----: | -------- |
| reqRules | string | 填,reqRules和respRules至少填一个 | - | 请求转换器配置,指定转换操作类型以及请求头、请求查询参数、请求体的转换规则 |
| respRules | string | 选填reqRules和respRules至少填一个 | - | 响应转换器配置,指定转换操作类型以及响应头、响应体的转换规则 |
`rules`中每一项的配置字段说明如下:
`reqRules``respRules`中每一项的配置字段说明如下:
| 名称 | 数据类型 | 填写要求 | 默认值 | 描述 |
| -------- | -------- | -------- | -------- | -------- |
| :----: | :----: | :----: | :----: | -------- |
| operate | string | 必填,可选值为 `remove`, `rename`, `replace`, `add`, `append`, `map`, `dedupe` | - | 指定转换操作类型,支持的操作类型有删除 (remove)、重命名 (rename)、更新 (replace)、添加 (add)、追加 (append)、映射 (map)、去重 (dedupe),当存在多项不同类型的转换规则时,按照上述操作类型顺序依次执行 |
| mapSource | string | 选填,可选值为`headers`, `querys`,`body` | - | 仅在operate为`map`时有效。指定映射来源,若不填该字段,则默认映射来源为自身 |
| headers | array of object | 选填 | - | 指定请求/响应头转换规则 |
| querys | array of object | 选填 | - | 指定请求查询参数转换规则 |
| body | array of object | 选填 | - | 指定请求/响应体参数转换规则,请求体转换允许 content-type 为 `application/json`, `application/x-www-form-urlencoded`, `multipart/form-data`;响应体转换仅允许 content-type 为 `application/json` |
@@ -21,9 +22,20 @@
`headers`, `querys`, `body`中每一项的配置字段说明如下:
| 名称 | 数据类型 | 填写要求 | 默认值 | 描述 |
| -------- | -------- | -------- | -------- |---------------------------------------------------|
| key | string | 选填 | - | 指定键,详见[转换操作类型](#转换操作类型) |
| value | string | 选填 | - | 指定值,详见[转换操作类型](#转换操作类型) |
| :----: | :----: | :----: | -------- |---------------------------------------------------|
| key | string | 选填 | - | 在operate为`remove`时使用,用法详见[转换操作类型](#转换操作类型) |
| oldKey | string | 选填 | - |在operate为`rename`时使用,用法详见[转换操作类型](#转换操作类型) |
| newKey | string | 选填 | - | 在operate为`rename`时使用,用法详见[转换操作类型](#转换操作类型) |
| key | string | 选填 | - | 在operate为`replace`时使用,用法详见[转换操作类型](#转换操作类型) |
| newValue | string | 选填 | - | 在operate为`replace`时使用,用法详见[转换操作类型](#转换操作类型) |
| key | string | 选填 | - | 在operate为`add`时使用,用法详见[转换操作类型](#转换操作类型) |
| value | string | 选填 | - | 在operate为`add`时使用,用法详见[转换操作类型](#转换操作类型) |
| key | string | 选填 | - | 在operate为`append`时使用,用法详见[转换操作类型](#转换操作类型) |
| appendValue | string | 选填 | - | 在operate为`append`时使用,用法详见[转换操作类型](#转换操作类型) |
| fromKey | string | 选填 | - | 在operate为`map`时使用,用法详见[转换操作类型](#转换操作类型) |
| toKey | string | 选填 | - | 在operate为`map`时使用,用法详见[转换操作类型](#转换操作类型) |
| key | string | 选填 | - | 在operate为`dedupe`时使用,用法详见[转换操作类型](#转换操作类型) |
| strategy | string | 选填 | - | 在operate为`dedupe`时使用,用法详见[转换操作类型](#转换操作类型) |
| value_type | string | 选填,可选值为 `object`, `boolean`, `number`, `string` | string | 当`content-type: application/json`时,该字段指定请求/响应体参数的值类型 |
| host_pattern | string | 选填 | - | 指定请求主机名匹配规则,当转换操作类型为 `replace`, `add`, `append` 时有效 |
| path_pattern | string | 选填 | - | 指定请求路径匹配规则,当转换操作类型为 `replace`, `add`, `append` 时有效 |
@@ -32,8 +44,8 @@
* `request transformer` 支持以下转换对象请求头部、请求查询参数、请求体application/json, application/x-www-form-urlencoded, multipart/form-data
* `response transformer` 支持以下转换对象响应头部、响应体application/json
* 转换操作类型的执行顺序remove → rename → replace → add → append → map → dedupe
* 插件支持双向转换能力,即单个插件能够完成对请求和响应都做转换
* 转换操作类型的执行顺序,为配置文件中编写的顺序,如remove → rename → replace → add → append → map → dedupe或者dedupe → map → append → add → replace → rename → remove等
* 当转换对象为 headers 时,` key` 不区分大小写;当为 headers 且为 `rename`, `map` 操作时,`value` 也不区分大小写(因为此时该字段具有 key 含义);而 querys 和 body 的 `key`, `value` 字段均区分大小写
* `value_type` 仅对 content-type 为 application/json 的请求/响应体有效
* `host_pattern``path_pathern` 支持 [RE2 语法](https://pkg.go.dev/regexp/syntax),仅对 `replace`, `add`, `append` 操作有效,且在一项转换规则中两者只能选填其一,若均填写,则 `host_pattern` 生效,而 `path_pattern` 失效
@@ -43,7 +55,7 @@
# 转换操作类型
| 操作类型 | key 字段含义 | value 字段含义 | 描述 |
| ------------- | ----------------- |-----| ------------------------------------------------------------ |
| :----: | :----: | :----: | ------------------------------------------------------------ |
| 删除 remove | 目标 key |无需设置| 若存在指定的 `key`,则删除;否则无操作 |
| 重命名 rename | 目标 oldKey |新的 key 名称 newKey| 若存在指定的 `oldKey:value`,则将其键名重命名为 `newKey`,得到 `newKey:value`;否则无操作 |
| 更新 replace | 目标 key |新的 value 值 newValue| 若存在指定的 `key:value`,则将其 value 更新为 `newValue`,得到 `key:newValue`;否则无操作 |
@@ -62,19 +74,18 @@
### 转换请求头部
```yaml
type: request
rules:
reqRules:
- operate: remove
headers:
- key: X-remove
- operate: rename
headers:
- key: X-not-renamed
value: X-renamed
- oldKey: X-not-renamed
newKey: X-renamed
- operate: replace
headers:
- key: X-replace
value: replaced
newValue: replaced
- operate: add
headers:
- key: X-add-append
@@ -83,20 +94,20 @@ rules:
- operate: append
headers:
- key: X-add-append
value: path-$1
appendValue: path-$1
path_pattern: ^.*?\/(\w+)[\?]{0,1}.*$
- operate: map
headers:
- key: X-add-append
value: X-map
- fromKey: X-add-append
toKey: X-map
- operate: dedupe
headers:
- key: X-dedupe-first
value: RETAIN_FIRST
strategy: RETAIN_FIRST
- key: X-dedupe-last
value: RETAIN_LAST
strategy: RETAIN_LAST
- key: X-dedupe-unique
value: RETAIN_UNIQUE
strategy: RETAIN_UNIQUE
```
发送请求
@@ -131,19 +142,18 @@ $ curl -v console.higress.io/get -H 'host: foo.bar.com' \
### 转换请求查询参数
```yaml
type: request
rules:
reqRules:
- operate: remove
querys:
- key: k1
- operate: rename
querys:
- key: k2
value: k2-new
- oldKey: k2
newKey: k2-new
- operate: replace
querys:
- key: k2-new
value: v2-new
newValue: v2-new
- operate: add
querys:
- key: k3
@@ -152,15 +162,15 @@ rules:
- operate: append
querys:
- key: k3
value: v32
appendValue: v32
- operate: map
querys:
- key: k3
value: k4
- fromKey: k3
toKey: k4
- operate: dedupe
querys:
- key: k4
value: RETAIN_FIRST
strategy: RETAIN_FIRST
```
发送请求
@@ -186,19 +196,18 @@ $ curl -v "console.higress.io/get?k1=v11&k1=v12&k2=v2"
### 转换请求体
```yaml
type: request
rules:
reqRules:
- operate: remove
body:
- key: a1
- operate: rename
body:
- key: a2
value: a2-new
- oldKey: a2
newKey: a2-new
- operate: replace
body:
- key: a3
value: t3-new
newValue: t3-new
value_type: string
- operate: add
body:
@@ -208,17 +217,17 @@ rules:
- operate: append
body:
- key: a1-new
value: t1-$1-append
appendValue: t1-$1-append
value_type: string
host_pattern: ^(.*)\.com$
- operate: map
body:
- key: a1-new
value: a4
- fromKey: a1-new
toKey: a4
- operate: dedupe
body:
- key: a4
value: RETAIN_FIRST
strategy: RETAIN_FIRST
```
发送请求:
@@ -313,8 +322,7 @@ $ curl -v -X POST console.higress.io/post -H 'host: foo.bar.com' \
1.通常情况下,指定的 key 中含有 `.` 表示嵌套含义,如下:
```yaml
type: response
rules:
respRules:
- operate: add
body:
- key: foo.bar
@@ -339,8 +347,7 @@ $ curl -v console.higress.io/get
> 当使用双引号括住字符串时使用 `\\.` 进行转义
```yaml
type: response
rules:
respRules:
- operate: add
body:
- key: foo\.bar
@@ -378,8 +385,7 @@ $ curl -v console.higress.io/get
1.移除 `user` 第一个元素:
```yaml
type: request
rules:
reqRules:
- operate: remove
body:
- key: users.0
@@ -409,12 +415,11 @@ $ curl -v -X POST console.higress.io/post \
2.将 `users` 第一个元素的 key 为 `123` 重命名为 `msg`:
```yaml
type: request
rules:
reqRules:
- operate: rename
body:
- key: users.0.123
value: users.0.first
- oldKey: users.0.123
newKey: users.0.first
```
```bash
@@ -466,12 +471,11 @@ $ curl -v -X POST console.higress.io/post \
```
```yaml
type: request
rules:
reqRules:
- operate: replace
body:
- key: users.#.age
value: 20
newValue: 20
```
```bash

View File

File diff suppressed because it is too large Load Diff

View File

@@ -59,6 +59,15 @@ func isValidOperation(op string) bool {
}
}
func isValidMapSource(source string) bool {
switch source {
case "headers", "querys", "body":
return true
default:
return false
}
}
func parseQueryByPath(path string) (map[string][]string, error) {
u, err := url.Parse(path)
if err != nil {

View File

@@ -2,12 +2,12 @@ package wasmplugin
import (
"errors"
"github.com/corazawaf/coraza/v3/debuglog"
"strconv"
"strings"
"github.com/alibaba/higress/plugins/wasm-go/pkg/wrapper"
"github.com/corazawaf/coraza/v3"
"github.com/corazawaf/coraza/v3/debuglog"
ctypes "github.com/corazawaf/coraza/v3/types"
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
@@ -76,6 +76,15 @@ func onHttpRequestHeaders(ctx wrapper.HttpContext, config WafConfig, log wrapper
tx := ctx.GetContext("tx").(ctypes.Transaction)
protocol, err := proxywasm.GetProperty([]string{"request", "protocol"})
if err != nil {
// TODO(anuraaga): HTTP protocol is commonly required in WAF rules, we should probably
// fail fast here, but proxytest does not support properties yet.
protocol = []byte("HTTP/2.0")
}
ctx.SetContext("httpProtocol", string(protocol))
// Note the pseudo-header :path includes the query.
// See https://httpwg.org/specs/rfc9113.html#rfc.section.8.3.1
uri, err := proxywasm.GetHttpRequestHeader(":path")
@@ -103,15 +112,6 @@ func onHttpRequestHeaders(ctx wrapper.HttpContext, config WafConfig, log wrapper
return types.ActionContinue
}
protocol, err := proxywasm.GetProperty([]string{"request", "protocol"})
if err != nil {
// TODO(anuraaga): HTTP protocol is commonly required in WAF rules, we should probably
// fail fast here, but proxytest does not support properties yet.
protocol = []byte("HTTP/2.0")
}
ctx.SetContext("httpProtocol", string(protocol))
tx.ProcessURI(uri, method, string(protocol))
hs, err := proxywasm.GetHttpRequestHeaders()

View File

@@ -116,7 +116,7 @@ func (p *NacosAddressProvider) GetNacosAddress(oldAddress string) <-chan string
var addr string
p.cond.L.Lock()
defer p.cond.L.Unlock()
log.Debugf("get nacos address, p.nacosAddr, oldAddress", p.nacosAddr, oldAddress)
log.Debugf("get nacos address, p.nacosAddr: %s, oldAddress: %s", p.nacosAddr, oldAddress)
for p.nacosAddr == oldAddress || p.nacosAddr == "" {
if p.isStop.Load() {
return

View File

@@ -26,9 +26,7 @@ import (
"github.com/nacos-group/nacos-sdk-go/model"
"github.com/nacos-group/nacos-sdk-go/vo"
"istio.io/api/networking/v1alpha3"
versionedclient "istio.io/client-go/pkg/clientset/versioned"
"istio.io/pkg/log"
ctrl "sigs.k8s.io/controller-runtime"
apiv1 "github.com/alibaba/higress/api/networking/v1"
"github.com/alibaba/higress/pkg/common"
@@ -61,7 +59,6 @@ type watcher struct {
cache memory.Cache
mutex *sync.Mutex
stop chan struct{}
client *versionedclient.Clientset
isStop bool
updateCacheWhenEmpty bool
authOption provider.AuthOption
@@ -79,18 +76,6 @@ func NewWatcher(cache memory.Cache, opts ...WatcherOption) (provider.Watcher, er
stop: make(chan struct{}),
}
config, err := ctrl.GetConfig()
if err != nil {
return nil, err
}
ic, err := versionedclient.NewForConfig(config)
if err != nil {
log.Errorf("can not new istio client, err:%v", err)
return nil, err
}
w.client = ic
w.NacosRefreshInterval = int64(DefaultRefreshInterval)
for _, opt := range opts {

View File

@@ -43,6 +43,26 @@ And after that, you should add your defined HigressConformanceTest to `e2e_test.
You can understand it quickly just by looking at codes in `test/ingress/conformance/tests/httproute-simple-same-namespace.go` and `test/ingress/conformance/tests/httproute-simple-same-namespace.yaml`, and try to write one.
### How to Implement Test Environment Reusability
The test environment reusability is primarily achieved through the following targets in the Makefile:
1. **make higress-conformance-test:** Used to run the entire Conformance testing process, including setting up the test environment, executing test cases, and cleaning up the test environment.
- **make higress-conformance-test-prepare:** Can be used to set up the environment for deployments such as higress-controller, higress-gateway, etc.
- **make run-higress-e2e-test:** Used to run the test cases.
- **make run-higress-e2e-test-setup:** Can be used to install the basic resources required for the test cases, such as nacos, dubbo, etc.
- **make run-higress-e2e-test-run:** Used to execute the test cases.
- **make run-higress-e2e-test-clean:** Can be used to clean up the basic resources installed during the setup phase of the test cases.
- **make higress-conformance-test-clean:** Used to clean up the test environment for deployments like higress-controller, higress-gateway, etc.
2. **make higress-wasmplugin-test:** Used to run the entire WasmPlugin testing process, including setting up the test environment, compiling WasmPlugin plugins, executing test cases, and cleaning up the test environment.
- **make higress-wasmplugin-test-prepare:** Can be used to set up the environment for deployments such as higress-controller, higress-gateway, and compile WasmPlugin plugins.
- **make run-higress-e2e-test-wasmplugin:** Used to run the test cases.
- **make run-higress-e2e-test-wasmplugin-setup:** Can be used to install the basic resources required for the test cases, such as nacos, dubbo, etc.
- **make run-higress-e2e-test-wasmplugin-run:** Used to execute the test cases.
- **make run-higress-e2e-test-wasmplugin-clean:** Can be used to clean up the basic resources installed during the setup phase of the test cases.
- **make higress-wasmplugin-test-clean:** Used to clean up the test environment for deployments like higress-controller, higress-gateway, etc.
## Gateway APIs Conformance Tests
Gateway API Conformance tests are based on the suite provided by `kubernetes-sig/gateway-api`, we can reuse that,

76
test/README_CN.md Normal file
View File

@@ -0,0 +1,76 @@
# Higress E2E 测试
Higress E2E 测试主要关注两个部分:
+ Ingress API 的一致性测试
+ Gateway API 的一致性测试
## Ingress API 一致性测试
### 架构
![ingress-arch](./e2e/arch.png)
### 工作流程
![ingress-workflow](./e2e/pipeline.png)
Higress 提供了运行 Ingress API 一致性测试和 wasmplugin 测试的 make 目标,
+ API 测试: `make higress-conformance-test`
+ WasmPlugin 测试: `make higress-wasmplugin-test`
+ 为测试构建所有 GO WasmPlugins: `make higress-wasmplugin-test`
+ 仅为一个 GO WasmPlugin 构建测试: `PLUGIN_NAME=request-block make higress-wasmplugin-test`
+ 仅为一个 CPP WasmPlugin 构建测试: `PLUGIN_TYPE=CPP PLUGIN_NAME=key_auth make higress-wasmplugin-test`
可以分为以下步骤:
1. delete-cluster: 检查是否有未删除的 kind 集群。
2. create-cluster: 创建一个新的 kind 集群。
3. docker-build: 构建 higress 的开发镜像。
4. kube-load-image: 将 dev higress-controller 镜像加载到 kind 集群中。
5. install-dev: 使用 helm 安装带有 dev 镜像的 higress-controller并安装最新的 higress-gateway、istiod。
6. run-e2e-test:
1. 设置一致性测试套件,例如在 `e2e_test.go` / `higressTests Slice` 中定义我们想要运行的一致性测试。我们选择打开的每个测试都在 `test/ingress/conformance/tests` 中定义。
2. 准备资源并将它们安装到集群中,例如后端服务/部署。
3.`e2e_test.go` / `higressTests Slice` 中加载我们选择打开的一致性测试,并逐个运行它们,如果不符合预期,则失败。
### 如何编写测试用例
要添加新的测试用例,首先需要在 `test/ingress/conformance/tests` 中添加 `xxx.go``xxx.yaml``xxx.yaml` 是您需要在集群中应用的 Ingress 资源,`xxx.go` 定义了 HigressConformanceTest。
然后,您应该将您定义的 HigressConformanceTest 添加到 `e2e_test.go` / `higressTests Slice` 中。
通过查看 `test/ingress/conformance/tests/httproute-simple-same-namespace.go``test/ingress/conformance/tests/httproute-simple-same-namespace.yaml` 中的代码,您可以快速了解并尝试编写一个测试用例。
### 如何实现测试环境的复用
主要通过 Makefile 中的以下几个目标实现:
1. **make higress-conformance-test:** 用于运行整个 Conformance 测试流程,包括搭建测试环境、运行测试用例、清理测试环境。
- **make higress-conformance-test-prepare:** 可用于搭建 higress-controller、higress-gateway 等 deployment 的环境。
- **make run-higress-e2e-test:** 可用于运行测试用例。
- **make run-higress-e2e-test-setup:** 可用于安装测试用例所需的基础资源,例如 nacos、dubbo 等。
- **make run-higress-e2e-test-run:** 可用于运行测试用例。
- **make run-higress-e2e-test-clean:** 可用于清理测试用例在 setup 阶段所安装的基础资源。
- **make higress-conformance-test-clean:** 可用于清理 higress-controller、higress-gateway 等 deployment 的测试环境。
2. **make higress-wasmplugin-test:** 用于运行整个 WasmPlugin 测试流程,包括搭建测试环境、编译 WasmPlugin 插件、运行测试用例、清理测试环境。
- **make higress-wasmplugin-test-prepare:** 可用于搭建 higress-controller、higress-gateway 等 deployment 的环境,并编译 WasmPlugin 插件。
- **make run-higress-e2e-test-wasmplugin:** 可用于运行测试用例。
- **make run-higress-e2e-test-wasmplugin-setup:** 可用于安装测试用例所需的基础资源,例如 nacos、dubbo 等。
- **make run-higress-e2e-test-wasmplugin-run:** 可用于运行测试用例。
- **make run-higress-e2e-test-wasmplugin-clean:** 可用于清理测试用例在 setup 阶段所安装的基础资源。
- **make higress-wasmplugin-test-clean:** 可用于清理 higress-controller、higress-gateway 等 deployment 的测试环境。
## Gateway API 一致性测试
Gateway API 一致性测试基于 `kubernetes-sig/gateway-api` 提供的套件,我们可以重复使用它,并决定我们需要打开哪些 Gateway API 的一致性测试。
此 API 包含一系列广泛的功能和用例,并已得到广泛实现。
这个大的功能集和各种实现的结合需要明确的一致性定义和测试,以确保在任何地方使用该 API 时都提供一致的体验。
Gateway API 包括一组一致性测试。这些测试创建具有指定 GatewayClass 的一系列 Gateways 和 Routes并测试实现是否符合 API 规范。
每个发布版本都包含一组一致性测试,随着 API 的演进,这些测试将不断扩展。
目前,一致性测试覆盖了标准通道中的大多数核心功能,以及一些扩展功能。

Some files were not shown because too many files have changed in this diff Show More