mirror of
https://github.com/alibaba/higress.git
synced 2026-02-25 21:21:01 +08:00
Compare commits
42 Commits
v1.3.3
...
v1.3.4-rc.
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
021387c9d3 | ||
|
|
ea0a694d81 | ||
|
|
8028fe03ca | ||
|
|
a68dde0b91 | ||
|
|
48c3db85c4 | ||
|
|
7e85065832 | ||
|
|
7967f5db70 | ||
|
|
5026973d59 | ||
|
|
7097eef6ba | ||
|
|
fae222806b | ||
|
|
c63cdb62ea | ||
|
|
e43f5d106f | ||
|
|
29c95ea557 | ||
|
|
73d5cc3f1d | ||
|
|
c1ddbcef7c | ||
|
|
dd39c87311 | ||
|
|
612c94dd8a | ||
|
|
e67ed481cf | ||
|
|
ccea33655f | ||
|
|
ad4cfdbd40 | ||
|
|
3598c21da0 | ||
|
|
a624351f84 | ||
|
|
c41264816e | ||
|
|
acd80d2528 | ||
|
|
073c10df77 | ||
|
|
90f89cf588 | ||
|
|
879192cf99 | ||
|
|
d3d000753d | ||
|
|
b8a01113e3 | ||
|
|
0bb9e6dd89 | ||
|
|
ecdd077c72 | ||
|
|
e971faeb0b | ||
|
|
77013e28b6 | ||
|
|
9faa5f37d1 | ||
|
|
665d9fa943 | ||
|
|
a71ecf41d1 | ||
|
|
b825f9176f | ||
|
|
d35d23e2d5 | ||
|
|
6d1e09c146 | ||
|
|
87c39d393f | ||
|
|
c97260c4a9 | ||
|
|
5e509e7032 |
6
.github/workflows/build-and-test-plugin.yaml
vendored
6
.github/workflows/build-and-test-plugin.yaml
vendored
@@ -52,11 +52,9 @@ jobs:
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |-
|
||||
envoy
|
||||
istio
|
||||
.git/modules
|
||||
key: ${{ runner.os }}-submodules-new-${{ github.run_id }}
|
||||
restore-keys: ${{ runner.os }}-submodules-new
|
||||
key: ${{ runner.os }}-submodules-cache-${{ github.run_id }}
|
||||
restore-keys: ${{ runner.os }}-submodules-cache
|
||||
|
||||
- run: git stash # restore patch
|
||||
|
||||
|
||||
18
.github/workflows/build-and-test.yaml
vendored
18
.github/workflows/build-and-test.yaml
vendored
@@ -36,11 +36,9 @@ jobs:
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |-
|
||||
envoy
|
||||
istio
|
||||
.git/modules
|
||||
key: ${{ runner.os }}-submodules-new-${{ github.run_id }}
|
||||
restore-keys: ${{ runner.os }}-submodules-new
|
||||
key: ${{ runner.os }}-submodules-cache-${{ github.run_id }}
|
||||
restore-keys: ${{ runner.os }}-submodules-cache
|
||||
|
||||
- run: git stash # restore patch
|
||||
|
||||
@@ -82,11 +80,9 @@ jobs:
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |-
|
||||
envoy
|
||||
istio
|
||||
.git/modules
|
||||
key: ${{ runner.os }}-submodules-new-${{ github.run_id }}
|
||||
restore-keys: ${{ runner.os }}-submodules-new
|
||||
key: ${{ runner.os }}-submodules-cache-${{ github.run_id }}
|
||||
restore-keys: ${{ runner.os }}-submodules-cache
|
||||
|
||||
- run: git stash # restore patch
|
||||
|
||||
@@ -130,11 +126,9 @@ jobs:
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |-
|
||||
envoy
|
||||
istio
|
||||
.git/modules
|
||||
key: ${{ runner.os }}-submodules-new-${{ github.run_id }}
|
||||
restore-keys: ${{ runner.os }}-submodules-new
|
||||
key: ${{ runner.os }}-submodules-cache-${{ github.run_id }}
|
||||
restore-keys: ${{ runner.os }}-submodules-cache
|
||||
|
||||
- run: git stash # restore patch
|
||||
|
||||
|
||||
30
.github/workflows/build-image-and-push.yaml
vendored
30
.github/workflows/build-image-and-push.yaml
vendored
@@ -20,6 +20,16 @@ jobs:
|
||||
with:
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Free Up GitHub Actions Ubuntu Runner Disk Space 🔧
|
||||
uses: jlumbroso/free-disk-space@main
|
||||
with:
|
||||
tool-cache: false
|
||||
android: true
|
||||
dotnet: true
|
||||
haskell: true
|
||||
large-packages: true
|
||||
swap-storage: true
|
||||
|
||||
- name: "Setup Go"
|
||||
uses: actions/setup-go@v3
|
||||
with:
|
||||
@@ -86,6 +96,16 @@ jobs:
|
||||
with:
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Free Up GitHub Actions Ubuntu Runner Disk Space 🔧
|
||||
uses: jlumbroso/free-disk-space@main
|
||||
with:
|
||||
tool-cache: false
|
||||
android: true
|
||||
dotnet: true
|
||||
haskell: true
|
||||
large-packages: true
|
||||
swap-storage: true
|
||||
|
||||
- name: "Setup Go"
|
||||
uses: actions/setup-go@v3
|
||||
with:
|
||||
@@ -153,6 +173,16 @@ jobs:
|
||||
with:
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Free Up GitHub Actions Ubuntu Runner Disk Space 🔧
|
||||
uses: jlumbroso/free-disk-space@main
|
||||
with:
|
||||
tool-cache: false
|
||||
android: true
|
||||
dotnet: true
|
||||
haskell: true
|
||||
large-packages: true
|
||||
swap-storage: true
|
||||
|
||||
- name: "Setup Go"
|
||||
uses: actions/setup-go@v3
|
||||
with:
|
||||
|
||||
@@ -26,6 +26,7 @@ header:
|
||||
- 'VERSION'
|
||||
- 'tools/'
|
||||
- 'test/README.md'
|
||||
- 'test/README_CN.md'
|
||||
- 'cmd/hgctl/config/testdata/config'
|
||||
- 'pkg/cmd/hgctl/manifests'
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
/api @johnlanni @CH3CHO
|
||||
/envoy @gengleilei @johnlanni
|
||||
/istio @SpecialYang @johnlanni
|
||||
/pkg @SpecialYang @johnlanni @CH3CHO @Xunzhuo
|
||||
/pkg @SpecialYang @johnlanni @CH3CHO
|
||||
/plugins @johnlanni @WeixinX
|
||||
/registry @NameHaibinZhang @2456868764 @johnlanni
|
||||
/test @Xunzhuo @2456868764 @CH3CHO
|
||||
|
||||
@@ -73,6 +73,7 @@
|
||||
* [分支定义](#分支定义)
|
||||
* [提交规则](#提交规则)
|
||||
* [PR说明](#PR说明)
|
||||
* [开发前准备](#开发前准备)
|
||||
|
||||
### 工作区准备
|
||||
|
||||
@@ -168,6 +169,12 @@ git config --get user.email
|
||||
|
||||
PR 是更改 Higress 项目文件的唯一方法。为了帮助审查人更好地理解你的目的,PR 描述不能太详细。我们鼓励贡献者遵循 [PR 模板](./.github/PULL_REQUEST_TEMPLATE.md) 来完成拉取请求。
|
||||
|
||||
### 开发前准备
|
||||
|
||||
```shell
|
||||
make prebuild && go mod tidy
|
||||
```
|
||||
|
||||
## 测试用例贡献
|
||||
|
||||
任何测试用例都会受到欢迎。目前,Higress 功能测试用例是高优先级的。
|
||||
|
||||
@@ -169,6 +169,12 @@ No matter commit message, or commit content, we do take more emphasis on code re
|
||||
|
||||
PR is the only way to make change to Higress project files. To help reviewers better get your purpose, PR description could not be too detailed. We encourage contributors to follow the [PR template](./.github/PULL_REQUEST_TEMPLATE.md) to finish the pull request.
|
||||
|
||||
### Pre-development preparation
|
||||
|
||||
```shell
|
||||
make prebuild && go mod tidy
|
||||
```
|
||||
|
||||
## Test case contribution
|
||||
|
||||
Any test case would be welcomed. Currently, Higress function test cases are high priority.
|
||||
|
||||
@@ -15,7 +15,7 @@ GO_LDFLAGS += -X $(VERSION_PACKAGE).higressVersion=$(shell cat VERSION) \
|
||||
|
||||
GO ?= go
|
||||
|
||||
export GOPROXY ?= https://proxy.golang.com.cn,direct
|
||||
export GOPROXY ?= https://proxy.golang.org,direct
|
||||
|
||||
TARGET_ARCH ?= amd64
|
||||
|
||||
@@ -138,11 +138,11 @@ export ENVOY_TAR_PATH:=/home/package/envoy.tar.gz
|
||||
|
||||
external/package/envoy-amd64.tar.gz:
|
||||
# cd external/proxy; BUILD_WITH_CONTAINER=1 make test_release
|
||||
cd external/package; wget "https://github.com/alibaba/higress/releases/download/v1.3.2/envoy-amd64.tar.gz"
|
||||
cd external/package; wget -O envoy-amd64.tar.gz "https://github.com/alibaba/higress/releases/download/v1.3.4-rc.1/envoy-symbol-amd64.tar.gz"
|
||||
|
||||
external/package/envoy-arm64.tar.gz:
|
||||
# cd external/proxy; BUILD_WITH_CONTAINER=1 make test_release
|
||||
cd external/package; wget "https://github.com/alibaba/higress/releases/download/v1.3.2/envoy-arm64.tar.gz"
|
||||
cd external/package; wget -O envoy-arm64.tar.gz "https://github.com/alibaba/higress/releases/download/v1.3.4-rc.1/envoy-symbol-arm64.tar.gz"
|
||||
|
||||
build-pilot:
|
||||
cd external/istio; rm -rf out/linux_amd64; GOOS_LOCAL=linux TARGET_OS=linux TARGET_ARCH=amd64 BUILD_WITH_CONTAINER=1 make build-linux
|
||||
@@ -154,13 +154,13 @@ build-pilot-local:
|
||||
build-gateway: prebuild external/package/envoy-amd64.tar.gz external/package/envoy-arm64.tar.gz build-pilot
|
||||
cd external/istio; BUILD_WITH_CONTAINER=1 BUILDX_PLATFORM=true DOCKER_BUILD_VARIANTS=default DOCKER_TARGETS="docker.proxyv2" make docker
|
||||
|
||||
build-gateway-local: prebuild external/package/envoy-amd64.tar.gz external/package/envoy-arm64.tar.gz build-pilot
|
||||
build-gateway-local: prebuild external/package/envoy-amd64.tar.gz external/package/envoy-arm64.tar.gz
|
||||
cd external/istio; rm -rf out/linux_${GOARCH_LOCAL}; GOOS_LOCAL=linux TARGET_OS=linux BUILD_WITH_CONTAINER=1 BUILDX_PLATFORM=false DOCKER_BUILD_VARIANTS=default DOCKER_TARGETS="docker.proxyv2" make docker
|
||||
|
||||
build-istio: prebuild build-pilot
|
||||
cd external/istio; BUILD_WITH_CONTAINER=1 BUILDX_PLATFORM=true DOCKER_BUILD_VARIANTS=default DOCKER_TARGETS="docker.pilot" make docker
|
||||
|
||||
build-istio-local: prebuild build-pilot-local
|
||||
build-istio-local: prebuild
|
||||
cd external/istio; rm -rf out/linux_${GOARCH_LOCAL}; GOOS_LOCAL=linux TARGET_OS=linux BUILD_WITH_CONTAINER=1 BUILDX_PLATFORM=false DOCKER_BUILD_VARIANTS=default DOCKER_TARGETS="docker.pilot" make docker
|
||||
|
||||
build-wasmplugins:
|
||||
@@ -177,13 +177,13 @@ install: pre-install
|
||||
cd helm/higress; helm dependency build
|
||||
helm install higress helm/higress -n higress-system --create-namespace --set 'global.local=true'
|
||||
|
||||
ENVOY_LATEST_IMAGE_TAG ?= sha-53ff28c
|
||||
ISTIO_LATEST_IMAGE_TAG ?= sha-53ff28c
|
||||
ENVOY_LATEST_IMAGE_TAG ?= sha-a68dde0
|
||||
ISTIO_LATEST_IMAGE_TAG ?= sha-a68dde0
|
||||
|
||||
install-dev: pre-install
|
||||
helm install higress helm/core -n higress-system --create-namespace --set 'controller.tag=$(TAG)' --set 'gateway.replicas=1' --set 'pilot.tag=$(ISTIO_LATEST_IMAGE_TAG)' --set 'gateway.tag=$(ENVOY_LATEST_IMAGE_TAG)' --set 'global.local=true'
|
||||
install-dev-wasmplugin: build-wasmplugins pre-install
|
||||
helm install higress helm/core -n higress-system --create-namespace --set 'controller.tag=$(TAG)' --set 'gateway.replicas=1' --set 'pilot.tag=$(ISTIO_LATEST_IMAGE_TAG)' --set 'gateway.tag=$(ENVOY_LATEST_IMAGE_TAG)' --set 'global.local=true' --set 'global.volumeWasmPlugins=true'
|
||||
helm install higress helm/core -n higress-system --create-namespace --set 'controller.tag=$(TAG)' --set 'gateway.replicas=1' --set 'pilot.tag=$(ISTIO_LATEST_IMAGE_TAG)' --set 'gateway.tag=$(ENVOY_LATEST_IMAGE_TAG)' --set 'global.local=true' --set 'global.volumeWasmPlugins=true' --set 'global.onlyPushRouteCluster=false'
|
||||
|
||||
uninstall:
|
||||
helm uninstall higress -n higress-system
|
||||
@@ -233,14 +233,30 @@ include tools/lint.mk
|
||||
.PHONY: gateway-conformance-test
|
||||
gateway-conformance-test:
|
||||
|
||||
# higress-conformance-test-prepare prepares the environment for higress conformance tests.
|
||||
.PHONY: higress-conformance-test-prepare
|
||||
higress-conformance-test-prepare: $(tools/kind) delete-cluster create-cluster docker-build kube-load-image install-dev
|
||||
|
||||
# higress-conformance-test runs ingress api conformance tests.
|
||||
.PHONY: higress-conformance-test
|
||||
higress-conformance-test: $(tools/kind) delete-cluster create-cluster docker-build kube-load-image install-dev run-higress-e2e-test delete-cluster
|
||||
|
||||
# higress-conformance-test-clean cleans the environment for higress conformance tests.
|
||||
.PHONY: higress-conformance-test-clean
|
||||
higress-conformance-test-clean: $(tools/kind) delete-cluster
|
||||
|
||||
# higress-wasmplugin-test-prepare prepares the environment for higress wasmplugin tests.
|
||||
.PHONY: higress-wasmplugin-test-prepare
|
||||
higress-wasmplugin-test-prepare: $(tools/kind) delete-cluster create-cluster docker-build kube-load-image install-dev-wasmplugin
|
||||
|
||||
# higress-wasmplugin-test runs ingress wasmplugin tests.
|
||||
.PHONY: higress-wasmplugin-test
|
||||
higress-wasmplugin-test: $(tools/kind) delete-cluster create-cluster docker-build kube-load-image install-dev-wasmplugin run-higress-e2e-test-wasmplugin delete-cluster
|
||||
|
||||
# higress-wasmplugin-test-clean cleans the environment for higress wasmplugin tests.
|
||||
.PHONY: higress-wasmplugin-test-clean
|
||||
higress-wasmplugin-test-clean: $(tools/kind) delete-cluster
|
||||
|
||||
# create-cluster creates a kube cluster with kind.
|
||||
.PHONY: create-cluster
|
||||
create-cluster: $(tools/kind)
|
||||
@@ -270,6 +286,17 @@ kube-load-image: $(tools/kind) ## Install the Higress image to a kind cluster us
|
||||
tools/hack/kind-load-image.sh docker.io/alihigress/httpbin 1.0.2
|
||||
tools/hack/kind-load-image.sh docker.io/charlie1380/eureka-registry-provider v0.3.0
|
||||
tools/hack/kind-load-image.sh docker.io/bitinit/eureka latest
|
||||
|
||||
# run-higress-e2e-test-setup starts to setup ingress e2e tests.
|
||||
.PHONT: run-higress-e2e-test-setup
|
||||
run-higress-e2e-test-setup:
|
||||
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
|
||||
@echo -e "\n\033[36mWaiting higress-controller to be ready...\033[0m\n"
|
||||
kubectl wait --timeout=10m -n higress-system deployment/higress-controller --for=condition=Available
|
||||
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
|
||||
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
|
||||
go test -v -tags conformance ./test/e2e/e2e_test.go --ingress-class=higress --debug=true --test-area=setup
|
||||
|
||||
# run-higress-e2e-test starts to run ingress e2e tests.
|
||||
.PHONY: run-higress-e2e-test
|
||||
run-higress-e2e-test:
|
||||
@@ -278,9 +305,39 @@ run-higress-e2e-test:
|
||||
kubectl wait --timeout=10m -n higress-system deployment/higress-controller --for=condition=Available
|
||||
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
|
||||
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
|
||||
go test -v -tags conformance ./test/e2e/e2e_test.go --ingress-class=higress --debug=true
|
||||
go test -v -tags conformance ./test/e2e/e2e_test.go --ingress-class=higress --debug=true --test-area=all
|
||||
|
||||
# run-higress-e2e-test starts to run ingress e2e tests.
|
||||
# run-higress-e2e-test-run starts to run ingress e2e conformance tests.
|
||||
.PHONY: run-higress-e2e-test-run
|
||||
run-higress-e2e-test-run:
|
||||
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
|
||||
@echo -e "\n\033[36mWaiting higress-controller to be ready...\033[0m\n"
|
||||
kubectl wait --timeout=10m -n higress-system deployment/higress-controller --for=condition=Available
|
||||
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
|
||||
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
|
||||
go test -v -tags conformance ./test/e2e/e2e_test.go --ingress-class=higress --debug=true --test-area=run
|
||||
|
||||
# run-higress-e2e-test-clean starts to clean ingress e2e tests.
|
||||
.PHONY: run-higress-e2e-test-clean
|
||||
run-higress-e2e-test-clean:
|
||||
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
|
||||
@echo -e "\n\033[36mWaiting higress-controller to be ready...\033[0m\n"
|
||||
kubectl wait --timeout=10m -n higress-system deployment/higress-controller --for=condition=Available
|
||||
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
|
||||
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
|
||||
go test -v -tags conformance ./test/e2e/e2e_test.go --ingress-class=higress --debug=true --test-area=clean
|
||||
|
||||
# run-higress-e2e-test-wasmplugin-setup starts to prepare ingress e2e tests.
|
||||
.PHONY: run-higress-e2e-test-wasmplugin-setup
|
||||
run-higress-e2e-test-wasmplugin-setup:
|
||||
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
|
||||
@echo -e "\n\033[36mWaiting higress-controller to be ready...\033[0m\n"
|
||||
kubectl wait --timeout=10m -n higress-system deployment/higress-controller --for=condition=Available
|
||||
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
|
||||
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
|
||||
go test -v -tags conformance ./test/e2e/e2e_test.go -isWasmPluginTest=true -wasmPluginType=$(PLUGIN_TYPE) -wasmPluginName=$(PLUGIN_NAME) --ingress-class=higress --debug=true --test-area=setup
|
||||
|
||||
# run-higress-e2e-test-wasmplugin starts to run ingress e2e tests.
|
||||
.PHONY: run-higress-e2e-test-wasmplugin
|
||||
run-higress-e2e-test-wasmplugin:
|
||||
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
|
||||
@@ -288,4 +345,24 @@ run-higress-e2e-test-wasmplugin:
|
||||
kubectl wait --timeout=10m -n higress-system deployment/higress-controller --for=condition=Available
|
||||
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
|
||||
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
|
||||
go test -v -tags conformance ./test/e2e/e2e_test.go -isWasmPluginTest=true -wasmPluginType=$(PLUGIN_TYPE) -wasmPluginName=$(PLUGIN_NAME) --ingress-class=higress --debug=true
|
||||
go test -v -tags conformance ./test/e2e/e2e_test.go -isWasmPluginTest=true -wasmPluginType=$(PLUGIN_TYPE) -wasmPluginName=$(PLUGIN_NAME) --ingress-class=higress --debug=true --test-area=all
|
||||
|
||||
# run-higress-e2e-test-wasmplugin-run starts to run ingress e2e conformance tests.
|
||||
.PHONY: run-higress-e2e-test-wasmplugin-run
|
||||
run-higress-e2e-test-wasmplugin-run:
|
||||
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
|
||||
@echo -e "\n\033[36mWaiting higress-controller to be ready...\033[0m\n"
|
||||
kubectl wait --timeout=10m -n higress-system deployment/higress-controller --for=condition=Available
|
||||
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
|
||||
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
|
||||
go test -v -tags conformance ./test/e2e/e2e_test.go -isWasmPluginTest=true -wasmPluginType=$(PLUGIN_TYPE) -wasmPluginName=$(PLUGIN_NAME) --ingress-class=higress --debug=true --test-area=run
|
||||
|
||||
# run-higress-e2e-test-wasmplugin-clean starts to clean ingress e2e tests.
|
||||
.PHONY: run-higress-e2e-test-wasmplugin-clean
|
||||
run-higress-e2e-test-wasmplugin-clean:
|
||||
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
|
||||
@echo -e "\n\033[36mWaiting higress-controller to be ready...\033[0m\n"
|
||||
kubectl wait --timeout=10m -n higress-system deployment/higress-controller --for=condition=Available
|
||||
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
|
||||
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
|
||||
go test -v -tags conformance ./test/e2e/e2e_test.go -isWasmPluginTest=true -wasmPluginType=$(PLUGIN_TYPE) -wasmPluginName=$(PLUGIN_NAME) --ingress-class=higress --debug=true --test-area=clean
|
||||
|
||||
@@ -2,7 +2,11 @@ codecov:
|
||||
require_ci_to_pass: yes
|
||||
coverage:
|
||||
status:
|
||||
patch: no
|
||||
patch:
|
||||
default:
|
||||
target: 50%
|
||||
threshold: 0%
|
||||
if_ci_failed: error # success, failure, error, ignore
|
||||
project:
|
||||
default:
|
||||
target: auto
|
||||
@@ -17,4 +21,4 @@ ignore:
|
||||
comment:
|
||||
layout: "reach,diff,flags,tree"
|
||||
behavior: default
|
||||
require_changes: no
|
||||
require_changes: no
|
||||
49
envoy/1.20/patches/envoy/20240110-fix-srds.patch
Normal file
49
envoy/1.20/patches/envoy/20240110-fix-srds.patch
Normal file
@@ -0,0 +1,49 @@
|
||||
diff -Naur envoy/source/common/router/BUILD envoy-new/source/common/router/BUILD
|
||||
--- envoy/source/common/router/BUILD 2024-01-10 20:10:14.505600746 +0800
|
||||
+++ envoy-new/source/common/router/BUILD 2024-01-10 20:07:25.960379955 +0800
|
||||
@@ -212,6 +212,7 @@
|
||||
"//envoy/router:rds_interface",
|
||||
"//envoy/router:scopes_interface",
|
||||
"//envoy/thread_local:thread_local_interface",
|
||||
+ "//source/common/http:header_utility_lib",
|
||||
"@envoy_api//envoy/config/route/v3:pkg_cc_proto",
|
||||
"@envoy_api//envoy/extensions/filters/network/http_connection_manager/v3:pkg_cc_proto",
|
||||
],
|
||||
diff -Naur envoy/source/common/router/scoped_config_impl.cc envoy-new/source/common/router/scoped_config_impl.cc
|
||||
--- envoy/source/common/router/scoped_config_impl.cc 2024-01-10 20:10:14.529600924 +0800
|
||||
+++ envoy-new/source/common/router/scoped_config_impl.cc 2024-01-10 20:09:50.161422411 +0800
|
||||
@@ -3,6 +3,8 @@
|
||||
#include "envoy/config/route/v3/scoped_route.pb.h"
|
||||
#include "envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.pb.h"
|
||||
|
||||
+#include "source/common/http/header_utility.h"
|
||||
+
|
||||
namespace Envoy {
|
||||
namespace Router {
|
||||
|
||||
@@ -74,18 +76,20 @@
|
||||
HostValueExtractorImpl::computeFragment(const Http::HeaderMap& headers,
|
||||
const StreamInfo::StreamInfo&,
|
||||
ReComputeCbPtr& recompute) const {
|
||||
- auto fragment = computeFragment(headers);
|
||||
auto host = static_cast<const Http::RequestHeaderMap&>(headers).getHostValue();
|
||||
+ auto port_start = Http::HeaderUtility::getPortStart(host);
|
||||
+ if (port_start != absl::string_view::npos) {
|
||||
+ host = host.substr(0, port_start);
|
||||
+ }
|
||||
*recompute = [this, host, recompute]() mutable -> std::unique_ptr<ScopeKeyFragmentBase> {
|
||||
return reComputeHelper(std::string(host), recompute, 0);
|
||||
};
|
||||
- return fragment;
|
||||
+ return std::make_unique<StringKeyFragment>(host);
|
||||
}
|
||||
|
||||
std::unique_ptr<ScopeKeyFragmentBase>
|
||||
-HostValueExtractorImpl::computeFragment(const Http::HeaderMap& headers) const {
|
||||
- return std::make_unique<StringKeyFragment>(
|
||||
- static_cast<const Http::RequestHeaderMap&>(headers).getHostValue());
|
||||
+HostValueExtractorImpl::computeFragment(const Http::HeaderMap&) const {
|
||||
+ return nullptr;
|
||||
}
|
||||
|
||||
std::unique_ptr<ScopeKeyFragmentBase>
|
||||
65
envoy/1.20/patches/envoy/20240111-fix-srds-compute.patch
Normal file
65
envoy/1.20/patches/envoy/20240111-fix-srds-compute.patch
Normal file
@@ -0,0 +1,65 @@
|
||||
diff -Naur envoy/source/common/router/scoped_config_impl.cc envoy-new/source/common/router/scoped_config_impl.cc
|
||||
--- envoy/source/common/router/scoped_config_impl.cc 2024-01-11 16:23:55.407881263 +0800
|
||||
+++ envoy-new/source/common/router/scoped_config_impl.cc 2024-01-11 16:23:42.311786814 +0800
|
||||
@@ -53,21 +53,26 @@
|
||||
}
|
||||
|
||||
std::unique_ptr<ScopeKeyFragmentBase>
|
||||
-HostValueExtractorImpl::reComputeHelper(const std::string& host, ReComputeCbPtr& next_recompute,
|
||||
+HostValueExtractorImpl::reComputeHelper(const std::string& host,
|
||||
+ ReComputeCbWeakPtr& weak_next_recompute,
|
||||
uint32_t recompute_seq) const {
|
||||
if (recompute_seq == max_recompute_num_) {
|
||||
ENVOY_LOG_MISC(warn,
|
||||
"recompute host fragment failed, maximum number of recalculations exceeded");
|
||||
return nullptr;
|
||||
}
|
||||
+ auto next_recompute = weak_next_recompute.lock();
|
||||
+ if (!next_recompute) {
|
||||
+ return nullptr;
|
||||
+ }
|
||||
if (host == "*") {
|
||||
*next_recompute = nullptr;
|
||||
return nullptr;
|
||||
}
|
||||
auto masked_host = maskFirstDNSLabel(host);
|
||||
*next_recompute = [this, masked_host, recompute_seq,
|
||||
- next_recompute]() mutable -> std::unique_ptr<ScopeKeyFragmentBase> {
|
||||
- return reComputeHelper(masked_host, next_recompute, recompute_seq + 1);
|
||||
+ weak_next_recompute]() mutable -> std::unique_ptr<ScopeKeyFragmentBase> {
|
||||
+ return reComputeHelper(masked_host, weak_next_recompute, recompute_seq + 1);
|
||||
};
|
||||
return std::make_unique<StringKeyFragment>(masked_host);
|
||||
}
|
||||
@@ -81,8 +86,9 @@
|
||||
if (port_start != absl::string_view::npos) {
|
||||
host = host.substr(0, port_start);
|
||||
}
|
||||
- *recompute = [this, host, recompute]() mutable -> std::unique_ptr<ScopeKeyFragmentBase> {
|
||||
- return reComputeHelper(std::string(host), recompute, 0);
|
||||
+ *recompute = [this, host, weak_recompute = ReComputeCbWeakPtr(recompute)]() mutable
|
||||
+ -> std::unique_ptr<ScopeKeyFragmentBase> {
|
||||
+ return reComputeHelper(std::string(host), weak_recompute, 0);
|
||||
};
|
||||
return std::make_unique<StringKeyFragment>(host);
|
||||
}
|
||||
diff -Naur envoy/source/common/router/scoped_config_impl.h envoy-new/source/common/router/scoped_config_impl.h
|
||||
--- envoy/source/common/router/scoped_config_impl.h 2024-01-11 16:23:55.407881263 +0800
|
||||
+++ envoy-new/source/common/router/scoped_config_impl.h 2024-01-11 16:23:42.311786814 +0800
|
||||
@@ -25,6 +25,7 @@
|
||||
#if defined(ALIMESH)
|
||||
using ReComputeCb = std::function<std::unique_ptr<ScopeKeyFragmentBase>()>;
|
||||
using ReComputeCbPtr = std::shared_ptr<ReComputeCb>;
|
||||
+using ReComputeCbWeakPtr = std::weak_ptr<ReComputeCb>;
|
||||
#endif
|
||||
|
||||
/**
|
||||
@@ -83,7 +84,7 @@
|
||||
|
||||
private:
|
||||
std::unique_ptr<ScopeKeyFragmentBase> reComputeHelper(const std::string& host,
|
||||
- ReComputeCbPtr& next_recompute,
|
||||
+ ReComputeCbWeakPtr& weak_next_recompute,
|
||||
uint32_t recompute_seq) const;
|
||||
|
||||
static constexpr uint32_t DefaultMaxRecomputeNum = 100;
|
||||
1445
envoy/1.20/patches/envoy/20240116-fix-cve.patch
Normal file
1445
envoy/1.20/patches/envoy/20240116-fix-cve.patch
Normal file
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
2
go.mod
2
go.mod
@@ -15,7 +15,7 @@ replace github.com/imdario/mergo => github.com/imdario/mergo v0.3.5
|
||||
|
||||
require (
|
||||
github.com/AlecAivazis/survey/v2 v2.3.6
|
||||
github.com/agiledragon/gomonkey/v2 v2.9.0
|
||||
github.com/agiledragon/gomonkey/v2 v2.11.0
|
||||
github.com/avast/retry-go/v4 v4.3.4
|
||||
github.com/compose-spec/compose-go v1.8.2
|
||||
github.com/docker/cli v20.10.20+incompatible
|
||||
|
||||
4
go.sum
4
go.sum
@@ -160,8 +160,8 @@ github.com/VividCortex/ewma v1.1.1/go.mod h1:2Tkkvm3sRDVXaiyucHiACn4cqf7DpdyLvmx
|
||||
github.com/VividCortex/gohistogram v1.0.0/go.mod h1:Pf5mBqqDxYaXu3hDrrU+w6nw50o/4+TcAqDqk/vUH7g=
|
||||
github.com/afex/hystrix-go v0.0.0-20180502004556-fa1af6a1f4f5/go.mod h1:SkGFH1ia65gfNATL8TAiHDNxPzPdmEL5uirI2Uyuz6c=
|
||||
github.com/agext/levenshtein v1.2.1/go.mod h1:JEDfjyjHDjOF/1e4FlBE/PkbqA9OfWu2ki2W0IB5558=
|
||||
github.com/agiledragon/gomonkey/v2 v2.9.0 h1:PDiKKybR596O6FHW+RVSG0Z7uGCBNbmbUXh3uCNQ7Hc=
|
||||
github.com/agiledragon/gomonkey/v2 v2.9.0/go.mod h1:ap1AmDzcVOAz1YpeJ3TCzIgstoaWLA6jbbgxfB4w2iY=
|
||||
github.com/agiledragon/gomonkey/v2 v2.11.0 h1:5oxSgA+tC1xuGsrIorR+sYiziYltmJyEZ9qA25b6l5U=
|
||||
github.com/agiledragon/gomonkey/v2 v2.11.0/go.mod h1:ap1AmDzcVOAz1YpeJ3TCzIgstoaWLA6jbbgxfB4w2iY=
|
||||
github.com/agl/ed25519 v0.0.0-20170116200512-5312a6153412/go.mod h1:WPjqKcmVOxf0XSf3YxCJs6N6AOSrOx3obionmG7T0y0=
|
||||
github.com/ahmetb/gen-crd-api-reference-docs v0.3.0/go.mod h1:TdjdkYhlOifCQWPs1UdTma97kQQMozf5h26hTuG70u8=
|
||||
github.com/alecthomas/jsonschema v0.0.0-20180308105923-f2c93856175a/go.mod h1:qpebaTNSsyUn5rPSJMsfqEtDw71TTggXM6stUDI16HA=
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
apiVersion: v2
|
||||
appVersion: 1.3.3
|
||||
appVersion: 1.3.4-rc.2
|
||||
description: Helm chart for deploying higress gateways
|
||||
icon: https://higress.io/img/higress_logo_small.png
|
||||
home: http://higress.io/
|
||||
@@ -10,4 +10,4 @@ name: higress-core
|
||||
sources:
|
||||
- http://github.com/alibaba/higress
|
||||
type: application
|
||||
version: 1.3.3
|
||||
version: 1.3.4-rc.2
|
||||
|
||||
@@ -154,6 +154,11 @@ spec:
|
||||
type: array
|
||||
httpPath:
|
||||
type: string
|
||||
paramFromEntireBody:
|
||||
properties:
|
||||
paramType:
|
||||
type: string
|
||||
type: object
|
||||
params:
|
||||
items:
|
||||
properties:
|
||||
|
||||
@@ -70,6 +70,10 @@ spec:
|
||||
periodSeconds: 3
|
||||
timeoutSeconds: 5
|
||||
env:
|
||||
- name: DEFAULT_UPSTREAM_CONCURRENCY_THRESHOLD
|
||||
value: "{{ .Values.global.defaultUpstreamConcurrencyThreshold }}"
|
||||
- name: ISTIO_GPRC_MAXRECVMSGSIZE
|
||||
value: "{{ .Values.global.xdsMaxRecvMsgSize }}"
|
||||
- name: ENBALE_SCOPED_RDS
|
||||
value: "{{ .Values.global.enableSRDS }}"
|
||||
- name: ON_DEMAND_RDS
|
||||
|
||||
@@ -175,15 +175,15 @@ spec:
|
||||
protocol: TCP
|
||||
{{- end }}
|
||||
readinessProbe:
|
||||
failureThreshold: 30
|
||||
failureThreshold: {{ .Values.gateway.readinessFailureThreshold }}
|
||||
httpGet:
|
||||
path: /healthz/ready
|
||||
port: 15021
|
||||
scheme: HTTP
|
||||
initialDelaySeconds: 1
|
||||
periodSeconds: 2
|
||||
successThreshold: 1
|
||||
timeoutSeconds: 3
|
||||
initialDelaySeconds: {{ .Values.gateway.readinessInitialDelaySeconds }}
|
||||
periodSeconds: {{ .Values.gateway.readinessPeriodSeconds }}
|
||||
successThreshold: {{ .Values.gateway.readinessSuccessThreshold }}
|
||||
timeoutSeconds: {{ .Values.gateway.readinessTimeoutSeconds }}
|
||||
{{- if not (or .Values.global.local .Values.global.kind) }}
|
||||
resources:
|
||||
{{- toYaml .Values.gateway.resources | nindent 12 }}
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
revision: ""
|
||||
global:
|
||||
enableSRDS: false
|
||||
onDemandRDS: true
|
||||
hostRDSMergeSubset: true
|
||||
xdsMaxRecvMsgSize: 104857600
|
||||
defaultUpstreamConcurrencyThreshold: 10000
|
||||
enableSRDS: true
|
||||
onDemandRDS: false
|
||||
hostRDSMergeSubset: false
|
||||
onlyPushRouteCluster: true
|
||||
# IngressClass filters which ingress resources the higress controller watches.
|
||||
# The default ingress class is higress.
|
||||
@@ -13,7 +15,7 @@ global:
|
||||
# resources in the k8s cluster.
|
||||
ingressClass: "higress"
|
||||
watchNamespace: ""
|
||||
disableAlpnH2: true
|
||||
disableAlpnH2: false
|
||||
enableStatus: true
|
||||
# whether to use autoscaling/v2 template for HPA settings
|
||||
# for internal usage only, not to be configured by users.
|
||||
@@ -151,12 +153,18 @@ global:
|
||||
# The number of successive failed probes before indicating readiness failure.
|
||||
readinessFailureThreshold: 30
|
||||
|
||||
# The number of successive successed probes before indicating readiness success.
|
||||
readinessSuccessThreshold: 30
|
||||
|
||||
# The initial delay for readiness probes in seconds.
|
||||
readinessInitialDelaySeconds: 1
|
||||
|
||||
# The period between readiness probes.
|
||||
readinessPeriodSeconds: 2
|
||||
|
||||
# The readiness timeout seconds
|
||||
readinessTimeoutSeconds: 3
|
||||
|
||||
# Resources for the sidecar.
|
||||
resources:
|
||||
requests:
|
||||
@@ -373,6 +381,21 @@ gateway:
|
||||
replicas: 2
|
||||
image: gateway
|
||||
|
||||
# The number of successive failed probes before indicating readiness failure.
|
||||
readinessFailureThreshold: 30
|
||||
|
||||
# The number of successive successed probes before indicating readiness success.
|
||||
readinessSuccessThreshold: 1
|
||||
|
||||
# The initial delay for readiness probes in seconds.
|
||||
readinessInitialDelaySeconds: 1
|
||||
|
||||
# The period between readiness probes.
|
||||
readinessPeriodSeconds: 2
|
||||
|
||||
# The readiness timeout seconds
|
||||
readinessTimeoutSeconds: 3
|
||||
|
||||
hub: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress
|
||||
tag: ""
|
||||
# revision declares which revision this gateway is a part of
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
dependencies:
|
||||
- name: higress-core
|
||||
repository: file://../core
|
||||
version: 1.3.3
|
||||
version: 1.3.4-rc.2
|
||||
- name: higress-console
|
||||
repository: https://higress.io/helm-charts/
|
||||
version: 1.3.1
|
||||
digest: sha256:585666df5da403450c5e586a71388bc0d029354b1100b20a50616f56711fa171
|
||||
generated: "2024-01-08T21:40:10.446936+08:00"
|
||||
version: 1.3.2
|
||||
digest: sha256:aa10fa4ae01c7e7f8297db5fe6f4752aa6526e2411d46c93abddbb4e7e84d36d
|
||||
generated: "2024-02-01T19:33:51.268079+08:00"
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
apiVersion: v2
|
||||
appVersion: 1.3.3
|
||||
appVersion: 1.3.4-rc.2
|
||||
description: Helm chart for deploying Higress gateways
|
||||
icon: https://higress.io/img/higress_logo_small.png
|
||||
home: http://higress.io/
|
||||
@@ -12,9 +12,9 @@ sources:
|
||||
dependencies:
|
||||
- name: higress-core
|
||||
repository: "file://../core"
|
||||
version: 1.3.3
|
||||
version: 1.3.4-rc.2
|
||||
- name: higress-console
|
||||
repository: "https://higress.io/helm-charts/"
|
||||
version: 1.3.1
|
||||
version: 1.3.2
|
||||
type: application
|
||||
version: 1.3.3
|
||||
version: 1.3.4-rc.2
|
||||
|
||||
373
istio/1.12/patches/istio/20240115-optimize-rds-cache.patch
Normal file
373
istio/1.12/patches/istio/20240115-optimize-rds-cache.patch
Normal file
@@ -0,0 +1,373 @@
|
||||
diff -Naur istio/pilot/pkg/model/push_context.go istio-new/pilot/pkg/model/push_context.go
|
||||
--- istio/pilot/pkg/model/push_context.go 2024-01-15 20:46:45.000000000 +0800
|
||||
+++ istio-new/pilot/pkg/model/push_context.go 2024-01-15 19:20:45.000000000 +0800
|
||||
@@ -96,6 +96,9 @@
|
||||
publicByGateway map[string][]config.Config
|
||||
// root vs namespace/name ->delegate vs virtualservice gvk/namespace/name
|
||||
delegates map[ConfigKey][]ConfigKey
|
||||
+ // Added by ingress
|
||||
+ byHost map[string][]config.Config
|
||||
+ // End added by ingress
|
||||
}
|
||||
|
||||
func newVirtualServiceIndex() virtualServiceIndex {
|
||||
@@ -104,6 +107,9 @@
|
||||
privateByNamespaceAndGateway: map[string]map[string][]config.Config{},
|
||||
exportedToNamespaceByGateway: map[string]map[string][]config.Config{},
|
||||
delegates: map[ConfigKey][]ConfigKey{},
|
||||
+ // Added by ingress
|
||||
+ byHost: map[string][]config.Config{},
|
||||
+ // End added by ingress
|
||||
}
|
||||
}
|
||||
|
||||
@@ -857,6 +863,13 @@
|
||||
return res
|
||||
}
|
||||
|
||||
+// Added by ingress
|
||||
+func (ps *PushContext) VirtualServicesForHost(proxy *Proxy, host string) []config.Config {
|
||||
+ return ps.virtualServiceIndex.byHost[host]
|
||||
+}
|
||||
+
|
||||
+// End added by ingress
|
||||
+
|
||||
// DelegateVirtualServicesConfigKey lists all the delegate virtual services configkeys associated with the provided virtual services
|
||||
func (ps *PushContext) DelegateVirtualServicesConfigKey(vses []config.Config) []ConfigKey {
|
||||
var out []ConfigKey
|
||||
@@ -1468,6 +1481,11 @@
|
||||
for _, virtualService := range vservices {
|
||||
ns := virtualService.Namespace
|
||||
rule := virtualService.Spec.(*networking.VirtualService)
|
||||
+ // Added by ingress
|
||||
+ for _, host := range rule.Hosts {
|
||||
+ ps.virtualServiceIndex.byHost[host] = append(ps.virtualServiceIndex.byHost[host], virtualService)
|
||||
+ }
|
||||
+ // End added by ingress
|
||||
gwNames := getGatewayNames(rule)
|
||||
if len(rule.ExportTo) == 0 {
|
||||
// No exportTo in virtualService. Use the global default
|
||||
diff -Naur istio/pilot/pkg/networking/core/v1alpha3/gateway.go istio-new/pilot/pkg/networking/core/v1alpha3/gateway.go
|
||||
--- istio/pilot/pkg/networking/core/v1alpha3/gateway.go 2024-01-15 20:46:45.000000000 +0800
|
||||
+++ istio-new/pilot/pkg/networking/core/v1alpha3/gateway.go 2024-01-15 20:04:05.000000000 +0800
|
||||
@@ -28,6 +28,7 @@
|
||||
route "github.com/envoyproxy/go-control-plane/envoy/config/route/v3"
|
||||
hcm "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/network/http_connection_manager/v3"
|
||||
tls "github.com/envoyproxy/go-control-plane/envoy/extensions/transport_sockets/tls/v3"
|
||||
+ discovery "github.com/envoyproxy/go-control-plane/envoy/service/discovery/v3"
|
||||
"github.com/hashicorp/go-multierror"
|
||||
|
||||
meshconfig "istio.io/api/mesh/v1alpha1"
|
||||
@@ -35,6 +36,7 @@
|
||||
"istio.io/istio/pilot/pkg/features"
|
||||
"istio.io/istio/pilot/pkg/model"
|
||||
istionetworking "istio.io/istio/pilot/pkg/networking"
|
||||
+ "istio.io/istio/pilot/pkg/networking/core/v1alpha3/envoyfilter"
|
||||
"istio.io/istio/pilot/pkg/networking/core/v1alpha3/extension"
|
||||
"istio.io/istio/pilot/pkg/networking/core/v1alpha3/mseingress"
|
||||
istio_route "istio.io/istio/pilot/pkg/networking/core/v1alpha3/route"
|
||||
@@ -423,8 +425,15 @@
|
||||
gatewayName string
|
||||
}
|
||||
|
||||
-func (configgen *ConfigGeneratorImpl) buildHostRDSConfig(node *model.Proxy, push *model.PushContext,
|
||||
- routeName string) *route.RouteConfiguration {
|
||||
+func (configgen *ConfigGeneratorImpl) buildHostRDSConfig(
|
||||
+ node *model.Proxy,
|
||||
+ req *model.PushRequest,
|
||||
+ routeName string,
|
||||
+ vsCache map[int][]virtualServiceContext,
|
||||
+ efw *model.EnvoyFilterWrapper,
|
||||
+ efKeys []string,
|
||||
+) (*discovery.Resource, bool) {
|
||||
+ push := req.Push
|
||||
var (
|
||||
hostRDSPort string
|
||||
hostRDSHost string
|
||||
@@ -432,7 +441,7 @@
|
||||
portAndHost := strings.SplitN(strings.TrimPrefix(routeName, constants.HigressHostRDSNamePrefix), ".", 2)
|
||||
if len(portAndHost) != 2 {
|
||||
log.Errorf("Invalid route %s when using Higress hostRDS", routeName)
|
||||
- return nil
|
||||
+ return nil, false
|
||||
}
|
||||
hostRDSPort = portAndHost[0]
|
||||
hostRDSHost = portAndHost[1]
|
||||
@@ -441,10 +450,24 @@
|
||||
rdsPort, err := strconv.Atoi(hostRDSPort)
|
||||
if err != nil {
|
||||
log.Errorf("Invalid port %s of route %s when using Higress hostRDS", hostRDSPort, routeName)
|
||||
- return nil
|
||||
+ return nil, false
|
||||
+ }
|
||||
+
|
||||
+ routeCache := &istio_route.Cache{
|
||||
+ RouteName: routeName,
|
||||
+ ProxyVersion: node.Metadata.IstioVersion,
|
||||
+ ListenerPort: rdsPort,
|
||||
+ // Use same host vs to cache, although the cache can be cleared when the port is different, this can be accepted
|
||||
+ VirtualServices: push.VirtualServicesForHost(node, hostRDSHost),
|
||||
+ EnvoyFilterKeys: efKeys,
|
||||
+ }
|
||||
+
|
||||
+ resource, exist := configgen.Cache.Get(routeCache)
|
||||
+ if exist {
|
||||
+ return resource, true
|
||||
}
|
||||
+
|
||||
listenerPort := uint32(rdsPort)
|
||||
- globalHTTPFilters := mseingress.ExtractGlobalHTTPFilters(node, push)
|
||||
|
||||
isH3DiscoveryNeeded := false
|
||||
|
||||
@@ -457,9 +480,9 @@
|
||||
break
|
||||
}
|
||||
}
|
||||
-
|
||||
gatewayRoutes := make(map[string]map[string][]*route.Route)
|
||||
gatewayVirtualServices := make(map[string][]config.Config)
|
||||
+ var listenerVirtualServices []virtualServiceContext
|
||||
var selectedVirtualServices []virtualServiceContext
|
||||
var vHost *route.VirtualHost
|
||||
serverIterator := func(mergedServers map[model.ServerPort]*model.MergedServers) {
|
||||
@@ -478,31 +501,8 @@
|
||||
gatewayVirtualServices[gatewayName] = virtualServices
|
||||
}
|
||||
for _, virtualService := range virtualServices {
|
||||
- hostMatch := false
|
||||
- var selectHost string
|
||||
- virtualServiceHosts := host.NewNames(virtualService.Spec.(*networking.VirtualService).Hosts)
|
||||
- for _, hostname := range virtualServiceHosts {
|
||||
- // exact match
|
||||
- if hostname == host.Name(hostRDSHost) {
|
||||
- hostMatch = true
|
||||
- selectHost = hostRDSHost
|
||||
- break
|
||||
- }
|
||||
- if features.HostRDSMergeSubset {
|
||||
- // subset match
|
||||
- if host.Name(hostRDSHost).SubsetOf(hostname) {
|
||||
- hostMatch = true
|
||||
- selectHost = string(hostname)
|
||||
- }
|
||||
- }
|
||||
- }
|
||||
- if !hostMatch {
|
||||
- continue
|
||||
- }
|
||||
- copiedVS := virtualService.DeepCopy()
|
||||
- copiedVS.Spec.(*networking.VirtualService).Hosts = []string{selectHost}
|
||||
- selectedVirtualServices = append(selectedVirtualServices, virtualServiceContext{
|
||||
- virtualService: copiedVS,
|
||||
+ listenerVirtualServices = append(listenerVirtualServices, virtualServiceContext{
|
||||
+ virtualService: virtualService,
|
||||
server: server,
|
||||
gatewayName: gatewayName,
|
||||
})
|
||||
@@ -510,15 +510,63 @@
|
||||
}
|
||||
}
|
||||
}
|
||||
- serverIterator(merged.MergedServers)
|
||||
- serverIterator(merged.MergedQUICTransportServers)
|
||||
- // Sort by subset
|
||||
- // before: ["*.abc.com", "*.com", "www.abc.com"]
|
||||
- // after: ["www.abc.com", "*.abc.com", "*.com"]
|
||||
- sort.SliceStable(selectedVirtualServices, func(i, j int) bool {
|
||||
- return host.Name(selectedVirtualServices[i].virtualService.Spec.(*networking.VirtualService).Hosts[0]).SubsetOf(
|
||||
- host.Name(selectedVirtualServices[j].virtualService.Spec.(*networking.VirtualService).Hosts[0]))
|
||||
- })
|
||||
+ var vsExists bool
|
||||
+ if listenerVirtualServices, vsExists = vsCache[rdsPort]; !vsExists {
|
||||
+ serverIterator(merged.MergedServers)
|
||||
+ serverIterator(merged.MergedQUICTransportServers)
|
||||
+ vsCache[rdsPort] = listenerVirtualServices
|
||||
+ }
|
||||
+ for _, vsCtx := range listenerVirtualServices {
|
||||
+ virtualService := vsCtx.virtualService
|
||||
+ hostMatch := false
|
||||
+ var selectHost string
|
||||
+ for _, hostname := range virtualService.Spec.(*networking.VirtualService).Hosts {
|
||||
+ // exact match
|
||||
+ if hostname == hostRDSHost {
|
||||
+ hostMatch = true
|
||||
+ selectHost = hostRDSHost
|
||||
+ break
|
||||
+ }
|
||||
+ if features.HostRDSMergeSubset {
|
||||
+ // subset match
|
||||
+ if host.Name(hostRDSHost).SubsetOf(host.Name(hostname)) {
|
||||
+ hostMatch = true
|
||||
+ selectHost = hostname
|
||||
+ }
|
||||
+ }
|
||||
+ }
|
||||
+ if !hostMatch {
|
||||
+ continue
|
||||
+ }
|
||||
+ if len(virtualService.Spec.(*networking.VirtualService).Hosts) > 1 {
|
||||
+ copiedVS := &networking.VirtualService{}
|
||||
+ copiedVS = virtualService.Spec.(*networking.VirtualService)
|
||||
+ copiedVS.Hosts = []string{selectHost}
|
||||
+ selectedVirtualServices = append(selectedVirtualServices, virtualServiceContext{
|
||||
+ virtualService: config.Config{
|
||||
+ Meta: virtualService.Meta,
|
||||
+ Spec: copiedVS,
|
||||
+ Status: virtualService.Status,
|
||||
+ },
|
||||
+ server: vsCtx.server,
|
||||
+ gatewayName: vsCtx.gatewayName,
|
||||
+ })
|
||||
+ } else {
|
||||
+ selectedVirtualServices = append(selectedVirtualServices, vsCtx)
|
||||
+ }
|
||||
+ }
|
||||
+ if features.HostRDSMergeSubset {
|
||||
+ // Sort by subset
|
||||
+ // before: ["*.abc.com", "*.com", "www.abc.com"]
|
||||
+ // after: ["www.abc.com", "*.abc.com", "*.com"]
|
||||
+ sort.SliceStable(selectedVirtualServices, func(i, j int) bool {
|
||||
+ return host.Name(selectedVirtualServices[i].virtualService.Spec.(*networking.VirtualService).Hosts[0]).SubsetOf(
|
||||
+ host.Name(selectedVirtualServices[j].virtualService.Spec.(*networking.VirtualService).Hosts[0]))
|
||||
+ })
|
||||
+ }
|
||||
+
|
||||
+ globalHTTPFilters := mseingress.ExtractGlobalHTTPFilters(node, push)
|
||||
+
|
||||
port := int(listenerPort)
|
||||
for _, ctx := range selectedVirtualServices {
|
||||
virtualService := ctx.virtualService
|
||||
@@ -605,25 +653,42 @@
|
||||
ValidateClusters: proto.BoolFalse,
|
||||
}
|
||||
|
||||
- return routeCfg
|
||||
+ routeCfg = envoyfilter.ApplyRouteConfigurationPatches(networking.EnvoyFilter_GATEWAY, node, efw, routeCfg)
|
||||
+ resource = &discovery.Resource{
|
||||
+ Name: routeName,
|
||||
+ Resource: util.MessageToAny(routeCfg),
|
||||
+ }
|
||||
+
|
||||
+ if features.EnableRDSCaching {
|
||||
+ configgen.Cache.Add(routeCache, req, resource)
|
||||
+ }
|
||||
+
|
||||
+ return resource, false
|
||||
}
|
||||
|
||||
// End added by ingress
|
||||
|
||||
-func (configgen *ConfigGeneratorImpl) buildGatewayHTTPRouteConfig(node *model.Proxy, push *model.PushContext,
|
||||
- routeName string) *route.RouteConfiguration {
|
||||
+// Modifed by ingress
|
||||
+func (configgen *ConfigGeneratorImpl) buildGatewayHTTPRouteConfig(
|
||||
+ node *model.Proxy,
|
||||
+ req *model.PushRequest,
|
||||
+ routeName string,
|
||||
+ vsCache map[int][]virtualServiceContext,
|
||||
+ efw *model.EnvoyFilterWrapper,
|
||||
+ efKeys []string,
|
||||
+) (*discovery.Resource, bool) {
|
||||
if node.MergedGateway == nil {
|
||||
log.Warnf("buildGatewayRoutes: no gateways for router %v", node.ID)
|
||||
- return &route.RouteConfiguration{
|
||||
- Name: routeName,
|
||||
- VirtualHosts: []*route.VirtualHost{},
|
||||
- ValidateClusters: proto.BoolFalse,
|
||||
- }
|
||||
+ return nil, false
|
||||
}
|
||||
-
|
||||
// Added by ingress
|
||||
+ push := req.Push
|
||||
if strings.HasPrefix(routeName, constants.HigressHostRDSNamePrefix) {
|
||||
- return configgen.buildHostRDSConfig(node, push, routeName)
|
||||
+ resource, cacheHit := configgen.buildHostRDSConfig(node, req, routeName, vsCache, efw, efKeys)
|
||||
+ if resource == nil {
|
||||
+ return nil, false
|
||||
+ }
|
||||
+ return resource, cacheHit
|
||||
}
|
||||
// End added by ingress
|
||||
|
||||
@@ -636,7 +701,7 @@
|
||||
|
||||
// This can happen when a gateway has recently been deleted. Envoy will still request route
|
||||
// information due to the draining of listeners, so we should not return an error.
|
||||
- return nil
|
||||
+ return nil, false
|
||||
}
|
||||
|
||||
servers := merged.ServersByRouteName[routeName]
|
||||
@@ -768,9 +833,16 @@
|
||||
ValidateClusters: proto.BoolFalse,
|
||||
}
|
||||
|
||||
- return routeCfg
|
||||
+ routeCfg = envoyfilter.ApplyRouteConfigurationPatches(networking.EnvoyFilter_GATEWAY, node, efw, routeCfg)
|
||||
+ resource := &discovery.Resource{
|
||||
+ Name: routeName,
|
||||
+ Resource: util.MessageToAny(routeCfg),
|
||||
+ }
|
||||
+ return resource, false
|
||||
}
|
||||
|
||||
+// End modified by ingress
|
||||
+
|
||||
// hashRouteList returns a hash of a list of pointers
|
||||
func hashRouteList(r []*route.Route) uint64 {
|
||||
hash := md5.New()
|
||||
diff -Naur istio/pilot/pkg/networking/core/v1alpha3/httproute.go istio-new/pilot/pkg/networking/core/v1alpha3/httproute.go
|
||||
--- istio/pilot/pkg/networking/core/v1alpha3/httproute.go 2024-01-15 20:46:41.000000000 +0800
|
||||
+++ istio-new/pilot/pkg/networking/core/v1alpha3/httproute.go 2024-01-15 10:29:09.000000000 +0800
|
||||
@@ -78,17 +78,30 @@
|
||||
routeConfigurations = append(routeConfigurations, rc)
|
||||
}
|
||||
case model.Router:
|
||||
+ // Modified by ingress
|
||||
+ vsCache := make(map[int][]virtualServiceContext)
|
||||
+ envoyfilterKeys := efw.Keys()
|
||||
for _, routeName := range routeNames {
|
||||
- rc := configgen.buildGatewayHTTPRouteConfig(node, req.Push, routeName)
|
||||
- if rc != nil {
|
||||
- rc = envoyfilter.ApplyRouteConfigurationPatches(networking.EnvoyFilter_GATEWAY, node, efw, rc)
|
||||
- resource := &discovery.Resource{
|
||||
+ rc, cached := configgen.buildGatewayHTTPRouteConfig(node, req, routeName, vsCache, efw, envoyfilterKeys)
|
||||
+ if cached && !features.EnableUnsafeAssertions {
|
||||
+ hit++
|
||||
+ } else {
|
||||
+ miss++
|
||||
+ }
|
||||
+ if rc == nil {
|
||||
+ emptyRoute := &route.RouteConfiguration{
|
||||
+ Name: routeName,
|
||||
+ VirtualHosts: []*route.VirtualHost{},
|
||||
+ ValidateClusters: proto.BoolFalse,
|
||||
+ }
|
||||
+ rc = &discovery.Resource{
|
||||
Name: routeName,
|
||||
- Resource: util.MessageToAny(rc),
|
||||
+ Resource: util.MessageToAny(emptyRoute),
|
||||
}
|
||||
- routeConfigurations = append(routeConfigurations, resource)
|
||||
}
|
||||
+ routeConfigurations = append(routeConfigurations, rc)
|
||||
}
|
||||
+ // End modified by ingress
|
||||
}
|
||||
if !features.EnableRDSCaching {
|
||||
return routeConfigurations, model.DefaultXdsLogDetails
|
||||
diff -Naur istio/pilot/pkg/xds/discovery.go istio-new/pilot/pkg/xds/discovery.go
|
||||
--- istio/pilot/pkg/xds/discovery.go 2024-01-15 20:46:45.000000000 +0800
|
||||
+++ istio-new/pilot/pkg/xds/discovery.go 2024-01-12 19:56:02.000000000 +0800
|
||||
@@ -392,6 +392,9 @@
|
||||
// ConfigUpdate implements ConfigUpdater interface, used to request pushes.
|
||||
// It replaces the 'clear cache' from v1.
|
||||
func (s *DiscoveryServer) ConfigUpdate(req *model.PushRequest) {
|
||||
+ if req.Full {
|
||||
+ log.Infof("full push happen, reason:%v", req.Reason)
|
||||
+ }
|
||||
inboundConfigUpdates.Increment()
|
||||
s.InboundUpdates.Inc()
|
||||
s.pushChannel <- req
|
||||
60
istio/1.12/patches/istio/20240201-optimize-default-arg.patch
Normal file
60
istio/1.12/patches/istio/20240201-optimize-default-arg.patch
Normal file
@@ -0,0 +1,60 @@
|
||||
diff -Naur istio/pilot/cmd/pilot-agent/status/util/stats.go istio-new/pilot/cmd/pilot-agent/status/util/stats.go
|
||||
--- istio/pilot/cmd/pilot-agent/status/util/stats.go 2024-02-01 10:20:13.000000000 +0800
|
||||
+++ istio-new/pilot/cmd/pilot-agent/status/util/stats.go 2024-01-31 22:44:53.000000000 +0800
|
||||
@@ -73,7 +73,7 @@
|
||||
localHostAddr = "localhost"
|
||||
}
|
||||
|
||||
- readinessURL := fmt.Sprintf("http://%s:%d/stats?usedonly&filter=%s", localHostAddr, adminPort, readyStatsRegex)
|
||||
+ readinessURL := fmt.Sprintf("http://%s:%d/stats?usedonly", localHostAddr, adminPort)
|
||||
stats, err := http.DoHTTPGetWithTimeout(readinessURL, readinessTimeout)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
@@ -105,7 +105,7 @@
|
||||
localHostAddr = "localhost"
|
||||
}
|
||||
|
||||
- stats, err := http.DoHTTPGet(fmt.Sprintf("http://%s:%d/stats?usedonly&filter=%s", localHostAddr, adminPort, updateStatsRegex))
|
||||
+ stats, err := http.DoHTTPGet(fmt.Sprintf("http://%s:%d/stats?usedonly", localHostAddr, adminPort))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
diff -Naur istio/pilot/pkg/features/pilot.go istio-new/pilot/pkg/features/pilot.go
|
||||
--- istio/pilot/pkg/features/pilot.go 2024-02-01 10:20:17.000000000 +0800
|
||||
+++ istio-new/pilot/pkg/features/pilot.go 2024-02-01 10:16:18.000000000 +0800
|
||||
@@ -575,6 +575,8 @@
|
||||
"If enabled, each host in virtualservice will have an independent RDS, which is used with SRDS").Get()
|
||||
OnDemandRDS = env.RegisterBoolVar("ON_DEMAND_RDS", false,
|
||||
"If enabled, the on demand filter will be added to the HCM filters").Get()
|
||||
+ DefaultUpstreamConcurrencyThreshold = env.RegisterIntVar("DEFAULT_UPSTREAM_CONCURRENCY_THRESHOLD", 1000000,
|
||||
+ "The default threshold of max_requests/max_pending_requests/max_connections of circuit breaker").Get()
|
||||
// End added by ingress
|
||||
)
|
||||
|
||||
diff -Naur istio/pilot/pkg/networking/core/v1alpha3/cluster.go istio-new/pilot/pkg/networking/core/v1alpha3/cluster.go
|
||||
--- istio/pilot/pkg/networking/core/v1alpha3/cluster.go 2024-02-01 10:20:17.000000000 +0800
|
||||
+++ istio-new/pilot/pkg/networking/core/v1alpha3/cluster.go 2024-02-01 10:16:05.000000000 +0800
|
||||
@@ -61,6 +61,7 @@
|
||||
|
||||
// getDefaultCircuitBreakerThresholds returns a copy of the default circuit breaker thresholds for the given traffic direction.
|
||||
func getDefaultCircuitBreakerThresholds() *cluster.CircuitBreakers_Thresholds {
|
||||
+ // Modified by ingress
|
||||
return &cluster.CircuitBreakers_Thresholds{
|
||||
// DefaultMaxRetries specifies the default for the Envoy circuit breaker parameter max_retries. This
|
||||
// defines the maximum number of parallel retries a given Envoy will allow to the upstream cluster. Envoy defaults
|
||||
@@ -68,11 +69,12 @@
|
||||
// where multiple endpoints in a cluster are terminated. In these scenarios the circuit breaker can kick
|
||||
// in before Pilot is able to deliver an updated endpoint list to Envoy, leading to client-facing 503s.
|
||||
MaxRetries: &wrappers.UInt32Value{Value: math.MaxUint32},
|
||||
- MaxRequests: &wrappers.UInt32Value{Value: math.MaxUint32},
|
||||
- MaxConnections: &wrappers.UInt32Value{Value: math.MaxUint32},
|
||||
- MaxPendingRequests: &wrappers.UInt32Value{Value: math.MaxUint32},
|
||||
+ MaxRequests: &wrappers.UInt32Value{Value: uint32(features.DefaultUpstreamConcurrencyThreshold)},
|
||||
+ MaxConnections: &wrappers.UInt32Value{Value: uint32(features.DefaultUpstreamConcurrencyThreshold)},
|
||||
+ MaxPendingRequests: &wrappers.UInt32Value{Value: uint32(features.DefaultUpstreamConcurrencyThreshold)},
|
||||
TrackRemaining: true,
|
||||
}
|
||||
+ // End modified by ingress
|
||||
}
|
||||
|
||||
// BuildClusters returns the list of clusters for the given proxy. This is the CDS output
|
||||
@@ -0,0 +1,88 @@
|
||||
diff -Naur istio/pilot/pkg/networking/core/v1alpha3/gateway.go istio-new/pilot/pkg/networking/core/v1alpha3/gateway.go
|
||||
--- istio/pilot/pkg/networking/core/v1alpha3/gateway.go 2024-02-01 13:53:17.000000000 +0800
|
||||
+++ istio-new/pilot/pkg/networking/core/v1alpha3/gateway.go 2024-02-01 13:52:11.000000000 +0800
|
||||
@@ -501,6 +501,16 @@
|
||||
gatewayVirtualServices[gatewayName] = virtualServices
|
||||
}
|
||||
for _, virtualService := range virtualServices {
|
||||
+ virtualServiceHosts := host.NewNames(virtualService.Spec.(*networking.VirtualService).Hosts)
|
||||
+ serverHosts := host.NamesForNamespace(server.Hosts, virtualService.Namespace)
|
||||
+
|
||||
+ // We have two cases here:
|
||||
+ // 1. virtualService hosts are 1.foo.com, 2.foo.com, 3.foo.com and server hosts are ns/*.foo.com
|
||||
+ // 2. virtualService hosts are *.foo.com, and server hosts are ns/1.foo.com, ns/2.foo.com, ns/3.foo.com
|
||||
+ intersectingHosts := serverHosts.Intersection(virtualServiceHosts)
|
||||
+ if len(intersectingHosts) == 0 {
|
||||
+ continue
|
||||
+ }
|
||||
listenerVirtualServices = append(listenerVirtualServices, virtualServiceContext{
|
||||
virtualService: virtualService,
|
||||
server: server,
|
||||
@@ -615,22 +625,24 @@
|
||||
|
||||
// check all hostname if is not exist with HttpsRedirect set to true
|
||||
// create VirtualHost to redirect
|
||||
- for _, hostname := range server.Hosts {
|
||||
- if !server.GetTls().GetHttpsRedirect() {
|
||||
- continue
|
||||
- }
|
||||
- if vHost != nil && host.Name(hostname) == host.Name(hostRDSHost) {
|
||||
+ if server.GetTls().GetHttpsRedirect() {
|
||||
+ if vHost != nil {
|
||||
vHost.RequireTls = route.VirtualHost_ALL
|
||||
- continue
|
||||
+ } else {
|
||||
+ vHost = &route.VirtualHost{
|
||||
+ Name: util.DomainName(hostRDSHost, port),
|
||||
+ Domains: buildGatewayVirtualHostDomains(hostRDSHost, port),
|
||||
+ IncludeRequestAttemptCount: true,
|
||||
+ RequireTls: route.VirtualHost_ALL,
|
||||
+ }
|
||||
}
|
||||
- vHost = &route.VirtualHost{
|
||||
- Name: util.DomainName(hostname, port),
|
||||
- Domains: buildGatewayVirtualHostDomains(hostname, port),
|
||||
- IncludeRequestAttemptCount: true,
|
||||
- RequireTls: route.VirtualHost_ALL,
|
||||
+ } else if vHost != nil {
|
||||
+ mode := server.GetTls().GetMode()
|
||||
+ if mode == networking.ServerTLSSettings_MUTUAL ||
|
||||
+ mode == networking.ServerTLSSettings_ISTIO_MUTUAL {
|
||||
+ vHost.AllowServerNames = append(vHost.AllowServerNames, server.Hosts...)
|
||||
}
|
||||
}
|
||||
-
|
||||
}
|
||||
var virtualHosts []*route.VirtualHost
|
||||
if vHost == nil {
|
||||
@@ -642,6 +654,30 @@
|
||||
Routes: []*route.Route{},
|
||||
}}
|
||||
} else {
|
||||
+ sort.SliceStable(vHost.AllowServerNames, func(i, j int) bool {
|
||||
+ hostI := vHost.AllowServerNames[i]
|
||||
+ hostJ := vHost.AllowServerNames[j]
|
||||
+ if host.Name(hostI).SubsetOf(host.Name(hostJ)) {
|
||||
+ return true
|
||||
+ }
|
||||
+ return hostI < hostJ
|
||||
+ })
|
||||
+ var uniqueServerNames []string
|
||||
+ hasAllCatch := false
|
||||
+ for i, name := range vHost.AllowServerNames {
|
||||
+ if name == "*" {
|
||||
+ hasAllCatch = true
|
||||
+ break
|
||||
+ }
|
||||
+ if i == 0 || vHost.AllowServerNames[i-1] != name {
|
||||
+ uniqueServerNames = append(uniqueServerNames, name)
|
||||
+ }
|
||||
+ }
|
||||
+ if hasAllCatch {
|
||||
+ vHost.AllowServerNames = nil
|
||||
+ } else {
|
||||
+ vHost.AllowServerNames = uniqueServerNames
|
||||
+ }
|
||||
vHost.Routes = istio_route.CombineVHostRoutes(vHost.Routes)
|
||||
virtualHosts = append(virtualHosts, vHost)
|
||||
}
|
||||
41
istio/1.12/patches/istio/20240202-fix-rds-cache.patch
Normal file
41
istio/1.12/patches/istio/20240202-fix-rds-cache.patch
Normal file
@@ -0,0 +1,41 @@
|
||||
diff -Naur istio/pilot/pkg/xds/discovery.go istio-new/pilot/pkg/xds/discovery.go
|
||||
--- istio/pilot/pkg/xds/discovery.go 2024-02-02 16:26:49.000000000 +0800
|
||||
+++ istio-new/pilot/pkg/xds/discovery.go 2024-02-02 15:38:53.000000000 +0800
|
||||
@@ -18,6 +18,7 @@
|
||||
"context"
|
||||
"fmt"
|
||||
"strconv"
|
||||
+ "strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@@ -41,6 +42,7 @@
|
||||
"istio.io/istio/pilot/pkg/util/sets"
|
||||
v3 "istio.io/istio/pilot/pkg/xds/v3"
|
||||
"istio.io/istio/pkg/cluster"
|
||||
+ "istio.io/istio/pkg/config/constants"
|
||||
"istio.io/istio/pkg/security"
|
||||
)
|
||||
|
||||
@@ -332,6 +334,21 @@
|
||||
} else {
|
||||
// Otherwise, just clear the updated configs
|
||||
s.Cache.Clear(req.ConfigsUpdated)
|
||||
+ //Added by ingress
|
||||
+ trimKeyMap := make(map[model.ConfigKey]struct{})
|
||||
+ for configKey := range req.ConfigsUpdated {
|
||||
+ if strings.HasPrefix(configKey.Name, constants.IstioIngressGatewayName+"-") {
|
||||
+ trimKeyMap[model.ConfigKey{
|
||||
+ Kind: configKey.Kind,
|
||||
+ Name: strings.TrimPrefix(configKey.Name, constants.IstioIngressGatewayName+"-"),
|
||||
+ Namespace: configKey.Namespace,
|
||||
+ }] = struct{}{}
|
||||
+ }
|
||||
+ }
|
||||
+ if len(trimKeyMap) > 0 {
|
||||
+ s.Cache.Clear(trimKeyMap)
|
||||
+ }
|
||||
+ //End added by ingress
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
diff -Naur istio/pilot/cmd/pilot-agent/status/util/stats.go istio-new/pilot/cmd/pilot-agent/status/util/stats.go
|
||||
--- istio/pilot/cmd/pilot-agent/status/util/stats.go 2024-02-04 18:48:18.000000000 +0800
|
||||
+++ istio-new/pilot/cmd/pilot-agent/status/util/stats.go 2024-02-04 09:35:42.000000000 +0800
|
||||
@@ -37,7 +37,7 @@
|
||||
updateStatsRegex = "^(cluster_manager\\.cds|listener_manager\\.lds)\\.(update_success|update_rejected)$"
|
||||
)
|
||||
|
||||
-var readinessTimeout = time.Second * 3 // Default Readiness timeout. It is set the same in helm charts.
|
||||
+var readinessTimeout = time.Second * 60 // Default Readiness timeout. It is set the same in helm charts.
|
||||
|
||||
type stat struct {
|
||||
name string
|
||||
@@ -105,7 +105,7 @@
|
||||
localHostAddr = "localhost"
|
||||
}
|
||||
|
||||
- stats, err := http.DoHTTPGet(fmt.Sprintf("http://%s:%d/stats?usedonly", localHostAddr, adminPort))
|
||||
+ stats, err := http.DoHTTPGetWithTimeout(fmt.Sprintf("http://%s:%d/stats?usedonly", localHostAddr, adminPort), readinessTimeout)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -584,7 +584,7 @@ func locateChart(cpOpts *action.ChartPathOptions, name string, settings *cli.Env
|
||||
return fileAbsPath, nil
|
||||
}
|
||||
|
||||
func ParseLatestVersion(repoUrl string, version string) (string, error) {
|
||||
func ParseLatestVersion(repoUrl string, version string, devel bool) (string, error) {
|
||||
|
||||
cpOpts := &action.ChartPathOptions{
|
||||
RepoURL: repoUrl,
|
||||
@@ -632,7 +632,16 @@ func ParseLatestVersion(repoUrl string, version string) (string, error) {
|
||||
|
||||
// get higress helm chart latest version
|
||||
if entries, ok := indexFile.Entries[RepoChartIndexYamlHigressIndex]; ok {
|
||||
return entries[0].AppVersion, nil
|
||||
if devel {
|
||||
return entries[0].AppVersion, nil
|
||||
}
|
||||
|
||||
if chatVersion, err := indexFile.Get(RepoChartIndexYamlHigressIndex, ""); err != nil {
|
||||
return "", errors.New("can't find higress latest version")
|
||||
} else {
|
||||
return chatVersion.Version, nil
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
return "", errors.New("can't find higress latest version")
|
||||
|
||||
@@ -52,6 +52,8 @@ type InstallArgs struct {
|
||||
Set []string
|
||||
// ManifestsPath is a path to a ManifestsPath and profiles directory in the local filesystem with a release tgz.
|
||||
ManifestsPath string
|
||||
// Devel if set true when version is latest, it will get latest version, otherwise it will get latest stable version
|
||||
Devel bool
|
||||
}
|
||||
|
||||
func (a *InstallArgs) String() string {
|
||||
@@ -67,6 +69,7 @@ func addInstallFlags(cmd *cobra.Command, args *InstallArgs) {
|
||||
cmd.PersistentFlags().StringSliceVarP(&args.InFilenames, "filename", "f", nil, filenameFlagHelpStr)
|
||||
cmd.PersistentFlags().StringArrayVarP(&args.Set, "set", "s", nil, setFlagHelpStr)
|
||||
cmd.PersistentFlags().StringVarP(&args.ManifestsPath, "manifests", "d", "", manifestsFlagHelpStr)
|
||||
cmd.PersistentFlags().BoolVar(&args.Devel, "devel", false, "use development versions (alpha, beta, and release candidate releases), If version is set, this is ignored")
|
||||
}
|
||||
|
||||
// --manifests is an alias for --set installPackagePath=
|
||||
@@ -141,7 +144,7 @@ func install(writer io.Writer, iArgs *InstallArgs) error {
|
||||
return err
|
||||
}
|
||||
|
||||
err = installManifests(profile, writer)
|
||||
err = installManifests(profile, writer, iArgs.Devel)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to install manifests: %v", err)
|
||||
}
|
||||
@@ -192,8 +195,8 @@ func promptProfileName(writer io.Writer) string {
|
||||
|
||||
}
|
||||
|
||||
func installManifests(profile *helm.Profile, writer io.Writer) error {
|
||||
installer, err := installer.NewInstaller(profile, writer, false)
|
||||
func installManifests(profile *helm.Profile, writer io.Writer, devel bool) error {
|
||||
installer, err := installer.NewInstaller(profile, writer, false, devel, installer.InstallInstallerMode)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -52,6 +52,8 @@ type ComponentOptions struct {
|
||||
Quiet bool
|
||||
// Capabilities
|
||||
Capabilities *chartutil.Capabilities
|
||||
// devel
|
||||
Devel bool
|
||||
}
|
||||
|
||||
type ComponentOption func(*ComponentOptions)
|
||||
@@ -98,6 +100,12 @@ func WithQuiet() ComponentOption {
|
||||
}
|
||||
}
|
||||
|
||||
func WithDevel(devel bool) ComponentOption {
|
||||
return func(opts *ComponentOptions) {
|
||||
opts.Devel = devel
|
||||
}
|
||||
}
|
||||
|
||||
func renderComponentManifest(spec any, renderer helm.Renderer, addOn bool, name ComponentName, namespace string) (string, error) {
|
||||
var valsBytes []byte
|
||||
var valsYaml string
|
||||
|
||||
@@ -52,7 +52,7 @@ func (h *HigressComponent) Run() error {
|
||||
// Parse latest version
|
||||
if h.opts.Version == helm.RepoLatestVersion {
|
||||
|
||||
latestVersion, err := helm.ParseLatestVersion(h.opts.RepoURL, h.opts.Version)
|
||||
latestVersion, err := helm.ParseLatestVersion(h.opts.RepoURL, h.opts.Version, h.opts.Devel)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -28,6 +28,8 @@ import (
|
||||
"k8s.io/client-go/util/homedir"
|
||||
)
|
||||
|
||||
type InstallerMode int32
|
||||
|
||||
const (
|
||||
HgctlHomeDirPath = ".hgctl"
|
||||
StandaloneInstalledPath = "higress-standalone"
|
||||
@@ -37,20 +39,26 @@ const (
|
||||
DefaultIstioNamespace = "istio-system"
|
||||
)
|
||||
|
||||
const (
|
||||
InstallInstallerMode InstallerMode = iota
|
||||
UpgradeInstallerMode
|
||||
UninstallInstallerMode
|
||||
)
|
||||
|
||||
type Installer interface {
|
||||
Install() error
|
||||
UnInstall() error
|
||||
Upgrade() error
|
||||
}
|
||||
|
||||
func NewInstaller(profile *helm.Profile, writer io.Writer, quiet bool) (Installer, error) {
|
||||
func NewInstaller(profile *helm.Profile, writer io.Writer, quiet bool, devel bool, installerMode InstallerMode) (Installer, error) {
|
||||
switch profile.Global.Install {
|
||||
case helm.InstallK8s, helm.InstallLocalK8s:
|
||||
cliClient, err := kubernetes.NewCLIClient(options.DefaultConfigFlags.ToRawKubeConfigLoader())
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to build kubernetes client: %w", err)
|
||||
}
|
||||
installer, err := NewK8sInstaller(profile, cliClient, writer, quiet)
|
||||
installer, err := NewK8sInstaller(profile, cliClient, writer, quiet, devel, installerMode)
|
||||
return installer, err
|
||||
case helm.InstallLocalDocker:
|
||||
installer, err := NewDockerInstaller(profile, writer, quiet)
|
||||
|
||||
@@ -254,7 +254,7 @@ func (o *K8sInstaller) isNamespacedObject(obj *object.K8sObject) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
func NewK8sInstaller(profile *helm.Profile, cli kubernetes.CLIClient, writer io.Writer, quiet bool) (*K8sInstaller, error) {
|
||||
func NewK8sInstaller(profile *helm.Profile, cli kubernetes.CLIClient, writer io.Writer, quiet bool, devel bool, installerMode InstallerMode) (*K8sInstaller, error) {
|
||||
if profile == nil {
|
||||
return nil, errors.New("install profile is empty")
|
||||
}
|
||||
@@ -267,14 +267,20 @@ func NewK8sInstaller(profile *helm.Profile, cli kubernetes.CLIClient, writer io.
|
||||
}
|
||||
fmt.Fprintf(writer, "%s\n", capabilities.KubeVersion.Version)
|
||||
// initialize components
|
||||
higressVersion := profile.Charts.Higress.Version
|
||||
if installerMode == UninstallInstallerMode {
|
||||
// uninstall
|
||||
higressVersion = profile.HigressVersion
|
||||
}
|
||||
components := make(map[ComponentName]Component)
|
||||
opts := []ComponentOption{
|
||||
WithComponentNamespace(profile.Global.Namespace),
|
||||
WithComponentChartPath(profile.InstallPackagePath),
|
||||
WithComponentVersion(profile.Charts.Higress.Version),
|
||||
WithComponentVersion(higressVersion),
|
||||
WithComponentRepoURL(profile.Charts.Higress.Url),
|
||||
WithComponentChartName(profile.Charts.Higress.Name),
|
||||
WithComponentCapabilities(capabilities),
|
||||
WithDevel(devel),
|
||||
}
|
||||
if quiet {
|
||||
opts = append(opts, WithQuiet())
|
||||
|
||||
@@ -37,6 +37,8 @@ type ManifestArgs struct {
|
||||
Set []string
|
||||
// ManifestsPath is a path to a ManifestsPath and profiles directory in the local filesystem with a release tgz.
|
||||
ManifestsPath string
|
||||
// Devel if set true when version is latest, it will get latest version, otherwise it will get latest stable version
|
||||
Devel bool
|
||||
}
|
||||
|
||||
func (a *ManifestArgs) String() string {
|
||||
@@ -70,6 +72,7 @@ func addManifestFlags(cmd *cobra.Command, args *ManifestArgs) {
|
||||
cmd.PersistentFlags().StringSliceVarP(&args.InFilenames, "filename", "f", nil, filenameFlagHelpStr)
|
||||
cmd.PersistentFlags().StringArrayVarP(&args.Set, "set", "s", nil, setFlagHelpStr)
|
||||
cmd.PersistentFlags().StringVarP(&args.ManifestsPath, "manifests", "d", "", manifestsFlagHelpStr)
|
||||
cmd.PersistentFlags().BoolVar(&args.Devel, "devel", false, "use development versions (alpha, beta, and release candidate releases), If version is set, this is ignored")
|
||||
}
|
||||
|
||||
// newManifestGenerateCmd generates a higress install manifest and applies it to a cluster
|
||||
@@ -113,20 +116,20 @@ func generate(writer io.Writer, iArgs *ManifestArgs) error {
|
||||
return err
|
||||
}
|
||||
|
||||
err = genManifests(profile, writer)
|
||||
err = genManifests(profile, writer, iArgs.Devel)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to install manifests: %v", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func genManifests(profile *helm.Profile, writer io.Writer) error {
|
||||
func genManifests(profile *helm.Profile, writer io.Writer, devel bool) error {
|
||||
cliClient, err := kubernetes.NewCLIClient(options.DefaultConfigFlags.ToRawKubeConfigLoader())
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to build kubernetes client: %w", err)
|
||||
}
|
||||
|
||||
op, err := installer.NewK8sInstaller(profile, cliClient, writer, true)
|
||||
op, err := installer.NewK8sInstaller(profile, cliClient, writer, true, devel, installer.InstallInstallerMode)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -126,7 +126,7 @@ func promptUninstall(writer io.Writer) bool {
|
||||
}
|
||||
|
||||
func uninstallManifests(profile *helm.Profile, writer io.Writer, uiArgs *uninstallArgs) error {
|
||||
installer, err := installer.NewInstaller(profile, writer, false)
|
||||
installer, err := installer.NewInstaller(profile, writer, false, false, installer.UninstallInstallerMode)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -37,6 +37,7 @@ func addUpgradeFlags(cmd *cobra.Command, args *upgradeArgs) {
|
||||
cmd.PersistentFlags().StringSliceVarP(&args.InFilenames, "filename", "f", nil, filenameFlagHelpStr)
|
||||
cmd.PersistentFlags().StringArrayVarP(&args.Set, "set", "s", nil, setFlagHelpStr)
|
||||
cmd.PersistentFlags().StringVarP(&args.ManifestsPath, "manifests", "d", "", manifestsFlagHelpStr)
|
||||
cmd.PersistentFlags().BoolVar(&args.Devel, "devel", false, "use development versions (alpha, beta, and release candidate releases), If version is set, this is ignored")
|
||||
}
|
||||
|
||||
// newUpgradeCmd upgrades Istio control plane in-place with eligibility checks.
|
||||
@@ -91,7 +92,7 @@ func upgrade(writer io.Writer, iArgs *InstallArgs) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
err = upgradeManifests(profile, writer)
|
||||
err = upgradeManifests(profile, writer, iArgs.Devel)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -120,8 +121,8 @@ func promptUpgrade(writer io.Writer) bool {
|
||||
}
|
||||
}
|
||||
|
||||
func upgradeManifests(profile *helm.Profile, writer io.Writer) error {
|
||||
installer, err := installer.NewInstaller(profile, writer, false)
|
||||
func upgradeManifests(profile *helm.Profile, writer io.Writer, devel bool) error {
|
||||
installer, err := installer.NewInstaller(profile, writer, false, devel, installer.UpgradeInstallerMode)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -19,6 +19,7 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
@@ -474,9 +475,6 @@ func (m *IngressConfig) convertVirtualService(configs []common.WrapperConfig) []
|
||||
gateways := []string{m.namespace + "/" +
|
||||
common.CreateConvertedName(m.clusterId, cleanHost),
|
||||
common.CreateConvertedName(constants.IstioIngressGatewayName, cleanHost)}
|
||||
if host != "*" {
|
||||
gateways = append(gateways, m.namespace+"/"+common.CreateConvertedName(m.clusterId, common.CleanHost("*")))
|
||||
}
|
||||
|
||||
wrapperVS, exist := convertOptions.VirtualServices[host]
|
||||
if !exist {
|
||||
@@ -673,6 +671,18 @@ func (m *IngressConfig) convertDestinationRule(configs []common.WrapperConfig) [
|
||||
|
||||
out := make([]config.Config, 0, len(destinationRules))
|
||||
for _, dr := range destinationRules {
|
||||
sort.SliceStable(dr.DestinationRule.TrafficPolicy.PortLevelSettings, func(i, j int) bool {
|
||||
portI := dr.DestinationRule.TrafficPolicy.PortLevelSettings[i].Port
|
||||
portJ := dr.DestinationRule.TrafficPolicy.PortLevelSettings[j].Port
|
||||
if portI == nil && portJ == nil {
|
||||
return true
|
||||
} else if portI == nil {
|
||||
return true
|
||||
} else if portJ == nil {
|
||||
return false
|
||||
}
|
||||
return portI.Number < portJ.Number
|
||||
})
|
||||
drName := util.CreateDestinationRuleName(m.clusterId, dr.ServiceKey.Namespace, dr.ServiceKey.Name)
|
||||
out = append(out, config.Config{
|
||||
Meta: config.Meta{
|
||||
|
||||
@@ -314,9 +314,6 @@ func (m *KIngressConfig) convertVirtualService(configs []common.WrapperConfig) [
|
||||
gateways := []string{m.namespace + "/" +
|
||||
common.CreateConvertedName(m.clusterId, cleanHost),
|
||||
common.CreateConvertedName(constants.IstioIngressGatewayName, cleanHost)}
|
||||
if host != "*" {
|
||||
gateways = append(gateways, m.namespace+"/"+common.CreateConvertedName(m.clusterId, common.CleanHost("*")))
|
||||
}
|
||||
|
||||
wrapperVS, exist := convertOptions.VirtualServices[host]
|
||||
if !exist {
|
||||
|
||||
@@ -83,7 +83,7 @@ func (i *Ingress) NeedRegexMatch(path string) bool {
|
||||
if i.Rewrite == nil {
|
||||
return false
|
||||
}
|
||||
if strings.ContainsAny(path, `\.+*?()|[]{}^$`) {
|
||||
if i.Rewrite.RewriteTarget != "" && strings.ContainsAny(path, `\.+*?()|[]{}^$`) {
|
||||
return true
|
||||
}
|
||||
if strings.ContainsAny(i.Rewrite.RewriteTarget, `$\`) {
|
||||
|
||||
@@ -67,6 +67,15 @@ func TestNeedRegexMatch(t *testing.T) {
|
||||
inputPath: "/.*",
|
||||
expect: true,
|
||||
},
|
||||
{
|
||||
input: &Ingress{
|
||||
Rewrite: &RewriteConfig{
|
||||
UseRegex: false,
|
||||
},
|
||||
},
|
||||
inputPath: "/.",
|
||||
expect: false,
|
||||
},
|
||||
{
|
||||
input: &Ingress{
|
||||
Rewrite: &RewriteConfig{
|
||||
|
||||
@@ -23,13 +23,14 @@ import (
|
||||
)
|
||||
|
||||
const (
|
||||
exact = "exact"
|
||||
regex = "regex"
|
||||
prefix = "prefix"
|
||||
MatchMethod = "match-method"
|
||||
MatchQuery = "match-query"
|
||||
MatchHeader = "match-header"
|
||||
sep = " "
|
||||
exact = "exact"
|
||||
regex = "regex"
|
||||
prefix = "prefix"
|
||||
MatchMethod = "match-method"
|
||||
MatchQuery = "match-query"
|
||||
MatchHeader = "match-header"
|
||||
MatchPseudoHeader = "match-pseudo-header"
|
||||
sep = " "
|
||||
)
|
||||
|
||||
var (
|
||||
@@ -56,6 +57,24 @@ func (m match) Parse(annotations Annotations, config *Ingress, _ *GlobalContext)
|
||||
IngressLog.Errorf("parse headers error %v within ingress %s/%s", err, config.Namespace, config.Name)
|
||||
}
|
||||
|
||||
var pseudoHeaderMatches map[string]map[string]string
|
||||
if pseudoHeaderMatches, err = m.matchByHeaderOrQueryParma(annotations, MatchPseudoHeader, pseudoHeaderMatches); err != nil {
|
||||
IngressLog.Errorf("parse headers error %v within ingress %s/%s", err, config.Namespace, config.Name)
|
||||
}
|
||||
if pseudoHeaderMatches != nil && len(pseudoHeaderMatches) > 0 {
|
||||
if config.Match.Headers == nil {
|
||||
config.Match.Headers = make(map[string]map[string]string)
|
||||
}
|
||||
for typ, mmap := range pseudoHeaderMatches {
|
||||
if config.Match.Headers[typ] == nil {
|
||||
config.Match.Headers[typ] = make(map[string]string)
|
||||
}
|
||||
for k, v := range mmap {
|
||||
config.Match.Headers[typ][":"+k] = v
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if config.Match.QueryParams, err = m.matchByHeaderOrQueryParma(annotations, MatchQuery, config.Match.QueryParams); err != nil {
|
||||
IngressLog.Errorf("parse query params error %v within ingress %s/%s", err, config.Namespace, config.Name)
|
||||
}
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
package annotations
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/google/go-cmp/cmp"
|
||||
@@ -112,11 +113,47 @@ func TestMatch_ParseHeaders(t *testing.T) {
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
typ: "exact",
|
||||
key: ":method",
|
||||
value: "GET",
|
||||
expect: map[string]map[string]string{
|
||||
exact: {
|
||||
":method": "GET",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
typ: "prefix",
|
||||
key: ":path",
|
||||
value: "/foo",
|
||||
expect: map[string]map[string]string{
|
||||
prefix: {
|
||||
":path": "/foo",
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
typ: "regex",
|
||||
key: ":authority",
|
||||
value: "test\\d+\\.com",
|
||||
expect: map[string]map[string]string{
|
||||
regex: {
|
||||
":authority": "test\\d+\\.com",
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range testCases {
|
||||
t.Run("", func(t *testing.T) {
|
||||
key := buildHigressAnnotationKey(tt.typ + "-" + MatchHeader + "-" + tt.key)
|
||||
matchKeyword := MatchHeader
|
||||
headerKey := tt.key
|
||||
if strings.HasPrefix(headerKey, ":") {
|
||||
headerKey = strings.TrimPrefix(headerKey, ":")
|
||||
matchKeyword = MatchPseudoHeader
|
||||
}
|
||||
key := buildHigressAnnotationKey(tt.typ + "-" + matchKeyword + "-" + headerKey)
|
||||
input := Annotations{key: tt.value}
|
||||
config := &Ingress{}
|
||||
_ = parser.Parse(input, config, nil)
|
||||
|
||||
@@ -65,7 +65,12 @@ func (u upstreamTLS) Parse(annotations Annotations, config *Ingress, _ *GlobalCo
|
||||
}
|
||||
|
||||
defer func() {
|
||||
config.UpstreamTLS = upstreamTLSConfig
|
||||
if upstreamTLSConfig.BackendProtocol == defaultBackendProtocol {
|
||||
// no need destination rule when use HTTP protocol
|
||||
config.UpstreamTLS = nil
|
||||
} else {
|
||||
config.UpstreamTLS = upstreamTLSConfig
|
||||
}
|
||||
}()
|
||||
|
||||
if proto, err := annotations.ParseStringASAP(backendProtocol); err == nil {
|
||||
|
||||
@@ -33,6 +33,12 @@ func TestUpstreamTLSParse(t *testing.T) {
|
||||
input: Annotations{},
|
||||
expect: nil,
|
||||
},
|
||||
{
|
||||
input: Annotations{
|
||||
buildNginxAnnotationKey(backendProtocol): "HTTP",
|
||||
},
|
||||
expect: nil,
|
||||
},
|
||||
{
|
||||
input: Annotations{
|
||||
buildNginxAnnotationKey(proxySSLSecret): "",
|
||||
|
||||
@@ -36,7 +36,6 @@ func ValidateBackendResource(resource *v1.TypedLocalObjectReference) bool {
|
||||
if resource == nil || resource.APIGroup == nil ||
|
||||
*resource.APIGroup != netv1.SchemeGroupVersion.Group ||
|
||||
resource.Kind != "McpBridge" || resource.Name != "default" {
|
||||
IngressLog.Warnf("invalid mcpbridge resource: %v", resource)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
|
||||
@@ -37,6 +37,7 @@ type HigressConfig struct {
|
||||
Tracing *Tracing `json:"tracing,omitempty"`
|
||||
Gzip *Gzip `json:"gzip,omitempty"`
|
||||
Downstream *Downstream `json:"downstream,omitempty"`
|
||||
Upstream *Upstream `json:"upstream,omitempty"`
|
||||
DisableXEnvoyHeaders bool `json:"disableXEnvoyHeaders,omitempty"`
|
||||
AddXRealIpHeader bool `json:"addXRealIpHeader,omitempty"`
|
||||
}
|
||||
@@ -47,6 +48,7 @@ func NewDefaultHigressConfig() *HigressConfig {
|
||||
Tracing: NewDefaultTracing(),
|
||||
Gzip: NewDefaultGzip(),
|
||||
Downstream: globalOption.Downstream,
|
||||
Upstream: globalOption.Upstream,
|
||||
DisableXEnvoyHeaders: globalOption.DisableXEnvoyHeaders,
|
||||
AddXRealIpHeader: globalOption.AddXRealIpHeader,
|
||||
}
|
||||
|
||||
@@ -15,7 +15,6 @@
|
||||
package configmap
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"sync/atomic"
|
||||
@@ -38,19 +37,22 @@ const (
|
||||
minInitialConnectionWindowSize = 65535
|
||||
maxInitialConnectionWindowSize = 2147483647
|
||||
|
||||
defaultIdleTimeout = 180
|
||||
defaultMaxRequestHeadersKb = 60
|
||||
defaultConnectionBufferLimits = 32768
|
||||
defaultMaxConcurrentStreams = 100
|
||||
defaultInitialStreamWindowSize = 65535
|
||||
defaultInitialConnectionWindowSize = 1048576
|
||||
defaultAddXRealIpHeader = false
|
||||
defaultDisableXEnvoyHeaders = false
|
||||
defaultIdleTimeout = 180
|
||||
defaultUpStreamIdleTimeout = 10
|
||||
defaultUpStreamConnectionBufferLimits = 10485760
|
||||
defaultMaxRequestHeadersKb = 60
|
||||
defaultConnectionBufferLimits = 32768
|
||||
defaultMaxConcurrentStreams = 100
|
||||
defaultInitialStreamWindowSize = 65535
|
||||
defaultInitialConnectionWindowSize = 1048576
|
||||
defaultAddXRealIpHeader = false
|
||||
defaultDisableXEnvoyHeaders = false
|
||||
)
|
||||
|
||||
// Global configures the behavior of the downstream connection, x-real-ip header and x-envoy headers.
|
||||
type Global struct {
|
||||
Downstream *Downstream `json:"downstream,omitempty"`
|
||||
Upstream *Upstream `json:"upstream,omitempty"`
|
||||
AddXRealIpHeader bool `json:"addXRealIpHeader,omitempty"`
|
||||
DisableXEnvoyHeaders bool `json:"disableXEnvoyHeaders,omitempty"`
|
||||
}
|
||||
@@ -58,7 +60,7 @@ type Global struct {
|
||||
// Downstream configures the behavior of the downstream connection.
|
||||
type Downstream struct {
|
||||
// IdleTimeout limits the time that a connection may be idle and stream idle.
|
||||
IdleTimeout uint32 `json:"idleTimeout,omitempty"`
|
||||
IdleTimeout uint32 `json:"idleTimeout"`
|
||||
// MaxRequestHeadersKb limits the size of request headers allowed.
|
||||
MaxRequestHeadersKb uint32 `json:"maxRequestHeadersKb,omitempty"`
|
||||
// ConnectionBufferLimits configures the buffer size limits for connections.
|
||||
@@ -67,6 +69,14 @@ type Downstream struct {
|
||||
Http2 *Http2 `json:"http2,omitempty"`
|
||||
}
|
||||
|
||||
// Upstream configures the behavior of the upstream connection.
|
||||
type Upstream struct {
|
||||
// IdleTimeout limits the time that a connection may be idle on the upstream.
|
||||
IdleTimeout uint32 `json:"idleTimeout"`
|
||||
// ConnectionBufferLimits configures the buffer size limits for connections.
|
||||
ConnectionBufferLimits uint32 `json:"connectionBufferLimits,omitempty"`
|
||||
}
|
||||
|
||||
// Http2 configures HTTP/2 specific options.
|
||||
type Http2 struct {
|
||||
// MaxConcurrentStreams limits the number of concurrent streams allowed.
|
||||
@@ -125,7 +135,7 @@ func compareGlobal(old *Global, new *Global) (Result, error) {
|
||||
return ResultDelete, nil
|
||||
}
|
||||
|
||||
if new.Downstream == nil && !new.AddXRealIpHeader && !new.DisableXEnvoyHeaders {
|
||||
if new.Downstream == nil && new.Upstream == nil && !new.AddXRealIpHeader && !new.DisableXEnvoyHeaders {
|
||||
return ResultDelete, nil
|
||||
}
|
||||
|
||||
@@ -139,18 +149,30 @@ func compareGlobal(old *Global, new *Global) (Result, error) {
|
||||
// deepCopyGlobal deep copies the global option.
|
||||
func deepCopyGlobal(global *Global) (*Global, error) {
|
||||
newGlobal := NewDefaultGlobalOption()
|
||||
bytes, err := json.Marshal(global)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
if global.Downstream != nil {
|
||||
newGlobal.Downstream.IdleTimeout = global.Downstream.IdleTimeout
|
||||
newGlobal.Downstream.MaxRequestHeadersKb = global.Downstream.MaxRequestHeadersKb
|
||||
newGlobal.Downstream.ConnectionBufferLimits = global.Downstream.ConnectionBufferLimits
|
||||
if global.Downstream.Http2 != nil {
|
||||
newGlobal.Downstream.Http2.MaxConcurrentStreams = global.Downstream.Http2.MaxConcurrentStreams
|
||||
newGlobal.Downstream.Http2.InitialStreamWindowSize = global.Downstream.Http2.InitialStreamWindowSize
|
||||
newGlobal.Downstream.Http2.InitialConnectionWindowSize = global.Downstream.Http2.InitialConnectionWindowSize
|
||||
}
|
||||
}
|
||||
err = json.Unmarshal(bytes, newGlobal)
|
||||
return newGlobal, err
|
||||
if global.Upstream != nil {
|
||||
newGlobal.Upstream.IdleTimeout = global.Upstream.IdleTimeout
|
||||
newGlobal.Upstream.ConnectionBufferLimits = global.Upstream.ConnectionBufferLimits
|
||||
}
|
||||
newGlobal.AddXRealIpHeader = global.AddXRealIpHeader
|
||||
newGlobal.DisableXEnvoyHeaders = global.DisableXEnvoyHeaders
|
||||
return newGlobal, nil
|
||||
}
|
||||
|
||||
// NewDefaultGlobalOption returns a default global config.
|
||||
func NewDefaultGlobalOption() *Global {
|
||||
return &Global{
|
||||
Downstream: NewDefaultDownstream(),
|
||||
Upstream: NewDefaultUpStream(),
|
||||
AddXRealIpHeader: defaultAddXRealIpHeader,
|
||||
DisableXEnvoyHeaders: defaultDisableXEnvoyHeaders,
|
||||
}
|
||||
@@ -166,6 +188,14 @@ func NewDefaultDownstream() *Downstream {
|
||||
}
|
||||
}
|
||||
|
||||
// NewDefaultUpStream returns a default upstream config.
|
||||
func NewDefaultUpStream() *Upstream {
|
||||
return &Upstream{
|
||||
IdleTimeout: defaultUpStreamIdleTimeout,
|
||||
ConnectionBufferLimits: defaultUpStreamConnectionBufferLimits,
|
||||
}
|
||||
}
|
||||
|
||||
// NewDefaultHttp2 returns a default http2 config.
|
||||
func NewDefaultHttp2() *Http2 {
|
||||
return &Http2{
|
||||
@@ -215,12 +245,14 @@ func (g *GlobalOptionController) GetName() string {
|
||||
func (g *GlobalOptionController) AddOrUpdateHigressConfig(name util.ClusterNamespacedName, old *HigressConfig, new *HigressConfig) error {
|
||||
newGlobal := &Global{
|
||||
Downstream: new.Downstream,
|
||||
Upstream: new.Upstream,
|
||||
AddXRealIpHeader: new.AddXRealIpHeader,
|
||||
DisableXEnvoyHeaders: new.DisableXEnvoyHeaders,
|
||||
}
|
||||
|
||||
oldGlobal := &Global{
|
||||
Downstream: old.Downstream,
|
||||
Upstream: old.Upstream,
|
||||
AddXRealIpHeader: old.AddXRealIpHeader,
|
||||
DisableXEnvoyHeaders: old.DisableXEnvoyHeaders,
|
||||
}
|
||||
@@ -264,6 +296,7 @@ func (g *GlobalOptionController) ValidHigressConfig(higressConfig *HigressConfig
|
||||
|
||||
global := &Global{
|
||||
Downstream: higressConfig.Downstream,
|
||||
Upstream: higressConfig.Upstream,
|
||||
AddXRealIpHeader: higressConfig.AddXRealIpHeader,
|
||||
DisableXEnvoyHeaders: higressConfig.DisableXEnvoyHeaders,
|
||||
}
|
||||
@@ -306,6 +339,18 @@ func (g *GlobalOptionController) ConstructEnvoyFilters() ([]*config.Config, erro
|
||||
downstreamConfig := g.generateDownstreamEnvoyFilter(downstreamStruct, bufferLimitStruct, namespace)
|
||||
configPatch = append(configPatch, downstreamConfig...)
|
||||
|
||||
if global.Upstream == nil {
|
||||
return generateEnvoyFilter(namespace, configPatch), nil
|
||||
}
|
||||
|
||||
upstreamStruct := g.constructUpstream(global.Upstream)
|
||||
bufferLimitStruct = g.constructUpstreamBufferLimit(global.Upstream)
|
||||
if len(upstreamStruct) == 0 {
|
||||
return generateEnvoyFilter(namespace, configPatch), nil
|
||||
}
|
||||
upstreamConfig := g.generateUpstreamEnvoyFilter(upstreamStruct, bufferLimitStruct, namespace)
|
||||
configPatch = append(configPatch, upstreamConfig...)
|
||||
|
||||
return generateEnvoyFilter(namespace, configPatch), nil
|
||||
}
|
||||
|
||||
@@ -365,6 +410,32 @@ func (g *GlobalOptionController) generateDownstreamEnvoyFilter(downstreamValueSt
|
||||
return downstreamConfig
|
||||
}
|
||||
|
||||
func (g *GlobalOptionController) generateUpstreamEnvoyFilter(upstreamValueStruct string, bufferLimit string, namespace string) []*networking.EnvoyFilter_EnvoyConfigObjectPatch {
|
||||
upstreamConfig := []*networking.EnvoyFilter_EnvoyConfigObjectPatch{
|
||||
{
|
||||
ApplyTo: networking.EnvoyFilter_CLUSTER,
|
||||
Match: &networking.EnvoyFilter_EnvoyConfigObjectMatch{
|
||||
Context: networking.EnvoyFilter_GATEWAY,
|
||||
},
|
||||
Patch: &networking.EnvoyFilter_Patch{
|
||||
Operation: networking.EnvoyFilter_Patch_MERGE,
|
||||
Value: util.BuildPatchStruct(upstreamValueStruct),
|
||||
},
|
||||
},
|
||||
{
|
||||
ApplyTo: networking.EnvoyFilter_CLUSTER,
|
||||
Match: &networking.EnvoyFilter_EnvoyConfigObjectMatch{
|
||||
Context: networking.EnvoyFilter_GATEWAY,
|
||||
},
|
||||
Patch: &networking.EnvoyFilter_Patch{
|
||||
Operation: networking.EnvoyFilter_Patch_MERGE,
|
||||
Value: util.BuildPatchStruct(bufferLimit),
|
||||
},
|
||||
},
|
||||
}
|
||||
return upstreamConfig
|
||||
}
|
||||
|
||||
// generateAddXRealIpHeaderEnvoyFilter generates the add x-real-ip header envoy filter.
|
||||
func (g *GlobalOptionController) generateAddXRealIpHeaderEnvoyFilter(addXRealIpHeaderStruct string, namespace string) []*networking.EnvoyFilter_EnvoyConfigObjectPatch {
|
||||
addXRealIpHeaderConfig := []*networking.EnvoyFilter_EnvoyConfigObjectPatch{
|
||||
@@ -460,6 +531,32 @@ func (g *GlobalOptionController) constructDownstream(downstream *Downstream) str
|
||||
return downstreamConfig
|
||||
}
|
||||
|
||||
// constructUpstream constructs the upstream config.
|
||||
func (g *GlobalOptionController) constructUpstream(upstream *Upstream) string {
|
||||
upstreamConfig := ""
|
||||
idleTimeout := upstream.IdleTimeout
|
||||
|
||||
upstreamConfig = fmt.Sprintf(`
|
||||
{
|
||||
"common_http_protocol_options": {
|
||||
"idleTimeout": "%ds"
|
||||
}
|
||||
}
|
||||
`, idleTimeout)
|
||||
|
||||
return upstreamConfig
|
||||
}
|
||||
|
||||
// constructUpstreamBufferLimit constructs the upstream buffer limit config.
|
||||
func (g *GlobalOptionController) constructUpstreamBufferLimit(upstream *Upstream) string {
|
||||
upstreamBufferLimitStruct := fmt.Sprintf(`
|
||||
{
|
||||
"per_connection_buffer_limit_bytes": %d
|
||||
}
|
||||
`, upstream.ConnectionBufferLimits)
|
||||
return upstreamBufferLimitStruct
|
||||
}
|
||||
|
||||
// constructAddXRealIpHeader constructs the add x-real-ip header config.
|
||||
func (g *GlobalOptionController) constructAddXRealIpHeader() string {
|
||||
addXRealIpHeaderStruct := fmt.Sprintf(`
|
||||
|
||||
@@ -41,6 +41,7 @@ func Test_validGlobal(t *testing.T) {
|
||||
name: "downstream nil",
|
||||
global: &Global{
|
||||
Downstream: nil,
|
||||
Upstream: NewDefaultUpStream(),
|
||||
AddXRealIpHeader: true,
|
||||
DisableXEnvoyHeaders: true,
|
||||
},
|
||||
@@ -131,35 +132,27 @@ func Test_deepCopyGlobal(t *testing.T) {
|
||||
name: "deep copy 2",
|
||||
global: &Global{
|
||||
Downstream: &Downstream{
|
||||
IdleTimeout: 1,
|
||||
IdleTimeout: 0,
|
||||
MaxRequestHeadersKb: 9600,
|
||||
ConnectionBufferLimits: 4096,
|
||||
Http2: NewDefaultHttp2(),
|
||||
},
|
||||
Upstream: &Upstream{
|
||||
IdleTimeout: 10,
|
||||
},
|
||||
AddXRealIpHeader: true,
|
||||
DisableXEnvoyHeaders: true,
|
||||
},
|
||||
want: &Global{
|
||||
Downstream: &Downstream{
|
||||
IdleTimeout: 1,
|
||||
IdleTimeout: 0,
|
||||
MaxRequestHeadersKb: 9600,
|
||||
ConnectionBufferLimits: 4096,
|
||||
Http2: NewDefaultHttp2(),
|
||||
},
|
||||
AddXRealIpHeader: true,
|
||||
DisableXEnvoyHeaders: true,
|
||||
},
|
||||
wantErr: nil,
|
||||
},
|
||||
{
|
||||
name: "deep copy 3",
|
||||
global: &Global{
|
||||
Downstream: &Downstream{},
|
||||
AddXRealIpHeader: true,
|
||||
DisableXEnvoyHeaders: true,
|
||||
},
|
||||
want: &Global{
|
||||
Downstream: NewDefaultDownstream(),
|
||||
Upstream: &Upstream{
|
||||
IdleTimeout: 10,
|
||||
},
|
||||
AddXRealIpHeader: true,
|
||||
DisableXEnvoyHeaders: true,
|
||||
},
|
||||
@@ -203,7 +196,13 @@ func Test_AddOrUpdateHigressConfig(t *testing.T) {
|
||||
old: NewDefaultHigressConfig(),
|
||||
new: &HigressConfig{
|
||||
Downstream: &Downstream{
|
||||
IdleTimeout: 1,
|
||||
IdleTimeout: 1,
|
||||
MaxRequestHeadersKb: defaultMaxRequestHeadersKb,
|
||||
ConnectionBufferLimits: defaultConnectionBufferLimits,
|
||||
Http2: NewDefaultHttp2(),
|
||||
},
|
||||
Upstream: &Upstream{
|
||||
IdleTimeout: 10,
|
||||
},
|
||||
AddXRealIpHeader: true,
|
||||
DisableXEnvoyHeaders: true,
|
||||
@@ -217,6 +216,9 @@ func Test_AddOrUpdateHigressConfig(t *testing.T) {
|
||||
ConnectionBufferLimits: defaultConnectionBufferLimits,
|
||||
Http2: NewDefaultHttp2(),
|
||||
},
|
||||
Upstream: &Upstream{
|
||||
IdleTimeout: 10,
|
||||
},
|
||||
AddXRealIpHeader: true,
|
||||
DisableXEnvoyHeaders: true,
|
||||
},
|
||||
@@ -225,6 +227,7 @@ func Test_AddOrUpdateHigressConfig(t *testing.T) {
|
||||
name: "delete and push",
|
||||
old: &HigressConfig{
|
||||
Downstream: NewDefaultDownstream(),
|
||||
Upstream: NewDefaultUpStream(),
|
||||
AddXRealIpHeader: defaultAddXRealIpHeader,
|
||||
DisableXEnvoyHeaders: defaultDisableXEnvoyHeaders,
|
||||
},
|
||||
@@ -233,6 +236,7 @@ func Test_AddOrUpdateHigressConfig(t *testing.T) {
|
||||
wantEventPush: "push",
|
||||
wantGlobal: &Global{
|
||||
Downstream: NewDefaultDownstream(),
|
||||
Upstream: NewDefaultUpStream(),
|
||||
AddXRealIpHeader: defaultAddXRealIpHeader,
|
||||
DisableXEnvoyHeaders: defaultDisableXEnvoyHeaders,
|
||||
},
|
||||
|
||||
@@ -1279,8 +1279,10 @@ func createRuleKey(annots map[string]string, hostAndPath string) string {
|
||||
if idx := strings.Index(k, annotations.MatchHeader); idx != -1 {
|
||||
key := k[start:idx] + k[idx+len(annotations.MatchHeader)+1:]
|
||||
headers = append(headers, [2]string{key, val})
|
||||
}
|
||||
if idx := strings.Index(k, annotations.MatchQuery); idx != -1 {
|
||||
} else if idx := strings.Index(k, annotations.MatchPseudoHeader); idx != -1 {
|
||||
key := k[start:idx] + ":" + k[idx+len(annotations.MatchPseudoHeader)+1:]
|
||||
headers = append(headers, [2]string{key, val})
|
||||
} else if idx := strings.Index(k, annotations.MatchQuery); idx != -1 {
|
||||
key := k[start:idx] + k[idx+len(annotations.MatchQuery)+1:]
|
||||
params = append(params, [2]string{key, val})
|
||||
}
|
||||
|
||||
@@ -1302,15 +1302,18 @@ func TestCreateRuleKey(t *testing.T) {
|
||||
}
|
||||
|
||||
annots := annotations.Annotations{
|
||||
buildHigressAnnotationKey(annotations.MatchMethod): "GET PUT",
|
||||
buildHigressAnnotationKey("exact-" + annotations.MatchHeader + "-abc"): "123",
|
||||
buildHigressAnnotationKey("prefix-" + annotations.MatchHeader + "-def"): "456",
|
||||
buildHigressAnnotationKey("exact-" + annotations.MatchQuery + "-region"): "beijing",
|
||||
buildHigressAnnotationKey("prefix-" + annotations.MatchQuery + "-user-id"): "user-",
|
||||
buildHigressAnnotationKey(annotations.MatchMethod): "GET PUT",
|
||||
buildHigressAnnotationKey("exact-" + annotations.MatchHeader + "-abc"): "123",
|
||||
buildHigressAnnotationKey("prefix-" + annotations.MatchHeader + "-def"): "456",
|
||||
buildHigressAnnotationKey("exact-" + annotations.MatchPseudoHeader + "-authority"): "foo.bar.com",
|
||||
buildHigressAnnotationKey("prefix-" + annotations.MatchPseudoHeader + "-scheme"): "htt",
|
||||
buildHigressAnnotationKey("exact-" + annotations.MatchQuery + "-region"): "beijing",
|
||||
buildHigressAnnotationKey("prefix-" + annotations.MatchQuery + "-user-id"): "user-",
|
||||
}
|
||||
expect := "higress.com-prefix-/foo" + sep + //host-pathType-path
|
||||
"GET PUT" + sep + // method
|
||||
"exact-abc\t123" + "\n" + "prefix-def\t456" + sep + // header
|
||||
"exact-:authority\tfoo.bar.com" + "\n" + "exact-abc\t123" + "\n" +
|
||||
"prefix-:scheme\thtt" + "\n" + "prefix-def\t456" + sep + // header
|
||||
"exact-region\tbeijing" + "\n" + "prefix-user-id\tuser-" + sep // params
|
||||
|
||||
key := createRuleKey(annots, wrapperHttpRoute.PathFormat())
|
||||
|
||||
@@ -1226,8 +1226,10 @@ func createRuleKey(annots map[string]string, hostAndPath string) string {
|
||||
if idx := strings.Index(k, annotations.MatchHeader); idx != -1 {
|
||||
key := k[start:idx] + k[idx+len(annotations.MatchHeader)+1:]
|
||||
headers = append(headers, [2]string{key, val})
|
||||
}
|
||||
if idx := strings.Index(k, annotations.MatchQuery); idx != -1 {
|
||||
} else if idx := strings.Index(k, annotations.MatchPseudoHeader); idx != -1 {
|
||||
key := k[start:idx] + ":" + k[idx+len(annotations.MatchPseudoHeader)+1:]
|
||||
headers = append(headers, [2]string{key, val})
|
||||
} else if idx := strings.Index(k, annotations.MatchQuery); idx != -1 {
|
||||
key := k[start:idx] + k[idx+len(annotations.MatchQuery)+1:]
|
||||
params = append(params, [2]string{key, val})
|
||||
}
|
||||
|
||||
@@ -699,8 +699,10 @@ func createRuleKey(annots map[string]string, hostAndPath string) string {
|
||||
if idx := strings.Index(k, annotations.MatchHeader); idx != -1 {
|
||||
key := k[start:idx] + k[idx+len(annotations.MatchHeader)+1:]
|
||||
headers = append(headers, [2]string{key, val})
|
||||
}
|
||||
if idx := strings.Index(k, annotations.MatchQuery); idx != -1 {
|
||||
} else if idx := strings.Index(k, annotations.MatchPseudoHeader); idx != -1 {
|
||||
key := k[start:idx] + ":" + k[idx+len(annotations.MatchPseudoHeader)+1:]
|
||||
headers = append(headers, [2]string{key, val})
|
||||
} else if idx := strings.Index(k, annotations.MatchQuery); idx != -1 {
|
||||
key := k[start:idx] + k[idx+len(annotations.MatchQuery)+1:]
|
||||
params = append(params, [2]string{key, val})
|
||||
}
|
||||
|
||||
@@ -581,15 +581,18 @@ func TestCreateRuleKey(t *testing.T) {
|
||||
}
|
||||
|
||||
annots := annotations.Annotations{
|
||||
buildHigressAnnotationKey(annotations.MatchMethod): "GET PUT",
|
||||
buildHigressAnnotationKey("exact-" + annotations.MatchHeader + "-abc"): "123",
|
||||
buildHigressAnnotationKey("prefix-" + annotations.MatchHeader + "-def"): "456",
|
||||
buildHigressAnnotationKey("exact-" + annotations.MatchQuery + "-region"): "beijing",
|
||||
buildHigressAnnotationKey("prefix-" + annotations.MatchQuery + "-user-id"): "user-",
|
||||
buildHigressAnnotationKey(annotations.MatchMethod): "GET PUT",
|
||||
buildHigressAnnotationKey("exact-" + annotations.MatchHeader + "-abc"): "123",
|
||||
buildHigressAnnotationKey("prefix-" + annotations.MatchHeader + "-def"): "456",
|
||||
buildHigressAnnotationKey("exact-" + annotations.MatchPseudoHeader + "-authority"): "foo.bar.com",
|
||||
buildHigressAnnotationKey("prefix-" + annotations.MatchPseudoHeader + "-scheme"): "htt",
|
||||
buildHigressAnnotationKey("exact-" + annotations.MatchQuery + "-region"): "beijing",
|
||||
buildHigressAnnotationKey("prefix-" + annotations.MatchQuery + "-user-id"): "user-",
|
||||
}
|
||||
expect := "higress.com-prefix-/foo" + sep + //host-pathType-path
|
||||
"GET PUT" + sep + // method
|
||||
"exact-abc\t123" + "\n" + "prefix-def\t456" + sep + // header
|
||||
"exact-:authority\tfoo.bar.com" + "\n" + "exact-abc\t123" + "\n" +
|
||||
"prefix-:scheme\thtt" + "\n" + "prefix-def\t456" + sep + // header
|
||||
"exact-region\tbeijing" + "\n" + "prefix-user-id\tuser-" + sep // params
|
||||
|
||||
key := createRuleKey(annots, wrapperHttpRoute.PathFormat())
|
||||
|
||||
23
plugins/wasm-go/.devcontainer/Dockerfile
Normal file
23
plugins/wasm-go/.devcontainer/Dockerfile
Normal file
@@ -0,0 +1,23 @@
|
||||
FROM higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/gateway:1.3.1
|
||||
|
||||
FROM ubuntu:20.04
|
||||
|
||||
RUN apt -y update \
|
||||
&& apt install -y --no-install-recommends python3-pip net-tools vim wget make curl git 2>&1 \
|
||||
&& apt install -y --reinstall ca-certificates \
|
||||
&& apt-get autoremove -y && apt-get clean \
|
||||
&& rm -rf /tmp/* /var/tmp/* \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
ENV PATH=/opt/tinygo/bin:/opt/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||
|
||||
RUN wget --no-check-certificate https://github.com/tinygo-org/tinygo/releases/download/v0.29.0/tinygo0.29.0.linux-amd64.tar.gz \
|
||||
&& tar -zvxf tinygo0.29.0.linux-amd64.tar.gz -C /opt \
|
||||
&& rm tinygo0.29.0.linux-amd64.tar.gz
|
||||
|
||||
RUN wget --no-check-certificate https://go.dev/dl/go1.19.linux-amd64.tar.gz \
|
||||
&& tar -zvxf go1.19.linux-amd64.tar.gz -C /opt \
|
||||
&& rm go1.19.linux-amd64.tar.gz \
|
||||
&& go install -v golang.org/x/tools/gopls@latest
|
||||
|
||||
COPY --from=0 /usr/local/bin/envoy /usr/local/bin/envoy
|
||||
21
plugins/wasm-go/.devcontainer/devcontainer.json
Normal file
21
plugins/wasm-go/.devcontainer/devcontainer.json
Normal file
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"name": "Wasm Go Dev",
|
||||
// "dockerFile": "Dockerfile",
|
||||
"image": "liuxr25/wasm-go:tinygo-0.29.0",
|
||||
"runArgs": [
|
||||
"--user=root"
|
||||
],
|
||||
"customizations": {
|
||||
"vscode": {
|
||||
"settings": {
|
||||
"terminal.integrated.shell.linux": "/bin/bash",
|
||||
"python.pythonPath": "/usr/bin/python3"
|
||||
},
|
||||
"extensions": [
|
||||
"ms-python.python",
|
||||
"golang.go"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
76
plugins/wasm-go/.devcontainer/gen_config.py
Normal file
76
plugins/wasm-go/.devcontainer/gen_config.py
Normal file
@@ -0,0 +1,76 @@
|
||||
import json
|
||||
import sys
|
||||
|
||||
|
||||
plugin_name = sys.argv[1]
|
||||
|
||||
with open("extensions/"+plugin_name+"/config.json", "r") as f:
|
||||
plugin_config = json.load(f)
|
||||
|
||||
config = f'''static_resources:
|
||||
listeners:
|
||||
- address:
|
||||
socket_address:
|
||||
address: 0.0.0.0
|
||||
port_value: 8080
|
||||
filter_chains:
|
||||
- filters:
|
||||
- name: envoy.filters.network.http_connection_manager
|
||||
typed_config:
|
||||
'@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
|
||||
codec_type: AUTO
|
||||
stat_prefix: ingress_http
|
||||
route_config:
|
||||
name: test
|
||||
virtual_hosts:
|
||||
- name: direct_response_service
|
||||
domains:
|
||||
- "*"
|
||||
routes:
|
||||
- match:
|
||||
prefix: "/"
|
||||
direct_response:
|
||||
status: 200
|
||||
body:
|
||||
inline_string: "hello world\\n"
|
||||
# - match:
|
||||
# prefix: "/"
|
||||
# route:
|
||||
# cluster: service-backend
|
||||
http_filters:
|
||||
- name: {plugin_name}
|
||||
typed_config:
|
||||
"@type": type.googleapis.com/udpa.type.v1.TypedStruct
|
||||
type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
|
||||
value:
|
||||
config:
|
||||
name: wasmdemo
|
||||
vm_config:
|
||||
runtime: envoy.wasm.runtime.v8
|
||||
code:
|
||||
local:
|
||||
filename: ./extensions/{plugin_name}/main.wasm
|
||||
configuration:
|
||||
"@type": "type.googleapis.com/google.protobuf.StringValue"
|
||||
value: '{json.dumps(plugin_config)}'
|
||||
- name: envoy.filters.http.router
|
||||
typed_config:
|
||||
'@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
|
||||
# clusters:
|
||||
# - name: service-backend
|
||||
# connect_timeout: 600s
|
||||
# type: STATIC
|
||||
# lb_policy: ROUND_ROBIN
|
||||
# load_assignment:
|
||||
# cluster_name: service-backend
|
||||
# endpoints:
|
||||
# - lb_endpoints:
|
||||
# - endpoint:
|
||||
# address:
|
||||
# socket_address:
|
||||
# address: 127.0.0.1
|
||||
# port_value: 8000
|
||||
'''
|
||||
|
||||
with open("extensions/"+plugin_name+"/config.yaml", "w") as f:
|
||||
f.write(config)
|
||||
@@ -60,3 +60,16 @@ builder:
|
||||
.
|
||||
@echo ""
|
||||
@echo "image: ${BUILDER}"
|
||||
|
||||
local-build:
|
||||
tinygo build -scheduler=none -target=wasi -gc=custom -tags='custommalloc nottinygc_finalizer' \
|
||||
-o extensions/${PLUGIN_NAME}/main.wasm \
|
||||
extensions/${PLUGIN_NAME}/main.go
|
||||
@echo ""
|
||||
@echo "wasm: extensions/${PLUGIN_NAME}/main.wasm"
|
||||
|
||||
local-run:
|
||||
python3 .devcontainer/gen_config.py ${PLUGIN_NAME}
|
||||
envoy -c extensions/${PLUGIN_NAME}/config.yaml --concurrency 0 --log-level info --component-log-level wasm:debug
|
||||
|
||||
local-all: local-build local-run
|
||||
58
plugins/wasm-go/extensions/bot-detect/README.md
Normal file
58
plugins/wasm-go/extensions/bot-detect/README.md
Normal file
@@ -0,0 +1,58 @@
|
||||
<p>
|
||||
<a href="README_EN.md"> English </a> | 中文
|
||||
</p>
|
||||
|
||||
# 功能说明
|
||||
`bot-detect`插件可以用于识别并阻止互联网爬虫对站点资源的爬取
|
||||
|
||||
# 配置字段
|
||||
|
||||
| 名称 | 数据类型 | 填写要求 | 默认值 | 描述 |
|
||||
| -------- | -------- | -------- | -------- | -------- |
|
||||
| allow | array of string | 选填 | - | 配置匹配 User-Agent 请求头的正则表达式,匹配命中时将允许其访问 |
|
||||
| deny | array of string | 选填 | - | 配置匹配 User-Agent 请求头的正则表达式,匹配命中时将屏蔽请求 |
|
||||
| blocked_code | number | 选填 | 403 | 配置请求被屏蔽时返回的 HTTP 状态码 |
|
||||
| blocked_message | string | 选填 | - | 配置请求被屏蔽时返回的 HTTP 应答 Body |
|
||||
|
||||
`allow` 和 `deny` 字段可以均不配置,则执行默认的爬虫判断逻辑,通过配置 `allow` 字段可以将原本命中默认爬虫判断逻辑的请求放行,通过配置 `deny` 字段可以增加额外的爬虫判断逻辑。
|
||||
|
||||
默认的爬虫判断正则表达式集合如下:
|
||||
|
||||
```bash
|
||||
# Bots General matcher 'name/0.0'
|
||||
(?:\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}(?:[Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50}))[/ ](\d+)(?:\.(\d+)(?:\.(\d+)|)|)
|
||||
# Bots General matcher 'name 0.0'
|
||||
(?:\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}(?:[Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50})) (\d+)(?:\.(\d+)(?:\.(\d+)|)|)
|
||||
# Bots containing spider|scrape|bot(but not CUBOT)|Crawl
|
||||
((?:[A-z0-9]{1,50}|[A-z\-]{1,50} ?|)(?: the |)(?:[Ss][Pp][Ii][Dd][Ee][Rr]|[Ss]crape|[Cc][Rr][Aa][Ww][Ll])[A-z0-9]{0,50})(?:(?:[ /]| v)(\d+)(?:\.(\d+)|)(?:\.(\d+)|)|)
|
||||
# Bots Pattern '/name-0.0'
|
||||
/((?:Ant-)?Nutch|[A-z]+[Bb]ot|[A-z]+[Ss]pider|Axtaris|fetchurl|Isara|ShopSalad|Tailsweep)[ \-](\d+)(?:\.(\d+)(?:\.(\d+))?)?
|
||||
# Bots Pattern 'name/0.0'
|
||||
\b(008|Altresium|Argus|BaiduMobaider|BoardReader|DNSGroup|DataparkSearch|EDI|Goodzer|Grub|INGRID|Infohelfer|LinkedInBot|LOOQ|Nutch|OgScrper|PathDefender|Peew|PostPost|Steeler|Twitterbot|VSE|WebCrunch|WebZIP|Y!J-BR[A-Z]|YahooSeeker|envolk|sproose|wminer)/(\d+)(?:\.(\d+)|)(?:\.(\d+)|)
|
||||
# More bots
|
||||
(CSimpleSpider|Cityreview Robot|CrawlDaddy|CrawlFire|Finderbots|Index crawler|Job Roboter|KiwiStatus Spider|Lijit Crawler|QuerySeekerSpider|ScollSpider|Trends Crawler|USyd-NLP-Spider|SiteCat Webbot|BotName\/\$BotVersion|123metaspider-Bot|1470\.net crawler|50\.nu|8bo Crawler Bot|Aboundex|Accoona-[A-z]{1,30}-Agent|AdsBot-Google(?:-[a-z]{1,30}|)|altavista|AppEngine-Google|archive.{0,30}\.org_bot|archiver|Ask Jeeves|[Bb]ai[Dd]u[Ss]pider(?:-[A-Za-z]{1,30})(?:-[A-Za-z]{1,30}|)|bingbot|BingPreview|blitzbot|BlogBridge|Bloglovin|BoardReader Blog Indexer|BoardReader Favicon Fetcher|boitho.com-dc|BotSeer|BUbiNG|\b\w{0,30}favicon\w{0,30}\b|\bYeti(?:-[a-z]{1,30}|)|Catchpoint(?: bot|)|[Cc]harlotte|Checklinks|clumboot|Comodo HTTP\(S\) Crawler|Comodo-Webinspector-Crawler|ConveraCrawler|CRAWL-E|CrawlConvera|Daumoa(?:-feedfetcher|)|Feed Seeker Bot|Feedbin|findlinks|Flamingo_SearchEngine|FollowSite Bot|furlbot|Genieo|gigabot|GomezAgent|gonzo1|(?:[a-zA-Z]{1,30}-|)Googlebot(?:-[a-zA-Z]{1,30}|)|Google SketchUp|grub-client|gsa-crawler|heritrix|HiddenMarket|holmes|HooWWWer|htdig|ia_archiver|ICC-Crawler|Icarus6j|ichiro(?:/mobile|)|IconSurf|IlTrovatore(?:-Setaccio|)|InfuzApp|Innovazion Crawler|InternetArchive|IP2[a-z]{1,30}Bot|jbot\b|KaloogaBot|Kraken|Kurzor|larbin|LEIA|LesnikBot|Linguee Bot|LinkAider|LinkedInBot|Lite Bot|Llaut|lycos|Mail\.RU_Bot|masscan|masidani_bot|Mediapartners-Google|Microsoft .{0,30} Bot|mogimogi|mozDex|MJ12bot|msnbot(?:-media {0,2}|)|msrbot|Mtps Feed Aggregation System|netresearch|Netvibes|NewsGator[^/]{0,30}|^NING|Nutch[^/]{0,30}|Nymesis|ObjectsSearch|OgScrper|Orbiter|OOZBOT|PagePeeker|PagesInventory|PaxleFramework|Peeplo Screenshot Bot|PlantyNet_WebRobot|Pompos|Qwantify|Read%20Later|Reaper|RedCarpet|Retreiver|Riddler|Rival IQ|scooter|Scrapy|Scrubby|searchsight|seekbot|semanticdiscovery|SemrushBot|Simpy|SimplePie|SEOstats|SimpleRSS|SiteCon|Slackbot-LinkExpanding|Slack-ImgProxy|Slurp|snappy|Speedy Spider|Squrl Java|Stringer|TheUsefulbot|ThumbShotsBot|Thumbshots\.ru|Tiny Tiny RSS|Twitterbot|WhatsApp|URL2PNG|Vagabondo|VoilaBot|^vortex|Votay bot|^voyager|WASALive.Bot|Web-sniffer|WebThumb|WeSEE:[A-z]{1,30}|WhatWeb|WIRE|WordPress|Wotbox|www\.almaden\.ibm\.com|Xenu(?:.s|) Link Sleuth|Xerka [A-z]{1,30}Bot|yacy(?:bot|)|YahooSeeker|Yahoo! Slurp|Yandex\w{1,30}|YodaoBot(?:-[A-z]{1,30}|)|YottaaMonitor|Yowedo|^Zao|^Zao-Crawler|ZeBot_www\.ze\.bz|ZooShot|ZyBorg)(?:[ /]v?(\d+)(?:\.(\d+)(?:\.(\d+)|)|)|)
|
||||
```
|
||||
|
||||
# 配置示例
|
||||
|
||||
## 放行原本命中爬虫规则的请求
|
||||
```yaml
|
||||
allow:
|
||||
- ".*Go-http-client.*"
|
||||
```
|
||||
|
||||
若不作该配置,默认的 Golang 网络库请求会被视做爬虫,被禁止访问
|
||||
|
||||
|
||||
## 增加爬虫判断
|
||||
```yaml
|
||||
deny:
|
||||
- "spd-tools.*"
|
||||
```
|
||||
|
||||
根据该配置,下列请求将被禁止访问:
|
||||
|
||||
```bash
|
||||
curl http://example.com -H 'User-Agent: spd-tools/1.1'
|
||||
curl http://exmaple.com -H 'User-Agent: spd-tools'
|
||||
```
|
||||
58
plugins/wasm-go/extensions/bot-detect/README_EN.md
Normal file
58
plugins/wasm-go/extensions/bot-detect/README_EN.md
Normal file
@@ -0,0 +1,58 @@
|
||||
<p>
|
||||
English | <a href="README.md">中文</a>
|
||||
</p>
|
||||
|
||||
# Description
|
||||
`bot-detect` plugin can be used to identify and prevent web crawlers from crawling websites.
|
||||
|
||||
# Configuration Fields
|
||||
|
||||
| Name | Type | Requirement | Default Value | Description |
|
||||
| -------- | -------- | -------- | -------- | -------- |
|
||||
| allow | array of string | Optional | - | A regular expression to match the User-Agent request header and will allow access if the match hits |
|
||||
| deny | array of string | Optional | - | A regular expression to match the User-Agent request header and will block the request if the match hits |
|
||||
| blocked_code | number | Optional | 403 | The HTTP status code returned when a request is blocked |
|
||||
| blocked_message | string | Optional | - | The HTTP response Body returned when a request is blocked |
|
||||
|
||||
If field `allow` and field `deny` are not configured at the same time, the default logic to identify crawlers will be executed. By configuring the `allow` field, requests that would otherwise hit the default logic can be allowed. The judgement can be extended by configuring the `deny` field
|
||||
|
||||
The default set of crawler judgment regular expressions is as follows:
|
||||
|
||||
```bash
|
||||
# Bots General matcher 'name/0.0'
|
||||
(?:\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}(?:[Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50}))[/ ](\d+)(?:\.(\d+)(?:\.(\d+)|)|)
|
||||
# Bots General matcher 'name 0.0'
|
||||
(?:\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}(?:[Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50})) (\d+)(?:\.(\d+)(?:\.(\d+)|)|)
|
||||
# Bots containing spider|scrape|bot(but not CUBOT)|Crawl
|
||||
((?:[A-z0-9]{1,50}|[A-z\-]{1,50} ?|)(?: the |)(?:[Ss][Pp][Ii][Dd][Ee][Rr]|[Ss]crape|[Cc][Rr][Aa][Ww][Ll])[A-z0-9]{0,50})(?:(?:[ /]| v)(\d+)(?:\.(\d+)|)(?:\.(\d+)|)|)
|
||||
# Bots Pattern '/name-0.0'
|
||||
/((?:Ant-)?Nutch|[A-z]+[Bb]ot|[A-z]+[Ss]pider|Axtaris|fetchurl|Isara|ShopSalad|Tailsweep)[ \-](\d+)(?:\.(\d+)(?:\.(\d+))?)?
|
||||
# Bots Pattern 'name/0.0'
|
||||
\b(008|Altresium|Argus|BaiduMobaider|BoardReader|DNSGroup|DataparkSearch|EDI|Goodzer|Grub|INGRID|Infohelfer|LinkedInBot|LOOQ|Nutch|OgScrper|PathDefender|Peew|PostPost|Steeler|Twitterbot|VSE|WebCrunch|WebZIP|Y!J-BR[A-Z]|YahooSeeker|envolk|sproose|wminer)/(\d+)(?:\.(\d+)|)(?:\.(\d+)|)
|
||||
# More bots
|
||||
(CSimpleSpider|Cityreview Robot|CrawlDaddy|CrawlFire|Finderbots|Index crawler|Job Roboter|KiwiStatus Spider|Lijit Crawler|QuerySeekerSpider|ScollSpider|Trends Crawler|USyd-NLP-Spider|SiteCat Webbot|BotName\/\$BotVersion|123metaspider-Bot|1470\.net crawler|50\.nu|8bo Crawler Bot|Aboundex|Accoona-[A-z]{1,30}-Agent|AdsBot-Google(?:-[a-z]{1,30}|)|altavista|AppEngine-Google|archive.{0,30}\.org_bot|archiver|Ask Jeeves|[Bb]ai[Dd]u[Ss]pider(?:-[A-Za-z]{1,30})(?:-[A-Za-z]{1,30}|)|bingbot|BingPreview|blitzbot|BlogBridge|Bloglovin|BoardReader Blog Indexer|BoardReader Favicon Fetcher|boitho.com-dc|BotSeer|BUbiNG|\b\w{0,30}favicon\w{0,30}\b|\bYeti(?:-[a-z]{1,30}|)|Catchpoint(?: bot|)|[Cc]harlotte|Checklinks|clumboot|Comodo HTTP\(S\) Crawler|Comodo-Webinspector-Crawler|ConveraCrawler|CRAWL-E|CrawlConvera|Daumoa(?:-feedfetcher|)|Feed Seeker Bot|Feedbin|findlinks|Flamingo_SearchEngine|FollowSite Bot|furlbot|Genieo|gigabot|GomezAgent|gonzo1|(?:[a-zA-Z]{1,30}-|)Googlebot(?:-[a-zA-Z]{1,30}|)|Google SketchUp|grub-client|gsa-crawler|heritrix|HiddenMarket|holmes|HooWWWer|htdig|ia_archiver|ICC-Crawler|Icarus6j|ichiro(?:/mobile|)|IconSurf|IlTrovatore(?:-Setaccio|)|InfuzApp|Innovazion Crawler|InternetArchive|IP2[a-z]{1,30}Bot|jbot\b|KaloogaBot|Kraken|Kurzor|larbin|LEIA|LesnikBot|Linguee Bot|LinkAider|LinkedInBot|Lite Bot|Llaut|lycos|Mail\.RU_Bot|masscan|masidani_bot|Mediapartners-Google|Microsoft .{0,30} Bot|mogimogi|mozDex|MJ12bot|msnbot(?:-media {0,2}|)|msrbot|Mtps Feed Aggregation System|netresearch|Netvibes|NewsGator[^/]{0,30}|^NING|Nutch[^/]{0,30}|Nymesis|ObjectsSearch|OgScrper|Orbiter|OOZBOT|PagePeeker|PagesInventory|PaxleFramework|Peeplo Screenshot Bot|PlantyNet_WebRobot|Pompos|Qwantify|Read%20Later|Reaper|RedCarpet|Retreiver|Riddler|Rival IQ|scooter|Scrapy|Scrubby|searchsight|seekbot|semanticdiscovery|SemrushBot|Simpy|SimplePie|SEOstats|SimpleRSS|SiteCon|Slackbot-LinkExpanding|Slack-ImgProxy|Slurp|snappy|Speedy Spider|Squrl Java|Stringer|TheUsefulbot|ThumbShotsBot|Thumbshots\.ru|Tiny Tiny RSS|Twitterbot|WhatsApp|URL2PNG|Vagabondo|VoilaBot|^vortex|Votay bot|^voyager|WASALive.Bot|Web-sniffer|WebThumb|WeSEE:[A-z]{1,30}|WhatWeb|WIRE|WordPress|Wotbox|www\.almaden\.ibm\.com|Xenu(?:.s|) Link Sleuth|Xerka [A-z]{1,30}Bot|yacy(?:bot|)|YahooSeeker|Yahoo! Slurp|Yandex\w{1,30}|YodaoBot(?:-[A-z]{1,30}|)|YottaaMonitor|Yowedo|^Zao|^Zao-Crawler|ZeBot_www\.ze\.bz|ZooShot|ZyBorg)(?:[ /]v?(\d+)(?:\.(\d+)(?:\.(\d+)|)|)|)
|
||||
```
|
||||
|
||||
# Configuration Samples
|
||||
|
||||
## Release Requests that would otherwise Hit the Crawler Rules
|
||||
```yaml
|
||||
allow:
|
||||
- ".*Go-http-client.*"
|
||||
```
|
||||
|
||||
Without this configuration, the default Golang web library request will be treated as a crawler and access will be denied.
|
||||
|
||||
|
||||
## Add Crawler Judgement
|
||||
```yaml
|
||||
deny:
|
||||
- "spd-tools.*"
|
||||
```
|
||||
|
||||
According to this configuration, the following requests will be denied:
|
||||
|
||||
```bash
|
||||
curl http://example.com -H 'User-Agent: spd-tools/1.1'
|
||||
curl http://exmaple.com -H 'User-Agent: spd-tools'
|
||||
```
|
||||
1
plugins/wasm-go/extensions/bot-detect/VERSION
Normal file
1
plugins/wasm-go/extensions/bot-detect/VERSION
Normal file
@@ -0,0 +1 @@
|
||||
1.0.0
|
||||
30
plugins/wasm-go/extensions/bot-detect/botdetect.yaml
Normal file
30
plugins/wasm-go/extensions/bot-detect/botdetect.yaml
Normal file
@@ -0,0 +1,30 @@
|
||||
apiVersion: extensions.higress.io/v1alpha1
|
||||
kind: WasmPlugin
|
||||
metadata:
|
||||
annotations:
|
||||
higress.io/wasm-plugin-description: 用于识别并阻止互联网爬虫对站点资源的爬取
|
||||
higress.io/wasm-plugin-title: Bot Detect
|
||||
creationTimestamp: '2024-01-03T10:34:36Z'
|
||||
generation: 2
|
||||
labels:
|
||||
higress.io/resource-definer: higress
|
||||
higress.io/wasm-plugin-built-in: 'true'
|
||||
higress.io/wasm-plugin-category: custom
|
||||
higress.io/wasm-plugin-name: bot-detect
|
||||
higress.io/wasm-plugin-version: 1.0.0
|
||||
name: bot-detect
|
||||
namespace: higress-system
|
||||
spec:
|
||||
defaultConfigDisable: true
|
||||
matchRules:
|
||||
- config:
|
||||
blocked_code: 401
|
||||
blocked_message: a bot
|
||||
deny:
|
||||
- Chrome
|
||||
configDisable: false
|
||||
ingress:
|
||||
- test
|
||||
phase: AUTHN
|
||||
priority: 310
|
||||
url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/20240103/bot-detect:1.0.0
|
||||
@@ -0,0 +1,68 @@
|
||||
/*
|
||||
* Copyright (c) 2022 Alibaba Group Holding Ltd.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package config
|
||||
|
||||
import (
|
||||
regexp "github.com/wasilibs/go-re2"
|
||||
)
|
||||
|
||||
var DefaultBotRegex = []*regexp.Regexp{
|
||||
regexp.MustCompile(`(\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}([Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50}))[/ ](\d+)(\.(\d+)(\.(\d+)|)|)`),
|
||||
regexp.MustCompile(`((\/[A-Za-z0-9\.]+|) {0,5}([A-Za-z0-9 \-_\!\[\]:]{0,50}([Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]{0,50})) (\d+)(\.(\d+)(\.(\d+)|)|))`),
|
||||
regexp.MustCompile(`((([A-z0-9]{1,50}|[A-z\-]{1,} ?|)( the |)(?:[Ss][Pp][Ii][Dd][Ee][Rr]|[Ss]crape|[Cc][Rr][Aa][Ww][Ll])[A-z0-9]{0,50})(([ /]| v)(\d+)(\.(\d+)|)(\.(\d+)|)|))`),
|
||||
regexp.MustCompile(`((Ant-)?Nutch|[A-z]+[Bb]ot|[A-z]+[Ss]pider|Axtaris|fetchurl|Isara|ShopSalad|Tailsweep)[ \-](\d+)(\.(\d+)(\.(\d+))?)?`),
|
||||
regexp.MustCompile(`\b(008|Altresium|Argus|BaiduMobaider|BoardReader|DNSGroup|DataparkSearch|EDI|Goodzer|Grub|INGRID|Infohelfer|LinkedInBot|LOOQ|Nutch|OgScrper|PathDefender|Peew|PostPost|Steeler|Twitterbot|VSE|WebCrunch|WebZIP|Y!J-BR[A-Z]|YahooSeeker|envolk|sproose|wminer)/(\d+)(?:\.(\d+)|)(?:\.(\d+)|)`),
|
||||
regexp.MustCompile(`((CSimpleSpider|Cityreview Robot|CrawlDaddy|CrawlFire|Finderbots|Index crawler|Job Roboter|KiwiStatus Spider|Lijit Crawler|QuerySeekerSpider|ScollSpider|Trends Crawler|USyd-NLP-Spider|SiteCat Webbot|BotName\/\$BotVersion|123metaspider-Bot|1470\.net crawler|50\.nu|8bo Crawler Bot|Aboundex|Accoona-[A-z]{1,30}-Agent|AdsBot-Google(?:-[a-z]{1,30}|)|altavista|AppEngine-Google|archive.{0,30}\.org_bot|archiver|Ask Jeeves|[Bb]ai[Dd]u[Ss]pider(?:-[A-Za-z]{1,30})(?:-[A-Za-z]{1,30}|)|bingbot|BingPreview|blitzbot|BlogBridge|Bloglovin|BoardReader Blog Indexer|BoardReader Favicon Fetcher|boitho.com-dc|BotSeer|BUbiNG|\b\w{0,30}favicon\w{0,30}\b|\bYeti(?:-[a-z]{1,30}|)|Catchpoint(?: bot|)|[Cc]harlotte|Checklinks|clumboot|Comodo HTTP\(S\) Crawler|Comodo-Webinspector-Crawler|ConveraCrawler|CRAWL-E|CrawlConvera|Daumoa(?:-feedfetcher|)|Feed Seeker Bot|Feedbin|findlinks|Flamingo_SearchEngine|FollowSite Bot|furlbot|Genieo|gigabot|GomezAgent|gonzo1|(?:[a-zA-Z]{1,30}-|)Googlebot(?:-[a-zA-Z]{1,30}|)|Google SketchUp|grub-client|gsa-crawler|heritrix|HiddenMarket|holmes|HooWWWer|htdig|ia_archiver|ICC-Crawler|Icarus6j|ichiro(?:/mobile|)|IconSurf|IlTrovatore(?:-Setaccio|)|InfuzApp|Innovazion Crawler|InternetArchive|IP2[a-z]{1,30}Bot|jbot\b|KaloogaBot|Kraken|Kurzor|larbin|LEIA|LesnikBot|Linguee Bot|LinkAider|LinkedInBot|Lite Bot|Llaut|lycos|Mail\.RU_Bot|masscan|masidani_bot|Mediapartners-Google|Microsoft .{0,30} Bot|mogimogi|mozDex|MJ12bot|msnbot(?:-media {0,2}|)|msrbot|Mtps Feed Aggregation System|netresearch|Netvibes|NewsGator[^/]{0,30}|^NING|Nutch[^/]{0,30}|Nymesis|ObjectsSearch|OgScrper|Orbiter|OOZBOT|PagePeeker|PagesInventory|PaxleFramework|Peeplo Screenshot Bot|PlantyNet_WebRobot|Pompos|Qwantify|Read%20Later|Reaper|RedCarpet|Retreiver|Riddler|Rival IQ|scooter|Scrapy|Scrubby|searchsight|seekbot|semanticdiscovery|SemrushBot|Simpy|SimplePie|SEOstats|SimpleRSS|SiteCon|Slackbot-LinkExpanding|Slack-ImgProxy|Slurp|snappy|Speedy Spider|Squrl Java|Stringer|TheUsefulbot|ThumbShotsBot|Thumbshots\.ru|Tiny Tiny RSS|Twitterbot|WhatsApp|URL2PNG|Vagabondo|VoilaBot|^vortex|Votay bot|^voyager|WASALive.Bot|Web-sniffer|WebThumb|WeSEE:[A-z]{1,30}|WhatWeb|WIRE|WordPress|Wotbox|www\.almaden\.ibm\.com|Xenu(?:.s|) Link Sleuth|Xerka [A-z]{1,30}Bot|yacy(?:bot|)|YahooSeeker|Yahoo! Slurp|Yandex\w{1,30}|YodaoBot(?:-[A-z]{1,30}|)|YottaaMonitor|Yowedo|^Zao|^Zao-Crawler|ZeBot_www\.ze\.bz|ZooShot|ZyBorg)(?:[ /]v?(\d+)(?:\.(\d+)(?:\.(\d+)|)|)|))`),
|
||||
}
|
||||
|
||||
type BotDetectConfig struct {
|
||||
BlockedCode uint32 `json:"blocked_code"`
|
||||
BlockedMessage string `json:"blocked_message"`
|
||||
Allow []*regexp.Regexp `json:"allow"`
|
||||
Deny []*regexp.Regexp `json:"deny"`
|
||||
}
|
||||
|
||||
func (bdc *BotDetectConfig) FillDefaultValue() {
|
||||
if bdc.BlockedCode == 0 {
|
||||
bdc.BlockedCode = 403
|
||||
}
|
||||
if bdc.BlockedMessage == "" {
|
||||
bdc.BlockedMessage = "Invalid User-Agent"
|
||||
}
|
||||
}
|
||||
|
||||
func (bdc *BotDetectConfig) Process(ua string) (bool, string) {
|
||||
if ua == "" {
|
||||
return false, "can not be empty"
|
||||
}
|
||||
for _, allowRule := range bdc.Allow {
|
||||
if allowRule.MatchString(ua) {
|
||||
return true, ""
|
||||
}
|
||||
}
|
||||
for _, denyRule := range bdc.Deny {
|
||||
if denyRule.MatchString(ua) {
|
||||
return false, denyRule.String()
|
||||
}
|
||||
}
|
||||
for _, defaultRule := range DefaultBotRegex {
|
||||
if defaultRule.MatchString(ua) {
|
||||
return false, defaultRule.String()
|
||||
}
|
||||
}
|
||||
return true, ""
|
||||
}
|
||||
@@ -0,0 +1,138 @@
|
||||
/*
|
||||
* Copyright (c) 2022 Alibaba Group Holding Ltd.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package config
|
||||
|
||||
import (
|
||||
"github.com/stretchr/testify/assert"
|
||||
regexp "github.com/wasilibs/go-re2"
|
||||
"log"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func toRegexMatch(regexs []string) []*regexp.Regexp {
|
||||
re := make([]*regexp.Regexp, 0)
|
||||
for _, regex := range regexs {
|
||||
c, err := regexp.Compile(regex)
|
||||
if err != nil {
|
||||
log.Default().Fatal(err.Error())
|
||||
}
|
||||
re = append(re, c)
|
||||
}
|
||||
return re
|
||||
}
|
||||
|
||||
func TestBotDetectConfig_ProcessTest(t *testing.T) {
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
ua string
|
||||
allow []string
|
||||
deny []string
|
||||
blockCode uint32
|
||||
blockMessage string
|
||||
want bool
|
||||
}{
|
||||
{
|
||||
"test empty bot detect",
|
||||
"",
|
||||
[]string{},
|
||||
[]string{},
|
||||
401,
|
||||
"bot has been blocked",
|
||||
false,
|
||||
},
|
||||
{
|
||||
"test default bot detect",
|
||||
"Ant-Tailsweep-1",
|
||||
[]string{},
|
||||
[]string{},
|
||||
401,
|
||||
"bot has been blocked",
|
||||
false,
|
||||
},
|
||||
{
|
||||
"test default bot detect",
|
||||
"indexer/1.2",
|
||||
[]string{},
|
||||
[]string{},
|
||||
401,
|
||||
"bot has been blocked",
|
||||
false,
|
||||
},
|
||||
{
|
||||
"test default bot detect",
|
||||
"indexer/1.1.0",
|
||||
[]string{},
|
||||
[]string{},
|
||||
401,
|
||||
"bot has been blocked",
|
||||
false,
|
||||
},
|
||||
{
|
||||
"test default bot detect",
|
||||
"YottaaMonitor",
|
||||
[]string{},
|
||||
[]string{},
|
||||
401,
|
||||
"bot has been blocked",
|
||||
false,
|
||||
},
|
||||
{
|
||||
"test allow bot detect",
|
||||
"BaiduMobaider",
|
||||
[]string{"BaiduMobaider"},
|
||||
[]string{},
|
||||
401,
|
||||
"bot has been blocked",
|
||||
true,
|
||||
},
|
||||
{
|
||||
"test deny bot detect",
|
||||
"Chrome",
|
||||
[]string{},
|
||||
[]string{"Chrome"},
|
||||
401,
|
||||
"bot has been blocked",
|
||||
false,
|
||||
},
|
||||
{
|
||||
"test allow and deny bot detect",
|
||||
"SameBotDetect",
|
||||
[]string{"SameBotDetect"},
|
||||
[]string{"SameBotDetect"},
|
||||
401,
|
||||
"bot has been blocked",
|
||||
true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
bdc := BotDetectConfig{
|
||||
BlockedCode: test.blockCode,
|
||||
BlockedMessage: test.blockMessage,
|
||||
Allow: toRegexMatch(test.allow),
|
||||
Deny: toRegexMatch(test.deny),
|
||||
}
|
||||
actual, _ := bdc.Process(test.ua)
|
||||
assert.Equal(t, test.want, actual, "")
|
||||
})
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
23
plugins/wasm-go/extensions/bot-detect/go.mod
Normal file
23
plugins/wasm-go/extensions/bot-detect/go.mod
Normal file
@@ -0,0 +1,23 @@
|
||||
module bot-detect
|
||||
|
||||
go 1.19
|
||||
|
||||
require (
|
||||
github.com/alibaba/higress/plugins/wasm-go v1.3.2
|
||||
github.com/stretchr/testify v1.8.0
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c
|
||||
github.com/tidwall/gjson v1.14.3
|
||||
github.com/wasilibs/go-re2 v1.4.1
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/davecgh/go-spew v1.1.1 // indirect
|
||||
github.com/google/uuid v1.3.0 // indirect
|
||||
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520 // indirect
|
||||
github.com/magefile/mage v1.15.0 // indirect
|
||||
github.com/pmezard/go-difflib v1.0.0 // indirect
|
||||
github.com/tetratelabs/wazero v1.6.0 // indirect
|
||||
github.com/tidwall/match v1.1.1 // indirect
|
||||
github.com/tidwall/pretty v1.2.0 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
)
|
||||
36
plugins/wasm-go/extensions/bot-detect/go.sum
Normal file
36
plugins/wasm-go/extensions/bot-detect/go.sum
Normal file
@@ -0,0 +1,36 @@
|
||||
github.com/alibaba/higress/plugins/wasm-go v1.3.2 h1:OKFo9zK7PFxvtSq9TmT8TwI6xqmNq5LZXfDBqPLOgkw=
|
||||
github.com/alibaba/higress/plugins/wasm-go v1.3.2/go.mod h1:WZ/68vwe8qWhusa6C4/gMwUqas0jvHWSOa1bp8iK8F4=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
|
||||
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520 h1:IHDghbGQ2DTIXHBHxWfqCYQW1fKjyJ/I7W1pMyUDeEA=
|
||||
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520/go.mod h1:Nz8ORLaFiLWotg6GeKlJMhv8cci8mM43uEnLA5t8iew=
|
||||
github.com/magefile/mage v1.15.0 h1:BvGheCMAsG3bWUDbZ8AyXXpCNwU9u5CB6sM+HNb9HYg=
|
||||
github.com/magefile/mage v1.15.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
|
||||
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
|
||||
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c h1:OCUFXVGixHLfNjg6/QYEhv+jHJ5mRGhpEUVFv9eWPJE=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c/go.mod h1:5t/pWFNJ9eMyu/K/Z+OeGhDJ9sN9eCo8fc2pyM/Qjg4=
|
||||
github.com/tetratelabs/wazero v1.6.0 h1:z0H1iikCdP8t+q341xqepY4EWvHEw8Es7tlqiVzlP3g=
|
||||
github.com/tetratelabs/wazero v1.6.0/go.mod h1:0U0G41+ochRKoPKCJlh0jMg1CHkyfK8kDqiirMmKY8A=
|
||||
github.com/tidwall/gjson v1.14.3 h1:9jvXn7olKEHU1S9vwoMGliaT8jq1vJ7IH/n9zD9Dnlw=
|
||||
github.com/tidwall/gjson v1.14.3/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
|
||||
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
|
||||
github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
|
||||
github.com/tidwall/pretty v1.2.0 h1:RWIZEg2iJ8/g6fDDYzMpobmaoGh5OLl4AXtGUGPcqCs=
|
||||
github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
|
||||
github.com/wasilibs/go-re2 v1.4.1 h1:E5+9O1M8UoGeqLB2A9omeoaWImqpuYDs9cKwvTJq/Oo=
|
||||
github.com/wasilibs/go-re2 v1.4.1/go.mod h1:ynB8eCwd9JsqUnsk8WlPDk6cEeme8BguZmnqOSURE4Y=
|
||||
github.com/wasilibs/nottinygc v0.4.0 h1:h1TJMihMC4neN6Zq+WKpLxgd9xCFMw7O9ETLwY2exJQ=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
104
plugins/wasm-go/extensions/bot-detect/main.go
Normal file
104
plugins/wasm-go/extensions/bot-detect/main.go
Normal file
@@ -0,0 +1,104 @@
|
||||
/*
|
||||
* Copyright (c) 2022 Alibaba Group Holding Ltd.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"bot-detect/config"
|
||||
"github.com/alibaba/higress/plugins/wasm-go/pkg/wrapper"
|
||||
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
|
||||
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
|
||||
"github.com/tidwall/gjson"
|
||||
regexp "github.com/wasilibs/go-re2"
|
||||
)
|
||||
|
||||
func main() {
|
||||
wrapper.SetCtx(
|
||||
"bot-detect",
|
||||
wrapper.ParseConfigBy(parseConfig),
|
||||
wrapper.ProcessRequestHeadersBy(onHttpRequestHeaders),
|
||||
)
|
||||
}
|
||||
|
||||
func parseConfig(json gjson.Result, botDetectConfig *config.BotDetectConfig, log wrapper.Log) error {
|
||||
log.Debug("parseConfig()")
|
||||
|
||||
if json.Get("blocked_code").Exists() {
|
||||
botDetectConfig.BlockedCode = uint32(int(json.Get("blocked_code").Int()))
|
||||
}
|
||||
|
||||
if json.Get("blocked_message").Exists() {
|
||||
botDetectConfig.BlockedMessage = json.Get("blocked_message").String()
|
||||
}
|
||||
|
||||
allowRules := make([]gjson.Result, 0)
|
||||
denyRules := make([]gjson.Result, 0)
|
||||
|
||||
allowRulesValue := json.Get("allow")
|
||||
if allowRulesValue.Exists() && allowRulesValue.IsArray() {
|
||||
allowRules = json.Get("allow").Array()
|
||||
}
|
||||
|
||||
denyRulesValue := json.Get("deny")
|
||||
if denyRulesValue.Exists() && denyRulesValue.IsArray() {
|
||||
denyRules = json.Get("deny").Array()
|
||||
}
|
||||
|
||||
for _, allowRule := range allowRules {
|
||||
c, err := regexp.Compile(allowRule.String())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
botDetectConfig.Allow = append(botDetectConfig.Allow, c)
|
||||
}
|
||||
|
||||
for _, denyRule := range denyRules {
|
||||
c, err := regexp.Compile(denyRule.String())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
botDetectConfig.Deny = append(botDetectConfig.Deny, c)
|
||||
}
|
||||
|
||||
// Fill default values
|
||||
botDetectConfig.FillDefaultValue()
|
||||
log.Debugf("botDetectConfig:%+v", botDetectConfig)
|
||||
return nil
|
||||
|
||||
}
|
||||
|
||||
func onHttpRequestHeaders(ctx wrapper.HttpContext, botDetectConfig config.BotDetectConfig, log wrapper.Log) types.Action {
|
||||
log.Debug("onHttpRequestHeaders()")
|
||||
//// Get user-agent header
|
||||
ua, err := proxywasm.GetHttpRequestHeader("user-agent")
|
||||
if err != nil {
|
||||
log.Warnf("failed to get user-agent: %v", err)
|
||||
return types.ActionPause
|
||||
}
|
||||
host := ctx.Host()
|
||||
scheme := ctx.Scheme()
|
||||
path := ctx.Path()
|
||||
method := ctx.Method()
|
||||
|
||||
if ok, rule := botDetectConfig.Process(ua); !ok {
|
||||
proxywasm.SendHttpResponse(botDetectConfig.BlockedCode, nil, []byte(botDetectConfig.BlockedMessage), -1)
|
||||
log.Debugf("scheme:%s, host:%s, method:%s, path:%s user-agent:%s has been blocked by rule:%s", scheme, host, method, path, ua, rule)
|
||||
return types.ActionPause
|
||||
}
|
||||
|
||||
log.Debugf("scheme:%s, host:%s, method:%s, path:%s user-agent:%s has been passed", scheme, host, method, path, ua)
|
||||
return types.ActionContinue
|
||||
}
|
||||
@@ -5,7 +5,7 @@ go 1.18
|
||||
replace github.com/alibaba/higress/plugins/wasm-go => ../..
|
||||
|
||||
require (
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0
|
||||
github.com/tidwall/gjson v1.14.3
|
||||
)
|
||||
|
||||
|
||||
@@ -7,8 +7,8 @@ github.com/magefile/mage v1.14.0 h1:6QDX3g6z1YvJ4olPhT1wksUcSa/V0a1B+pJb73fBjyo=
|
||||
github.com/magefile/mage v1.14.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c h1:OCUFXVGixHLfNjg6/QYEhv+jHJ5mRGhpEUVFv9eWPJE=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c/go.mod h1:5t/pWFNJ9eMyu/K/Z+OeGhDJ9sN9eCo8fc2pyM/Qjg4=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0 h1:kS7BvMKN+FiptV4pfwiNX8e3q14evxAWkhYbxt8EI1M=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0/go.mod h1:qkW5MBz2jch2u8bS59wws65WC+Gtx3x0aPUX5JL7CXI=
|
||||
github.com/tidwall/gjson v1.14.3 h1:9jvXn7olKEHU1S9vwoMGliaT8jq1vJ7IH/n9zD9Dnlw=
|
||||
github.com/tidwall/gjson v1.14.3/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
|
||||
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
|
||||
|
||||
@@ -6,7 +6,7 @@ replace github.com/alibaba/higress/plugins/wasm-go => ../..
|
||||
|
||||
require (
|
||||
github.com/alibaba/higress/plugins/wasm-go v0.0.0
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0
|
||||
)
|
||||
|
||||
require (
|
||||
|
||||
@@ -7,8 +7,8 @@ github.com/magefile/mage v1.14.0 h1:6QDX3g6z1YvJ4olPhT1wksUcSa/V0a1B+pJb73fBjyo=
|
||||
github.com/magefile/mage v1.14.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c h1:OCUFXVGixHLfNjg6/QYEhv+jHJ5mRGhpEUVFv9eWPJE=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c/go.mod h1:5t/pWFNJ9eMyu/K/Z+OeGhDJ9sN9eCo8fc2pyM/Qjg4=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0 h1:kS7BvMKN+FiptV4pfwiNX8e3q14evxAWkhYbxt8EI1M=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0/go.mod h1:qkW5MBz2jch2u8bS59wws65WC+Gtx3x0aPUX5JL7CXI=
|
||||
github.com/tidwall/gjson v1.14.3 h1:9jvXn7olKEHU1S9vwoMGliaT8jq1vJ7IH/n9zD9Dnlw=
|
||||
github.com/tidwall/gjson v1.14.3/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
|
||||
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
|
||||
|
||||
@@ -6,7 +6,7 @@ replace github.com/alibaba/higress/plugins/wasm-go => ../..
|
||||
|
||||
require (
|
||||
github.com/alibaba/higress/plugins/wasm-go v0.0.0
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0
|
||||
github.com/tidwall/gjson v1.14.3
|
||||
)
|
||||
|
||||
|
||||
@@ -7,8 +7,8 @@ github.com/magefile/mage v1.14.0 h1:6QDX3g6z1YvJ4olPhT1wksUcSa/V0a1B+pJb73fBjyo=
|
||||
github.com/magefile/mage v1.14.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c h1:OCUFXVGixHLfNjg6/QYEhv+jHJ5mRGhpEUVFv9eWPJE=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c/go.mod h1:5t/pWFNJ9eMyu/K/Z+OeGhDJ9sN9eCo8fc2pyM/Qjg4=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0 h1:kS7BvMKN+FiptV4pfwiNX8e3q14evxAWkhYbxt8EI1M=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0/go.mod h1:qkW5MBz2jch2u8bS59wws65WC+Gtx3x0aPUX5JL7CXI=
|
||||
github.com/tidwall/gjson v1.14.3 h1:9jvXn7olKEHU1S9vwoMGliaT8jq1vJ7IH/n9zD9Dnlw=
|
||||
github.com/tidwall/gjson v1.14.3/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
|
||||
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
|
||||
|
||||
@@ -5,7 +5,7 @@ go 1.19
|
||||
require (
|
||||
github.com/alibaba/higress/plugins/wasm-go v0.0.0-20230807053545-d307d0e755f1
|
||||
github.com/go-jose/go-jose/v3 v3.0.0
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0
|
||||
github.com/tidwall/gjson v1.14.3
|
||||
golang.org/x/oauth2 v0.11.0
|
||||
)
|
||||
|
||||
@@ -20,8 +20,8 @@ github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZN
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c h1:OCUFXVGixHLfNjg6/QYEhv+jHJ5mRGhpEUVFv9eWPJE=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c/go.mod h1:5t/pWFNJ9eMyu/K/Z+OeGhDJ9sN9eCo8fc2pyM/Qjg4=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0 h1:kS7BvMKN+FiptV4pfwiNX8e3q14evxAWkhYbxt8EI1M=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0/go.mod h1:qkW5MBz2jch2u8bS59wws65WC+Gtx3x0aPUX5JL7CXI=
|
||||
github.com/tidwall/gjson v1.14.3 h1:9jvXn7olKEHU1S9vwoMGliaT8jq1vJ7IH/n9zD9Dnlw=
|
||||
github.com/tidwall/gjson v1.14.3/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
|
||||
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
|
||||
|
||||
101
plugins/wasm-go/extensions/opa/README.md
Normal file
101
plugins/wasm-go/extensions/opa/README.md
Normal file
@@ -0,0 +1,101 @@
|
||||
# 功能说明
|
||||
|
||||
该插件实现了 `OPA` 策略控制
|
||||
|
||||
# 该教程使用k8s,[k8s配置文件](../../../../test/e2e/conformance/tests/go-wasm-opa.yaml)
|
||||
|
||||
支持client `k8s,nacos,ip,route` 策略去访问
|
||||
|
||||
## 配置字段
|
||||
|
||||
| 字段 | 数据类型 | 填写要求 | 默认值 | 描述 |
|
||||
|---------------|--------|------|-----|--------------------------------------|
|
||||
| policy | string | 必填 | - | opa 策略 |
|
||||
| timeout | string | 必填 | - | 访问超时时间设置 |
|
||||
| serviceSource | string | 必填 | - | k8s,nacos,ip,route |
|
||||
| host | string | 非必填 | - | 服务主机(serviceSource为`ip`必填) |
|
||||
| serviceName | string | 非必填 | - | 服务名称(serviceSource为`k8s,nacos,ip`必填) |
|
||||
| servicePort | string | 非必填 | - | 服务端口(serviceSource为`k8s,nacos,ip`必填) |
|
||||
| namespace | string | 非必填 | - | 服务端口(serviceSource为`k8s,nacos`必填) |
|
||||
|
||||
这是一个用于OPA认证配置的表格,确保在提供所有必要的信息时遵循上述指导。
|
||||
|
||||
## 配置示例
|
||||
|
||||
```yaml
|
||||
serviceSource: k8s
|
||||
serviceName: opa
|
||||
servicePort: 8181
|
||||
namespace: higress-backend
|
||||
policy: example1
|
||||
timeout: 5s
|
||||
```
|
||||
|
||||
# 在宿主机上执行OPA的流程
|
||||
|
||||
## 启动opa服务
|
||||
|
||||
```shell
|
||||
docker run -d --name opa -p 8181:8181 openpolicyagent/opa:0.35.0 run -s
|
||||
```
|
||||
|
||||
## 创建opa策略
|
||||
|
||||
```shell
|
||||
curl -X PUT '127.0.0.1:8181/v1/policies/example1' \
|
||||
-H 'Content-Type: text/plain' \
|
||||
-d 'package example1
|
||||
|
||||
import input.request
|
||||
|
||||
default allow = false
|
||||
|
||||
allow {
|
||||
# HTTP method must GET
|
||||
request.method == "GET"
|
||||
}'
|
||||
```
|
||||
|
||||
## 查询策略
|
||||
|
||||
```shell
|
||||
curl -X POST '127.0.0.1:8181/v1/data/example1/allow' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"input":{"request":{"method":"GET"}}}'
|
||||
```
|
||||
|
||||
# 测试插件
|
||||
|
||||
## 打包 WASM 插件
|
||||
|
||||
> 在 `wasm-go` 目录下把Dockerfile文件改成`PLUGIN_NAME=opa`,然后执行以下命令
|
||||
|
||||
```shell
|
||||
docker build -t build-wasm-opa --build-arg GOPROXY=https://goproxy.cn,direct --platform=linux/amd64 .
|
||||
```
|
||||
|
||||
## 拷贝插件
|
||||
|
||||
> 在当前的目录执行以下命令,将插件拷贝当前的目录
|
||||
|
||||
```shell
|
||||
docker cp wasm-opa:/plugin.wasm .
|
||||
```
|
||||
|
||||
## 运行插件
|
||||
|
||||
> 运行前修改envoy.yaml 这两个字段 `OPA_SERVER` `OPA_PORT` 替换宿主机上的IP和端口
|
||||
|
||||
```shell
|
||||
docker compose up
|
||||
```
|
||||
|
||||
## 使用curl测试插件
|
||||
|
||||
```shell
|
||||
curl http://127.0.0.1:10000/get -X GET -v
|
||||
```
|
||||
|
||||
```shell
|
||||
curl http://127.0.0.1:10000/get -X POST -v
|
||||
```
|
||||
1
plugins/wasm-go/extensions/opa/VERSION
Normal file
1
plugins/wasm-go/extensions/opa/VERSION
Normal file
@@ -0,0 +1 @@
|
||||
1.0.0
|
||||
82
plugins/wasm-go/extensions/opa/config.go
Normal file
82
plugins/wasm-go/extensions/opa/config.go
Normal file
@@ -0,0 +1,82 @@
|
||||
// Copyright (c) 2022 Alibaba Group Holding Ltd.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"strings"
|
||||
|
||||
"github.com/alibaba/higress/plugins/wasm-go/pkg/wrapper"
|
||||
"github.com/tidwall/gjson"
|
||||
)
|
||||
|
||||
type OpaConfig struct {
|
||||
policy string
|
||||
timeout uint32
|
||||
|
||||
client wrapper.HttpClient
|
||||
}
|
||||
|
||||
func Client(json gjson.Result) (wrapper.HttpClient, error) {
|
||||
serviceSource := strings.TrimSpace(json.Get("serviceSource").String())
|
||||
serviceName := strings.TrimSpace(json.Get("serviceName").String())
|
||||
servicePort := json.Get("servicePort").Int()
|
||||
|
||||
host := strings.TrimSpace(json.Get("host").String())
|
||||
if host == "" {
|
||||
if serviceName == "" || servicePort == 0 {
|
||||
return nil, errors.New("invalid service config")
|
||||
}
|
||||
}
|
||||
|
||||
var namespace string
|
||||
if serviceSource == "k8s" || serviceSource == "nacos" {
|
||||
if namespace = strings.TrimSpace(json.Get("namespace").String()); namespace == "" {
|
||||
return nil, errors.New("namespace not allow empty")
|
||||
}
|
||||
}
|
||||
|
||||
switch serviceSource {
|
||||
case "k8s":
|
||||
return wrapper.NewClusterClient(wrapper.K8sCluster{
|
||||
ServiceName: serviceName,
|
||||
Namespace: namespace,
|
||||
Port: servicePort,
|
||||
}), nil
|
||||
case "nacos":
|
||||
return wrapper.NewClusterClient(wrapper.NacosCluster{
|
||||
ServiceName: serviceName,
|
||||
NamespaceID: namespace,
|
||||
Port: servicePort,
|
||||
}), nil
|
||||
case "ip":
|
||||
return wrapper.NewClusterClient(wrapper.StaticIpCluster{
|
||||
ServiceName: serviceName,
|
||||
Host: host,
|
||||
Port: servicePort,
|
||||
}), nil
|
||||
case "dns":
|
||||
return wrapper.NewClusterClient(wrapper.DnsCluster{
|
||||
ServiceName: serviceName,
|
||||
Port: servicePort,
|
||||
Domain: json.Get("domain").String(),
|
||||
}), nil
|
||||
case "route":
|
||||
return wrapper.NewClusterClient(wrapper.RouteCluster{
|
||||
Host: host,
|
||||
}), nil
|
||||
}
|
||||
return nil, errors.New("unknown service source: " + serviceSource)
|
||||
}
|
||||
50
plugins/wasm-go/extensions/opa/config_test.go
Normal file
50
plugins/wasm-go/extensions/opa/config_test.go
Normal file
@@ -0,0 +1,50 @@
|
||||
// Copyright (c) 2022 Alibaba Group Holding Ltd.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/alibaba/higress/plugins/wasm-go/pkg/wrapper"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/tidwall/gjson"
|
||||
)
|
||||
|
||||
func TestConfig(t *testing.T) {
|
||||
json := gjson.Result{Type: gjson.JSON, Raw: `{"serviceSource": "k8s","serviceName": "opa","servicePort": 8181,"namespace": "example1","policy": "example1","timeout": "5s"}`}
|
||||
config := &OpaConfig{}
|
||||
assert.NoError(t, parseConfig(json, config, wrapper.Log{}))
|
||||
assert.Equal(t, config.policy, "example1")
|
||||
assert.Equal(t, config.timeout, uint32(5000))
|
||||
assert.NotNil(t, config.client)
|
||||
|
||||
type tt struct {
|
||||
raw string
|
||||
result bool
|
||||
}
|
||||
|
||||
tests := []tt{
|
||||
{raw: `{}`, result: false},
|
||||
{raw: `{"policy": "example1","timeout": "5s"}`, result: false},
|
||||
{raw: `{"serviceSource": "route","host": "example.com","policy": "example1","timeout": "5s"}`, result: true},
|
||||
{raw: `{"serviceSource": "nacos","serviceName": "opa","servicePort": 8181,"policy": "example1","timeout": "5s"}`, result: false},
|
||||
{raw: `{"serviceSource": "nacos","serviceName": "opa","servicePort": 8181,"namespace": "example1","policy": "example1","timeout": "5s"}`, result: true},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
json = gjson.Result{Type: gjson.JSON, Raw: test.raw}
|
||||
assert.Equal(t, parseConfig(json, config, wrapper.Log{}) == nil, test.result)
|
||||
}
|
||||
}
|
||||
16
plugins/wasm-go/extensions/opa/docker-compose.yaml
Normal file
16
plugins/wasm-go/extensions/opa/docker-compose.yaml
Normal file
@@ -0,0 +1,16 @@
|
||||
version: '3.7'
|
||||
services:
|
||||
envoy:
|
||||
image: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/gateway:1.3.1
|
||||
entrypoint: /usr/local/bin/envoy
|
||||
command: -c /etc/envoy/envoy.yaml --component-log-level wasm:debug
|
||||
networks:
|
||||
- wasmtest
|
||||
ports:
|
||||
- "10000:10000"
|
||||
volumes:
|
||||
- ./envoy.yaml:/etc/envoy/envoy.yaml
|
||||
- ./plugin.wasm:/etc/envoy/plugin.wasm
|
||||
|
||||
networks:
|
||||
wasmtest: { }
|
||||
69
plugins/wasm-go/extensions/opa/envoy.yaml
Normal file
69
plugins/wasm-go/extensions/opa/envoy.yaml
Normal file
@@ -0,0 +1,69 @@
|
||||
admin:
|
||||
address:
|
||||
socket_address:
|
||||
protocol: TCP
|
||||
address: 0.0.0.0
|
||||
port_value: 9901
|
||||
static_resources:
|
||||
listeners:
|
||||
- name: listener_0
|
||||
address:
|
||||
socket_address:
|
||||
protocol: TCP
|
||||
address: 0.0.0.0
|
||||
port_value: 10000
|
||||
filter_chains:
|
||||
- filters:
|
||||
- name: envoy.filters.network.http_connection_manager
|
||||
typed_config:
|
||||
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
|
||||
stat_prefix: ingress_http
|
||||
route_config:
|
||||
name: local_route
|
||||
virtual_hosts:
|
||||
- name: local_service
|
||||
domains: [ "*" ]
|
||||
routes:
|
||||
- match:
|
||||
prefix: "/"
|
||||
route:
|
||||
cluster: opa-server
|
||||
http_filters:
|
||||
- name: wasmdemo
|
||||
typed_config:
|
||||
"@type": type.googleapis.com/udpa.type.v1.TypedStruct
|
||||
type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
|
||||
value:
|
||||
config:
|
||||
name: wasmdemo
|
||||
vm_config:
|
||||
runtime: envoy.wasm.runtime.v8
|
||||
code:
|
||||
local:
|
||||
filename: /etc/envoy/plugin.wasm
|
||||
configuration:
|
||||
"@type": "type.googleapis.com/google.protobuf.StringValue"
|
||||
value: |
|
||||
{
|
||||
"serviceSource": "route",
|
||||
"host": "OPA_SERVER:OPA_PORT",
|
||||
"policy": "example1",
|
||||
"timeout": "5s"
|
||||
}
|
||||
- name: envoy.filters.http.router
|
||||
clusters:
|
||||
- name: opa-server
|
||||
connect_timeout: 0.5s
|
||||
type: STRICT_DNS
|
||||
lb_policy: ROUND_ROBIN
|
||||
dns_refresh_rate: 5s
|
||||
dns_lookup_family: V4_ONLY
|
||||
load_assignment:
|
||||
cluster_name: opa-server
|
||||
endpoints:
|
||||
- lb_endpoints:
|
||||
- endpoint:
|
||||
address:
|
||||
socket_address:
|
||||
address: OPA_SERVER # opa server Host IP
|
||||
port_value: OPA_PORT # opa server Host PORT
|
||||
23
plugins/wasm-go/extensions/opa/go.mod
Normal file
23
plugins/wasm-go/extensions/opa/go.mod
Normal file
@@ -0,0 +1,23 @@
|
||||
module github.com/alibaba/higress/plugins/wasm-go/extensions/opa
|
||||
|
||||
go 1.19
|
||||
|
||||
require (
|
||||
github.com/alibaba/higress/plugins/wasm-go v0.0.0
|
||||
github.com/stretchr/testify v1.8.4
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0
|
||||
github.com/tidwall/gjson v1.14.4
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/davecgh/go-spew v1.1.1 // indirect
|
||||
github.com/google/uuid v1.5.0 // indirect
|
||||
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520 // indirect
|
||||
github.com/magefile/mage v1.14.0 // indirect
|
||||
github.com/pmezard/go-difflib v1.0.0 // indirect
|
||||
github.com/tidwall/match v1.1.1 // indirect
|
||||
github.com/tidwall/pretty v1.2.0 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
)
|
||||
|
||||
replace github.com/alibaba/higress/plugins/wasm-go => ../..
|
||||
24
plugins/wasm-go/extensions/opa/go.sum
Normal file
24
plugins/wasm-go/extensions/opa/go.sum
Normal file
@@ -0,0 +1,24 @@
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/google/uuid v1.5.0 h1:1p67kYwdtXjb0gL0BPiP1Av9wiZPo5A8z2cWkTZ+eyU=
|
||||
github.com/google/uuid v1.5.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520 h1:IHDghbGQ2DTIXHBHxWfqCYQW1fKjyJ/I7W1pMyUDeEA=
|
||||
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520/go.mod h1:Nz8ORLaFiLWotg6GeKlJMhv8cci8mM43uEnLA5t8iew=
|
||||
github.com/magefile/mage v1.14.0 h1:6QDX3g6z1YvJ4olPhT1wksUcSa/V0a1B+pJb73fBjyo=
|
||||
github.com/magefile/mage v1.14.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
|
||||
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0 h1:kS7BvMKN+FiptV4pfwiNX8e3q14evxAWkhYbxt8EI1M=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0/go.mod h1:qkW5MBz2jch2u8bS59wws65WC+Gtx3x0aPUX5JL7CXI=
|
||||
github.com/tidwall/gjson v1.14.4 h1:uo0p8EbA09J7RQaflQ1aBRffTR7xedD2bcIVSYxLnkM=
|
||||
github.com/tidwall/gjson v1.14.4/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
|
||||
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
|
||||
github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
|
||||
github.com/tidwall/pretty v1.2.0 h1:RWIZEg2iJ8/g6fDDYzMpobmaoGh5OLl4AXtGUGPcqCs=
|
||||
github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
134
plugins/wasm-go/extensions/opa/main.go
Normal file
134
plugins/wasm-go/extensions/opa/main.go
Normal file
@@ -0,0 +1,134 @@
|
||||
// Copyright (c) 2022 Alibaba Group Holding Ltd.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/alibaba/higress/plugins/wasm-go/pkg/wrapper"
|
||||
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
|
||||
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
|
||||
"github.com/tidwall/gjson"
|
||||
)
|
||||
|
||||
func main() {
|
||||
wrapper.SetCtx(
|
||||
"opa",
|
||||
wrapper.ParseConfigBy(parseConfig),
|
||||
wrapper.ProcessRequestHeadersBy(onHttpRequestHeaders),
|
||||
wrapper.ProcessRequestBodyBy(onHttpRequestBody),
|
||||
)
|
||||
}
|
||||
|
||||
type Metadata struct {
|
||||
Input map[string]interface{} `json:"input"`
|
||||
}
|
||||
|
||||
func parseConfig(json gjson.Result, config *OpaConfig, log wrapper.Log) error {
|
||||
policy := json.Get("policy").String()
|
||||
if strings.TrimSpace(policy) == "" {
|
||||
return errors.New("policy not allow empty")
|
||||
}
|
||||
|
||||
timeout := json.Get("timeout").String()
|
||||
if strings.TrimSpace(timeout) == "" {
|
||||
return errors.New("timeout not allow empty")
|
||||
}
|
||||
|
||||
duration, err := time.ParseDuration(timeout)
|
||||
if err != nil {
|
||||
return errors.New("timeout parse fail: " + err.Error())
|
||||
}
|
||||
|
||||
var uint32Duration uint32
|
||||
|
||||
if duration.Milliseconds() > int64(^uint32(0)) {
|
||||
} else {
|
||||
uint32Duration = uint32(duration.Milliseconds())
|
||||
}
|
||||
config.timeout = uint32Duration
|
||||
|
||||
client, err := Client(json)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
config.client = client
|
||||
config.policy = policy
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func onHttpRequestHeaders(ctx wrapper.HttpContext, config OpaConfig, log wrapper.Log) types.Action {
|
||||
return opaCall(ctx, config, nil, log)
|
||||
}
|
||||
|
||||
func onHttpRequestBody(ctx wrapper.HttpContext, config OpaConfig, body []byte, log wrapper.Log) types.Action {
|
||||
return opaCall(ctx, config, body, log)
|
||||
}
|
||||
|
||||
func opaCall(ctx wrapper.HttpContext, config OpaConfig, body []byte, log wrapper.Log) types.Action {
|
||||
request := make(map[string]interface{}, 6)
|
||||
headers, _ := proxywasm.GetHttpRequestHeaders()
|
||||
|
||||
request["method"] = ctx.Method()
|
||||
request["scheme"] = ctx.Scheme()
|
||||
request["path"] = ctx.Path()
|
||||
request["headers"] = headers
|
||||
if len(body) != 0 {
|
||||
request["body"] = body
|
||||
}
|
||||
parse, _ := url.Parse(ctx.Path())
|
||||
query, _ := url.ParseQuery(parse.RawQuery)
|
||||
request["query"] = query
|
||||
|
||||
data, _ := json.Marshal(Metadata{Input: map[string]interface{}{"request": request}})
|
||||
if err := config.client.Post(fmt.Sprintf("/v1/data/%s/allow", config.policy),
|
||||
[][2]string{{"Content-Type", "application/json"}},
|
||||
data, rspCall, config.timeout); err != nil {
|
||||
log.Errorf("client opa fail %v", err)
|
||||
return types.ActionPause
|
||||
}
|
||||
return types.ActionPause
|
||||
}
|
||||
|
||||
func rspCall(statusCode int, _ http.Header, responseBody []byte) {
|
||||
if statusCode != http.StatusOK {
|
||||
proxywasm.SendHttpResponse(uint32(statusCode), nil, []byte("opa state not is 200"), -1)
|
||||
return
|
||||
}
|
||||
var rsp map[string]interface{}
|
||||
if err := json.Unmarshal(responseBody, &rsp); err != nil {
|
||||
proxywasm.SendHttpResponse(http.StatusInternalServerError, nil, []byte(fmt.Sprintf("opa parse rsp fail %+v", err)), -1)
|
||||
return
|
||||
}
|
||||
|
||||
result, ok := rsp["result"].(bool)
|
||||
if !ok {
|
||||
proxywasm.SendHttpResponse(http.StatusInternalServerError, nil, []byte("rsp type conversion fail"), -1)
|
||||
return
|
||||
}
|
||||
|
||||
if !result {
|
||||
proxywasm.SendHttpResponse(http.StatusUnauthorized, nil, []byte("opa server not allowed"), -1)
|
||||
return
|
||||
}
|
||||
proxywasm.ResumeHttpRequest()
|
||||
}
|
||||
@@ -6,7 +6,7 @@ replace github.com/alibaba/higress/plugins/wasm-go => ../..
|
||||
|
||||
require (
|
||||
github.com/alibaba/higress/plugins/wasm-go v0.0.0
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0
|
||||
github.com/tidwall/gjson v1.14.3
|
||||
)
|
||||
|
||||
|
||||
@@ -7,8 +7,8 @@ github.com/magefile/mage v1.14.0 h1:6QDX3g6z1YvJ4olPhT1wksUcSa/V0a1B+pJb73fBjyo=
|
||||
github.com/magefile/mage v1.14.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c h1:OCUFXVGixHLfNjg6/QYEhv+jHJ5mRGhpEUVFv9eWPJE=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.19.1-0.20220822060051-f9d179a57f8c/go.mod h1:5t/pWFNJ9eMyu/K/Z+OeGhDJ9sN9eCo8fc2pyM/Qjg4=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0 h1:kS7BvMKN+FiptV4pfwiNX8e3q14evxAWkhYbxt8EI1M=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0/go.mod h1:qkW5MBz2jch2u8bS59wws65WC+Gtx3x0aPUX5JL7CXI=
|
||||
github.com/tidwall/gjson v1.14.3 h1:9jvXn7olKEHU1S9vwoMGliaT8jq1vJ7IH/n9zD9Dnlw=
|
||||
github.com/tidwall/gjson v1.14.3/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
|
||||
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
|
||||
|
||||
146
plugins/wasm-go/extensions/request-validation/README.md
Normal file
146
plugins/wasm-go/extensions/request-validation/README.md
Normal file
@@ -0,0 +1,146 @@
|
||||
# 功能说明
|
||||
`request-validation`插件用于提前验证向上游服务转发的请求。该插件使用`JSON Schema`机制进行数据验证,可以验证请求的body及header数据。
|
||||
|
||||
# 配置字段
|
||||
|
||||
| 名称 | 数据类型 | 填写要求 | 默认值 | 描述 |
|
||||
| -------- | -------- | -------- |-----| -------- |
|
||||
|header_schema|object|选填| - |配置用于验证请求header的JSON Schema|
|
||||
|body_schema|object|选填| - |配置用于验证请求body的JSON Schema|
|
||||
|rejected_code|number|选填| 403 |配置请求被拒绝时返回的HTTP状态码|
|
||||
|rejected_msg|string|选填| - |配置请求被拒绝时返回的HTTP应答Body|
|
||||
|enable_swagger|bool|选填| false |配置是否开启swagger文档验证|
|
||||
|enable_oas3|bool|选填| false |配置是否开启OAS3文档验证|
|
||||
|
||||
**校验规则对header和body是一样的,下面以body为例说明**
|
||||
|
||||
# 配置示例
|
||||
|
||||
## 枚举(Enum)验证
|
||||
```yaml
|
||||
body_schema:
|
||||
type: object
|
||||
required:
|
||||
- enum_payload
|
||||
properties:
|
||||
enum_payload:
|
||||
type: string
|
||||
enum:
|
||||
- "enum_string_1"
|
||||
- "enum_string_2"
|
||||
default: "enum_string_1"
|
||||
```
|
||||
|
||||
## 布尔(Boolean)验证
|
||||
```yaml
|
||||
body_schema:
|
||||
type: object
|
||||
required:
|
||||
- boolean_payload
|
||||
properties:
|
||||
boolean_payload:
|
||||
type: boolean
|
||||
default: true
|
||||
```
|
||||
|
||||
## 数字范围(Number or Integer)验证
|
||||
```yaml
|
||||
body_schema:
|
||||
type: object
|
||||
required:
|
||||
- integer_payload
|
||||
properties:
|
||||
integer_payload:
|
||||
type: integer
|
||||
minimum: 1
|
||||
maximum: 10
|
||||
```
|
||||
|
||||
## 字符串长度(String)验证
|
||||
```yaml
|
||||
body_schema:
|
||||
type: object
|
||||
required:
|
||||
- string_payload
|
||||
properties:
|
||||
string_payload:
|
||||
type: string
|
||||
minLength: 1
|
||||
maxLength: 10
|
||||
```
|
||||
|
||||
## 正则表达式(Regex)验证
|
||||
```yaml
|
||||
body_schema:
|
||||
type: object
|
||||
required:
|
||||
- regex_payload
|
||||
properties:
|
||||
regex_payload:
|
||||
type: string
|
||||
minLength: 1
|
||||
maxLength: 10
|
||||
pattern: "^[a-zA-Z0-9_]+$"
|
||||
```
|
||||
|
||||
## 数组(Array)验证
|
||||
```yaml
|
||||
body_schema:
|
||||
type: object
|
||||
required:
|
||||
- array_payload
|
||||
properties:
|
||||
array_payload:
|
||||
type: array
|
||||
minItems: 1
|
||||
items:
|
||||
type: integer
|
||||
minimum: 1
|
||||
maximum: 10
|
||||
uniqueItems: true
|
||||
default: [1, 2, 3]
|
||||
```
|
||||
|
||||
## 多字段组合(Combined)验证
|
||||
```yaml
|
||||
body_schema:
|
||||
type: object
|
||||
required:
|
||||
- boolean_payload
|
||||
- array_payload
|
||||
- regex_payload
|
||||
properties:
|
||||
boolean_payload:
|
||||
type: boolean
|
||||
array_payload:
|
||||
type: array
|
||||
minItems: 1
|
||||
items:
|
||||
type: integer
|
||||
minimum: 1
|
||||
maximum: 10
|
||||
uniqueItems: true
|
||||
default: [1, 2, 3]
|
||||
regex_payload:
|
||||
type: string
|
||||
minLength: 1
|
||||
maxLength: 10
|
||||
pattern: "^[a-zA-Z0-9_]+$"
|
||||
```
|
||||
|
||||
## 自定义拒绝信息
|
||||
```yaml
|
||||
body_schema:
|
||||
type: object
|
||||
required:
|
||||
- boolean_payload
|
||||
properties:
|
||||
boolean_payload:
|
||||
type: boolean
|
||||
rejected_code: 403
|
||||
rejected_msg: "请求被拒绝"
|
||||
```
|
||||
|
||||
# 本地调试
|
||||
|
||||
参考[使用 GO 语言开发 WASM 插件](https://higress.io/zh-cn/docs/user/wasm-go#%E4%B8%89%E6%9C%AC%E5%9C%B0%E8%B0%83%E8%AF%95)
|
||||
1
plugins/wasm-go/extensions/request-validation/VERSION
Normal file
1
plugins/wasm-go/extensions/request-validation/VERSION
Normal file
@@ -0,0 +1 @@
|
||||
1.0.0
|
||||
18
plugins/wasm-go/extensions/request-validation/go.mod
Normal file
18
plugins/wasm-go/extensions/request-validation/go.mod
Normal file
@@ -0,0 +1,18 @@
|
||||
module github.com/alibaba/higress/plugins/wasm-go/extensions/request-validation
|
||||
|
||||
go 1.19
|
||||
|
||||
require (
|
||||
github.com/alibaba/higress/plugins/wasm-go v1.3.1
|
||||
github.com/santhosh-tekuri/jsonschema v1.2.4
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0
|
||||
github.com/tidwall/gjson v1.17.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/google/uuid v1.3.0 // indirect
|
||||
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520 // indirect
|
||||
github.com/magefile/mage v1.14.0 // indirect
|
||||
github.com/tidwall/match v1.1.1 // indirect
|
||||
github.com/tidwall/pretty v1.2.0 // indirect
|
||||
)
|
||||
22
plugins/wasm-go/extensions/request-validation/go.sum
Normal file
22
plugins/wasm-go/extensions/request-validation/go.sum
Normal file
@@ -0,0 +1,22 @@
|
||||
github.com/alibaba/higress/plugins/wasm-go v1.3.1 h1:d+t4W2NyqmqUz6DPZENflODfkLgdVlTfyso+nq0fSkg=
|
||||
github.com/alibaba/higress/plugins/wasm-go v1.3.1/go.mod h1:WZ/68vwe8qWhusa6C4/gMwUqas0jvHWSOa1bp8iK8F4=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
|
||||
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520 h1:IHDghbGQ2DTIXHBHxWfqCYQW1fKjyJ/I7W1pMyUDeEA=
|
||||
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520/go.mod h1:Nz8ORLaFiLWotg6GeKlJMhv8cci8mM43uEnLA5t8iew=
|
||||
github.com/magefile/mage v1.14.0 h1:6QDX3g6z1YvJ4olPhT1wksUcSa/V0a1B+pJb73fBjyo=
|
||||
github.com/magefile/mage v1.14.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/santhosh-tekuri/jsonschema v1.2.4 h1:hNhW8e7t+H1vgY+1QeEQpveR6D4+OwKPXCfD2aieJis=
|
||||
github.com/santhosh-tekuri/jsonschema v1.2.4/go.mod h1:TEAUOeZSmIxTTuHatJzrvARHiuO9LYd+cIxzgEHCQI4=
|
||||
github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0 h1:kS7BvMKN+FiptV4pfwiNX8e3q14evxAWkhYbxt8EI1M=
|
||||
github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0/go.mod h1:qkW5MBz2jch2u8bS59wws65WC+Gtx3x0aPUX5JL7CXI=
|
||||
github.com/tidwall/gjson v1.17.0 h1:/Jocvlh98kcTfpN2+JzGQWQcqrPQwDrVEMApx/M5ZwM=
|
||||
github.com/tidwall/gjson v1.17.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
|
||||
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
|
||||
github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
|
||||
github.com/tidwall/pretty v1.2.0 h1:RWIZEg2iJ8/g6fDDYzMpobmaoGh5OLl4AXtGUGPcqCs=
|
||||
github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
199
plugins/wasm-go/extensions/request-validation/main.go
Normal file
199
plugins/wasm-go/extensions/request-validation/main.go
Normal file
@@ -0,0 +1,199 @@
|
||||
// Copyright (c) 2022 Alibaba Group Holding Ltd.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/alibaba/higress/plugins/wasm-go/pkg/wrapper"
|
||||
"github.com/santhosh-tekuri/jsonschema"
|
||||
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
|
||||
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
|
||||
"github.com/tidwall/gjson"
|
||||
)
|
||||
|
||||
const (
|
||||
defaultHeaderSchema = "header"
|
||||
defaultBodySchema = "body"
|
||||
defaultRejectedCode = 403
|
||||
)
|
||||
|
||||
func main() {
|
||||
wrapper.SetCtx(
|
||||
"request-validation",
|
||||
wrapper.ProcessRequestHeadersBy(onHttpRequestHeaders),
|
||||
wrapper.ProcessRequestBodyBy(onHttpRequestBody),
|
||||
wrapper.ParseConfigBy(parseConfig),
|
||||
)
|
||||
}
|
||||
|
||||
// Config is the config for request validation.
|
||||
type Config struct {
|
||||
// compiler is the compiler for json schema.
|
||||
compiler *jsonschema.Compiler
|
||||
// rejectedCode is the code for rejected request.
|
||||
rejectedCode uint32
|
||||
// rejectedMsg is the message for rejected request.
|
||||
rejectedMsg string
|
||||
// draft is the draft version of json schema.
|
||||
draft *jsonschema.Draft
|
||||
// enableBodySchema is the flag for enable body schema.
|
||||
enableBodySchema bool
|
||||
// enableHeaderSchema is the flag for enable header schema.
|
||||
enableHeaderSchema bool
|
||||
}
|
||||
|
||||
func parseConfig(result gjson.Result, config *Config, log wrapper.Log) error {
|
||||
headerSchema := result.Get("header_schema").String()
|
||||
bodySchema := result.Get("body_schema").String()
|
||||
enableSwagger := result.Get("enable_swagger").Bool()
|
||||
enableOas3 := result.Get("enable_oas3").Bool()
|
||||
code := result.Get("rejected_code").Int()
|
||||
msg := result.Get("rejected_msg").String()
|
||||
|
||||
// set config default value
|
||||
config.enableBodySchema = false
|
||||
config.enableHeaderSchema = false
|
||||
|
||||
// check enable_swagger and enable_oas3
|
||||
if enableSwagger && enableOas3 {
|
||||
return fmt.Errorf("enable_swagger and enable_oas3 can not be true at the same time")
|
||||
}
|
||||
|
||||
// set draft version
|
||||
if enableSwagger {
|
||||
config.draft = jsonschema.Draft4
|
||||
}
|
||||
if enableOas3 {
|
||||
config.draft = jsonschema.Draft7
|
||||
}
|
||||
if !enableSwagger && !enableOas3 {
|
||||
config.draft = jsonschema.Draft7
|
||||
}
|
||||
|
||||
// create compiler
|
||||
compiler := jsonschema.NewCompiler()
|
||||
compiler.Draft = config.draft
|
||||
config.compiler = compiler
|
||||
|
||||
// add header schema to compiler
|
||||
if headerSchema != "" {
|
||||
err := config.compiler.AddResource(defaultHeaderSchema, strings.NewReader(headerSchema))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
config.enableHeaderSchema = true
|
||||
}
|
||||
|
||||
// add body schema to compiler
|
||||
if bodySchema != "" {
|
||||
err := config.compiler.AddResource(defaultBodySchema, strings.NewReader(bodySchema))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
config.enableBodySchema = true
|
||||
}
|
||||
|
||||
// check rejected_code is valid
|
||||
if code != 0 && code > 100 && code < 600 {
|
||||
config.rejectedCode = uint32(code)
|
||||
} else {
|
||||
config.rejectedCode = defaultRejectedCode
|
||||
}
|
||||
config.rejectedMsg = msg
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func onHttpRequestHeaders(ctx wrapper.HttpContext, config Config, log wrapper.Log) types.Action {
|
||||
if !config.enableHeaderSchema {
|
||||
return types.ActionContinue
|
||||
}
|
||||
|
||||
// get headers
|
||||
headers, err := proxywasm.GetHttpRequestHeaders()
|
||||
if err != nil {
|
||||
log.Errorf("get request headers failed: %v", err)
|
||||
return types.ActionContinue
|
||||
}
|
||||
|
||||
// covert to schema
|
||||
schema := make(map[string]interface{})
|
||||
for _, header := range headers {
|
||||
schema[header[0]] = header[1]
|
||||
}
|
||||
|
||||
// convert to json string
|
||||
schemaBytes, err := json.Marshal(schema)
|
||||
if err != nil {
|
||||
log.Errorf("marshal schema failed: %v", err)
|
||||
return types.ActionContinue
|
||||
}
|
||||
|
||||
// validate
|
||||
document := strings.NewReader(string(schemaBytes))
|
||||
compile, err := config.compiler.Compile(defaultHeaderSchema)
|
||||
if err != nil {
|
||||
log.Errorf("compile schema failed: %v", err)
|
||||
return types.ActionContinue
|
||||
}
|
||||
err = compile.Validate(document)
|
||||
if err != nil {
|
||||
log.Errorf("validate request headers failed: %v", err)
|
||||
proxywasm.SendHttpResponse(config.rejectedCode, nil, []byte(config.rejectedMsg), -1)
|
||||
return types.ActionPause
|
||||
}
|
||||
|
||||
return types.ActionContinue
|
||||
}
|
||||
|
||||
func onHttpRequestBody(ctx wrapper.HttpContext, config Config, body []byte, log wrapper.Log) types.Action {
|
||||
if !config.enableBodySchema {
|
||||
return types.ActionContinue
|
||||
}
|
||||
|
||||
// covert to schema
|
||||
schema := make(map[string]interface{})
|
||||
err := json.Unmarshal(body, &schema)
|
||||
if err != nil {
|
||||
log.Errorf("unmarshal body failed: %v", err)
|
||||
return types.ActionContinue
|
||||
}
|
||||
|
||||
// convert to json string
|
||||
schemaBytes, err := json.Marshal(schema)
|
||||
if err != nil {
|
||||
log.Errorf("marshal schema failed: %v", err)
|
||||
return types.ActionContinue
|
||||
}
|
||||
|
||||
// validate
|
||||
document := strings.NewReader(string(schemaBytes))
|
||||
compile, err := config.compiler.Compile(defaultBodySchema)
|
||||
if err != nil {
|
||||
log.Errorf("compile schema failed: %v", err)
|
||||
return types.ActionContinue
|
||||
}
|
||||
err = compile.Validate(document)
|
||||
if err != nil {
|
||||
log.Errorf("validate request body failed: %v", err)
|
||||
proxywasm.SendHttpResponse(config.rejectedCode, nil, []byte(config.rejectedMsg), -1)
|
||||
return types.ActionPause
|
||||
}
|
||||
|
||||
return types.ActionContinue
|
||||
}
|
||||
@@ -5,15 +5,16 @@
|
||||
# 配置字段
|
||||
|
||||
| 名称 | 数据类型 | 填写要求 | 默认值 | 描述 |
|
||||
| -------- | -------- | -------- | -------- | -------- |
|
||||
| type | string | 必填,可选值为 `request`, `response` | - | 指定转换器类型 |
|
||||
| rules | array of object | 选填 | - | 指定转换操作类型以及请求/响应头、请求查询参数、请求/响应体参数的转换规则 |
|
||||
| :----: | :----: | :----: | :----: | -------- |
|
||||
| reqRules | string | 选填,reqRules和respRules至少填一个 | - | 请求转换器配置,指定转换操作类型以及请求头、请求查询参数、请求体的转换规则 |
|
||||
| respRules | string | 选填,reqRules和respRules至少填一个 | - | 响应转换器配置,指定转换操作类型以及响应头、响应体的转换规则 |
|
||||
|
||||
`rules`中每一项的配置字段说明如下:
|
||||
`reqRules`和`respRules`中每一项的配置字段说明如下:
|
||||
|
||||
| 名称 | 数据类型 | 填写要求 | 默认值 | 描述 |
|
||||
| -------- | -------- | -------- | -------- | -------- |
|
||||
| :----: | :----: | :----: | :----: | -------- |
|
||||
| operate | string | 必填,可选值为 `remove`, `rename`, `replace`, `add`, `append`, `map`, `dedupe` | - | 指定转换操作类型,支持的操作类型有删除 (remove)、重命名 (rename)、更新 (replace)、添加 (add)、追加 (append)、映射 (map)、去重 (dedupe),当存在多项不同类型的转换规则时,按照上述操作类型顺序依次执行 |
|
||||
| mapSource | string | 选填,可选值为`headers`, `querys`,`body` | - | 仅在operate为`map`时有效。指定映射来源,若不填该字段,则默认映射来源为自身 |
|
||||
| headers | array of object | 选填 | - | 指定请求/响应头转换规则 |
|
||||
| querys | array of object | 选填 | - | 指定请求查询参数转换规则 |
|
||||
| body | array of object | 选填 | - | 指定请求/响应体参数转换规则,请求体转换允许 content-type 为 `application/json`, `application/x-www-form-urlencoded`, `multipart/form-data`;响应体转换仅允许 content-type 为 `application/json` |
|
||||
@@ -21,9 +22,20 @@
|
||||
`headers`, `querys`, `body`中每一项的配置字段说明如下:
|
||||
|
||||
| 名称 | 数据类型 | 填写要求 | 默认值 | 描述 |
|
||||
| -------- | -------- | -------- | -------- |---------------------------------------------------|
|
||||
| key | string | 选填 | - | 指定键,详见[转换操作类型](#转换操作类型) |
|
||||
| value | string | 选填 | - | 指定值,详见[转换操作类型](#转换操作类型) |
|
||||
| :----: | :----: | :----: | -------- |---------------------------------------------------|
|
||||
| key | string | 选填 | - | 在operate为`remove`时使用,用法详见[转换操作类型](#转换操作类型) |
|
||||
| oldKey | string | 选填 | - |在operate为`rename`时使用,用法详见[转换操作类型](#转换操作类型) |
|
||||
| newKey | string | 选填 | - | 在operate为`rename`时使用,用法详见[转换操作类型](#转换操作类型) |
|
||||
| key | string | 选填 | - | 在operate为`replace`时使用,用法详见[转换操作类型](#转换操作类型) |
|
||||
| newValue | string | 选填 | - | 在operate为`replace`时使用,用法详见[转换操作类型](#转换操作类型) |
|
||||
| key | string | 选填 | - | 在operate为`add`时使用,用法详见[转换操作类型](#转换操作类型) |
|
||||
| value | string | 选填 | - | 在operate为`add`时使用,用法详见[转换操作类型](#转换操作类型) |
|
||||
| key | string | 选填 | - | 在operate为`append`时使用,用法详见[转换操作类型](#转换操作类型) |
|
||||
| appendValue | string | 选填 | - | 在operate为`append`时使用,用法详见[转换操作类型](#转换操作类型) |
|
||||
| fromKey | string | 选填 | - | 在operate为`map`时使用,用法详见[转换操作类型](#转换操作类型) |
|
||||
| toKey | string | 选填 | - | 在operate为`map`时使用,用法详见[转换操作类型](#转换操作类型) |
|
||||
| key | string | 选填 | - | 在operate为`dedupe`时使用,用法详见[转换操作类型](#转换操作类型) |
|
||||
| strategy | string | 选填 | - | 在operate为`dedupe`时使用,用法详见[转换操作类型](#转换操作类型) |
|
||||
| value_type | string | 选填,可选值为 `object`, `boolean`, `number`, `string` | string | 当`content-type: application/json`时,该字段指定请求/响应体参数的值类型 |
|
||||
| host_pattern | string | 选填 | - | 指定请求主机名匹配规则,当转换操作类型为 `replace`, `add`, `append` 时有效 |
|
||||
| path_pattern | string | 选填 | - | 指定请求路径匹配规则,当转换操作类型为 `replace`, `add`, `append` 时有效 |
|
||||
@@ -32,8 +44,8 @@
|
||||
|
||||
* `request transformer` 支持以下转换对象:请求头部、请求查询参数、请求体(application/json, application/x-www-form-urlencoded, multipart/form-data)
|
||||
* `response transformer` 支持以下转换对象:响应头部、响应体(application/json)
|
||||
|
||||
* 转换操作类型的执行顺序:remove → rename → replace → add → append → map → dedupe
|
||||
* 插件支持双向转换能力,即单个插件能够完成对请求和响应都做转换
|
||||
* 转换操作类型的执行顺序,为配置文件中编写的顺序,如:remove → rename → replace → add → append → map → dedupe或者dedupe → map → append → add → replace → rename → remove等
|
||||
* 当转换对象为 headers 时,` key` 不区分大小写;当为 headers 且为 `rename`, `map` 操作时,`value` 也不区分大小写(因为此时该字段具有 key 含义);而 querys 和 body 的 `key`, `value` 字段均区分大小写
|
||||
* `value_type` 仅对 content-type 为 application/json 的请求/响应体有效
|
||||
* `host_pattern` 和 `path_pathern` 支持 [RE2 语法](https://pkg.go.dev/regexp/syntax),仅对 `replace`, `add`, `append` 操作有效,且在一项转换规则中两者只能选填其一,若均填写,则 `host_pattern` 生效,而 `path_pattern` 失效
|
||||
@@ -43,7 +55,7 @@
|
||||
# 转换操作类型
|
||||
|
||||
| 操作类型 | key 字段含义 | value 字段含义 | 描述 |
|
||||
| ------------- | ----------------- |-----| ------------------------------------------------------------ |
|
||||
| :----: | :----: | :----: | ------------------------------------------------------------ |
|
||||
| 删除 remove | 目标 key |无需设置| 若存在指定的 `key`,则删除;否则无操作 |
|
||||
| 重命名 rename | 目标 oldKey |新的 key 名称 newKey| 若存在指定的 `oldKey:value`,则将其键名重命名为 `newKey`,得到 `newKey:value`;否则无操作 |
|
||||
| 更新 replace | 目标 key |新的 value 值 newValue| 若存在指定的 `key:value`,则将其 value 更新为 `newValue`,得到 `key:newValue`;否则无操作 |
|
||||
@@ -62,19 +74,18 @@
|
||||
### 转换请求头部
|
||||
|
||||
```yaml
|
||||
type: request
|
||||
rules:
|
||||
reqRules:
|
||||
- operate: remove
|
||||
headers:
|
||||
- key: X-remove
|
||||
- operate: rename
|
||||
headers:
|
||||
- key: X-not-renamed
|
||||
value: X-renamed
|
||||
- oldKey: X-not-renamed
|
||||
newKey: X-renamed
|
||||
- operate: replace
|
||||
headers:
|
||||
- key: X-replace
|
||||
value: replaced
|
||||
newValue: replaced
|
||||
- operate: add
|
||||
headers:
|
||||
- key: X-add-append
|
||||
@@ -83,20 +94,20 @@ rules:
|
||||
- operate: append
|
||||
headers:
|
||||
- key: X-add-append
|
||||
value: path-$1
|
||||
appendValue: path-$1
|
||||
path_pattern: ^.*?\/(\w+)[\?]{0,1}.*$
|
||||
- operate: map
|
||||
headers:
|
||||
- key: X-add-append
|
||||
value: X-map
|
||||
- fromKey: X-add-append
|
||||
toKey: X-map
|
||||
- operate: dedupe
|
||||
headers:
|
||||
- key: X-dedupe-first
|
||||
value: RETAIN_FIRST
|
||||
strategy: RETAIN_FIRST
|
||||
- key: X-dedupe-last
|
||||
value: RETAIN_LAST
|
||||
strategy: RETAIN_LAST
|
||||
- key: X-dedupe-unique
|
||||
value: RETAIN_UNIQUE
|
||||
strategy: RETAIN_UNIQUE
|
||||
```
|
||||
|
||||
发送请求
|
||||
@@ -131,19 +142,18 @@ $ curl -v console.higress.io/get -H 'host: foo.bar.com' \
|
||||
### 转换请求查询参数
|
||||
|
||||
```yaml
|
||||
type: request
|
||||
rules:
|
||||
reqRules:
|
||||
- operate: remove
|
||||
querys:
|
||||
- key: k1
|
||||
- operate: rename
|
||||
querys:
|
||||
- key: k2
|
||||
value: k2-new
|
||||
- oldKey: k2
|
||||
newKey: k2-new
|
||||
- operate: replace
|
||||
querys:
|
||||
- key: k2-new
|
||||
value: v2-new
|
||||
newValue: v2-new
|
||||
- operate: add
|
||||
querys:
|
||||
- key: k3
|
||||
@@ -152,15 +162,15 @@ rules:
|
||||
- operate: append
|
||||
querys:
|
||||
- key: k3
|
||||
value: v32
|
||||
appendValue: v32
|
||||
- operate: map
|
||||
querys:
|
||||
- key: k3
|
||||
value: k4
|
||||
- fromKey: k3
|
||||
toKey: k4
|
||||
- operate: dedupe
|
||||
querys:
|
||||
- key: k4
|
||||
value: RETAIN_FIRST
|
||||
strategy: RETAIN_FIRST
|
||||
```
|
||||
|
||||
发送请求
|
||||
@@ -186,19 +196,18 @@ $ curl -v "console.higress.io/get?k1=v11&k1=v12&k2=v2"
|
||||
### 转换请求体
|
||||
|
||||
```yaml
|
||||
type: request
|
||||
rules:
|
||||
reqRules:
|
||||
- operate: remove
|
||||
body:
|
||||
- key: a1
|
||||
- operate: rename
|
||||
body:
|
||||
- key: a2
|
||||
value: a2-new
|
||||
- oldKey: a2
|
||||
newKey: a2-new
|
||||
- operate: replace
|
||||
body:
|
||||
- key: a3
|
||||
value: t3-new
|
||||
newValue: t3-new
|
||||
value_type: string
|
||||
- operate: add
|
||||
body:
|
||||
@@ -208,17 +217,17 @@ rules:
|
||||
- operate: append
|
||||
body:
|
||||
- key: a1-new
|
||||
value: t1-$1-append
|
||||
appendValue: t1-$1-append
|
||||
value_type: string
|
||||
host_pattern: ^(.*)\.com$
|
||||
- operate: map
|
||||
body:
|
||||
- key: a1-new
|
||||
value: a4
|
||||
- fromKey: a1-new
|
||||
toKey: a4
|
||||
- operate: dedupe
|
||||
body:
|
||||
- key: a4
|
||||
value: RETAIN_FIRST
|
||||
strategy: RETAIN_FIRST
|
||||
```
|
||||
|
||||
发送请求:
|
||||
@@ -313,8 +322,7 @@ $ curl -v -X POST console.higress.io/post -H 'host: foo.bar.com' \
|
||||
1.通常情况下,指定的 key 中含有 `.` 表示嵌套含义,如下:
|
||||
|
||||
```yaml
|
||||
type: response
|
||||
rules:
|
||||
respRules:
|
||||
- operate: add
|
||||
body:
|
||||
- key: foo.bar
|
||||
@@ -339,8 +347,7 @@ $ curl -v console.higress.io/get
|
||||
> 当使用双引号括住字符串时使用 `\\.` 进行转义
|
||||
|
||||
```yaml
|
||||
type: response
|
||||
rules:
|
||||
respRules:
|
||||
- operate: add
|
||||
body:
|
||||
- key: foo\.bar
|
||||
@@ -378,8 +385,7 @@ $ curl -v console.higress.io/get
|
||||
1.移除 `user` 第一个元素:
|
||||
|
||||
```yaml
|
||||
type: request
|
||||
rules:
|
||||
reqRules:
|
||||
- operate: remove
|
||||
body:
|
||||
- key: users.0
|
||||
@@ -409,12 +415,11 @@ $ curl -v -X POST console.higress.io/post \
|
||||
2.将 `users` 第一个元素的 key 为 `123` 重命名为 `msg`:
|
||||
|
||||
```yaml
|
||||
type: request
|
||||
rules:
|
||||
reqRules:
|
||||
- operate: rename
|
||||
body:
|
||||
- key: users.0.123
|
||||
value: users.0.first
|
||||
- oldKey: users.0.123
|
||||
newKey: users.0.first
|
||||
```
|
||||
|
||||
```bash
|
||||
@@ -466,12 +471,11 @@ $ curl -v -X POST console.higress.io/post \
|
||||
```
|
||||
|
||||
```yaml
|
||||
type: request
|
||||
rules:
|
||||
reqRules:
|
||||
- operate: replace
|
||||
body:
|
||||
- key: users.#.age
|
||||
value: 20
|
||||
newValue: 20
|
||||
```
|
||||
|
||||
```bash
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -59,6 +59,15 @@ func isValidOperation(op string) bool {
|
||||
}
|
||||
}
|
||||
|
||||
func isValidMapSource(source string) bool {
|
||||
switch source {
|
||||
case "headers", "querys", "body":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
func parseQueryByPath(path string) (map[string][]string, error) {
|
||||
u, err := url.Parse(path)
|
||||
if err != nil {
|
||||
|
||||
@@ -2,12 +2,12 @@ package wasmplugin
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"github.com/corazawaf/coraza/v3/debuglog"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/alibaba/higress/plugins/wasm-go/pkg/wrapper"
|
||||
"github.com/corazawaf/coraza/v3"
|
||||
"github.com/corazawaf/coraza/v3/debuglog"
|
||||
ctypes "github.com/corazawaf/coraza/v3/types"
|
||||
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
|
||||
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
|
||||
@@ -76,6 +76,15 @@ func onHttpRequestHeaders(ctx wrapper.HttpContext, config WafConfig, log wrapper
|
||||
|
||||
tx := ctx.GetContext("tx").(ctypes.Transaction)
|
||||
|
||||
protocol, err := proxywasm.GetProperty([]string{"request", "protocol"})
|
||||
if err != nil {
|
||||
// TODO(anuraaga): HTTP protocol is commonly required in WAF rules, we should probably
|
||||
// fail fast here, but proxytest does not support properties yet.
|
||||
protocol = []byte("HTTP/2.0")
|
||||
}
|
||||
|
||||
ctx.SetContext("httpProtocol", string(protocol))
|
||||
|
||||
// Note the pseudo-header :path includes the query.
|
||||
// See https://httpwg.org/specs/rfc9113.html#rfc.section.8.3.1
|
||||
uri, err := proxywasm.GetHttpRequestHeader(":path")
|
||||
@@ -103,15 +112,6 @@ func onHttpRequestHeaders(ctx wrapper.HttpContext, config WafConfig, log wrapper
|
||||
return types.ActionContinue
|
||||
}
|
||||
|
||||
protocol, err := proxywasm.GetProperty([]string{"request", "protocol"})
|
||||
if err != nil {
|
||||
// TODO(anuraaga): HTTP protocol is commonly required in WAF rules, we should probably
|
||||
// fail fast here, but proxytest does not support properties yet.
|
||||
protocol = []byte("HTTP/2.0")
|
||||
}
|
||||
|
||||
ctx.SetContext("httpProtocol", string(protocol))
|
||||
|
||||
tx.ProcessURI(uri, method, string(protocol))
|
||||
|
||||
hs, err := proxywasm.GetHttpRequestHeaders()
|
||||
|
||||
@@ -116,7 +116,7 @@ func (p *NacosAddressProvider) GetNacosAddress(oldAddress string) <-chan string
|
||||
var addr string
|
||||
p.cond.L.Lock()
|
||||
defer p.cond.L.Unlock()
|
||||
log.Debugf("get nacos address, p.nacosAddr, oldAddress", p.nacosAddr, oldAddress)
|
||||
log.Debugf("get nacos address, p.nacosAddr: %s, oldAddress: %s", p.nacosAddr, oldAddress)
|
||||
for p.nacosAddr == oldAddress || p.nacosAddr == "" {
|
||||
if p.isStop.Load() {
|
||||
return
|
||||
|
||||
@@ -26,9 +26,7 @@ import (
|
||||
"github.com/nacos-group/nacos-sdk-go/model"
|
||||
"github.com/nacos-group/nacos-sdk-go/vo"
|
||||
"istio.io/api/networking/v1alpha3"
|
||||
versionedclient "istio.io/client-go/pkg/clientset/versioned"
|
||||
"istio.io/pkg/log"
|
||||
ctrl "sigs.k8s.io/controller-runtime"
|
||||
|
||||
apiv1 "github.com/alibaba/higress/api/networking/v1"
|
||||
"github.com/alibaba/higress/pkg/common"
|
||||
@@ -61,7 +59,6 @@ type watcher struct {
|
||||
cache memory.Cache
|
||||
mutex *sync.Mutex
|
||||
stop chan struct{}
|
||||
client *versionedclient.Clientset
|
||||
isStop bool
|
||||
updateCacheWhenEmpty bool
|
||||
authOption provider.AuthOption
|
||||
@@ -79,18 +76,6 @@ func NewWatcher(cache memory.Cache, opts ...WatcherOption) (provider.Watcher, er
|
||||
stop: make(chan struct{}),
|
||||
}
|
||||
|
||||
config, err := ctrl.GetConfig()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
ic, err := versionedclient.NewForConfig(config)
|
||||
if err != nil {
|
||||
log.Errorf("can not new istio client, err:%v", err)
|
||||
return nil, err
|
||||
}
|
||||
w.client = ic
|
||||
|
||||
w.NacosRefreshInterval = int64(DefaultRefreshInterval)
|
||||
|
||||
for _, opt := range opts {
|
||||
|
||||
@@ -43,6 +43,26 @@ And after that, you should add your defined HigressConformanceTest to `e2e_test.
|
||||
|
||||
You can understand it quickly just by looking at codes in `test/ingress/conformance/tests/httproute-simple-same-namespace.go` and `test/ingress/conformance/tests/httproute-simple-same-namespace.yaml`, and try to write one.
|
||||
|
||||
### How to Implement Test Environment Reusability
|
||||
|
||||
The test environment reusability is primarily achieved through the following targets in the Makefile:
|
||||
|
||||
1. **make higress-conformance-test:** Used to run the entire Conformance testing process, including setting up the test environment, executing test cases, and cleaning up the test environment.
|
||||
- **make higress-conformance-test-prepare:** Can be used to set up the environment for deployments such as higress-controller, higress-gateway, etc.
|
||||
- **make run-higress-e2e-test:** Used to run the test cases.
|
||||
- **make run-higress-e2e-test-setup:** Can be used to install the basic resources required for the test cases, such as nacos, dubbo, etc.
|
||||
- **make run-higress-e2e-test-run:** Used to execute the test cases.
|
||||
- **make run-higress-e2e-test-clean:** Can be used to clean up the basic resources installed during the setup phase of the test cases.
|
||||
- **make higress-conformance-test-clean:** Used to clean up the test environment for deployments like higress-controller, higress-gateway, etc.
|
||||
|
||||
2. **make higress-wasmplugin-test:** Used to run the entire WasmPlugin testing process, including setting up the test environment, compiling WasmPlugin plugins, executing test cases, and cleaning up the test environment.
|
||||
- **make higress-wasmplugin-test-prepare:** Can be used to set up the environment for deployments such as higress-controller, higress-gateway, and compile WasmPlugin plugins.
|
||||
- **make run-higress-e2e-test-wasmplugin:** Used to run the test cases.
|
||||
- **make run-higress-e2e-test-wasmplugin-setup:** Can be used to install the basic resources required for the test cases, such as nacos, dubbo, etc.
|
||||
- **make run-higress-e2e-test-wasmplugin-run:** Used to execute the test cases.
|
||||
- **make run-higress-e2e-test-wasmplugin-clean:** Can be used to clean up the basic resources installed during the setup phase of the test cases.
|
||||
- **make higress-wasmplugin-test-clean:** Used to clean up the test environment for deployments like higress-controller, higress-gateway, etc.
|
||||
|
||||
## Gateway APIs Conformance Tests
|
||||
|
||||
Gateway API Conformance tests are based on the suite provided by `kubernetes-sig/gateway-api`, we can reuse that,
|
||||
|
||||
76
test/README_CN.md
Normal file
76
test/README_CN.md
Normal file
@@ -0,0 +1,76 @@
|
||||
# Higress E2E 测试
|
||||
|
||||
Higress E2E 测试主要关注两个部分:
|
||||
|
||||
+ Ingress API 的一致性测试
|
||||
+ Gateway API 的一致性测试
|
||||
|
||||
## Ingress API 一致性测试
|
||||
|
||||
### 架构
|
||||
|
||||

|
||||
|
||||
### 工作流程
|
||||
|
||||

|
||||
|
||||
Higress 提供了运行 Ingress API 一致性测试和 wasmplugin 测试的 make 目标,
|
||||
|
||||
+ API 测试: `make higress-conformance-test`
|
||||
+ WasmPlugin 测试: `make higress-wasmplugin-test`
|
||||
+ 为测试构建所有 GO WasmPlugins: `make higress-wasmplugin-test`
|
||||
+ 仅为一个 GO WasmPlugin 构建测试: `PLUGIN_NAME=request-block make higress-wasmplugin-test`
|
||||
+ 仅为一个 CPP WasmPlugin 构建测试: `PLUGIN_TYPE=CPP PLUGIN_NAME=key_auth make higress-wasmplugin-test`
|
||||
|
||||
可以分为以下步骤:
|
||||
|
||||
1. delete-cluster: 检查是否有未删除的 kind 集群。
|
||||
2. create-cluster: 创建一个新的 kind 集群。
|
||||
3. docker-build: 构建 higress 的开发镜像。
|
||||
4. kube-load-image: 将 dev higress-controller 镜像加载到 kind 集群中。
|
||||
5. install-dev: 使用 helm 安装带有 dev 镜像的 higress-controller,并安装最新的 higress-gateway、istiod。
|
||||
6. run-e2e-test:
|
||||
1. 设置一致性测试套件,例如在 `e2e_test.go` / `higressTests Slice` 中定义我们想要运行的一致性测试。我们选择打开的每个测试都在 `test/ingress/conformance/tests` 中定义。
|
||||
2. 准备资源并将它们安装到集群中,例如后端服务/部署。
|
||||
3. 在 `e2e_test.go` / `higressTests Slice` 中加载我们选择打开的一致性测试,并逐个运行它们,如果不符合预期,则失败。
|
||||
|
||||
### 如何编写测试用例
|
||||
|
||||
要添加新的测试用例,首先需要在 `test/ingress/conformance/tests` 中添加 `xxx.go` 和 `xxx.yaml`。`xxx.yaml` 是您需要在集群中应用的 Ingress 资源,`xxx.go` 定义了 HigressConformanceTest。
|
||||
|
||||
然后,您应该将您定义的 HigressConformanceTest 添加到 `e2e_test.go` / `higressTests Slice` 中。
|
||||
|
||||
通过查看 `test/ingress/conformance/tests/httproute-simple-same-namespace.go` 和 `test/ingress/conformance/tests/httproute-simple-same-namespace.yaml` 中的代码,您可以快速了解并尝试编写一个测试用例。
|
||||
|
||||
### 如何实现测试环境的复用
|
||||
|
||||
主要通过 Makefile 中的以下几个目标实现:
|
||||
|
||||
1. **make higress-conformance-test:** 用于运行整个 Conformance 测试流程,包括搭建测试环境、运行测试用例、清理测试环境。
|
||||
- **make higress-conformance-test-prepare:** 可用于搭建 higress-controller、higress-gateway 等 deployment 的环境。
|
||||
- **make run-higress-e2e-test:** 可用于运行测试用例。
|
||||
- **make run-higress-e2e-test-setup:** 可用于安装测试用例所需的基础资源,例如 nacos、dubbo 等。
|
||||
- **make run-higress-e2e-test-run:** 可用于运行测试用例。
|
||||
- **make run-higress-e2e-test-clean:** 可用于清理测试用例在 setup 阶段所安装的基础资源。
|
||||
- **make higress-conformance-test-clean:** 可用于清理 higress-controller、higress-gateway 等 deployment 的测试环境。
|
||||
|
||||
2. **make higress-wasmplugin-test:** 用于运行整个 WasmPlugin 测试流程,包括搭建测试环境、编译 WasmPlugin 插件、运行测试用例、清理测试环境。
|
||||
- **make higress-wasmplugin-test-prepare:** 可用于搭建 higress-controller、higress-gateway 等 deployment 的环境,并编译 WasmPlugin 插件。
|
||||
- **make run-higress-e2e-test-wasmplugin:** 可用于运行测试用例。
|
||||
- **make run-higress-e2e-test-wasmplugin-setup:** 可用于安装测试用例所需的基础资源,例如 nacos、dubbo 等。
|
||||
- **make run-higress-e2e-test-wasmplugin-run:** 可用于运行测试用例。
|
||||
- **make run-higress-e2e-test-wasmplugin-clean:** 可用于清理测试用例在 setup 阶段所安装的基础资源。
|
||||
- **make higress-wasmplugin-test-clean:** 可用于清理 higress-controller、higress-gateway 等 deployment 的测试环境。
|
||||
|
||||
## Gateway API 一致性测试
|
||||
|
||||
Gateway API 一致性测试基于 `kubernetes-sig/gateway-api` 提供的套件,我们可以重复使用它,并决定我们需要打开哪些 Gateway API 的一致性测试。
|
||||
|
||||
此 API 包含一系列广泛的功能和用例,并已得到广泛实现。
|
||||
这个大的功能集和各种实现的结合需要明确的一致性定义和测试,以确保在任何地方使用该 API 时都提供一致的体验。
|
||||
|
||||
Gateway API 包括一组一致性测试。这些测试创建具有指定 GatewayClass 的一系列 Gateways 和 Routes,并测试实现是否符合 API 规范。
|
||||
|
||||
每个发布版本都包含一组一致性测试,随着 API 的演进,这些测试将不断扩展。
|
||||
目前,一致性测试覆盖了标准通道中的大多数核心功能,以及一些扩展功能。
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user