Compare commits
49 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e9f34e1d19 | ||
|
|
ccdcebb24d | ||
|
|
7ddd75e34f | ||
|
|
aebea435fa | ||
|
|
ea788313c6 | ||
|
|
727edfcca8 | ||
|
|
cc1989a74b | ||
|
|
0196a90b69 | ||
|
|
9c3e3988e0 | ||
|
|
458e13c9e0 | ||
|
|
979859b232 | ||
|
|
b163e5f776 | ||
|
|
d062e18940 | ||
|
|
87c1e7ba4a | ||
|
|
5194c952f2 | ||
|
|
c1cc44d32f | ||
|
|
82fafe980d | ||
|
|
34752deca2 | ||
|
|
9e42e2c72a | ||
|
|
e531f5d786 | ||
|
|
10e75ac55d | ||
|
|
4a8d09dc89 | ||
|
|
116bc100a7 | ||
|
|
b1feaad9f7 | ||
|
|
4d372f8374 | ||
|
|
4b2c544c0d | ||
|
|
8131cb1a42 | ||
|
|
1dd6466261 | ||
|
|
dda08a2152 | ||
|
|
01c7121ee4 | ||
|
|
d939d7653c | ||
|
|
058cd5a24e | ||
|
|
db3f55ac4a | ||
|
|
a311a34537 | ||
|
|
e8fe2ea1c7 | ||
|
|
10302dd39c | ||
|
|
55a4483fcc | ||
|
|
4dd2412b78 | ||
|
|
387c714072 | ||
|
|
6a2d876d50 | ||
|
|
f5fb2c4f88 | ||
|
|
6f9676e259 | ||
|
|
2427ce2c1e | ||
|
|
02abe67fce | ||
|
|
ad39f4e82c | ||
|
|
243c89b459 | ||
|
|
11418cd6e0 | ||
|
|
b19c6200d2 | ||
|
|
5f6a06c100 |
144
README.md
@@ -1,89 +1,93 @@
|
||||
# kafka可视化管理平台
|
||||
一款轻量级的kafka可视化管理平台,安装配置快捷、简单易用。
|
||||
为了开发的省事,没有国际化支持,只支持中文展示。
|
||||
为了开发的省事,没有国际化支持,页面只支持中文展示。
|
||||
用过rocketmq-console吧,对,前端展示风格跟那个有点类似。
|
||||
## 安装包下载
|
||||
以下两种方式2选一,直接下载安装包或下载源码,手动打包
|
||||
* 点击下载(v1.0.1版本):[kafka-console-ui.tar.gz](https://github.com/xxd763795151/kafka-console-ui/releases/download/v1.0.1/kafka-console-ui.tar.gz) 或 [kafka-console-ui.zip](https://github.com/xxd763795151/kafka-console-ui/releases/download/v1.0.1/kafka-console-ui.zip)
|
||||
* 参考下面的打包部署,下载源码重新打包(提交的最新功能特性)
|
||||
|
||||
## 页面预览
|
||||
如果github能查看图片的话,可以点击[查看菜单页面](./document/overview/概览.md),查看每个页面的样子
|
||||
## 集群迁移支持说明
|
||||
当前主分支及日后版本不再提供消息同步、集群迁移的解决方案,如有需要,查看:[集群迁移说明](./document/datasync/集群迁移.md)
|
||||
## ACL说明
|
||||
acl配置说明,如果kafka集群启用了ACL,但是控制台没看到Acl菜单,可以查看:[Acl配置启用说明](./document/acl/Acl.md)
|
||||
## 功能支持
|
||||
* 多集群支持
|
||||
* 集群信息
|
||||
* Topic管理
|
||||
* 消费组管理
|
||||
* 基于SASL_SCRAM认证授权管理
|
||||
* 消息管理
|
||||
* ACL
|
||||
* 运维
|
||||

|
||||
## 技术栈
|
||||
* spring boot
|
||||
* java、scala
|
||||
* kafka
|
||||
* h2
|
||||
* vue
|
||||
## kafka版本
|
||||
* 当前使用的kafka 2.8.0
|
||||
## 监控
|
||||
仅提供运维管理功能,监控、告警需要配合其它组件,使用请查看:https://blog.csdn.net/x763795151/article/details/119705372
|
||||
# 打包、部署
|
||||
## 打包
|
||||
环境要求
|
||||
* maven 3.6+
|
||||
* jdk 8
|
||||
* git
|
||||
|
||||
功能明细看这个脑图:
|
||||

|
||||
|
||||
## 安装包下载
|
||||
点击下载(v1.0.4版本):[kafka-console-ui.zip](https://github.com/xxd763795151/kafka-console-ui/releases/download/v1.0.4/kafka-console-ui.zip)
|
||||
|
||||
如果安装包下载的比较慢,可以查看下面的源码打包说明,把代码下载下来,快速打包,不过最新main分支代码刚升级了kafka版本到3.2.0,还没有充分测试,如果需要稳定版本,可以下载 1.0.4-release分支代码
|
||||
|
||||
## 快速使用
|
||||
### Windows
|
||||
1. 解压缩zip安装包
|
||||
2. 进入bin目录(必须在bin目录下),双击执行`start.bat`启动
|
||||
3. 停止:直接关闭启动的命令行窗口即可
|
||||
|
||||
### Linux或Mac OS
|
||||
```
|
||||
git clone https://github.com/xxd763795151/kafka-console-ui.git
|
||||
cd kafka-console-ui
|
||||
# linux或mac执行
|
||||
sh package.sh
|
||||
# windows执行
|
||||
package.bat
|
||||
```
|
||||
打包成功,输出文件(以下2种归档类型):
|
||||
* target/kafka-console-ui.tar.gz
|
||||
* target/kafka-console-ui.zip
|
||||
## 部署
|
||||
### Mac OS 或 Linux
|
||||
```
|
||||
# 解压缩(以tar.gz为例)
|
||||
tar -zxvf kafka-console-ui.tar.gz
|
||||
# 解压缩
|
||||
unzip kafka-console-ui.zip
|
||||
# 进入解压缩后的目录
|
||||
cd kafka-console-ui
|
||||
# 编辑配置
|
||||
vim config/application.yml
|
||||
# 启动
|
||||
sh bin/start.sh
|
||||
# 停止
|
||||
sh bin/shutdown.sh
|
||||
```
|
||||
### Windows
|
||||
1.解压缩zip安装包
|
||||
2.编辑配置文件 :`config/application.yml`
|
||||
3.进入bin目录(必须在bin目录下),执行`start.bat`启动
|
||||
|
||||
启动完成,访问:http://127.0.0.1:7766
|
||||
### 访问地址
|
||||
启动完成,访问:http://127.0.0.1:7766
|
||||
|
||||
# 开发环境
|
||||
* jdk 8
|
||||
* idea
|
||||
* scala 2.13
|
||||
* maven >=3.6+
|
||||
* webstorm
|
||||
除了webstorm是开发前端的ide可以根据自己需要代替,jdk scala是必须有的。
|
||||
# 本地开发配置
|
||||
以我自己为例,开发环境里的工具准备好,然后代码clone到本地。
|
||||
## 后端配置
|
||||
1. 用idea打开项目
|
||||
2. 打开idea的Project Structure(Settings) -> Modules -> 设置src/main/scala为Sources,因为约定src/main/java是源码目录,所以这里要再加一个
|
||||
3. 打开idea的Project Structure(Settings) -> Libraries 添加scala sdk,然后选择本地下载的scala 2.13的目录,确定添加进来
|
||||
## 前端
|
||||
前端代码在工程的ui目录下,找个前端开发的ide打开进行开发即可。
|
||||
## 注意
|
||||
前后分离,直接启动后端如果未编译前端代码是没有前端页面的,可以先打包进行编译`sh package.sh`,然后再用idea启动,或者前端部分单独启动
|
||||
# 页面示例
|
||||
如果未启用ACL配置,不会显示ACL的菜单页面,所以导航栏上没有Acl这一项
|
||||
### 配置集群
|
||||
第一次启动,打开浏览器后,因为还没有配置kafka集群信息,所以页面右上角可能会有错误信息,比如:No Cluster Info或者是没有集群信息,请先切换集群之类的提示。
|
||||
|
||||

|
||||

|
||||

|
||||

|
||||
增加消息检索页面
|
||||

|
||||
集群配置如下:
|
||||
1. 点击页面上方导航栏的 [运维] 菜单
|
||||
2. 点击集群管理下的 [集群切换] 按钮
|
||||
3. 在弹框里点击 [新增集群]
|
||||
4. 然后输入kafka集群地址和一个名称(随便起个名字)
|
||||
5. 点击提交便增加成功了
|
||||
6. 增加成功可以看到会话框已经有这个集群信息,然后点击右侧的 [切换] 按钮,便切换该集群为当前集群
|
||||
|
||||
后续如果再增加其它集群,就可以按上面这个流程,如果想切换到哪个集群,点击切换按钮,便会切换到对应的集群,页面的右上角会显示当前是使用的哪个集群,如果不确定,可以刷新下页面。
|
||||
|
||||
在新增集群的时候,除了集群地址还可以输入集群的其它属性配置,比如请求超时,ACL配置等。如果开启了ACL,切换到该集群的时候,导航栏上便会出现ACL菜单,支持进行相关操作(目前是基于SASL_SCRAM认证授权管理支持的最完善,其它的我也没验证过,虽然是我开发的,但是我也没具体全部验证这一块功能,授权部分应该是通用的)
|
||||
|
||||
## kafka版本
|
||||
* 当前使用的kafka 3.2.0
|
||||
## 监控
|
||||
仅提供运维管理功能,监控、告警需要配合其它组件,如有需要,建议请查看:https://blog.csdn.net/x763795151/article/details/119705372
|
||||
|
||||
## 源码打包
|
||||
如果想通过源码打包,查看:[源码打包说明](./document/package/源码打包.md)
|
||||
|
||||
## 本地开发
|
||||
如果需要本地开发,开发环境配置查看:[本地开发](./document/develop/开发配置.md)
|
||||
|
||||
## 登录认证和权限
|
||||
目前主分支不支持登录认证,感谢@dongyinuo 同学开发了一版支持登录认证,及相关的按钮权限(主要有两个角色:管理员和普通开发人员)。
|
||||
在分支:feature/dongyinuo/20220501/devops 上。
|
||||
如果有需要使用管理台登录认证的,可以切换到这个分支上进行打包,打包方式看 源码打包 说明。
|
||||
默认登录账户:admin/kafka-console-ui521
|
||||
|
||||
## DockerCompose部署
|
||||
感谢@wdkang123 同学分享的部署方式,如果有需要请查看[DockerCompose部署方式](./document/deploy/docker部署.md)
|
||||
|
||||
## 联系方式
|
||||
+ 微信群
|
||||
<img src="./document/contact/weixin_contact.jpg" width="40%"/>
|
||||
|
||||
[//]: # (<img src="https://github.com/xxd763795151/kafka-console-ui/blob/main/document/contact/weixin_contact.jpeg" width="40%"/>)
|
||||
|
||||
+ 若联系方式失效, 请联系加一下微信, 说明意图
|
||||
- xxd763795151
|
||||
- wxid_7jy2ezljvebt12
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
|
||||
<id>rocketmq-reput</id>
|
||||
<formats>
|
||||
<format>tar.gz</format>
|
||||
<!-- <format>tar.gz</format>-->
|
||||
<format>zip</format>
|
||||
</formats>
|
||||
|
||||
|
||||
@@ -5,4 +5,4 @@ set JAVA_OPTS=-Xmx512m -Xms512m -Xmn256m -Xss256k
|
||||
set CONFIG_FILE=../config/application.yml
|
||||
set TARGET=../lib/kafka-console-ui.jar
|
||||
set DATA_DIR=..
|
||||
%JAVA_CMD% -jar %TARGET% --spring.config.location=%CONFIG_FILE% --data.dir=%DATA_DIR%
|
||||
"%JAVA_CMD%" -jar %TARGET% --spring.config.location=%CONFIG_FILE% --data.dir=%DATA_DIR%
|
||||
|
||||
|
Before Width: | Height: | Size: 42 KiB |
36
document/acl/Acl.md
Normal file
@@ -0,0 +1,36 @@
|
||||
# Acl配置启用说明
|
||||
## 前言
|
||||
可能有的同学是看了这篇文章来的:[如何通过可视化方式快捷管理kafka的acl配置](https://blog.csdn.net/x763795151/article/details/120200119)
|
||||
|
||||
这篇文章里可能说了是通过修改配置文件application.yml的方式来启用ACL,示例如下:
|
||||
```yaml
|
||||
kafka:
|
||||
config:
|
||||
# kafka broker地址,多个以逗号分隔
|
||||
bootstrap-server: 'localhost:9092'
|
||||
# 服务端是否启用acl,如果不启用,下面的几项都忽略即可
|
||||
enable-acl: true
|
||||
# 只支持2种安全协议SASL_PLAINTEXT和PLAINTEXT,启用acl则设置为SASL_PLAINTEXT,不启用acl不需关心这个配置
|
||||
security-protocol: SASL_PLAINTEXT
|
||||
sasl-mechanism: SCRAM-SHA-256
|
||||
# 超级管理员用户名,在broker上已经配置为超级管理员
|
||||
admin-username: admin
|
||||
# 超级管理员密码
|
||||
admin-password: admin
|
||||
# 启动自动创建配置的超级管理员用户
|
||||
admin-create: true
|
||||
# broker连接的zk地址
|
||||
zookeeper-addr: localhost:2181
|
||||
sasl-jaas-config: org.apache.kafka.common.security.scram.ScramLoginModule required username="${kafka.config.admin-username}" password="${kafka.config.admin-password}";
|
||||
```
|
||||
其中说明了kafka.config.enable-acl配置项需要为true。
|
||||
|
||||
注意:**现在不再支持这种方式了**
|
||||
## 新版本说明
|
||||
因为现在支持多集群配置,关于多集群配置,可以看主页说明的 配置集群 介绍。
|
||||
所以这里把这些额外的配置项都去掉了。
|
||||
|
||||
如果启用了ACL,在页面上新增集群的时候,在属性里配置集群的ACL相关信息,如下:
|
||||
如果控制台检测到属性里有ACL相关属性配置,切换到这个集群后,ACL菜单会自动出现的。
|
||||
|
||||
注意:只支持SASL。
|
||||
BIN
document/acl/新增集群.png
Normal file
|
After Width: | Height: | Size: 245 KiB |
BIN
document/contact/weixin_contact.jpg
Normal file
|
After Width: | Height: | Size: 205 KiB |
11
document/datasync/集群迁移.md
Normal file
@@ -0,0 +1,11 @@
|
||||
# 集群迁移
|
||||
可能是有的同学看了我以前发的解决云上、云下集群迁移的方案,好奇看到了这里。
|
||||
当时发的文章链接在这里:[kafka新老集群平滑迁移实践](https://blog.csdn.net/x763795151/article/details/121070563)
|
||||
|
||||
不过考虑到这些功能涉及到业务属性,已经在新的版本中都去掉了。
|
||||
|
||||
当前主分支及日后版本不再提供消息同步、集群迁移的解决方案,如有需要,可以使用single-data-sync分支的代码或者历史发布的v1.0.2(直接下载): [kafka-console-ui.zip](https://github.com/xxd763795151/kafka-console-ui/releases/download/v1.0.2/kafka-console-ui.zip) 的安装包。
|
||||
|
||||
v1.0.2版本及其之前的版本只支持单集群配置,但是对于SASL_SCRAM认证授权管理功能相当完善。
|
||||
|
||||
后续版本会支持多集群管理,并将v1.0.2之前的部分功能去掉或优化,目的是做为一个足够轻量的管理工具,不再涉及其它属性。
|
||||
189
document/deploy/docker部署.md
Normal file
@@ -0,0 +1,189 @@
|
||||
# Docker/DockerCompose部署
|
||||
|
||||
# 1.快速上手
|
||||
|
||||
## 1.1 镜像拉取
|
||||
|
||||
```shell
|
||||
docker pull wdkang/kafka-console-ui
|
||||
```
|
||||
|
||||
|
||||
|
||||
## 1.2 查看镜像
|
||||
|
||||
```shell
|
||||
docker images
|
||||
```
|
||||
|
||||
|
||||
|
||||
## 1.3 启动服务
|
||||
|
||||
由于Docker内不会对数据进行持久化 所以这里推荐将数据目录映射到实体机中
|
||||
|
||||
详见 **2.数据持久**
|
||||
|
||||
```shell
|
||||
docker run -d -p 7766:7766 wdkang/kafka-console-ui
|
||||
```
|
||||
|
||||
|
||||
|
||||
## 1.4 查看状态
|
||||
|
||||
```shell
|
||||
docker ps -a
|
||||
```
|
||||
|
||||
|
||||
|
||||
## 1.5 查看日志
|
||||
|
||||
```shell
|
||||
docker logs -f ${containerId}
|
||||
```
|
||||
|
||||
|
||||
|
||||
## 1.6 访问服务
|
||||
|
||||
```shell
|
||||
http://localhost:7766
|
||||
```
|
||||
|
||||
|
||||
|
||||
# 2. 数据持久
|
||||
|
||||
推荐对数据进行持久化
|
||||
|
||||
## 2.1 新建目录
|
||||
|
||||
```shell
|
||||
mkdir -p /home/kafka-console-ui/data /home/kafka-console-ui/log
|
||||
cd /home/kafka-console-ui
|
||||
```
|
||||
|
||||
## 2.2 启动服务
|
||||
|
||||
```shell
|
||||
docker run -d -p 7766:7766 -v $PWD/data:/app/data -v $PWD/log:/app/log wdkang/kafka-console-ui
|
||||
```
|
||||
|
||||
|
||||
|
||||
# 3.自主打包
|
||||
|
||||
## 3.1 构建镜像
|
||||
|
||||
**前置需求**
|
||||
|
||||
(可根据自身情况修改Dockerfile)
|
||||
|
||||
下载[kafka-console-ui.zip](https://github.com/xxd763795151/kafka-console-ui/releases)包
|
||||
|
||||
解压后 将Dockerfile放入文件夹的根目录
|
||||
|
||||
**Dockefile**
|
||||
|
||||
```dockerfile
|
||||
# jdk
|
||||
FROM openjdk:8-jdk-alpine
|
||||
# label
|
||||
LABEL by="https://github.com/xxd763795151/kafka-console-ui"
|
||||
# root
|
||||
RUN mkdir -p /app && cd /app
|
||||
WORKDIR /app
|
||||
# config log data
|
||||
RUN mkdir -p /app/config && mkdir -p /app/log && mkdir -p /app/data && mkdir -p /app/lib
|
||||
# add file
|
||||
ADD ./lib/kafka-console-ui.jar /app/lib
|
||||
ADD ./config /app/config
|
||||
# port
|
||||
EXPOSE 7766
|
||||
# start server
|
||||
CMD java -jar -Xmx512m -Xms512m -Xmn256m -Xss256k /app/lib/kafka-console-ui.jar --spring.config.location="/app/config/" --logging.home="/app/log" --data.dir="/app/data"
|
||||
|
||||
```
|
||||
|
||||
**进行打包**
|
||||
|
||||
在文件夹根目录下
|
||||
|
||||
(注意末尾有个点)
|
||||
|
||||
```shell
|
||||
docker build -t ${your_docker_hub_addr} .
|
||||
```
|
||||
|
||||
|
||||
|
||||
## 3.2 上传镜像
|
||||
|
||||
```shell
|
||||
docker push ${your_docker_hub_addr}
|
||||
```
|
||||
|
||||
|
||||
|
||||
# 4.容器编排
|
||||
|
||||
```dockerfile
|
||||
# docker-compose 编排
|
||||
version: '3'
|
||||
services:
|
||||
# 服务名
|
||||
kafka-console-ui:
|
||||
# 容器名
|
||||
container_name: "kafka-console-ui"
|
||||
# 端口
|
||||
ports:
|
||||
- "7766:7766"
|
||||
# 持久化
|
||||
volumes:
|
||||
- ./data:/app/data
|
||||
- ./log:/app/log
|
||||
# 防止读写文件有问题
|
||||
privileged: true
|
||||
user: root
|
||||
# 镜像地址
|
||||
image: "wdkang/kafka-console-ui"
|
||||
```
|
||||
|
||||
|
||||
|
||||
## 4.1 拉取镜像
|
||||
|
||||
```shell
|
||||
docker-compose pull kafka-console-ui
|
||||
```
|
||||
|
||||
|
||||
|
||||
## 4.2 构建启动
|
||||
|
||||
```shell
|
||||
docker-compose up --detach --build kafka-console-ui
|
||||
```
|
||||
|
||||
|
||||
|
||||
## 4.3 查看状态
|
||||
|
||||
```shell
|
||||
docker-compose ps -a
|
||||
```
|
||||
|
||||
|
||||
|
||||
## 4.3 停止服务
|
||||
|
||||
```shell
|
||||
docker-compose down
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
35
document/develop/开发配置.md
Normal file
@@ -0,0 +1,35 @@
|
||||
# 本地开发配置说明
|
||||
## 技术栈
|
||||
* spring boot
|
||||
* java、scala
|
||||
* kafka
|
||||
* h2
|
||||
* vue
|
||||
|
||||
## 开发环境
|
||||
* jdk 8
|
||||
* idea
|
||||
* scala 2.13
|
||||
* maven >=3.6+
|
||||
* webstorm
|
||||
* Node
|
||||
|
||||
除了webstorm是开发前端的ide可以根据自己需要代替,jdk scala是必须有的。
|
||||
|
||||
开发的时候,我本地用的node版本是v14.16.0,下载目录:https://nodejs.org/download/release/v14.16.0/ . 过高或过低版本是否适用,我也没测试过。
|
||||
|
||||
scala 2.13下载地址,在这个页面最下面:https://www.scala-lang.org/download/scala2.html
|
||||
## 克隆代码
|
||||
以我自己为例,开发环境里的工具准备好,然后代码clone到本地。
|
||||
## 后端配置
|
||||
1. 用idea打开项目
|
||||
2. 打开idea的Project Structure(Settings) -> Modules -> 设置src/main/scala为Sources,因为约定src/main/java是源码目录,所以这里要再加一个源码目录
|
||||
3. 打开idea的Settings -> plugins 搜索scala plugin并安装,然后应该是要重启idea生效,这一步必须在第4步之前
|
||||
4. 打开idea的Project Structure(Settings) -> Libraries 添加scala sdk,然后选择本地下载的scala 2.13的目录,确定添加进来(如果使用的idea可以直接勾选,也可以不用先下载到本地)
|
||||
## 前端
|
||||
前端代码在工程的ui目录下,找个前端开发的ide如web storm打开进行开发即可。
|
||||
|
||||
## 注意
|
||||
前后分离,直接启动后端工程的话,src/main/resources目录下可能是没有静态文件的,所以直接通过浏览器访问应该是没页面的。
|
||||
|
||||
可以先打包编译一下前端文件,比如执行:`sh package.sh`,然后再用idea启动。或者是后端用idea启动完成后,找个前端的ide 比如web storm打开工程的ui目录下的前端项目,单独启动,进行开发。
|
||||
BIN
document/img/功能特性-old.png
Normal file
|
After Width: | Height: | Size: 99 KiB |
BIN
document/img/功能特性.png
Normal file
|
After Width: | Height: | Size: 103 KiB |
BIN
document/overview/img/Broker配置.png
Normal file
|
After Width: | Height: | Size: 45 KiB |
BIN
document/overview/img/Topic.png
Normal file
|
After Width: | Height: | Size: 38 KiB |
BIN
document/overview/img/消息时间查询.png
Normal file
|
After Width: | Height: | Size: 75 KiB |
BIN
document/overview/img/消息详情.png
Normal file
|
After Width: | Height: | Size: 36 KiB |
BIN
document/overview/img/消费组.png
Normal file
|
After Width: | Height: | Size: 45 KiB |
BIN
document/overview/img/运维.png
Normal file
|
After Width: | Height: | Size: 58 KiB |
BIN
document/overview/img/集群.png
Normal file
|
After Width: | Height: | Size: 26 KiB |
BIN
document/overview/img/集群切换.png
Normal file
|
After Width: | Height: | Size: 57 KiB |
32
document/overview/概览.md
Normal file
@@ -0,0 +1,32 @@
|
||||
# 菜单预览
|
||||
如果未启用ACL配置,不会显示ACL的菜单页面,所以导航栏上没有Acl这一项
|
||||
## 集群
|
||||
* 展示集群列表
|
||||

|
||||
* 查看或修改集群配置
|
||||

|
||||
|
||||
|
||||
## Topic
|
||||
* Topic列表
|
||||

|
||||
|
||||
|
||||
## 消费组
|
||||
* 消费组列表
|
||||

|
||||
|
||||
|
||||
## 消息
|
||||
* 根据时间检索或过滤消息
|
||||

|
||||
|
||||
* 消息详情
|
||||

|
||||
|
||||
|
||||
## 运维
|
||||
* 运维页面
|
||||

|
||||
* 集群切换
|
||||

|
||||
34
document/package/源码打包.md
Normal file
@@ -0,0 +1,34 @@
|
||||
# 源码打包说明
|
||||
可以直接下载最新代码,进行打包,最新代码相比已经发布的安装包可能会包含最新的特性
|
||||
|
||||
## 环境要求
|
||||
* maven 3.6+
|
||||
* jdk 8
|
||||
* git(非必须)
|
||||
|
||||
maven是建议版本>=3.6版本,3.4+和3.5+我也没试过,3.3+的版本,在windows上我试了下打包有bug,可能把spring boot的application.yml打不到合适的目录。
|
||||
|
||||
如果3.6+在mac上也不行(也是上面这个问题),建议用最新版本试试。
|
||||
|
||||
## 源码下载
|
||||
```
|
||||
git clone https://github.com/xxd763795151/kafka-console-ui.git
|
||||
```
|
||||
或者直接在页面下载源码
|
||||
|
||||
## 打包
|
||||
我已经写了个简单的打包脚本,直接执行即可。
|
||||
### Windows
|
||||
```
|
||||
cd kafka-console-ui
|
||||
# windows执行
|
||||
package.bat
|
||||
```
|
||||
### Linux或Mac OS
|
||||
```
|
||||
cd kafka-console-ui
|
||||
# linux或mac执行
|
||||
sh package.sh
|
||||
```
|
||||
|
||||
打包完成,会在target目录下生成一个kafka-console-ui.zip的安装包
|
||||
|
Before Width: | Height: | Size: 492 KiB |
BIN
document/消息.png
|
Before Width: | Height: | Size: 400 KiB |
BIN
document/消费组.png
|
Before Width: | Height: | Size: 53 KiB |
BIN
document/运维.png
|
Before Width: | Height: | Size: 126 KiB |
BIN
document/集群.png
|
Before Width: | Height: | Size: 30 KiB |
16
pom.xml
@@ -10,7 +10,7 @@
|
||||
</parent>
|
||||
<groupId>com.xuxd</groupId>
|
||||
<artifactId>kafka-console-ui</artifactId>
|
||||
<version>1.0.2</version>
|
||||
<version>1.0.5</version>
|
||||
<name>kafka-console-ui</name>
|
||||
<description>Kafka console manage ui</description>
|
||||
<properties>
|
||||
@@ -21,7 +21,7 @@
|
||||
<ui.path>${project.basedir}/ui</ui.path>
|
||||
<frontend-maven-plugin.version>1.11.0</frontend-maven-plugin.version>
|
||||
<compiler.version>1.8</compiler.version>
|
||||
<kafka.version>2.8.0</kafka.version>
|
||||
<kafka.version>3.2.0</kafka.version>
|
||||
<maven.assembly.plugin.version>3.0.0</maven.assembly.plugin.version>
|
||||
<mybatis-plus-boot-starter.version>3.4.2</mybatis-plus-boot-starter.version>
|
||||
<scala.version>2.13.6</scala.version>
|
||||
@@ -76,6 +76,18 @@
|
||||
<groupId>org.apache.kafka</groupId>
|
||||
<artifactId>kafka_2.13</artifactId>
|
||||
<version>${kafka.version}</version>
|
||||
<exclusions>
|
||||
<exclusion>
|
||||
<groupId>com.typesafe.scala-logging</groupId>
|
||||
<artifactId>scala-logging_2.13</artifactId>
|
||||
</exclusion>
|
||||
</exclusions>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.typesafe.scala-logging</groupId>
|
||||
<artifactId>scala-logging_2.13</artifactId>
|
||||
<version>3.9.2</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
|
||||
@@ -3,11 +3,13 @@ package com.xuxd.kafka.console;
|
||||
import org.mybatis.spring.annotation.MapperScan;
|
||||
import org.springframework.boot.SpringApplication;
|
||||
import org.springframework.boot.autoconfigure.SpringBootApplication;
|
||||
import org.springframework.boot.web.servlet.ServletComponentScan;
|
||||
import org.springframework.scheduling.annotation.EnableScheduling;
|
||||
|
||||
@MapperScan("com.xuxd.kafka.console.dao")
|
||||
@SpringBootApplication
|
||||
@EnableScheduling
|
||||
@ServletComponentScan
|
||||
public class KafkaConsoleUiApplication {
|
||||
|
||||
public static void main(String[] args) {
|
||||
|
||||
@@ -1,18 +1,15 @@
|
||||
package com.xuxd.kafka.console.beans;
|
||||
|
||||
import java.util.Objects;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import org.apache.kafka.common.acl.AccessControlEntry;
|
||||
import org.apache.kafka.common.acl.AccessControlEntryFilter;
|
||||
import org.apache.kafka.common.acl.AclBinding;
|
||||
import org.apache.kafka.common.acl.AclBindingFilter;
|
||||
import org.apache.kafka.common.acl.AclOperation;
|
||||
import org.apache.kafka.common.acl.AclPermissionType;
|
||||
import org.apache.kafka.common.acl.*;
|
||||
import org.apache.kafka.common.resource.PatternType;
|
||||
import org.apache.kafka.common.resource.ResourcePattern;
|
||||
import org.apache.kafka.common.resource.ResourcePatternFilter;
|
||||
import org.apache.kafka.common.resource.ResourceType;
|
||||
import org.apache.kafka.common.security.auth.KafkaPrincipal;
|
||||
import org.apache.kafka.common.utils.SecurityUtils;
|
||||
|
||||
import java.util.Objects;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
@@ -41,7 +38,9 @@ public class AclEntry {
|
||||
entry.setResourceType(binding.pattern().resourceType().name());
|
||||
entry.setName(binding.pattern().name());
|
||||
entry.setPatternType(binding.pattern().patternType().name());
|
||||
entry.setPrincipal(KafkaPrincipal.fromString(binding.entry().principal()).getName());
|
||||
// entry.setPrincipal(KafkaPrincipal.fromString(binding.entry().principal()).getName());
|
||||
// 3.x版本使用该方法
|
||||
entry.setPrincipal(SecurityUtils.parseKafkaPrincipal(binding.entry().principal()).getName());
|
||||
entry.setHost(binding.entry().host());
|
||||
entry.setOperation(binding.entry().operation().name());
|
||||
entry.setPermissionType(binding.entry().permissionType().name());
|
||||
|
||||
@@ -8,7 +8,7 @@ import org.apache.kafka.common.Node;
|
||||
* @author xuxd
|
||||
* @date 2021-10-08 14:03:21
|
||||
**/
|
||||
public class BrokerNode {
|
||||
public class BrokerNode implements Comparable{
|
||||
|
||||
private int id;
|
||||
|
||||
@@ -80,4 +80,8 @@ public class BrokerNode {
|
||||
public void setController(boolean controller) {
|
||||
isController = controller;
|
||||
}
|
||||
|
||||
@Override public int compareTo(Object o) {
|
||||
return this.id - ((BrokerNode)o).id;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
package com.xuxd.kafka.console.beans;
|
||||
|
||||
import com.xuxd.kafka.console.beans.enums.FilterType;
|
||||
import org.apache.kafka.common.serialization.Deserializer;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
* @author xuxd
|
||||
* @date 2021-12-29 15:30:08
|
||||
**/
|
||||
public class MessageFilter {
|
||||
|
||||
private FilterType filterType = FilterType.NONE;
|
||||
|
||||
private Object searchContent = null;
|
||||
|
||||
private String headerKey = null;
|
||||
|
||||
private String headerValue = null;
|
||||
|
||||
private Deserializer deserializer = null;
|
||||
|
||||
private boolean isContainsValue = false;
|
||||
|
||||
public FilterType getFilterType() {
|
||||
return filterType;
|
||||
}
|
||||
|
||||
public void setFilterType(FilterType filterType) {
|
||||
this.filterType = filterType;
|
||||
}
|
||||
|
||||
public Object getSearchContent() {
|
||||
return searchContent;
|
||||
}
|
||||
|
||||
public void setSearchContent(Object searchContent) {
|
||||
this.searchContent = searchContent;
|
||||
}
|
||||
|
||||
public String getHeaderKey() {
|
||||
return headerKey;
|
||||
}
|
||||
|
||||
public void setHeaderKey(String headerKey) {
|
||||
this.headerKey = headerKey;
|
||||
}
|
||||
|
||||
public String getHeaderValue() {
|
||||
return headerValue;
|
||||
}
|
||||
|
||||
public void setHeaderValue(String headerValue) {
|
||||
this.headerValue = headerValue;
|
||||
}
|
||||
|
||||
public Deserializer getDeserializer() {
|
||||
return deserializer;
|
||||
}
|
||||
|
||||
public void setDeserializer(Deserializer deserializer) {
|
||||
this.deserializer = deserializer;
|
||||
}
|
||||
|
||||
public boolean isContainsValue() {
|
||||
return isContainsValue;
|
||||
}
|
||||
|
||||
public void setContainsValue(boolean containsValue) {
|
||||
isContainsValue = containsValue;
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
package com.xuxd.kafka.console.beans;
|
||||
|
||||
import com.xuxd.kafka.console.beans.enums.FilterType;
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
@@ -24,4 +25,12 @@ public class QueryMessage {
|
||||
private String keyDeserializer;
|
||||
|
||||
private String valueDeserializer;
|
||||
|
||||
private FilterType filter;
|
||||
|
||||
private String value;
|
||||
|
||||
private String headerKey;
|
||||
|
||||
private String headerValue;
|
||||
}
|
||||
|
||||
@@ -21,7 +21,7 @@ public class TopicPartition implements Comparable {
|
||||
}
|
||||
TopicPartition other = (TopicPartition) o;
|
||||
if (!this.topic.equals(other.getTopic())) {
|
||||
return this.compareTo(other);
|
||||
return this.topic.compareTo(other.topic);
|
||||
}
|
||||
|
||||
return this.partition - other.partition;
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
package com.xuxd.kafka.console.beans.dos;
|
||||
|
||||
import com.baomidou.mybatisplus.annotation.IdType;
|
||||
import com.baomidou.mybatisplus.annotation.TableId;
|
||||
import com.baomidou.mybatisplus.annotation.TableName;
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
* @author xuxd
|
||||
* @date 2021-12-31 09:54:24
|
||||
**/
|
||||
@Data
|
||||
@TableName("t_cluster_info")
|
||||
public class ClusterInfoDO {
|
||||
|
||||
@TableId(type = IdType.AUTO)
|
||||
private Long id;
|
||||
|
||||
private String clusterName;
|
||||
|
||||
private String address;
|
||||
|
||||
private String properties;
|
||||
|
||||
private String updateTime;
|
||||
}
|
||||
@@ -23,4 +23,6 @@ public class KafkaUserDO {
|
||||
private String password;
|
||||
|
||||
private String updateTime;
|
||||
|
||||
private Long clusterInfoId;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
package com.xuxd.kafka.console.beans.dto;
|
||||
|
||||
import com.xuxd.kafka.console.beans.dos.ClusterInfoDO;
|
||||
import com.xuxd.kafka.console.utils.ConvertUtil;
|
||||
import lombok.Data;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
* @author xuxd
|
||||
* @date 2022-01-04 20:19:03
|
||||
**/
|
||||
@Data
|
||||
public class ClusterInfoDTO {
|
||||
private Long id;
|
||||
|
||||
private String clusterName;
|
||||
|
||||
private String address;
|
||||
|
||||
private String properties;
|
||||
|
||||
private String updateTime;
|
||||
|
||||
public ClusterInfoDO to() {
|
||||
ClusterInfoDO infoDO = new ClusterInfoDO();
|
||||
infoDO.setId(id);
|
||||
infoDO.setClusterName(clusterName);
|
||||
infoDO.setAddress(address);
|
||||
|
||||
if (StringUtils.isNotBlank(properties)) {
|
||||
infoDO.setProperties(ConvertUtil.propertiesStr2JsonStr(properties));
|
||||
}
|
||||
|
||||
return infoDO;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
package com.xuxd.kafka.console.beans.dto;
|
||||
|
||||
import java.util.List;
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
* @author xuxd
|
||||
* @date 2022-02-15 19:08:13
|
||||
**/
|
||||
@Data
|
||||
public class ProposedAssignmentDTO {
|
||||
|
||||
private String topic;
|
||||
|
||||
private List<Integer> brokers;
|
||||
}
|
||||
@@ -1,8 +1,10 @@
|
||||
package com.xuxd.kafka.console.beans.dto;
|
||||
|
||||
import com.xuxd.kafka.console.beans.QueryMessage;
|
||||
import com.xuxd.kafka.console.beans.enums.FilterType;
|
||||
import java.util.Date;
|
||||
import lombok.Data;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
@@ -27,6 +29,14 @@ public class QueryMessageDTO {
|
||||
|
||||
private String valueDeserializer;
|
||||
|
||||
private String filter;
|
||||
|
||||
private String value;
|
||||
|
||||
private String headerKey;
|
||||
|
||||
private String headerValue;
|
||||
|
||||
public QueryMessage toQueryMessage() {
|
||||
QueryMessage queryMessage = new QueryMessage();
|
||||
queryMessage.setTopic(topic);
|
||||
@@ -45,6 +55,21 @@ public class QueryMessageDTO {
|
||||
queryMessage.setKeyDeserializer(keyDeserializer);
|
||||
queryMessage.setValueDeserializer(valueDeserializer);
|
||||
|
||||
if (StringUtils.isNotBlank(filter)) {
|
||||
queryMessage.setFilter(FilterType.valueOf(filter.toUpperCase()));
|
||||
} else {
|
||||
queryMessage.setFilter(FilterType.NONE);
|
||||
}
|
||||
if (StringUtils.isNotBlank(value)) {
|
||||
queryMessage.setValue(value.trim());
|
||||
}
|
||||
if (StringUtils.isNotBlank(headerKey)) {
|
||||
queryMessage.setHeaderKey(headerKey.trim());
|
||||
}
|
||||
if (StringUtils.isNotBlank(headerValue)) {
|
||||
queryMessage.setHeaderValue(headerValue.trim());
|
||||
}
|
||||
|
||||
return queryMessage;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
package com.xuxd.kafka.console.beans.enums;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
* @author xuxd
|
||||
* @date 2021-12-29 14:36:01
|
||||
**/
|
||||
public enum FilterType {
|
||||
NONE, BODY, HEADER
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
package com.xuxd.kafka.console.beans.vo;
|
||||
|
||||
import java.util.List;
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
* @author xuxd
|
||||
* @date 2022-01-22 16:24:58
|
||||
**/
|
||||
@Data
|
||||
public class BrokerApiVersionVO {
|
||||
|
||||
private int brokerId;
|
||||
|
||||
private String host;
|
||||
|
||||
private int supportNums;
|
||||
|
||||
private int unSupportNums;
|
||||
|
||||
private List<String> versionInfo;
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
package com.xuxd.kafka.console.beans.vo;
|
||||
|
||||
import com.xuxd.kafka.console.beans.dos.ClusterInfoDO;
|
||||
import com.xuxd.kafka.console.utils.ConvertUtil;
|
||||
import java.util.List;
|
||||
import lombok.Data;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
* @author xuxd
|
||||
* @date 2022-01-04 19:16:11
|
||||
**/
|
||||
@Data
|
||||
public class ClusterInfoVO {
|
||||
|
||||
private Long id;
|
||||
|
||||
private String clusterName;
|
||||
|
||||
private String address;
|
||||
|
||||
private List<String> properties;
|
||||
|
||||
private String updateTime;
|
||||
|
||||
public static ClusterInfoVO from(ClusterInfoDO infoDO) {
|
||||
ClusterInfoVO vo = new ClusterInfoVO();
|
||||
vo.setId(infoDO.getId());
|
||||
vo.setClusterName(infoDO.getClusterName());
|
||||
vo.setAddress(infoDO.getAddress());
|
||||
vo.setUpdateTime(infoDO.getUpdateTime());
|
||||
if (StringUtils.isNotBlank(infoDO.getProperties())) {
|
||||
vo.setProperties(ConvertUtil.jsonStr2List(infoDO.getProperties()));
|
||||
}
|
||||
return vo;
|
||||
|
||||
}
|
||||
}
|
||||
66
src/main/java/com/xuxd/kafka/console/boot/Bootstrap.java
Normal file
@@ -0,0 +1,66 @@
|
||||
package com.xuxd.kafka.console.boot;
|
||||
|
||||
import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
|
||||
import com.xuxd.kafka.console.beans.dos.ClusterInfoDO;
|
||||
import com.xuxd.kafka.console.config.KafkaConfig;
|
||||
import com.xuxd.kafka.console.dao.ClusterInfoMapper;
|
||||
import com.xuxd.kafka.console.utils.ConvertUtil;
|
||||
import java.util.List;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
import org.apache.commons.collections.CollectionUtils;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import org.springframework.beans.factory.ObjectProvider;
|
||||
import org.springframework.beans.factory.SmartInitializingSingleton;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
* @author xuxd
|
||||
* @date 2021-12-30 19:16:50
|
||||
**/
|
||||
@Slf4j
|
||||
@Component
|
||||
public class Bootstrap implements SmartInitializingSingleton {
|
||||
|
||||
public static final String DEFAULT_CLUSTER_NAME = "default";
|
||||
|
||||
private final KafkaConfig config;
|
||||
|
||||
private final ClusterInfoMapper clusterInfoMapper;
|
||||
|
||||
public Bootstrap(KafkaConfig config, ObjectProvider<ClusterInfoMapper> clusterInfoMapper) {
|
||||
this.config = config;
|
||||
this.clusterInfoMapper = clusterInfoMapper.getIfAvailable();
|
||||
}
|
||||
|
||||
private void initialize() {
|
||||
loadDefaultClusterConfig();
|
||||
}
|
||||
|
||||
private void loadDefaultClusterConfig() {
|
||||
log.info("load default kafka config.");
|
||||
if (StringUtils.isBlank(config.getBootstrapServer())) {
|
||||
return;
|
||||
}
|
||||
|
||||
QueryWrapper<ClusterInfoDO> clusterInfoDOQueryWrapper = new QueryWrapper<>();
|
||||
clusterInfoDOQueryWrapper.eq("cluster_name", DEFAULT_CLUSTER_NAME);
|
||||
List<Object> objects = clusterInfoMapper.selectObjs(clusterInfoDOQueryWrapper);
|
||||
if (CollectionUtils.isNotEmpty(objects)) {
|
||||
log.warn("default kafka cluster config has existed[any of cluster name or address].");
|
||||
return;
|
||||
}
|
||||
|
||||
ClusterInfoDO infoDO = new ClusterInfoDO();
|
||||
infoDO.setClusterName(DEFAULT_CLUSTER_NAME);
|
||||
infoDO.setAddress(config.getBootstrapServer().trim());
|
||||
infoDO.setProperties(ConvertUtil.toJsonString(config.getProperties()));
|
||||
clusterInfoMapper.insert(infoDO);
|
||||
log.info("Insert default config: {}", infoDO);
|
||||
}
|
||||
|
||||
@Override public void afterSingletonsInstantiated() {
|
||||
initialize();
|
||||
}
|
||||
}
|
||||
33
src/main/java/com/xuxd/kafka/console/cache/TimeBasedCache.java
vendored
Normal file
@@ -0,0 +1,33 @@
|
||||
package com.xuxd.kafka.console.cache;
|
||||
|
||||
import com.google.common.cache.CacheBuilder;
|
||||
import com.google.common.cache.CacheLoader;
|
||||
import com.google.common.cache.LoadingCache;
|
||||
import com.google.common.cache.RemovalListener;
|
||||
import kafka.console.KafkaConsole;
|
||||
|
||||
import java.util.concurrent.ExecutionException;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
public class TimeBasedCache<K, V> {
|
||||
private LoadingCache<K, V> cache;
|
||||
|
||||
private KafkaConsole console;
|
||||
|
||||
public TimeBasedCache(CacheLoader<K, V> loader, RemovalListener<K, V> listener) {
|
||||
cache = CacheBuilder.newBuilder()
|
||||
.maximumSize(50) // maximum 100 records can be cached
|
||||
.expireAfterAccess(30, TimeUnit.MINUTES) // cache will expire after 30 minutes of access
|
||||
.removalListener(listener)
|
||||
.build(loader);
|
||||
|
||||
}
|
||||
|
||||
public V get(K k) {
|
||||
try {
|
||||
return cache.get(k);
|
||||
} catch (ExecutionException e) {
|
||||
throw new RuntimeException("Get connection from cache error.", e);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
package com.xuxd.kafka.console.config;
|
||||
|
||||
import java.util.Properties;
|
||||
import org.apache.kafka.clients.CommonClientConfigs;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
* @author xuxd
|
||||
* @date 2021-12-30 15:46:55
|
||||
**/
|
||||
public class ContextConfig {
|
||||
|
||||
public static final int DEFAULT_REQUEST_TIMEOUT_MS = 5000;
|
||||
|
||||
private Long clusterInfoId;
|
||||
|
||||
private String clusterName;
|
||||
|
||||
private String bootstrapServer;
|
||||
|
||||
private int requestTimeoutMs = DEFAULT_REQUEST_TIMEOUT_MS;
|
||||
|
||||
private Properties properties = new Properties();
|
||||
|
||||
public String getBootstrapServer() {
|
||||
return bootstrapServer;
|
||||
}
|
||||
|
||||
public void setBootstrapServer(String bootstrapServer) {
|
||||
this.bootstrapServer = bootstrapServer;
|
||||
}
|
||||
|
||||
public int getRequestTimeoutMs() {
|
||||
return properties.containsKey(CommonClientConfigs.REQUEST_TIMEOUT_MS_CONFIG) ?
|
||||
Integer.parseInt(properties.getProperty(CommonClientConfigs.REQUEST_TIMEOUT_MS_CONFIG)) : requestTimeoutMs;
|
||||
}
|
||||
|
||||
public void setRequestTimeoutMs(int requestTimeoutMs) {
|
||||
this.requestTimeoutMs = requestTimeoutMs;
|
||||
}
|
||||
|
||||
public Properties getProperties() {
|
||||
return properties;
|
||||
}
|
||||
|
||||
public Long getClusterInfoId() {
|
||||
return clusterInfoId;
|
||||
}
|
||||
|
||||
public void setClusterInfoId(Long clusterInfoId) {
|
||||
this.clusterInfoId = clusterInfoId;
|
||||
}
|
||||
|
||||
public String getClusterName() {
|
||||
return clusterName;
|
||||
}
|
||||
|
||||
public void setClusterName(String clusterName) {
|
||||
this.clusterName = clusterName;
|
||||
}
|
||||
|
||||
public void setProperties(Properties properties) {
|
||||
this.properties = properties;
|
||||
}
|
||||
|
||||
@Override public String toString() {
|
||||
return "KafkaContextConfig{" +
|
||||
"bootstrapServer='" + bootstrapServer + '\'' +
|
||||
", requestTimeoutMs=" + requestTimeoutMs +
|
||||
", properties=" + properties +
|
||||
'}';
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
package com.xuxd.kafka.console.config;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
* @author xuxd
|
||||
* @date 2021-12-30 18:55:28
|
||||
**/
|
||||
public class ContextConfigHolder {
|
||||
|
||||
public static final ThreadLocal<ContextConfig> CONTEXT_CONFIG = new ThreadLocal<>();
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
package com.xuxd.kafka.console.config;
|
||||
|
||||
import java.util.Properties;
|
||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
|
||||
@@ -15,23 +16,15 @@ public class KafkaConfig {
|
||||
|
||||
private String bootstrapServer;
|
||||
|
||||
private int requestTimeoutMs;
|
||||
|
||||
private String securityProtocol;
|
||||
|
||||
private String saslMechanism;
|
||||
|
||||
private String saslJaasConfig;
|
||||
|
||||
private String adminUsername;
|
||||
|
||||
private String adminPassword;
|
||||
|
||||
private boolean adminCreate;
|
||||
|
||||
private String zookeeperAddr;
|
||||
|
||||
private boolean enableAcl;
|
||||
private Properties properties;
|
||||
|
||||
private boolean cacheAdminConnection;
|
||||
|
||||
private boolean cacheProducerConnection;
|
||||
|
||||
private boolean cacheConsumerConnection;
|
||||
|
||||
public String getBootstrapServer() {
|
||||
return bootstrapServer;
|
||||
@@ -41,62 +34,6 @@ public class KafkaConfig {
|
||||
this.bootstrapServer = bootstrapServer;
|
||||
}
|
||||
|
||||
public int getRequestTimeoutMs() {
|
||||
return requestTimeoutMs;
|
||||
}
|
||||
|
||||
public void setRequestTimeoutMs(int requestTimeoutMs) {
|
||||
this.requestTimeoutMs = requestTimeoutMs;
|
||||
}
|
||||
|
||||
public String getSecurityProtocol() {
|
||||
return securityProtocol;
|
||||
}
|
||||
|
||||
public void setSecurityProtocol(String securityProtocol) {
|
||||
this.securityProtocol = securityProtocol;
|
||||
}
|
||||
|
||||
public String getSaslMechanism() {
|
||||
return saslMechanism;
|
||||
}
|
||||
|
||||
public void setSaslMechanism(String saslMechanism) {
|
||||
this.saslMechanism = saslMechanism;
|
||||
}
|
||||
|
||||
public String getSaslJaasConfig() {
|
||||
return saslJaasConfig;
|
||||
}
|
||||
|
||||
public void setSaslJaasConfig(String saslJaasConfig) {
|
||||
this.saslJaasConfig = saslJaasConfig;
|
||||
}
|
||||
|
||||
public String getAdminUsername() {
|
||||
return adminUsername;
|
||||
}
|
||||
|
||||
public void setAdminUsername(String adminUsername) {
|
||||
this.adminUsername = adminUsername;
|
||||
}
|
||||
|
||||
public String getAdminPassword() {
|
||||
return adminPassword;
|
||||
}
|
||||
|
||||
public void setAdminPassword(String adminPassword) {
|
||||
this.adminPassword = adminPassword;
|
||||
}
|
||||
|
||||
public boolean isAdminCreate() {
|
||||
return adminCreate;
|
||||
}
|
||||
|
||||
public void setAdminCreate(boolean adminCreate) {
|
||||
this.adminCreate = adminCreate;
|
||||
}
|
||||
|
||||
public String getZookeeperAddr() {
|
||||
return zookeeperAddr;
|
||||
}
|
||||
@@ -105,11 +42,35 @@ public class KafkaConfig {
|
||||
this.zookeeperAddr = zookeeperAddr;
|
||||
}
|
||||
|
||||
public boolean isEnableAcl() {
|
||||
return enableAcl;
|
||||
public Properties getProperties() {
|
||||
return properties;
|
||||
}
|
||||
|
||||
public void setEnableAcl(boolean enableAcl) {
|
||||
this.enableAcl = enableAcl;
|
||||
public void setProperties(Properties properties) {
|
||||
this.properties = properties;
|
||||
}
|
||||
|
||||
public boolean isCacheAdminConnection() {
|
||||
return cacheAdminConnection;
|
||||
}
|
||||
|
||||
public void setCacheAdminConnection(boolean cacheAdminConnection) {
|
||||
this.cacheAdminConnection = cacheAdminConnection;
|
||||
}
|
||||
|
||||
public boolean isCacheProducerConnection() {
|
||||
return cacheProducerConnection;
|
||||
}
|
||||
|
||||
public void setCacheProducerConnection(boolean cacheProducerConnection) {
|
||||
this.cacheProducerConnection = cacheProducerConnection;
|
||||
}
|
||||
|
||||
public boolean isCacheConsumerConnection() {
|
||||
return cacheConsumerConnection;
|
||||
}
|
||||
|
||||
public void setCacheConsumerConnection(boolean cacheConsumerConnection) {
|
||||
this.cacheConsumerConnection = cacheConsumerConnection;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,8 +1,13 @@
|
||||
package com.xuxd.kafka.console.controller;
|
||||
|
||||
import com.xuxd.kafka.console.beans.dto.ClusterInfoDTO;
|
||||
import com.xuxd.kafka.console.service.ClusterService;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.web.bind.annotation.DeleteMapping;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.PostMapping;
|
||||
import org.springframework.web.bind.annotation.PutMapping;
|
||||
import org.springframework.web.bind.annotation.RequestBody;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
|
||||
@@ -23,4 +28,34 @@ public class ClusterController {
|
||||
public Object getClusterInfo() {
|
||||
return clusterService.getClusterInfo();
|
||||
}
|
||||
|
||||
@GetMapping("/info")
|
||||
public Object getClusterInfoList() {
|
||||
return clusterService.getClusterInfoList();
|
||||
}
|
||||
|
||||
@PostMapping("/info")
|
||||
public Object addClusterInfo(@RequestBody ClusterInfoDTO dto) {
|
||||
return clusterService.addClusterInfo(dto.to());
|
||||
}
|
||||
|
||||
@DeleteMapping("/info")
|
||||
public Object deleteClusterInfo(@RequestBody ClusterInfoDTO dto) {
|
||||
return clusterService.deleteClusterInfo(dto.getId());
|
||||
}
|
||||
|
||||
@PutMapping("/info")
|
||||
public Object updateClusterInfo(@RequestBody ClusterInfoDTO dto) {
|
||||
return clusterService.updateClusterInfo(dto.to());
|
||||
}
|
||||
|
||||
@GetMapping("/info/peek")
|
||||
public Object peekClusterInfo() {
|
||||
return clusterService.peekClusterInfo();
|
||||
}
|
||||
|
||||
@GetMapping("/info/api/version")
|
||||
public Object getBrokerApiVersionInfo() {
|
||||
return clusterService.getBrokerApiVersionInfo();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -36,7 +36,7 @@ public class ConfigController {
|
||||
this.configService = configService;
|
||||
}
|
||||
|
||||
@GetMapping
|
||||
@GetMapping("/console")
|
||||
public Object getConfig() {
|
||||
return ResponseData.create().data(configMap).success();
|
||||
}
|
||||
|
||||
@@ -1,14 +1,15 @@
|
||||
package com.xuxd.kafka.console.controller;
|
||||
|
||||
import com.xuxd.kafka.console.beans.QueryMessage;
|
||||
import com.xuxd.kafka.console.beans.ResponseData;
|
||||
import com.xuxd.kafka.console.beans.SendMessage;
|
||||
import com.xuxd.kafka.console.beans.dto.QueryMessageDTO;
|
||||
import com.xuxd.kafka.console.service.MessageService;
|
||||
import org.apache.commons.collections.CollectionUtils;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.PostMapping;
|
||||
import org.springframework.web.bind.annotation.RequestBody;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
import org.springframework.web.bind.annotation.*;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
@@ -52,4 +53,12 @@ public class MessageController {
|
||||
public Object resend(@RequestBody SendMessage message) {
|
||||
return messageService.resend(message);
|
||||
}
|
||||
|
||||
@DeleteMapping
|
||||
public Object delete(@RequestBody List<QueryMessage> messages) {
|
||||
if (CollectionUtils.isEmpty(messages)) {
|
||||
return ResponseData.create().failed("params is null");
|
||||
}
|
||||
return messageService.delete(messages);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,6 +2,7 @@ package com.xuxd.kafka.console.controller;
|
||||
|
||||
import com.xuxd.kafka.console.beans.TopicPartition;
|
||||
import com.xuxd.kafka.console.beans.dto.BrokerThrottleDTO;
|
||||
import com.xuxd.kafka.console.beans.dto.ProposedAssignmentDTO;
|
||||
import com.xuxd.kafka.console.beans.dto.ReplicationDTO;
|
||||
import com.xuxd.kafka.console.beans.dto.SyncDataDTO;
|
||||
import com.xuxd.kafka.console.service.OperationService;
|
||||
@@ -74,4 +75,9 @@ public class OperationController {
|
||||
public Object cancelReassignment(@RequestBody TopicPartition partition) {
|
||||
return operationService.cancelReassignment(new org.apache.kafka.common.TopicPartition(partition.getTopic(), partition.getPartition()));
|
||||
}
|
||||
|
||||
@PostMapping("/replication/reassignments/proposed")
|
||||
public Object proposedAssignments(@RequestBody ProposedAssignmentDTO dto) {
|
||||
return operationService.proposedAssignments(dto.getTopic(), dto.getBrokers());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -43,8 +43,8 @@ public class TopicController {
|
||||
}
|
||||
|
||||
@DeleteMapping
|
||||
public Object deleteTopic(@RequestParam String topic) {
|
||||
return topicService.deleteTopic(topic);
|
||||
public Object deleteTopic(@RequestBody List<String> topics) {
|
||||
return topicService.deleteTopics(topics);
|
||||
}
|
||||
|
||||
@GetMapping("/partition")
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
package com.xuxd.kafka.console.dao;
|
||||
|
||||
import com.baomidou.mybatisplus.core.mapper.BaseMapper;
|
||||
import com.xuxd.kafka.console.beans.dos.ClusterInfoDO;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
* @author xuxd
|
||||
* @date 2021-12-31 09:58:52
|
||||
**/
|
||||
public interface ClusterInfoMapper extends BaseMapper<ClusterInfoDO> {
|
||||
}
|
||||
@@ -1,9 +1,22 @@
|
||||
package com.xuxd.kafka.console.schedule;
|
||||
|
||||
import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
|
||||
import com.xuxd.kafka.console.beans.dos.ClusterInfoDO;
|
||||
import com.xuxd.kafka.console.beans.dos.KafkaUserDO;
|
||||
import com.xuxd.kafka.console.config.ContextConfig;
|
||||
import com.xuxd.kafka.console.config.ContextConfigHolder;
|
||||
import com.xuxd.kafka.console.dao.ClusterInfoMapper;
|
||||
import com.xuxd.kafka.console.dao.KafkaUserMapper;
|
||||
import com.xuxd.kafka.console.utils.ConvertUtil;
|
||||
import com.xuxd.kafka.console.utils.SaslUtil;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
import kafka.console.KafkaConfigConsole;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
import org.apache.commons.collections.CollectionUtils;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import org.springframework.beans.factory.ObjectProvider;
|
||||
import org.springframework.scheduling.annotation.Scheduled;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
@@ -21,25 +34,58 @@ public class KafkaAclSchedule {
|
||||
|
||||
private final KafkaConfigConsole configConsole;
|
||||
|
||||
public KafkaAclSchedule(KafkaUserMapper userMapper, KafkaConfigConsole configConsole) {
|
||||
this.userMapper = userMapper;
|
||||
this.configConsole = configConsole;
|
||||
private final ClusterInfoMapper clusterInfoMapper;
|
||||
|
||||
public KafkaAclSchedule(ObjectProvider<KafkaUserMapper> userMapper,
|
||||
ObjectProvider<KafkaConfigConsole> configConsole, ObjectProvider<ClusterInfoMapper> clusterInfoMapper) {
|
||||
this.userMapper = userMapper.getIfAvailable();
|
||||
this.configConsole = configConsole.getIfAvailable();
|
||||
this.clusterInfoMapper = clusterInfoMapper.getIfAvailable();
|
||||
}
|
||||
|
||||
@Scheduled(cron = "${cron.clear-dirty-user}")
|
||||
public void clearDirtyKafkaUser() {
|
||||
log.info("Start clear dirty data for kafka user from database.");
|
||||
Set<String> userSet = configConsole.getUserList(null);
|
||||
userMapper.selectList(null).forEach(u -> {
|
||||
if (!userSet.contains(u.getUsername())) {
|
||||
log.info("clear user: {} from database.", u.getUsername());
|
||||
try {
|
||||
userMapper.deleteById(u.getId());
|
||||
} catch (Exception e) {
|
||||
log.error("userMapper.deleteById error, user: " + u, e);
|
||||
try {
|
||||
log.info("Start clear dirty data for kafka user from database.");
|
||||
List<ClusterInfoDO> clusterInfoDOS = clusterInfoMapper.selectList(null);
|
||||
List<Long> clusterInfoIds = new ArrayList<>();
|
||||
for (ClusterInfoDO infoDO : clusterInfoDOS) {
|
||||
ContextConfig config = new ContextConfig();
|
||||
config.setClusterInfoId(infoDO.getId());
|
||||
config.setClusterName(infoDO.getClusterName());
|
||||
|
||||
config.setBootstrapServer(infoDO.getAddress());
|
||||
if (StringUtils.isNotBlank(infoDO.getProperties())) {
|
||||
config.setProperties(ConvertUtil.toProperties(infoDO.getProperties()));
|
||||
}
|
||||
ContextConfigHolder.CONTEXT_CONFIG.set(config);
|
||||
if (SaslUtil.isEnableSasl() && SaslUtil.isEnableScram()) {
|
||||
log.info("Start clear cluster: {}", infoDO.getClusterName());
|
||||
Set<String> userSet = configConsole.getUserList(null);
|
||||
QueryWrapper<KafkaUserDO> queryWrapper = new QueryWrapper<>();
|
||||
queryWrapper.eq("cluster_info_id", infoDO.getId());
|
||||
userMapper.selectList(queryWrapper).forEach(u -> {
|
||||
if (!userSet.contains(u.getUsername())) {
|
||||
log.info("clear user: {} from database.", u.getUsername());
|
||||
try {
|
||||
userMapper.deleteById(u.getId());
|
||||
} catch (Exception e) {
|
||||
log.error("userMapper.deleteById error, user: " + u, e);
|
||||
}
|
||||
}
|
||||
});
|
||||
clusterInfoIds.add(infoDO.getId());
|
||||
}
|
||||
}
|
||||
});
|
||||
log.info("Clear end.");
|
||||
if (CollectionUtils.isNotEmpty(clusterInfoIds)) {
|
||||
log.info("Clear the cluster id {}, which not found.", clusterInfoIds);
|
||||
QueryWrapper<KafkaUserDO> wrapper = new QueryWrapper<>();
|
||||
wrapper.notIn("cluster_info_id", clusterInfoIds);
|
||||
userMapper.delete(wrapper);
|
||||
}
|
||||
log.info("Clear end.");
|
||||
} finally {
|
||||
ContextConfigHolder.CONTEXT_CONFIG.remove();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
package com.xuxd.kafka.console.service;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
* @author xuxd
|
||||
* @date 2021-12-31 11:42:43
|
||||
**/
|
||||
public interface ClusterInfoService {
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
package com.xuxd.kafka.console.service;
|
||||
|
||||
import com.xuxd.kafka.console.beans.ResponseData;
|
||||
import com.xuxd.kafka.console.beans.dos.ClusterInfoDO;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
@@ -10,4 +11,16 @@ import com.xuxd.kafka.console.beans.ResponseData;
|
||||
**/
|
||||
public interface ClusterService {
|
||||
ResponseData getClusterInfo();
|
||||
|
||||
ResponseData getClusterInfoList();
|
||||
|
||||
ResponseData addClusterInfo(ClusterInfoDO infoDO);
|
||||
|
||||
ResponseData deleteClusterInfo(Long id);
|
||||
|
||||
ResponseData updateClusterInfo(ClusterInfoDO infoDO);
|
||||
|
||||
ResponseData peekClusterInfo();
|
||||
|
||||
ResponseData getBrokerApiVersionInfo();
|
||||
}
|
||||
|
||||
@@ -4,6 +4,8 @@ import com.xuxd.kafka.console.beans.QueryMessage;
|
||||
import com.xuxd.kafka.console.beans.ResponseData;
|
||||
import com.xuxd.kafka.console.beans.SendMessage;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
@@ -23,4 +25,6 @@ public interface MessageService {
|
||||
ResponseData send(SendMessage message);
|
||||
|
||||
ResponseData resend(SendMessage message);
|
||||
|
||||
ResponseData delete(List<QueryMessage> messages);
|
||||
}
|
||||
|
||||
@@ -30,4 +30,6 @@ public interface OperationService {
|
||||
ResponseData currentReassignments();
|
||||
|
||||
ResponseData cancelReassignment(TopicPartition partition);
|
||||
|
||||
ResponseData proposedAssignments(String topic, List<Integer> brokerList);
|
||||
}
|
||||
|
||||
@@ -4,6 +4,8 @@ import com.xuxd.kafka.console.beans.ReplicaAssignment;
|
||||
import com.xuxd.kafka.console.beans.ResponseData;
|
||||
import com.xuxd.kafka.console.beans.enums.TopicThrottleSwitch;
|
||||
import com.xuxd.kafka.console.beans.enums.TopicType;
|
||||
|
||||
import java.util.Collection;
|
||||
import java.util.List;
|
||||
import org.apache.kafka.clients.admin.NewTopic;
|
||||
|
||||
@@ -19,7 +21,7 @@ public interface TopicService {
|
||||
|
||||
ResponseData getTopicList(String topic, TopicType type);
|
||||
|
||||
ResponseData deleteTopic(String topic);
|
||||
ResponseData deleteTopics(Collection<String> topics);
|
||||
|
||||
ResponseData getTopicPartitionInfo(String topic);
|
||||
|
||||
|
||||
@@ -6,29 +6,37 @@ import com.xuxd.kafka.console.beans.CounterMap;
|
||||
import com.xuxd.kafka.console.beans.ResponseData;
|
||||
import com.xuxd.kafka.console.beans.dos.KafkaUserDO;
|
||||
import com.xuxd.kafka.console.beans.vo.KafkaUserDetailVO;
|
||||
import com.xuxd.kafka.console.config.KafkaConfig;
|
||||
import com.xuxd.kafka.console.config.ContextConfigHolder;
|
||||
import com.xuxd.kafka.console.dao.KafkaUserMapper;
|
||||
import com.xuxd.kafka.console.service.AclService;
|
||||
import com.xuxd.kafka.console.utils.SaslUtil;
|
||||
import java.util.Arrays;
|
||||
import java.util.Collections;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Properties;
|
||||
import java.util.Set;
|
||||
import java.util.stream.Collectors;
|
||||
import kafka.console.KafkaAclConsole;
|
||||
import kafka.console.KafkaConfigConsole;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import org.apache.kafka.clients.CommonClientConfigs;
|
||||
import org.apache.kafka.clients.admin.ScramMechanism;
|
||||
import org.apache.kafka.clients.admin.UserScramCredentialsDescription;
|
||||
import org.apache.kafka.common.acl.AclBinding;
|
||||
import org.apache.kafka.common.acl.AclOperation;
|
||||
import org.apache.kafka.common.config.SaslConfigs;
|
||||
import org.apache.kafka.common.security.auth.SecurityProtocol;
|
||||
import org.springframework.beans.factory.ObjectProvider;
|
||||
import org.springframework.beans.factory.SmartInitializingSingleton;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Service;
|
||||
import scala.Tuple2;
|
||||
|
||||
import static com.xuxd.kafka.console.utils.SaslUtil.isEnableSasl;
|
||||
import static com.xuxd.kafka.console.utils.SaslUtil.isEnableScram;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
@@ -37,7 +45,7 @@ import scala.Tuple2;
|
||||
**/
|
||||
@Slf4j
|
||||
@Service
|
||||
public class AclServiceImpl implements AclService, SmartInitializingSingleton {
|
||||
public class AclServiceImpl implements AclService {
|
||||
|
||||
@Autowired
|
||||
private KafkaConfigConsole configConsole;
|
||||
@@ -45,9 +53,6 @@ public class AclServiceImpl implements AclService, SmartInitializingSingleton {
|
||||
@Autowired
|
||||
private KafkaAclConsole aclConsole;
|
||||
|
||||
@Autowired
|
||||
private KafkaConfig kafkaConfig;
|
||||
|
||||
private final KafkaUserMapper kafkaUserMapper;
|
||||
|
||||
public AclServiceImpl(ObjectProvider<KafkaUserMapper> kafkaUserMapper) {
|
||||
@@ -64,15 +69,23 @@ public class AclServiceImpl implements AclService, SmartInitializingSingleton {
|
||||
}
|
||||
|
||||
@Override public ResponseData addOrUpdateUser(String name, String pass) {
|
||||
if (!isEnableSasl()) {
|
||||
return ResponseData.create().failed("Only support SASL protocol.");
|
||||
}
|
||||
if (!isEnableScram()) {
|
||||
return ResponseData.create().failed("Only support SASL_SCRAM.");
|
||||
}
|
||||
log.info("add or update user, username: {}, password: {}", name, pass);
|
||||
if (!configConsole.addOrUpdateUser(name, pass)) {
|
||||
Tuple2<Object, String> tuple2 = configConsole.addOrUpdateUser(name, pass);
|
||||
if (!(boolean) tuple2._1()) {
|
||||
log.error("add user to kafka failed.");
|
||||
return ResponseData.create().failed("add user to kafka failed");
|
||||
return ResponseData.create().failed("add user to kafka failed: " + tuple2._2());
|
||||
}
|
||||
// save user info to database.
|
||||
KafkaUserDO userDO = new KafkaUserDO();
|
||||
userDO.setUsername(name);
|
||||
userDO.setPassword(pass);
|
||||
userDO.setClusterInfoId(ContextConfigHolder.CONTEXT_CONFIG.get().getClusterInfoId());
|
||||
try {
|
||||
Map<String, Object> map = new HashMap<>();
|
||||
map.put("username", name);
|
||||
@@ -86,12 +99,24 @@ public class AclServiceImpl implements AclService, SmartInitializingSingleton {
|
||||
}
|
||||
|
||||
@Override public ResponseData deleteUser(String name) {
|
||||
if (!isEnableSasl()) {
|
||||
return ResponseData.create().failed("Only support SASL protocol.");
|
||||
}
|
||||
if (!isEnableScram()) {
|
||||
return ResponseData.create().failed("Only support SASL_SCRAM.");
|
||||
}
|
||||
log.info("delete user: {}", name);
|
||||
Tuple2<Object, String> tuple2 = configConsole.deleteUser(name);
|
||||
return (boolean) tuple2._1() ? ResponseData.create().success() : ResponseData.create().failed(tuple2._2());
|
||||
}
|
||||
|
||||
@Override public ResponseData deleteUserAndAuth(String name) {
|
||||
if (!isEnableSasl()) {
|
||||
return ResponseData.create().failed("Only support SASL protocol.");
|
||||
}
|
||||
if (!isEnableScram()) {
|
||||
return ResponseData.create().failed("Only support SASL_SCRAM.");
|
||||
}
|
||||
log.info("delete user and authority: {}", name);
|
||||
AclEntry entry = new AclEntry();
|
||||
entry.setPrincipal(name);
|
||||
@@ -120,7 +145,8 @@ public class AclServiceImpl implements AclService, SmartInitializingSingleton {
|
||||
Map<String, Object> resultMap = new HashMap<>();
|
||||
entryMap.forEach((k, v) -> {
|
||||
Map<String, List<AclEntry>> map = v.stream().collect(Collectors.groupingBy(e -> e.getResourceType() + "#" + e.getName()));
|
||||
if (k.equals(kafkaConfig.getAdminUsername())) {
|
||||
String username = SaslUtil.findUsername(ContextConfigHolder.CONTEXT_CONFIG.get().getProperties().getProperty(SaslConfigs.SASL_JAAS_CONFIG));
|
||||
if (k.equals(username)) {
|
||||
Map<String, Object> map2 = new HashMap<>(map);
|
||||
Map<String, Object> userMap = new HashMap<>();
|
||||
userMap.put("role", "admin");
|
||||
@@ -133,7 +159,8 @@ public class AclServiceImpl implements AclService, SmartInitializingSingleton {
|
||||
|
||||
detailList.values().forEach(u -> {
|
||||
if (!resultMap.containsKey(u.name()) && !u.credentialInfos().isEmpty()) {
|
||||
if (!u.name().equals(kafkaConfig.getAdminUsername())) {
|
||||
String username = SaslUtil.findUsername(ContextConfigHolder.CONTEXT_CONFIG.get().getProperties().getProperty(SaslConfigs.SASL_JAAS_CONFIG));
|
||||
if (!u.name().equals(username)) {
|
||||
resultMap.put(u.name(), Collections.emptyMap());
|
||||
} else {
|
||||
Map<String, Object> map2 = new HashMap<>();
|
||||
@@ -194,27 +221,29 @@ public class AclServiceImpl implements AclService, SmartInitializingSingleton {
|
||||
}
|
||||
Map<String, Object> param = new HashMap<>();
|
||||
param.put("username", username);
|
||||
param.put("cluster_info_id", ContextConfigHolder.CONTEXT_CONFIG.get().getClusterInfoId());
|
||||
List<KafkaUserDO> dos = kafkaUserMapper.selectByMap(param);
|
||||
if (dos.isEmpty()) {
|
||||
vo.setConsistencyDescription("Password is null.");
|
||||
} else {
|
||||
vo.setPassword(dos.stream().findFirst().get().getPassword());
|
||||
// check for consistency.
|
||||
boolean consistent = configConsole.isPassConsistent(username, vo.getPassword());
|
||||
vo.setConsistencyDescription(consistent ? "Consistent" : "Password is not consistent.");
|
||||
// boolean consistent = configConsole.isPassConsistent(username, vo.getPassword());
|
||||
// vo.setConsistencyDescription(consistent ? "Consistent" : "Password is not consistent.");
|
||||
vo.setConsistencyDescription("Can not check password consistent.");
|
||||
}
|
||||
|
||||
return ResponseData.create().data(vo).success();
|
||||
}
|
||||
|
||||
@Override public void afterSingletonsInstantiated() {
|
||||
if (kafkaConfig.isEnableAcl() && kafkaConfig.isAdminCreate()) {
|
||||
log.info("Start create admin user, username: {}, password: {}", kafkaConfig.getAdminUsername(), kafkaConfig.getAdminPassword());
|
||||
boolean done = configConsole.addOrUpdateUserWithZK(kafkaConfig.getAdminUsername(), kafkaConfig.getAdminPassword());
|
||||
if (!done) {
|
||||
log.error("Create admin failed.");
|
||||
throw new IllegalStateException();
|
||||
}
|
||||
}
|
||||
}
|
||||
// @Override public void afterSingletonsInstantiated() {
|
||||
// if (kafkaConfig.isEnableAcl() && kafkaConfig.isAdminCreate()) {
|
||||
// log.info("Start create admin user, username: {}, password: {}", kafkaConfig.getAdminUsername(), kafkaConfig.getAdminPassword());
|
||||
// boolean done = configConsole.addOrUpdateUserWithZK(kafkaConfig.getAdminUsername(), kafkaConfig.getAdminPassword());
|
||||
// if (!done) {
|
||||
// log.error("Create admin failed.");
|
||||
// throw new IllegalStateException();
|
||||
// }
|
||||
// }
|
||||
// }
|
||||
}
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
package com.xuxd.kafka.console.service.impl;
|
||||
|
||||
import com.xuxd.kafka.console.service.ClusterInfoService;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
* @author xuxd
|
||||
* @date 2021-12-31 11:42:59
|
||||
**/
|
||||
@Service
|
||||
public class ClusterInfoServiceImpl implements ClusterInfoService {
|
||||
}
|
||||
@@ -1,9 +1,24 @@
|
||||
package com.xuxd.kafka.console.service.impl;
|
||||
|
||||
import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
|
||||
import com.xuxd.kafka.console.beans.BrokerNode;
|
||||
import com.xuxd.kafka.console.beans.ClusterInfo;
|
||||
import com.xuxd.kafka.console.beans.ResponseData;
|
||||
import com.xuxd.kafka.console.beans.dos.ClusterInfoDO;
|
||||
import com.xuxd.kafka.console.beans.vo.BrokerApiVersionVO;
|
||||
import com.xuxd.kafka.console.beans.vo.ClusterInfoVO;
|
||||
import com.xuxd.kafka.console.dao.ClusterInfoMapper;
|
||||
import com.xuxd.kafka.console.service.ClusterService;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.stream.Collectors;
|
||||
import kafka.console.ClusterConsole;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
import org.apache.commons.collections.CollectionUtils;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import org.apache.kafka.clients.NodeApiVersions;
|
||||
import org.apache.kafka.common.Node;
|
||||
import org.springframework.beans.factory.ObjectProvider;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
/**
|
||||
@@ -12,13 +27,87 @@ import org.springframework.stereotype.Service;
|
||||
* @author xuxd
|
||||
* @date 2021-10-08 14:23:09
|
||||
**/
|
||||
@Slf4j
|
||||
@Service
|
||||
public class ClusterServiceImpl implements ClusterService {
|
||||
|
||||
@Autowired
|
||||
private ClusterConsole clusterConsole;
|
||||
private final ClusterConsole clusterConsole;
|
||||
|
||||
private final ClusterInfoMapper clusterInfoMapper;
|
||||
|
||||
public ClusterServiceImpl(ObjectProvider<ClusterConsole> clusterConsole,
|
||||
ObjectProvider<ClusterInfoMapper> clusterInfoMapper) {
|
||||
this.clusterConsole = clusterConsole.getIfAvailable();
|
||||
this.clusterInfoMapper = clusterInfoMapper.getIfAvailable();
|
||||
}
|
||||
|
||||
@Override public ResponseData getClusterInfo() {
|
||||
return ResponseData.create().data(clusterConsole.clusterInfo()).success();
|
||||
ClusterInfo clusterInfo = clusterConsole.clusterInfo();
|
||||
Set<BrokerNode> nodes = clusterInfo.getNodes();
|
||||
if (nodes == null) {
|
||||
log.error("集群节点信息为空,集群地址可能不正确或集群内没有活跃节点");
|
||||
return ResponseData.create().failed("集群节点信息为空,集群地址可能不正确或集群内没有活跃节点");
|
||||
}
|
||||
clusterInfo.setNodes(new TreeSet<>(nodes));
|
||||
return ResponseData.create().data(clusterInfo).success();
|
||||
}
|
||||
|
||||
@Override public ResponseData getClusterInfoList() {
|
||||
return ResponseData.create().data(clusterInfoMapper.selectList(null)
|
||||
.stream().map(ClusterInfoVO::from).collect(Collectors.toList())).success();
|
||||
}
|
||||
|
||||
@Override public ResponseData addClusterInfo(ClusterInfoDO infoDO) {
|
||||
QueryWrapper<ClusterInfoDO> queryWrapper = new QueryWrapper<>();
|
||||
queryWrapper.eq("cluster_name", infoDO.getClusterName());
|
||||
if (clusterInfoMapper.selectCount(queryWrapper) > 0) {
|
||||
return ResponseData.create().failed("cluster name exist.");
|
||||
}
|
||||
clusterInfoMapper.insert(infoDO);
|
||||
return ResponseData.create().success();
|
||||
}
|
||||
|
||||
@Override public ResponseData deleteClusterInfo(Long id) {
|
||||
clusterInfoMapper.deleteById(id);
|
||||
return ResponseData.create().success();
|
||||
}
|
||||
|
||||
@Override public ResponseData updateClusterInfo(ClusterInfoDO infoDO) {
|
||||
clusterInfoMapper.updateById(infoDO);
|
||||
return ResponseData.create().success();
|
||||
}
|
||||
|
||||
@Override public ResponseData peekClusterInfo() {
|
||||
List<ClusterInfoDO> dos = clusterInfoMapper.selectList(null);
|
||||
if (CollectionUtils.isEmpty(dos)) {
|
||||
return ResponseData.create().failed("No Cluster Info.");
|
||||
}
|
||||
return ResponseData.create().data(dos.stream().findFirst().map(ClusterInfoVO::from)).success();
|
||||
}
|
||||
|
||||
@Override public ResponseData getBrokerApiVersionInfo() {
|
||||
HashMap<Node, NodeApiVersions> map = clusterConsole.listBrokerVersionInfo();
|
||||
List<BrokerApiVersionVO> list = new ArrayList<>(map.size());
|
||||
map.forEach(((node, versions) -> {
|
||||
BrokerApiVersionVO vo = new BrokerApiVersionVO();
|
||||
vo.setBrokerId(node.id());
|
||||
vo.setHost(node.host() + ":" + node.port());
|
||||
vo.setSupportNums(versions.allSupportedApiVersions().size());
|
||||
String versionInfo = versions.toString(true);
|
||||
int from = 0;
|
||||
int count = 0;
|
||||
int index = -1;
|
||||
while ((index = versionInfo.indexOf("UNSUPPORTED", from)) >= 0 && from < versionInfo.length()) {
|
||||
count++;
|
||||
from = index + 1;
|
||||
}
|
||||
vo.setUnSupportNums(count);
|
||||
versionInfo = versionInfo.substring(1, versionInfo.length() - 2);
|
||||
vo.setVersionInfo(Arrays.asList(StringUtils.split(versionInfo, ",")));
|
||||
list.add(vo);
|
||||
}));
|
||||
Collections.sort(list, Comparator.comparingInt(BrokerApiVersionVO::getBrokerId));
|
||||
return ResponseData.create().data(list).success();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
package com.xuxd.kafka.console.service.impl;
|
||||
|
||||
import com.xuxd.kafka.console.beans.MessageFilter;
|
||||
import com.xuxd.kafka.console.beans.QueryMessage;
|
||||
import com.xuxd.kafka.console.beans.ResponseData;
|
||||
import com.xuxd.kafka.console.beans.SendMessage;
|
||||
import com.xuxd.kafka.console.beans.enums.FilterType;
|
||||
import com.xuxd.kafka.console.beans.vo.ConsumerRecordVO;
|
||||
import com.xuxd.kafka.console.beans.vo.MessageDetailVO;
|
||||
import com.xuxd.kafka.console.service.ConsumerService;
|
||||
@@ -22,6 +24,7 @@ import kafka.console.TopicConsole;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
import org.apache.commons.collections.CollectionUtils;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import org.apache.kafka.clients.admin.RecordsToDelete;
|
||||
import org.apache.kafka.clients.admin.TopicDescription;
|
||||
import org.apache.kafka.clients.consumer.ConsumerRecord;
|
||||
import org.apache.kafka.clients.producer.ProducerRecord;
|
||||
@@ -32,6 +35,7 @@ import org.apache.kafka.common.serialization.Deserializer;
|
||||
import org.apache.kafka.common.serialization.DoubleDeserializer;
|
||||
import org.apache.kafka.common.serialization.FloatDeserializer;
|
||||
import org.apache.kafka.common.serialization.IntegerDeserializer;
|
||||
import org.apache.kafka.common.serialization.LongDeserializer;
|
||||
import org.apache.kafka.common.serialization.StringDeserializer;
|
||||
import org.springframework.beans.BeansException;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
@@ -70,23 +74,83 @@ public class MessageServiceImpl implements MessageService, ApplicationContextAwa
|
||||
deserializerDict.put("Float", new FloatDeserializer());
|
||||
deserializerDict.put("Double", new DoubleDeserializer());
|
||||
deserializerDict.put("Byte", new BytesDeserializer());
|
||||
deserializerDict.put("Long", new LongDeserializer());
|
||||
}
|
||||
|
||||
public static String defaultDeserializer = "String";
|
||||
|
||||
@Override public ResponseData searchByTime(QueryMessage queryMessage) {
|
||||
int maxNums = 10000;
|
||||
int maxNums = 5000;
|
||||
|
||||
Object searchContent = null;
|
||||
String headerKey = null;
|
||||
String headerValue = null;
|
||||
MessageFilter filter = new MessageFilter();
|
||||
switch (queryMessage.getFilter()) {
|
||||
case BODY:
|
||||
if (StringUtils.isBlank(queryMessage.getValue())) {
|
||||
queryMessage.setFilter(FilterType.NONE);
|
||||
} else {
|
||||
if (StringUtils.isBlank(queryMessage.getValueDeserializer())) {
|
||||
queryMessage.setValueDeserializer(defaultDeserializer);
|
||||
}
|
||||
switch (queryMessage.getValueDeserializer()) {
|
||||
case "String":
|
||||
searchContent = String.valueOf(queryMessage.getValue());
|
||||
filter.setContainsValue(true);
|
||||
break;
|
||||
case "Integer":
|
||||
searchContent = Integer.valueOf(queryMessage.getValue());
|
||||
break;
|
||||
case "Float":
|
||||
searchContent = Float.valueOf(queryMessage.getValue());
|
||||
break;
|
||||
case "Double":
|
||||
searchContent = Double.valueOf(queryMessage.getValue());
|
||||
break;
|
||||
case "Long":
|
||||
searchContent = Long.valueOf(queryMessage.getValue());
|
||||
break;
|
||||
default:
|
||||
throw new IllegalArgumentException("Message body type not support.");
|
||||
}
|
||||
}
|
||||
break;
|
||||
case HEADER:
|
||||
headerKey = queryMessage.getHeaderKey();
|
||||
if (StringUtils.isBlank(headerKey)) {
|
||||
queryMessage.setFilter(FilterType.NONE);
|
||||
} else {
|
||||
if (StringUtils.isNotBlank(queryMessage.getHeaderValue())) {
|
||||
headerValue = String.valueOf(queryMessage.getHeaderValue());
|
||||
}
|
||||
}
|
||||
break;
|
||||
default:
|
||||
break;
|
||||
}
|
||||
|
||||
FilterType filterType = queryMessage.getFilter();
|
||||
Deserializer deserializer = deserializerDict.get(queryMessage.getValueDeserializer());
|
||||
filter.setFilterType(filterType);
|
||||
filter.setSearchContent(searchContent);
|
||||
filter.setDeserializer(deserializer);
|
||||
filter.setHeaderKey(headerKey);
|
||||
filter.setHeaderValue(headerValue);
|
||||
|
||||
Set<TopicPartition> partitions = getPartitions(queryMessage);
|
||||
long startTime = System.currentTimeMillis();
|
||||
List<ConsumerRecord<byte[], byte[]>> records = messageConsole.searchBy(partitions, queryMessage.getStartTime(), queryMessage.getEndTime(), maxNums);
|
||||
Tuple2<List<ConsumerRecord<byte[], byte[]>>, Object> tuple2 = messageConsole.searchBy(partitions, queryMessage.getStartTime(), queryMessage.getEndTime(), maxNums, filter);
|
||||
List<ConsumerRecord<byte[], byte[]>> records = tuple2._1();
|
||||
log.info("search message by time, cost time: {}", (System.currentTimeMillis() - startTime));
|
||||
List<ConsumerRecordVO> vos = records.stream().filter(record -> record.timestamp() <= queryMessage.getEndTime())
|
||||
.map(ConsumerRecordVO::fromConsumerRecord).collect(Collectors.toList());
|
||||
Map<String, Object> res = new HashMap<>();
|
||||
vos = vos.subList(0, Math.min(maxNums, vos.size()));
|
||||
res.put("maxNum", maxNums);
|
||||
res.put("realNum", vos.size());
|
||||
res.put("data", vos.subList(0, Math.min(maxNums, vos.size())));
|
||||
res.put("searchNum", tuple2._2());
|
||||
res.put("data", vos);
|
||||
return ResponseData.create().data(res).success();
|
||||
}
|
||||
|
||||
@@ -179,6 +243,18 @@ public class MessageServiceImpl implements MessageService, ApplicationContextAwa
|
||||
return success ? ResponseData.create().success("success: " + tuple2._2()) : ResponseData.create().failed(tuple2._2());
|
||||
}
|
||||
|
||||
@Override
|
||||
public ResponseData delete(List<QueryMessage> messages) {
|
||||
Map<TopicPartition, RecordsToDelete> params = new HashMap<>(messages.size(), 1f);
|
||||
|
||||
messages.forEach(message -> {
|
||||
params.put(new TopicPartition(message.getTopic(), message.getPartition()), RecordsToDelete.beforeOffset(message.getOffset()));
|
||||
});
|
||||
Tuple2<Object, String> tuple2 = messageConsole.delete(params);
|
||||
boolean success = (boolean) tuple2._1();
|
||||
return success ? ResponseData.create().success() : ResponseData.create().failed(tuple2._2());
|
||||
}
|
||||
|
||||
private Map<TopicPartition, ConsumerRecord<byte[], byte[]>> searchRecordByOffset(QueryMessage queryMessage) {
|
||||
Set<TopicPartition> partitions = getPartitions(queryMessage);
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package com.xuxd.kafka.console.service.impl;
|
||||
|
||||
import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
|
||||
import com.google.common.collect.Lists;
|
||||
import com.google.gson.Gson;
|
||||
import com.google.gson.JsonObject;
|
||||
import com.xuxd.kafka.console.beans.ResponseData;
|
||||
@@ -10,6 +11,7 @@ import com.xuxd.kafka.console.beans.vo.OffsetAlignmentVO;
|
||||
import com.xuxd.kafka.console.dao.MinOffsetAlignmentMapper;
|
||||
import com.xuxd.kafka.console.service.OperationService;
|
||||
import com.xuxd.kafka.console.utils.GsonUtil;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.HashMap;
|
||||
import java.util.HashSet;
|
||||
@@ -19,6 +21,7 @@ import java.util.Properties;
|
||||
import java.util.Set;
|
||||
import java.util.stream.Collectors;
|
||||
import kafka.console.OperationConsole;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import org.apache.kafka.clients.admin.PartitionReassignment;
|
||||
import org.apache.kafka.common.TopicPartition;
|
||||
import org.springframework.beans.factory.ObjectProvider;
|
||||
@@ -162,4 +165,21 @@ public class OperationServiceImpl implements OperationService {
|
||||
}
|
||||
return ResponseData.create().success();
|
||||
}
|
||||
|
||||
@Override public ResponseData proposedAssignments(String topic, List<Integer> brokerList) {
|
||||
Map<String, Object> params = new HashMap<>();
|
||||
params.put("version", 1);
|
||||
Map<String, String> topicMap = new HashMap<>(1, 1.0f);
|
||||
topicMap.put("topic", topic);
|
||||
params.put("topics", Lists.newArrayList(topicMap));
|
||||
List<String> list = brokerList.stream().map(String::valueOf).collect(Collectors.toList());
|
||||
Map<TopicPartition, List<Object>> assignments = operationConsole.proposedAssignments(gson.toJson(params), StringUtils.join(list, ","));
|
||||
List<CurrentReassignmentVO> res = new ArrayList<>(assignments.size());
|
||||
assignments.forEach((tp, replicas) -> {
|
||||
CurrentReassignmentVO vo = new CurrentReassignmentVO(tp.topic(), tp.partition(),
|
||||
replicas.stream().map(x -> (Integer) x).collect(Collectors.toList()), null, null);
|
||||
res.add(vo);
|
||||
});
|
||||
return ResponseData.create().data(res).success();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -9,16 +9,6 @@ import com.xuxd.kafka.console.beans.vo.TopicDescriptionVO;
|
||||
import com.xuxd.kafka.console.beans.vo.TopicPartitionVO;
|
||||
import com.xuxd.kafka.console.service.TopicService;
|
||||
import com.xuxd.kafka.console.utils.GsonUtil;
|
||||
import java.util.Calendar;
|
||||
import java.util.Collections;
|
||||
import java.util.Comparator;
|
||||
import java.util.HashMap;
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.atomic.AtomicLong;
|
||||
import java.util.stream.Collectors;
|
||||
import kafka.console.MessageConsole;
|
||||
import kafka.console.TopicConsole;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
@@ -33,6 +23,10 @@ import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Service;
|
||||
import scala.Tuple2;
|
||||
|
||||
import java.util.*;
|
||||
import java.util.concurrent.atomic.AtomicLong;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
@@ -87,8 +81,8 @@ public class TopicServiceImpl implements TopicService {
|
||||
return ResponseData.create().data(topicDescriptions.stream().map(d -> TopicDescriptionVO.from(d))).success();
|
||||
}
|
||||
|
||||
@Override public ResponseData deleteTopic(String topic) {
|
||||
Tuple2<Object, String> tuple2 = topicConsole.deleteTopic(topic);
|
||||
@Override public ResponseData deleteTopics(Collection<String> topics) {
|
||||
Tuple2<Object, String> tuple2 = topicConsole.deleteTopics(topics);
|
||||
return (Boolean) tuple2._1 ? ResponseData.create().success() : ResponseData.create().failed(tuple2._2);
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
package com.xuxd.kafka.console.service.interceptor;
|
||||
|
||||
import com.xuxd.kafka.console.beans.ResponseData;
|
||||
import com.xuxd.kafka.console.beans.dos.ClusterInfoDO;
|
||||
import com.xuxd.kafka.console.config.ContextConfig;
|
||||
import com.xuxd.kafka.console.config.ContextConfigHolder;
|
||||
import com.xuxd.kafka.console.dao.ClusterInfoMapper;
|
||||
import com.xuxd.kafka.console.utils.ConvertUtil;
|
||||
import java.io.IOException;
|
||||
import java.util.HashSet;
|
||||
import java.util.Set;
|
||||
import javax.servlet.Filter;
|
||||
import javax.servlet.FilterChain;
|
||||
import javax.servlet.ServletException;
|
||||
import javax.servlet.ServletRequest;
|
||||
import javax.servlet.ServletResponse;
|
||||
import javax.servlet.annotation.WebFilter;
|
||||
import javax.servlet.http.HttpServletRequest;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.http.MediaType;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
* @author xuxd
|
||||
* @date 2022-01-05 19:56:25
|
||||
**/
|
||||
@WebFilter(filterName = "context-set-filter", urlPatterns = {"/acl/*","/user/*","/cluster/*","/config/*","/consumer/*","/message/*","/topic/*","/op/*"})
|
||||
@Slf4j
|
||||
public class ContextSetFilter implements Filter {
|
||||
|
||||
private Set<String> excludes = new HashSet<>();
|
||||
|
||||
{
|
||||
excludes.add("/cluster/info/peek");
|
||||
excludes.add("/cluster/info");
|
||||
excludes.add("/config/console");
|
||||
}
|
||||
|
||||
@Autowired
|
||||
private ClusterInfoMapper clusterInfoMapper;
|
||||
|
||||
@Override public void doFilter(ServletRequest req, ServletResponse response,
|
||||
FilterChain chain) throws IOException, ServletException {
|
||||
try {
|
||||
HttpServletRequest request = (HttpServletRequest) req;
|
||||
String uri = request.getRequestURI();
|
||||
if (!excludes.contains(uri)) {
|
||||
String headerId = request.getHeader(Header.ID);
|
||||
if (StringUtils.isBlank(headerId)) {
|
||||
// ResponseData failed = ResponseData.create().failed("Cluster info is null.");
|
||||
ResponseData failed = ResponseData.create().failed("没有集群信息,请先切换集群");
|
||||
response.setContentType(MediaType.APPLICATION_JSON_UTF8_VALUE);
|
||||
response.getWriter().println(ConvertUtil.toJsonString(failed));
|
||||
return;
|
||||
} else {
|
||||
ClusterInfoDO infoDO = clusterInfoMapper.selectById(Long.valueOf(headerId));
|
||||
if (infoDO == null) {
|
||||
ResponseData failed = ResponseData.create().failed("该集群找不到信息,请切换一个有效集群");
|
||||
response.setContentType(MediaType.APPLICATION_JSON_UTF8_VALUE);
|
||||
response.getWriter().println(ConvertUtil.toJsonString(failed));
|
||||
return;
|
||||
}
|
||||
ContextConfig config = new ContextConfig();
|
||||
config.setClusterInfoId(infoDO.getId());
|
||||
config.setClusterName(infoDO.getClusterName());
|
||||
|
||||
config.setBootstrapServer(infoDO.getAddress());
|
||||
if (StringUtils.isNotBlank(infoDO.getProperties())) {
|
||||
config.setProperties(ConvertUtil.toProperties(infoDO.getProperties()));
|
||||
}
|
||||
ContextConfigHolder.CONTEXT_CONFIG.set(config);
|
||||
}
|
||||
}
|
||||
chain.doFilter(req, response);
|
||||
} finally {
|
||||
ContextConfigHolder.CONTEXT_CONFIG.remove();
|
||||
}
|
||||
}
|
||||
|
||||
interface Header {
|
||||
String ID = "X-Cluster-Info-Id";
|
||||
String NAME = "X-Cluster-Info-Name";
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package com.xuxd.kafka.console.interceptor;
|
||||
package com.xuxd.kafka.console.service.interceptor;
|
||||
|
||||
import com.xuxd.kafka.console.beans.ResponseData;
|
||||
import javax.servlet.http.HttpServletRequest;
|
||||
@@ -1,9 +1,15 @@
|
||||
package com.xuxd.kafka.console.utils;
|
||||
|
||||
import com.google.common.base.Preconditions;
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.IOException;
|
||||
import java.util.Arrays;
|
||||
import java.util.HashMap;
|
||||
import java.util.Iterator;
|
||||
import java.util.LinkedList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Properties;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
import org.springframework.util.ClassUtils;
|
||||
|
||||
@@ -35,6 +41,52 @@ public class ConvertUtil {
|
||||
}
|
||||
});
|
||||
}
|
||||
Iterator<Map.Entry<String, Object>> iterator = res.entrySet().iterator();
|
||||
while (iterator.hasNext()) {
|
||||
if (iterator.next().getValue() == null) {
|
||||
iterator.remove();
|
||||
}
|
||||
}
|
||||
return res;
|
||||
}
|
||||
|
||||
public static String toJsonString(Object src) {
|
||||
return GsonUtil.INSTANCE.get().toJson(src);
|
||||
}
|
||||
|
||||
public static Properties toProperties(String jsonStr) {
|
||||
return GsonUtil.INSTANCE.get().fromJson(jsonStr, Properties.class);
|
||||
}
|
||||
|
||||
public static String jsonStr2PropertiesStr(String jsonStr) {
|
||||
StringBuilder sb = new StringBuilder();
|
||||
Map<String, Object> map = GsonUtil.INSTANCE.get().fromJson(jsonStr, Map.class);
|
||||
map.keySet().forEach(k -> {
|
||||
sb.append(k).append("=").append(map.get(k).toString()).append(System.lineSeparator());
|
||||
});
|
||||
|
||||
return sb.toString();
|
||||
}
|
||||
|
||||
public static List<String> jsonStr2List(String jsonStr) {
|
||||
List<String> res = new LinkedList<>();
|
||||
Map<String, Object> map = GsonUtil.INSTANCE.get().fromJson(jsonStr, Map.class);
|
||||
map.forEach((k, v) -> {
|
||||
res.add(k + "=" + v);
|
||||
});
|
||||
|
||||
return res;
|
||||
}
|
||||
|
||||
public static String propertiesStr2JsonStr(String propertiesStr) {
|
||||
String res = "{}";
|
||||
try (ByteArrayInputStream baos = new ByteArrayInputStream(propertiesStr.getBytes())) {
|
||||
Properties properties = new Properties();
|
||||
properties.load(baos);
|
||||
res = toJsonString(properties);
|
||||
} catch (IOException e) {
|
||||
log.error("propertiesStr2JsonStr error.", e);
|
||||
}
|
||||
return res;
|
||||
}
|
||||
}
|
||||
|
||||
61
src/main/java/com/xuxd/kafka/console/utils/SaslUtil.java
Normal file
@@ -0,0 +1,61 @@
|
||||
package com.xuxd.kafka.console.utils;
|
||||
|
||||
import com.xuxd.kafka.console.config.ContextConfigHolder;
|
||||
import java.util.Properties;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
import org.apache.kafka.clients.CommonClientConfigs;
|
||||
import org.apache.kafka.clients.admin.ScramMechanism;
|
||||
import org.apache.kafka.common.config.SaslConfigs;
|
||||
import org.apache.kafka.common.security.auth.SecurityProtocol;
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
* @author xuxd
|
||||
* @date 2022-01-06 11:07:41
|
||||
**/
|
||||
public class SaslUtil {
|
||||
|
||||
public static final Pattern JAAS_PATTERN = Pattern.compile("^.*(username=\"(.*)\"[ \t]+).*$");
|
||||
|
||||
private SaslUtil() {
|
||||
}
|
||||
|
||||
public static String findUsername(String saslJaasConfig) {
|
||||
Matcher matcher = JAAS_PATTERN.matcher(saslJaasConfig);
|
||||
return matcher.find() ? matcher.group(2) : "";
|
||||
}
|
||||
|
||||
public static boolean isEnableSasl() {
|
||||
Properties properties = ContextConfigHolder.CONTEXT_CONFIG.get().getProperties();
|
||||
if (!properties.containsKey(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG)) {
|
||||
return false;
|
||||
}
|
||||
String s = properties.getProperty(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG);
|
||||
SecurityProtocol protocol = SecurityProtocol.valueOf(s);
|
||||
switch (protocol) {
|
||||
case SASL_SSL:
|
||||
case SASL_PLAINTEXT:
|
||||
return true;
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
public static boolean isEnableScram() {
|
||||
Properties properties = ContextConfigHolder.CONTEXT_CONFIG.get().getProperties();
|
||||
if (!properties.containsKey(SaslConfigs.SASL_MECHANISM)) {
|
||||
return false;
|
||||
}
|
||||
String s = properties.getProperty(SaslConfigs.SASL_MECHANISM);
|
||||
ScramMechanism mechanism = ScramMechanism.fromMechanismName(s);
|
||||
switch (mechanism) {
|
||||
case UNKNOWN:
|
||||
return false;
|
||||
default:
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -6,23 +6,20 @@ server:
|
||||
|
||||
kafka:
|
||||
config:
|
||||
# kafka broker地址,多个以逗号分隔
|
||||
bootstrap-server: 'localhost:9092'
|
||||
request-timeout-ms: 5000
|
||||
# 服务端是否启用acl,如果不启用,下面的所有配置都忽略即可,只用配置上面的Kafka集群地址就行了
|
||||
enable-acl: false
|
||||
# 只支持2种安全协议SASL_PLAINTEXT和PLAINTEXT,启用acl则设置为SASL_PLAINTEXT,不启用acl不需关心这个配置
|
||||
security-protocol: SASL_PLAINTEXT
|
||||
sasl-mechanism: SCRAM-SHA-256
|
||||
# 超级管理员用户名,在broker上已经配置为超级管理员
|
||||
admin-username: admin
|
||||
# 超级管理员密码
|
||||
admin-password: admin
|
||||
# 启动自动创建配置的超级管理员用户
|
||||
admin-create: false
|
||||
# broker连接的zk地址,如果启动自动创建配置的超级管理员用户则必须配置,否则忽略
|
||||
zookeeper-addr: 'localhost:2181'
|
||||
sasl-jaas-config: org.apache.kafka.common.security.scram.ScramLoginModule required username="${kafka.config.admin-username}" password="${kafka.config.admin-password}";
|
||||
# 如果不存在default集群,启动的时候默认会把这个加载进来(如果这里配置集群地址了),如果已经存在,则不加载
|
||||
# kafka broker地址,多个以逗号分隔,不是必须在这里配置,也可以启动之后,在页面上添加集群信息
|
||||
bootstrap-server:
|
||||
# 集群其它属性配置
|
||||
properties:
|
||||
# request.timeout.ms: 5000
|
||||
# 缓存连接,不缓存的情况下,每次请求建立连接. 即使每次请求建立连接,其实也很快,某些情况下开启ACL,查询可能很慢,可以设置连接缓存为true,
|
||||
# 或者想提高查询速度,也可以设置下面连接缓存为true
|
||||
# 缓存 admin client的连接
|
||||
cache-admin-connection: false
|
||||
# 缓存 producer的连接
|
||||
cache-producer-connection: false
|
||||
# 缓存 consumer的连接
|
||||
cache-consumer-connection: false
|
||||
|
||||
spring:
|
||||
application:
|
||||
@@ -46,6 +43,7 @@ spring:
|
||||
logging:
|
||||
home: ./
|
||||
|
||||
# 基于scram方案的acl,这里会记录创建的用户密码等信息,定时扫描,如果集群中已经不存在这些用户,就把这些信息从db中清除掉
|
||||
cron:
|
||||
# clear-dirty-user: 0 * * * * ?
|
||||
clear-dirty-user: 0 0 1 * * ?
|
||||
|
||||
@@ -1,23 +1,37 @@
|
||||
-- DROP TABLE IF EXISTS T_KAKFA_USER;
|
||||
|
||||
-- kafka ACL启用SASL_SCRAM中的用户
|
||||
CREATE TABLE IF NOT EXISTS T_KAFKA_USER
|
||||
(
|
||||
ID IDENTITY NOT NULL COMMENT '主键ID',
|
||||
USERNAME VARCHAR(50) NOT NULL DEFAULT '' COMMENT '姓名',
|
||||
PASSWORD VARCHAR(50) NOT NULL DEFAULT '' COMMENT '年龄',
|
||||
UPDATE_TIME TIMESTAMP NOT NULL DEFAULT NOW() COMMENT '更新时间',
|
||||
ID IDENTITY NOT NULL COMMENT '主键ID',
|
||||
USERNAME VARCHAR(50) NOT NULL DEFAULT '' COMMENT '用户名',
|
||||
PASSWORD VARCHAR(50) NOT NULL DEFAULT '' COMMENT '密码',
|
||||
UPDATE_TIME TIMESTAMP NOT NULL DEFAULT NOW() COMMENT '更新时间',
|
||||
CLUSTER_INFO_ID BIGINT NOT NULL COMMENT '集群信息里的集群ID',
|
||||
PRIMARY KEY (ID),
|
||||
UNIQUE (USERNAME)
|
||||
);
|
||||
|
||||
-- 消息同步解决方案中使用的位点对齐信息
|
||||
CREATE TABLE IF NOT EXISTS T_MIN_OFFSET_ALIGNMENT
|
||||
(
|
||||
ID IDENTITY NOT NULL COMMENT '主键ID',
|
||||
GROUP_ID VARCHAR(128) NOT NULL DEFAULT '' COMMENT 'groupId',
|
||||
TOPIC VARCHAR(128) NOT NULL DEFAULT '' COMMENT 'topic',
|
||||
THAT_OFFSET VARCHAR(512) NOT NULL DEFAULT '' COMMENT 'min offset for that kafka cluster',
|
||||
THIS_OFFSET VARCHAR(512) NOT NULL DEFAULT '' COMMENT 'min offset for this kafka cluster',
|
||||
UPDATE_TIME TIMESTAMP NOT NULL DEFAULT NOW() COMMENT '更新时间',
|
||||
ID IDENTITY NOT NULL COMMENT '主键ID',
|
||||
GROUP_ID VARCHAR(128) NOT NULL DEFAULT '' COMMENT 'groupId',
|
||||
TOPIC VARCHAR(128) NOT NULL DEFAULT '' COMMENT 'topic',
|
||||
THAT_OFFSET VARCHAR(512) NOT NULL DEFAULT '' COMMENT 'min offset for that kafka cluster',
|
||||
THIS_OFFSET VARCHAR(512) NOT NULL DEFAULT '' COMMENT 'min offset for this kafka cluster',
|
||||
UPDATE_TIME TIMESTAMP NOT NULL DEFAULT NOW() COMMENT '更新时间',
|
||||
PRIMARY KEY (ID),
|
||||
UNIQUE (GROUP_ID, TOPIC)
|
||||
);
|
||||
|
||||
-- 多集群管理,每个集群的配置信息
|
||||
CREATE TABLE IF NOT EXISTS T_CLUSTER_INFO
|
||||
(
|
||||
ID IDENTITY NOT NULL COMMENT '主键ID',
|
||||
CLUSTER_NAME VARCHAR(128) NOT NULL DEFAULT '' COMMENT '集群名',
|
||||
ADDRESS VARCHAR(256) NOT NULL DEFAULT '' COMMENT '集群地址',
|
||||
PROPERTIES VARCHAR(512) NOT NULL DEFAULT '' COMMENT '集群的其它属性配置',
|
||||
UPDATE_TIME TIMESTAMP NOT NULL DEFAULT NOW() COMMENT '更新时间',
|
||||
PRIMARY KEY (ID),
|
||||
UNIQUE (CLUSTER_NAME)
|
||||
);
|
||||
333
src/main/scala/kafka/console/BrokerApiVersion.scala
Normal file
@@ -0,0 +1,333 @@
|
||||
package kafka.console
|
||||
|
||||
import com.xuxd.kafka.console.config.ContextConfigHolder
|
||||
import kafka.utils.Implicits.MapExtensionMethods
|
||||
import kafka.utils.Logging
|
||||
import org.apache.kafka.clients._
|
||||
import org.apache.kafka.clients.admin.AdminClientConfig
|
||||
import org.apache.kafka.clients.consumer.internals.{ConsumerNetworkClient, RequestFuture}
|
||||
import org.apache.kafka.common.Node
|
||||
import org.apache.kafka.common.config.ConfigDef.ValidString.in
|
||||
import org.apache.kafka.common.config.ConfigDef.{Importance, Type}
|
||||
import org.apache.kafka.common.config.{AbstractConfig, ConfigDef}
|
||||
import org.apache.kafka.common.errors.AuthenticationException
|
||||
import org.apache.kafka.common.internals.ClusterResourceListeners
|
||||
import org.apache.kafka.common.message.ApiVersionsResponseData.ApiVersionCollection
|
||||
import org.apache.kafka.common.metrics.Metrics
|
||||
import org.apache.kafka.common.network.Selector
|
||||
import org.apache.kafka.common.protocol.Errors
|
||||
import org.apache.kafka.common.requests._
|
||||
import org.apache.kafka.common.utils.{KafkaThread, LogContext, Time}
|
||||
import org.slf4j.{Logger, LoggerFactory}
|
||||
|
||||
import java.io.IOException
|
||||
import java.util.Properties
|
||||
import java.util.concurrent.atomic.AtomicInteger
|
||||
import java.util.concurrent.{ConcurrentLinkedQueue, TimeUnit}
|
||||
import scala.jdk.CollectionConverters.{ListHasAsScala, MapHasAsJava, PropertiesHasAsScala, SetHasAsScala}
|
||||
import scala.util.{Failure, Success, Try}
|
||||
|
||||
/**
|
||||
* kafka-console-ui.
|
||||
*
|
||||
* Copy from {@link kafka.admin.BrokerApiVersionsCommand}.
|
||||
*
|
||||
* @author xuxd
|
||||
* @date 2022-01-22 15:15:57
|
||||
* */
|
||||
object BrokerApiVersion{
|
||||
|
||||
protected lazy val log : Logger = LoggerFactory.getLogger(this.getClass)
|
||||
|
||||
def listAllBrokerApiVersionInfo(): java.util.HashMap[Node, NodeApiVersions] = {
|
||||
val res = new java.util.HashMap[Node, NodeApiVersions]()
|
||||
val adminClient = createAdminClient()
|
||||
try {
|
||||
adminClient.awaitBrokers()
|
||||
val brokerMap = adminClient.listAllBrokerVersionInfo()
|
||||
brokerMap.forKeyValue {
|
||||
(broker, versionInfoOrError) =>
|
||||
versionInfoOrError match {
|
||||
case Success(v) => {
|
||||
res.put(broker, v)
|
||||
}
|
||||
case Failure(v) => log.error(s"${broker} -> ERROR: ${v}\n")
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
adminClient.close()
|
||||
}
|
||||
|
||||
res
|
||||
}
|
||||
|
||||
private def createAdminClient(): AdminClient = {
|
||||
val props = new Properties()
|
||||
props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, ContextConfigHolder.CONTEXT_CONFIG.get().getBootstrapServer())
|
||||
props.put(AdminClientConfig.REQUEST_TIMEOUT_MS_CONFIG, ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs())
|
||||
props.putAll(ContextConfigHolder.CONTEXT_CONFIG.get().getProperties())
|
||||
AdminClient.create(props)
|
||||
}
|
||||
|
||||
// org.apache.kafka.clients.admin.AdminClient doesn't currently expose a way to retrieve the supported api versions.
|
||||
// We inline the bits we need from kafka.admin.AdminClient so that we can delete it.
|
||||
private class AdminClient(val time: Time,
|
||||
val client: ConsumerNetworkClient,
|
||||
val bootstrapBrokers: List[Node]) extends Logging {
|
||||
|
||||
@volatile var running = true
|
||||
val pendingFutures = new ConcurrentLinkedQueue[RequestFuture[ClientResponse]]()
|
||||
|
||||
val networkThread = new KafkaThread("admin-client-network-thread", () => {
|
||||
try {
|
||||
while (running)
|
||||
client.poll(time.timer(Long.MaxValue))
|
||||
} catch {
|
||||
case t: Throwable =>
|
||||
error("admin-client-network-thread exited", t)
|
||||
} finally {
|
||||
pendingFutures.forEach { future =>
|
||||
try {
|
||||
future.raise(Errors.UNKNOWN_SERVER_ERROR)
|
||||
} catch {
|
||||
case _: IllegalStateException => // It is OK if the future has been completed
|
||||
}
|
||||
}
|
||||
pendingFutures.clear()
|
||||
}
|
||||
}, true)
|
||||
|
||||
networkThread.start()
|
||||
|
||||
private def send(target: Node,
|
||||
request: AbstractRequest.Builder[_ <: AbstractRequest]): AbstractResponse = {
|
||||
val future = client.send(target, request)
|
||||
pendingFutures.add(future)
|
||||
future.awaitDone(Long.MaxValue, TimeUnit.MILLISECONDS)
|
||||
pendingFutures.remove(future)
|
||||
if (future.succeeded())
|
||||
future.value().responseBody()
|
||||
else
|
||||
throw future.exception()
|
||||
}
|
||||
|
||||
private def sendAnyNode(request: AbstractRequest.Builder[_ <: AbstractRequest]): AbstractResponse = {
|
||||
bootstrapBrokers.foreach { broker =>
|
||||
try {
|
||||
return send(broker, request)
|
||||
} catch {
|
||||
case e: AuthenticationException =>
|
||||
throw e
|
||||
case e: Exception =>
|
||||
debug(s"Request ${request.apiKey()} failed against node $broker", e)
|
||||
}
|
||||
}
|
||||
throw new RuntimeException(s"Request ${request.apiKey()} failed on brokers $bootstrapBrokers")
|
||||
}
|
||||
|
||||
private def getApiVersions(node: Node): ApiVersionCollection = {
|
||||
val response = send(node, new ApiVersionsRequest.Builder()).asInstanceOf[ApiVersionsResponse]
|
||||
Errors.forCode(response.data.errorCode).maybeThrow()
|
||||
response.data.apiKeys
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait until there is a non-empty list of brokers in the cluster.
|
||||
*/
|
||||
def awaitBrokers(): Unit = {
|
||||
var nodes = List[Node]()
|
||||
val start = System.currentTimeMillis()
|
||||
val maxWait = 30 * 1000
|
||||
do {
|
||||
nodes = findAllBrokers()
|
||||
if (nodes.isEmpty) {
|
||||
Thread.sleep(50)
|
||||
}
|
||||
}
|
||||
while (nodes.isEmpty && (System.currentTimeMillis() - start < maxWait))
|
||||
}
|
||||
|
||||
private def findAllBrokers(): List[Node] = {
|
||||
val request = MetadataRequest.Builder.allTopics()
|
||||
val response = sendAnyNode(request).asInstanceOf[MetadataResponse]
|
||||
val errors = response.errors
|
||||
if (!errors.isEmpty) {
|
||||
log.info(s"Metadata request contained errors: $errors")
|
||||
}
|
||||
|
||||
// 在3.x版本中这个方法是buildCluster 代替cluster()了
|
||||
response.buildCluster.nodes.asScala.toList
|
||||
// response.cluster().nodes.asScala.toList
|
||||
}
|
||||
|
||||
def listAllBrokerVersionInfo(): Map[Node, Try[NodeApiVersions]] =
|
||||
findAllBrokers().map { broker =>
|
||||
broker -> Try[NodeApiVersions](new NodeApiVersions(getApiVersions(broker)))
|
||||
}.toMap
|
||||
|
||||
def close(): Unit = {
|
||||
running = false
|
||||
try {
|
||||
client.close()
|
||||
} catch {
|
||||
case e: IOException =>
|
||||
error("Exception closing nioSelector:", e)
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
private object AdminClient {
|
||||
val DefaultConnectionMaxIdleMs = 9 * 60 * 1000
|
||||
val DefaultRequestTimeoutMs = 5000
|
||||
val DefaultSocketConnectionSetupMs = CommonClientConfigs.SOCKET_CONNECTION_SETUP_TIMEOUT_MS_CONFIG
|
||||
val DefaultSocketConnectionSetupMaxMs = CommonClientConfigs.SOCKET_CONNECTION_SETUP_TIMEOUT_MAX_MS_CONFIG
|
||||
val DefaultMaxInFlightRequestsPerConnection = 100
|
||||
val DefaultReconnectBackoffMs = 50
|
||||
val DefaultReconnectBackoffMax = 50
|
||||
val DefaultSendBufferBytes = 128 * 1024
|
||||
val DefaultReceiveBufferBytes = 32 * 1024
|
||||
val DefaultRetryBackoffMs = 100
|
||||
|
||||
val AdminClientIdSequence = new AtomicInteger(1)
|
||||
val AdminConfigDef = {
|
||||
val config = new ConfigDef()
|
||||
.define(
|
||||
CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG,
|
||||
Type.LIST,
|
||||
Importance.HIGH,
|
||||
CommonClientConfigs.BOOTSTRAP_SERVERS_DOC)
|
||||
.define(CommonClientConfigs.CLIENT_DNS_LOOKUP_CONFIG,
|
||||
Type.STRING,
|
||||
ClientDnsLookup.USE_ALL_DNS_IPS.toString,
|
||||
in(ClientDnsLookup.USE_ALL_DNS_IPS.toString,
|
||||
ClientDnsLookup.RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY.toString),
|
||||
Importance.MEDIUM,
|
||||
CommonClientConfigs.CLIENT_DNS_LOOKUP_DOC)
|
||||
.define(
|
||||
CommonClientConfigs.SECURITY_PROTOCOL_CONFIG,
|
||||
ConfigDef.Type.STRING,
|
||||
CommonClientConfigs.DEFAULT_SECURITY_PROTOCOL,
|
||||
ConfigDef.Importance.MEDIUM,
|
||||
CommonClientConfigs.SECURITY_PROTOCOL_DOC)
|
||||
.define(
|
||||
CommonClientConfigs.REQUEST_TIMEOUT_MS_CONFIG,
|
||||
ConfigDef.Type.INT,
|
||||
DefaultRequestTimeoutMs,
|
||||
ConfigDef.Importance.MEDIUM,
|
||||
CommonClientConfigs.REQUEST_TIMEOUT_MS_DOC)
|
||||
.define(
|
||||
CommonClientConfigs.SOCKET_CONNECTION_SETUP_TIMEOUT_MS_CONFIG,
|
||||
ConfigDef.Type.LONG,
|
||||
CommonClientConfigs.DEFAULT_SOCKET_CONNECTION_SETUP_TIMEOUT_MS,
|
||||
ConfigDef.Importance.MEDIUM,
|
||||
CommonClientConfigs.SOCKET_CONNECTION_SETUP_TIMEOUT_MS_DOC)
|
||||
.define(
|
||||
CommonClientConfigs.SOCKET_CONNECTION_SETUP_TIMEOUT_MAX_MS_CONFIG,
|
||||
ConfigDef.Type.LONG,
|
||||
CommonClientConfigs.DEFAULT_SOCKET_CONNECTION_SETUP_TIMEOUT_MAX_MS,
|
||||
ConfigDef.Importance.MEDIUM,
|
||||
CommonClientConfigs.SOCKET_CONNECTION_SETUP_TIMEOUT_MAX_MS_DOC)
|
||||
.define(
|
||||
CommonClientConfigs.RETRY_BACKOFF_MS_CONFIG,
|
||||
ConfigDef.Type.LONG,
|
||||
DefaultRetryBackoffMs,
|
||||
ConfigDef.Importance.MEDIUM,
|
||||
CommonClientConfigs.RETRY_BACKOFF_MS_DOC)
|
||||
.withClientSslSupport()
|
||||
.withClientSaslSupport()
|
||||
config
|
||||
}
|
||||
|
||||
class AdminConfig(originals: Map[_, _]) extends AbstractConfig(AdminConfigDef, originals.asJava, false)
|
||||
|
||||
def create(props: Properties): AdminClient = {
|
||||
val properties = new Properties()
|
||||
val names = props.stringPropertyNames()
|
||||
for (name <- names.asScala.toSet) {
|
||||
properties.put(name, props.get(name).toString())
|
||||
}
|
||||
create(properties.asScala.toMap)
|
||||
}
|
||||
|
||||
def create(props: Map[String, _]): AdminClient = create(new AdminConfig(props))
|
||||
|
||||
def create(config: AdminConfig): AdminClient = {
|
||||
val clientId = "admin-" + AdminClientIdSequence.getAndIncrement()
|
||||
val logContext = new LogContext(s"[LegacyAdminClient clientId=$clientId] ")
|
||||
val time = Time.SYSTEM
|
||||
val metrics = new Metrics(time)
|
||||
val metadata = new Metadata(100L, 60 * 60 * 1000L, logContext,
|
||||
new ClusterResourceListeners)
|
||||
val channelBuilder = ClientUtils.createChannelBuilder(config, time, logContext)
|
||||
val requestTimeoutMs = config.getInt(CommonClientConfigs.REQUEST_TIMEOUT_MS_CONFIG)
|
||||
val connectionSetupTimeoutMs = config.getLong(CommonClientConfigs.SOCKET_CONNECTION_SETUP_TIMEOUT_MS_CONFIG)
|
||||
val connectionSetupTimeoutMaxMs = config.getLong(CommonClientConfigs.SOCKET_CONNECTION_SETUP_TIMEOUT_MAX_MS_CONFIG)
|
||||
val retryBackoffMs = config.getLong(CommonClientConfigs.RETRY_BACKOFF_MS_CONFIG)
|
||||
|
||||
val brokerUrls = config.getList(CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG)
|
||||
val clientDnsLookup = config.getString(CommonClientConfigs.CLIENT_DNS_LOOKUP_CONFIG)
|
||||
val brokerAddresses = ClientUtils.parseAndValidateAddresses(brokerUrls, clientDnsLookup)
|
||||
metadata.bootstrap(brokerAddresses)
|
||||
|
||||
val selector = new Selector(
|
||||
DefaultConnectionMaxIdleMs,
|
||||
metrics,
|
||||
time,
|
||||
"admin",
|
||||
channelBuilder,
|
||||
logContext)
|
||||
|
||||
// 版本不一样,这个地方的兼容性问题也不一样了
|
||||
// 3.x版本用这个
|
||||
val networkClient = new NetworkClient(
|
||||
selector,
|
||||
metadata,
|
||||
clientId,
|
||||
DefaultMaxInFlightRequestsPerConnection,
|
||||
DefaultReconnectBackoffMs,
|
||||
DefaultReconnectBackoffMax,
|
||||
DefaultSendBufferBytes,
|
||||
DefaultReceiveBufferBytes,
|
||||
requestTimeoutMs,
|
||||
connectionSetupTimeoutMs,
|
||||
connectionSetupTimeoutMaxMs,
|
||||
time,
|
||||
true,
|
||||
new ApiVersions,
|
||||
logContext)
|
||||
|
||||
// val networkClient = new NetworkClient(
|
||||
// selector,
|
||||
// metadata,
|
||||
// clientId,
|
||||
// DefaultMaxInFlightRequestsPerConnection,
|
||||
// DefaultReconnectBackoffMs,
|
||||
// DefaultReconnectBackoffMax,
|
||||
// DefaultSendBufferBytes,
|
||||
// DefaultReceiveBufferBytes,
|
||||
// requestTimeoutMs,
|
||||
// connectionSetupTimeoutMs,
|
||||
// connectionSetupTimeoutMaxMs,
|
||||
// ClientDnsLookup.USE_ALL_DNS_IPS,
|
||||
// time,
|
||||
// true,
|
||||
// new ApiVersions,
|
||||
// logContext)
|
||||
|
||||
val highLevelClient = new ConsumerNetworkClient(
|
||||
logContext,
|
||||
networkClient,
|
||||
metadata,
|
||||
time,
|
||||
retryBackoffMs,
|
||||
requestTimeoutMs,
|
||||
Integer.MAX_VALUE)
|
||||
|
||||
new AdminClient(
|
||||
time,
|
||||
highLevelClient,
|
||||
metadata.fetch.nodes.asScala.toList)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,12 +1,13 @@
|
||||
package kafka.console
|
||||
|
||||
import com.xuxd.kafka.console.beans.{BrokerNode, ClusterInfo}
|
||||
import com.xuxd.kafka.console.config.{ContextConfigHolder, KafkaConfig}
|
||||
import org.apache.kafka.clients.NodeApiVersions
|
||||
import org.apache.kafka.clients.admin.DescribeClusterResult
|
||||
import org.apache.kafka.common.Node
|
||||
|
||||
import java.util.Collections
|
||||
import java.util.concurrent.TimeUnit
|
||||
|
||||
import com.xuxd.kafka.console.beans.{BrokerNode, ClusterInfo}
|
||||
import com.xuxd.kafka.console.config.KafkaConfig
|
||||
import org.apache.kafka.clients.admin.DescribeClusterResult
|
||||
|
||||
import scala.jdk.CollectionConverters.{CollectionHasAsScala, SetHasAsJava, SetHasAsScala}
|
||||
|
||||
/**
|
||||
@@ -19,6 +20,7 @@ class ClusterConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaConf
|
||||
|
||||
def clusterInfo(): ClusterInfo = {
|
||||
withAdminClientAndCatchError(admin => {
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
val clusterResult: DescribeClusterResult = admin.describeCluster()
|
||||
val clusterInfo = new ClusterInfo
|
||||
clusterInfo.setClusterId(clusterResult.clusterId().get(timeoutMs, TimeUnit.MILLISECONDS))
|
||||
@@ -41,4 +43,8 @@ class ClusterConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaConf
|
||||
new ClusterInfo
|
||||
}).asInstanceOf[ClusterInfo]
|
||||
}
|
||||
|
||||
def listBrokerVersionInfo(): java.util.HashMap[Node, NodeApiVersions] = {
|
||||
BrokerApiVersion.listAllBrokerApiVersionInfo()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3,8 +3,7 @@ package kafka.console
|
||||
import java.util
|
||||
import java.util.Collections
|
||||
import java.util.concurrent.TimeUnit
|
||||
|
||||
import com.xuxd.kafka.console.config.KafkaConfig
|
||||
import com.xuxd.kafka.console.config.{ContextConfigHolder, KafkaConfig}
|
||||
import kafka.console.ConfigConsole.BrokerLoggerConfigType
|
||||
import kafka.server.ConfigType
|
||||
import org.apache.kafka.clients.admin.{AlterConfigOp, Config, ConfigEntry, DescribeConfigsOptions}
|
||||
@@ -69,6 +68,7 @@ class ConfigConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaConfi
|
||||
val configResource = new ConfigResource(getResourceTypeAndValidate(entityType, entityName), entityName)
|
||||
|
||||
val config = Map(configResource -> Collections.singletonList(new AlterConfigOp(entry, opType)).asInstanceOf[util.Collection[AlterConfigOp]])
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
admin.incrementalAlterConfigs(config.asJava).all().get(timeoutMs, TimeUnit.MILLISECONDS)
|
||||
(true, "")
|
||||
}, e => {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
package kafka.console
|
||||
|
||||
import com.xuxd.kafka.console.config.KafkaConfig
|
||||
import com.xuxd.kafka.console.config.{ContextConfigHolder, KafkaConfig}
|
||||
import org.apache.kafka.clients.admin.ListOffsetsResult.ListOffsetsResultInfo
|
||||
import org.apache.kafka.clients.admin._
|
||||
import org.apache.kafka.clients.consumer.{ConsumerConfig, OffsetAndMetadata, OffsetResetStrategy}
|
||||
@@ -75,6 +75,7 @@ class ConsumerConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaCon
|
||||
val endOffsets = commitOffsets.keySet.map { topicPartition =>
|
||||
topicPartition -> OffsetSpec.latest
|
||||
}.toMap
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
admin.listOffsets(endOffsets.asJava).all().get(timeoutMs, TimeUnit.MILLISECONDS)
|
||||
}, e => {
|
||||
log.error("listOffsets error.", e)
|
||||
@@ -166,6 +167,7 @@ class ConsumerConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaCon
|
||||
|
||||
def resetPartitionToTargetOffset(groupId: String, partition: TopicPartition, offset: Long): (Boolean, String) = {
|
||||
withAdminClientAndCatchError(admin => {
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
admin.alterConsumerGroupOffsets(groupId, Map(partition -> new OffsetAndMetadata(offset)).asJava).all().get(timeoutMs, TimeUnit.MILLISECONDS)
|
||||
(true, "")
|
||||
}, e => {
|
||||
@@ -178,7 +180,7 @@ class ConsumerConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaCon
|
||||
timestamp: java.lang.Long): (Boolean, String) = {
|
||||
withAdminClientAndCatchError(admin => {
|
||||
val logOffsets = getLogTimestampOffsets(admin, groupId, topicPartitions.asScala, timestamp)
|
||||
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
admin.alterConsumerGroupOffsets(groupId, logOffsets.asJava).all().get(timeoutMs, TimeUnit.MILLISECONDS)
|
||||
(true, "")
|
||||
}, e => {
|
||||
@@ -256,6 +258,7 @@ class ConsumerConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaCon
|
||||
val timestampOffsets = topicPartitions.map { topicPartition =>
|
||||
topicPartition -> OffsetSpec.forTimestamp(timestamp)
|
||||
}.toMap
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
val offsets = admin.listOffsets(
|
||||
timestampOffsets.asJava,
|
||||
new ListOffsetsOptions().timeoutMs(timeoutMs)
|
||||
@@ -280,6 +283,7 @@ class ConsumerConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaCon
|
||||
val endOffsets = topicPartitions.map { topicPartition =>
|
||||
topicPartition -> OffsetSpec.latest
|
||||
}.toMap
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
val offsets = admin.listOffsets(
|
||||
endOffsets.asJava,
|
||||
new ListOffsetsOptions().timeoutMs(timeoutMs)
|
||||
|
||||
@@ -3,9 +3,8 @@ package kafka.console
|
||||
import java.util
|
||||
import java.util.concurrent.TimeUnit
|
||||
import java.util.{Collections, List}
|
||||
|
||||
import com.xuxd.kafka.console.beans.AclEntry
|
||||
import com.xuxd.kafka.console.config.KafkaConfig
|
||||
import com.xuxd.kafka.console.config.{ContextConfigHolder, KafkaConfig}
|
||||
import org.apache.commons.lang3.StringUtils
|
||||
import org.apache.kafka.common.acl._
|
||||
import org.apache.kafka.common.resource.{ResourcePattern, ResourcePatternFilter, ResourceType}
|
||||
@@ -58,6 +57,7 @@ class KafkaAclConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaCon
|
||||
def addAcl(acls: List[AclBinding]): Boolean = {
|
||||
withAdminClient(adminClient => {
|
||||
try {
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
adminClient.createAcls(acls).all().get(timeoutMs, TimeUnit.MILLISECONDS)
|
||||
true
|
||||
} catch {
|
||||
@@ -100,6 +100,7 @@ class KafkaAclConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaCon
|
||||
def deleteAcl(entry: AclEntry, allResource: Boolean, allPrincipal: Boolean, allOperation: Boolean): Boolean = {
|
||||
withAdminClient(adminClient => {
|
||||
try {
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
val result = adminClient.deleteAcls(Collections.singleton(entry.toAclBindingFilter(allResource, allPrincipal, allOperation))).all().get(timeoutMs, TimeUnit.MILLISECONDS)
|
||||
log.info("delete acl: {}", result)
|
||||
true
|
||||
@@ -113,6 +114,7 @@ class KafkaAclConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaCon
|
||||
def deleteAcl(filters: util.Collection[AclBindingFilter]): Boolean = {
|
||||
withAdminClient(adminClient => {
|
||||
try {
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
val result = adminClient.deleteAcls(filters).all().get(timeoutMs, TimeUnit.MILLISECONDS)
|
||||
log.info("delete acl: {}", result)
|
||||
true
|
||||
|
||||
@@ -1,16 +1,16 @@
|
||||
package kafka.console
|
||||
|
||||
import com.xuxd.kafka.console.config.{ContextConfigHolder, KafkaConfig}
|
||||
import kafka.server.ConfigType
|
||||
import kafka.utils.Implicits.PropertiesOps
|
||||
import org.apache.kafka.clients.admin._
|
||||
import org.apache.kafka.common.config.SaslConfigs
|
||||
import org.apache.kafka.common.security.scram.internals.{ScramCredentialUtils, ScramFormatter}
|
||||
|
||||
import java.security.MessageDigest
|
||||
import java.util
|
||||
import java.util.concurrent.TimeUnit
|
||||
import java.util.{Properties, Set}
|
||||
|
||||
import com.xuxd.kafka.console.config.KafkaConfig
|
||||
import kafka.server.ConfigType
|
||||
import kafka.utils.Implicits.PropertiesOps
|
||||
import org.apache.kafka.clients.admin._
|
||||
import org.apache.kafka.common.security.scram.internals.{ScramCredentialUtils, ScramFormatter}
|
||||
|
||||
import scala.jdk.CollectionConverters.{CollectionHasAsScala, DictionaryHasAsScala, SeqHasAsJava}
|
||||
|
||||
/**
|
||||
@@ -35,31 +35,32 @@ class KafkaConfigConsole(config: KafkaConfig) extends KafkaConsole(config: Kafka
|
||||
}).asInstanceOf[util.Map[String, UserScramCredentialsDescription]]
|
||||
}
|
||||
|
||||
def addOrUpdateUser(name: String, pass: String): Boolean = {
|
||||
def addOrUpdateUser(name: String, pass: String): (Boolean, String) = {
|
||||
withAdminClient(adminClient => {
|
||||
try {
|
||||
adminClient.alterUserScramCredentials(util.Arrays.asList(
|
||||
new UserScramCredentialUpsertion(name,
|
||||
new ScramCredentialInfo(ScramMechanism.fromMechanismName(config.getSaslMechanism), defaultIterations), pass)))
|
||||
.all().get(timeoutMs, TimeUnit.MILLISECONDS)
|
||||
true
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
val mechanisms = ContextConfigHolder.CONTEXT_CONFIG.get().getProperties().getProperty(SaslConfigs.SASL_MECHANISM).split(",").toSeq
|
||||
val scrams = mechanisms.map(m => new UserScramCredentialUpsertion(name,
|
||||
new ScramCredentialInfo(ScramMechanism.fromMechanismName(m), defaultIterations), pass))
|
||||
adminClient.alterUserScramCredentials(scrams.asInstanceOf[Seq[UserScramCredentialAlteration]].asJava).all().get(timeoutMs, TimeUnit.MILLISECONDS)
|
||||
(true, "")
|
||||
} catch {
|
||||
case ex: Exception => log.error("addOrUpdateUser error", ex)
|
||||
false
|
||||
(false, ex.getMessage)
|
||||
}
|
||||
|
||||
}).asInstanceOf[Boolean]
|
||||
}).asInstanceOf[(Boolean, String)]
|
||||
}
|
||||
|
||||
def addOrUpdateUserWithZK(name: String, pass: String): Boolean = {
|
||||
withZKClient(adminZkClient => {
|
||||
try {
|
||||
val credential = new ScramFormatter(org.apache.kafka.common.security.scram.internals.ScramMechanism.forMechanismName(config.getSaslMechanism))
|
||||
val credential = new ScramFormatter(org.apache.kafka.common.security.scram.internals.ScramMechanism.forMechanismName(ContextConfigHolder.CONTEXT_CONFIG.get().getProperties().getProperty(SaslConfigs.SASL_MECHANISM)))
|
||||
.generateCredential(pass, defaultIterations)
|
||||
val credentialStr = ScramCredentialUtils.credentialToString(credential)
|
||||
|
||||
val userConfig: Properties = new Properties()
|
||||
userConfig.put(config.getSaslMechanism, credentialStr)
|
||||
userConfig.put(ContextConfigHolder.CONTEXT_CONFIG.get().getProperties().getProperty(SaslConfigs.SASL_MECHANISM), credentialStr)
|
||||
|
||||
val configs = adminZkClient.fetchEntityConfig(ConfigType.User, name)
|
||||
userConfig ++= configs
|
||||
@@ -101,6 +102,7 @@ class KafkaConfigConsole(config: KafkaConfig) extends KafkaConsole(config: Kafka
|
||||
// .all().get(timeoutMs, TimeUnit.MILLISECONDS)
|
||||
// all delete
|
||||
val userDetail = getUserDetailList(util.Collections.singletonList(name))
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
userDetail.values().asScala.foreach(u => {
|
||||
adminClient.alterUserScramCredentials(u.credentialInfos().asScala.map(s => new UserScramCredentialDeletion(u.name(), s.mechanism())
|
||||
.asInstanceOf[UserScramCredentialAlteration]).toList.asJava)
|
||||
|
||||
@@ -1,20 +1,21 @@
|
||||
package kafka.console
|
||||
|
||||
import com.xuxd.kafka.console.config.KafkaConfig
|
||||
import kafka.zk.{AdminZkClient, KafkaZkClient}
|
||||
import org.apache.kafka.clients.CommonClientConfigs
|
||||
import com.google.common.cache.{CacheLoader, RemovalListener, RemovalNotification}
|
||||
import com.xuxd.kafka.console.cache.TimeBasedCache
|
||||
import com.xuxd.kafka.console.config.{ContextConfigHolder, KafkaConfig}
|
||||
import kafka.zk.AdminZkClient
|
||||
import org.apache.kafka.clients.admin._
|
||||
import org.apache.kafka.clients.consumer.{ConsumerConfig, KafkaConsumer, OffsetAndMetadata}
|
||||
import org.apache.kafka.clients.producer.KafkaProducer
|
||||
import org.apache.kafka.common.TopicPartition
|
||||
import org.apache.kafka.common.config.SaslConfigs
|
||||
import org.apache.kafka.common.requests.ListOffsetsResponse
|
||||
import org.apache.kafka.common.serialization.{ByteArrayDeserializer, ByteArraySerializer, StringSerializer}
|
||||
import org.apache.kafka.common.utils.Time
|
||||
import org.slf4j.{Logger, LoggerFactory}
|
||||
|
||||
import java.util.Properties
|
||||
import java.util.concurrent.Executors
|
||||
import scala.collection.{Map, Seq}
|
||||
import scala.concurrent.{ExecutionContext, Future}
|
||||
import scala.jdk.CollectionConverters.{MapHasAsJava, MapHasAsScala}
|
||||
|
||||
/**
|
||||
@@ -25,15 +26,17 @@ import scala.jdk.CollectionConverters.{MapHasAsJava, MapHasAsScala}
|
||||
* */
|
||||
class KafkaConsole(config: KafkaConfig) {
|
||||
|
||||
protected val timeoutMs: Int = config.getRequestTimeoutMs
|
||||
// protected val timeoutMs: Int = config.getRequestTimeoutMs
|
||||
|
||||
protected def withAdminClient(f: Admin => Any): Any = {
|
||||
|
||||
val admin = createAdminClient()
|
||||
val admin = if (config.isCacheAdminConnection()) AdminCache.cache.get(ContextConfigHolder.CONTEXT_CONFIG.get().getBootstrapServer()) else createAdminClient()
|
||||
try {
|
||||
f(admin)
|
||||
} finally {
|
||||
admin.close()
|
||||
if (!config.isCacheAdminConnection) {
|
||||
admin.close()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -47,33 +50,40 @@ class KafkaConsole(config: KafkaConfig) {
|
||||
|
||||
protected def withConsumerAndCatchError(f: KafkaConsumer[Array[Byte], Array[Byte]] => Any, eh: Exception => Any,
|
||||
extra: Properties = new Properties()): Any = {
|
||||
val props = getProps()
|
||||
props.putAll(extra)
|
||||
props.put(ConsumerConfig.CLIENT_ID_CONFIG, String.valueOf(System.currentTimeMillis()))
|
||||
val consumer = new KafkaConsumer(props, new ByteArrayDeserializer, new ByteArrayDeserializer)
|
||||
// val props = getProps()
|
||||
// props.putAll(extra)
|
||||
// props.put(ConsumerConfig.CLIENT_ID_CONFIG, String.valueOf(System.currentTimeMillis()))
|
||||
// val consumer = new KafkaConsumer(props, new ByteArrayDeserializer, new ByteArrayDeserializer)
|
||||
ConsumerCache.setProperties(extra)
|
||||
val consumer = if (config.isCacheConsumerConnection) ConsumerCache.cache.get(ContextConfigHolder.CONTEXT_CONFIG.get().getBootstrapServer()) else KafkaConsole.createByteArrayKVConsumer(extra)
|
||||
|
||||
try {
|
||||
f(consumer)
|
||||
} catch {
|
||||
case er: Exception => eh(er)
|
||||
}
|
||||
finally {
|
||||
consumer.close()
|
||||
ConsumerCache.clearProperties()
|
||||
if (!config.isCacheConsumerConnection) {
|
||||
consumer.close()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
protected def withProducerAndCatchError(f: KafkaProducer[String, String] => Any, eh: Exception => Any,
|
||||
extra: Properties = new Properties()): Any = {
|
||||
val props = getProps()
|
||||
props.putAll(extra)
|
||||
props.put(ConsumerConfig.CLIENT_ID_CONFIG, String.valueOf(System.currentTimeMillis()))
|
||||
val producer = new KafkaProducer[String, String](props, new StringSerializer, new StringSerializer)
|
||||
ProducerCache.setProperties(extra)
|
||||
val producer = if (config.isCacheProducerConnection) ProducerCache.cache.get(ContextConfigHolder.CONTEXT_CONFIG.get().getBootstrapServer) else KafkaConsole.createProducer(extra)
|
||||
try {
|
||||
f(producer)
|
||||
} catch {
|
||||
case er: Exception => eh(er)
|
||||
}
|
||||
finally {
|
||||
producer.close()
|
||||
ProducerCache.clearProperties()
|
||||
if (!config.isCacheProducerConnection) {
|
||||
producer.close()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -81,7 +91,6 @@ class KafkaConsole(config: KafkaConfig) {
|
||||
extra: Properties = new Properties()): Any = {
|
||||
val props = getProps()
|
||||
props.putAll(extra)
|
||||
props.put(ConsumerConfig.CLIENT_ID_CONFIG, String.valueOf(System.currentTimeMillis()))
|
||||
val producer = new KafkaProducer[Array[Byte], Array[Byte]](props, new ByteArraySerializer, new ByteArraySerializer)
|
||||
try {
|
||||
f(producer)
|
||||
@@ -93,14 +102,17 @@ class KafkaConsole(config: KafkaConfig) {
|
||||
}
|
||||
}
|
||||
|
||||
@Deprecated
|
||||
protected def withZKClient(f: AdminZkClient => Any): Any = {
|
||||
val zkClient = KafkaZkClient(config.getZookeeperAddr, false, 30000, 30000, Int.MaxValue, Time.SYSTEM)
|
||||
val adminZkClient = new AdminZkClient(zkClient)
|
||||
try {
|
||||
f(adminZkClient)
|
||||
} finally {
|
||||
zkClient.close()
|
||||
}
|
||||
// val zkClient = KafkaZkClient(config.getZookeeperAddr, false, 30000, 30000, Int.MaxValue, Time.SYSTEM)
|
||||
// 3.x
|
||||
// val zkClient = KafkaZkClient(config.getZookeeperAddr, false, 30000, 30000, Int.MaxValue, Time.SYSTEM, new ZKClientConfig(), "KafkaZkClient")
|
||||
// val adminZkClient = new AdminZkClient(zkClient)
|
||||
// try {
|
||||
// f(adminZkClient)
|
||||
// } finally {
|
||||
// zkClient.close()
|
||||
// }
|
||||
}
|
||||
|
||||
protected def createAdminClient(props: Properties): Admin = {
|
||||
@@ -108,29 +120,52 @@ class KafkaConsole(config: KafkaConfig) {
|
||||
}
|
||||
|
||||
protected def withTimeoutMs[T <: AbstractOptions[T]](options: T) = {
|
||||
options.timeoutMs(timeoutMs)
|
||||
options.timeoutMs(ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs())
|
||||
}
|
||||
|
||||
private def createAdminClient(): Admin = {
|
||||
Admin.create(getProps())
|
||||
KafkaConsole.createAdminClient()
|
||||
}
|
||||
|
||||
private def getProps(): Properties = {
|
||||
val props: Properties = new Properties();
|
||||
props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, config.getBootstrapServer)
|
||||
props.put(AdminClientConfig.REQUEST_TIMEOUT_MS_CONFIG, config.getRequestTimeoutMs())
|
||||
if (config.isEnableAcl) {
|
||||
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, config.getSecurityProtocol())
|
||||
props.put(SaslConfigs.SASL_MECHANISM, config.getSaslMechanism())
|
||||
props.put(SaslConfigs.SASL_JAAS_CONFIG, config.getSaslJaasConfig())
|
||||
}
|
||||
props
|
||||
KafkaConsole.getProps()
|
||||
}
|
||||
}
|
||||
|
||||
object KafkaConsole {
|
||||
val log: Logger = LoggerFactory.getLogger(this.getClass)
|
||||
|
||||
def createAdminClient(): Admin = {
|
||||
Admin.create(getProps())
|
||||
}
|
||||
|
||||
def createByteArrayKVConsumer(extra: Properties) : KafkaConsumer[Array[Byte], Array[Byte]] = {
|
||||
val props = getProps()
|
||||
props.putAll(extra)
|
||||
props.put(ConsumerConfig.CLIENT_ID_CONFIG, String.valueOf(System.currentTimeMillis()))
|
||||
new KafkaConsumer(props, new ByteArrayDeserializer, new ByteArrayDeserializer)
|
||||
}
|
||||
|
||||
def createProducer(extra: Properties) : KafkaProducer[String, String] = {
|
||||
val props = getProps()
|
||||
props.putAll(extra)
|
||||
new KafkaProducer(props, new StringSerializer, new StringSerializer)
|
||||
}
|
||||
|
||||
def createByteArrayStringProducer(extra: Properties) : KafkaProducer[Array[Byte], Array[Byte]] = {
|
||||
val props = getProps()
|
||||
props.putAll(extra)
|
||||
new KafkaProducer(props, new ByteArraySerializer, new ByteArraySerializer)
|
||||
}
|
||||
|
||||
def getProps(): Properties = {
|
||||
val props: Properties = new Properties();
|
||||
props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, ContextConfigHolder.CONTEXT_CONFIG.get().getBootstrapServer())
|
||||
props.put(AdminClientConfig.REQUEST_TIMEOUT_MS_CONFIG, ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs())
|
||||
props.putAll(ContextConfigHolder.CONTEXT_CONFIG.get().getProperties())
|
||||
props
|
||||
}
|
||||
|
||||
def getCommittedOffsets(admin: Admin, groupId: String,
|
||||
timeoutMs: Integer): Map[TopicPartition, OffsetAndMetadata] = {
|
||||
admin.listConsumerGroupOffsets(
|
||||
@@ -180,4 +215,88 @@ object KafkaConsole {
|
||||
}.toMap
|
||||
res
|
||||
}
|
||||
implicit val ec = ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(2))
|
||||
}
|
||||
|
||||
object AdminCache {
|
||||
|
||||
private val log: Logger = LoggerFactory.getLogger(this.getClass)
|
||||
|
||||
private val cacheLoader = new CacheLoader[String, Admin] {
|
||||
override def load(key: String): Admin = KafkaConsole.createAdminClient()
|
||||
}
|
||||
|
||||
private val removeListener = new RemovalListener[String, Admin] {
|
||||
override def onRemoval(notification: RemovalNotification[String, Admin]): Unit = {
|
||||
Future {
|
||||
log.warn("Close expired admin connection: {}", notification.getKey)
|
||||
notification.getValue.close()
|
||||
log.warn("Close expired admin connection complete: {}", notification.getKey)
|
||||
}(KafkaConsole.ec)
|
||||
|
||||
}
|
||||
}
|
||||
val cache = new TimeBasedCache[String, Admin](cacheLoader, removeListener)
|
||||
}
|
||||
|
||||
object ConsumerCache {
|
||||
|
||||
private val log: Logger = LoggerFactory.getLogger(this.getClass)
|
||||
|
||||
private val threadLocal = new ThreadLocal[Properties]
|
||||
|
||||
private val cacheLoader = new CacheLoader[String, KafkaConsumer[Array[Byte], Array[Byte]]] {
|
||||
override def load(key: String): KafkaConsumer[Array[Byte], Array[Byte]] = KafkaConsole.createByteArrayKVConsumer(threadLocal.get())
|
||||
}
|
||||
|
||||
private val removeListener = new RemovalListener[String, KafkaConsumer[Array[Byte], Array[Byte]]] {
|
||||
override def onRemoval(notification: RemovalNotification[String, KafkaConsumer[Array[Byte], Array[Byte]]]): Unit = {
|
||||
Future {
|
||||
log.warn("Close expired consumer connection: {}", notification.getKey)
|
||||
notification.getValue.close()
|
||||
log.warn("Close expired consumer connection complete: {}", notification.getKey)
|
||||
}(KafkaConsole.ec)
|
||||
|
||||
}
|
||||
}
|
||||
val cache = new TimeBasedCache[String, KafkaConsumer[Array[Byte], Array[Byte]]](cacheLoader, removeListener)
|
||||
|
||||
def setProperties(props : Properties) : Unit = {
|
||||
threadLocal.set(props)
|
||||
}
|
||||
|
||||
def clearProperties() : Unit = {
|
||||
threadLocal.remove()
|
||||
}
|
||||
}
|
||||
|
||||
object ProducerCache {
|
||||
|
||||
private val log: Logger = LoggerFactory.getLogger(this.getClass)
|
||||
|
||||
private val threadLocal = new ThreadLocal[Properties]
|
||||
|
||||
private val cacheLoader = new CacheLoader[String, KafkaProducer[String, String]] {
|
||||
override def load(key: String): KafkaProducer[String, String] = KafkaConsole.createProducer(threadLocal.get())
|
||||
}
|
||||
|
||||
private val removeListener = new RemovalListener[String, KafkaProducer[String, String]] {
|
||||
override def onRemoval(notification: RemovalNotification[String, KafkaProducer[String, String]]): Unit = {
|
||||
Future {
|
||||
log.warn("Close expired producer connection: {}", notification.getKey)
|
||||
notification.getValue.close()
|
||||
log.warn("Close expired producer connection complete: {}", notification.getKey)
|
||||
}(KafkaConsole.ec)
|
||||
|
||||
}
|
||||
}
|
||||
val cache = new TimeBasedCache[String, KafkaProducer[String, String]](cacheLoader, removeListener)
|
||||
|
||||
def setProperties(props : Properties) : Unit = {
|
||||
threadLocal.set(props)
|
||||
}
|
||||
|
||||
def clearProperties() : Unit = {
|
||||
threadLocal.remove()
|
||||
}
|
||||
}
|
||||
@@ -1,13 +1,17 @@
|
||||
package kafka.console
|
||||
|
||||
import com.xuxd.kafka.console.config.KafkaConfig
|
||||
import com.xuxd.kafka.console.beans.MessageFilter
|
||||
import com.xuxd.kafka.console.beans.enums.FilterType
|
||||
import com.xuxd.kafka.console.config.{ContextConfigHolder, KafkaConfig}
|
||||
import org.apache.commons.lang3.StringUtils
|
||||
import org.apache.kafka.clients.admin.{DeleteRecordsOptions, RecordsToDelete}
|
||||
import org.apache.kafka.clients.consumer.{ConsumerConfig, ConsumerRecord}
|
||||
import org.apache.kafka.clients.producer.ProducerRecord
|
||||
import org.apache.kafka.common.TopicPartition
|
||||
|
||||
import java.time.Duration
|
||||
import java.util
|
||||
import java.util.Properties
|
||||
import java.util.{Properties}
|
||||
import scala.collection.immutable
|
||||
import scala.jdk.CollectionConverters.{CollectionHasAsScala, MapHasAsScala, SeqHasAsJava}
|
||||
|
||||
@@ -20,10 +24,11 @@ import scala.jdk.CollectionConverters.{CollectionHasAsScala, MapHasAsScala, SeqH
|
||||
class MessageConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaConfig) with Logging {
|
||||
|
||||
def searchBy(partitions: util.Collection[TopicPartition], startTime: Long, endTime: Long,
|
||||
maxNums: Int): util.List[ConsumerRecord[Array[Byte], Array[Byte]]] = {
|
||||
maxNums: Int, filter: MessageFilter): (util.List[ConsumerRecord[Array[Byte], Array[Byte]]], Int) = {
|
||||
var startOffTable: immutable.Map[TopicPartition, Long] = Map.empty
|
||||
var endOffTable: immutable.Map[TopicPartition, Long] = Map.empty
|
||||
withAdminClientAndCatchError(admin => {
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
val startTable = KafkaConsole.getLogTimestampOffsets(admin, partitions.asScala.toSeq, startTime, timeoutMs)
|
||||
startOffTable = startTable.map(t2 => (t2._1, t2._2.offset())).toMap
|
||||
|
||||
@@ -33,8 +38,44 @@ class MessageConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaConf
|
||||
log.error("getLogTimestampOffsets error.", e)
|
||||
throw new RuntimeException("getLogTimestampOffsets error", e)
|
||||
})
|
||||
|
||||
val headerValueBytes = if (StringUtils.isNotEmpty(filter.getHeaderValue())) filter.getHeaderValue().getBytes() else None
|
||||
|
||||
def filterMessage(record: ConsumerRecord[Array[Byte], Array[Byte]]): Boolean = {
|
||||
filter.getFilterType() match {
|
||||
case FilterType.BODY => {
|
||||
val body = filter.getDeserializer().deserialize(record.topic(), record.value())
|
||||
var contains = false
|
||||
if (filter.isContainsValue) {
|
||||
contains = body.asInstanceOf[String].contains(filter.getSearchContent().asInstanceOf[String])
|
||||
} else {
|
||||
contains = body.equals(filter.getSearchContent)
|
||||
}
|
||||
contains
|
||||
}
|
||||
case FilterType.HEADER => {
|
||||
if (StringUtils.isNotEmpty(filter.getHeaderKey()) && StringUtils.isNotEmpty(filter.getHeaderValue())) {
|
||||
val iterator = record.headers().headers(filter.getHeaderKey()).iterator()
|
||||
var contains = false
|
||||
while (iterator.hasNext() && !contains) {
|
||||
val next = iterator.next().value()
|
||||
contains = (next.sameElements(headerValueBytes.asInstanceOf[Array[Byte]]))
|
||||
}
|
||||
contains
|
||||
} else if (StringUtils.isNotEmpty(filter.getHeaderKey()) && StringUtils.isEmpty(filter.getHeaderValue())) {
|
||||
record.headers().headers(filter.getHeaderKey()).iterator().hasNext()
|
||||
} else {
|
||||
true
|
||||
}
|
||||
}
|
||||
case FilterType.NONE => true
|
||||
}
|
||||
}
|
||||
|
||||
var terminate: Boolean = (startOffTable == endOffTable)
|
||||
val res = new util.LinkedList[ConsumerRecord[Array[Byte], Array[Byte]]]()
|
||||
// 检索的消息条数
|
||||
var searchNums = 0
|
||||
// 如果最小和最大偏移一致,就结束
|
||||
if (!terminate) {
|
||||
|
||||
@@ -51,9 +92,10 @@ class MessageConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaConf
|
||||
// 1.所有查询分区达都到最大偏移的时候
|
||||
while (!terminate) {
|
||||
// 达到查询的最大条数
|
||||
if (res.size() >= maxNums) {
|
||||
if (searchNums >= maxNums) {
|
||||
terminate = true
|
||||
} else {
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
val records = consumer.poll(Duration.ofMillis(timeoutMs))
|
||||
|
||||
if (records.isEmpty) {
|
||||
@@ -67,6 +109,7 @@ class MessageConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaConf
|
||||
if (first.offset() >= endOff) {
|
||||
arrive.remove(tp)
|
||||
} else {
|
||||
searchNums += recordList.size()
|
||||
//
|
||||
// (String topic,
|
||||
// int partition,
|
||||
@@ -80,12 +123,12 @@ class MessageConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaConf
|
||||
// V value,
|
||||
// Headers headers,
|
||||
// Optional<Integer> leaderEpoch)
|
||||
val nullVList = recordList.asScala.map(record => new ConsumerRecord[Array[Byte], Array[Byte]](record.topic(),
|
||||
val nullVList = recordList.asScala.filter(filterMessage(_)).map(record => new ConsumerRecord[Array[Byte], Array[Byte]](record.topic(),
|
||||
record.partition(),
|
||||
record.offset(),
|
||||
record.timestamp(),
|
||||
record.timestampType(),
|
||||
record.checksum(),
|
||||
// record.checksum(),
|
||||
record.serializedKeySize(),
|
||||
record.serializedValueSize(),
|
||||
record.key(),
|
||||
@@ -114,7 +157,7 @@ class MessageConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaConf
|
||||
})
|
||||
}
|
||||
|
||||
res
|
||||
(res, searchNums)
|
||||
}
|
||||
|
||||
def searchBy(
|
||||
@@ -149,6 +192,7 @@ class MessageConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaConf
|
||||
|
||||
var terminate = tpSet.isEmpty
|
||||
while (!terminate) {
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
val records = consumer.poll(Duration.ofMillis(timeoutMs))
|
||||
val tps = new util.HashSet(tpSet).asScala
|
||||
for (tp <- tps) {
|
||||
@@ -193,4 +237,14 @@ class MessageConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaConf
|
||||
(false, e.getMessage)
|
||||
}).asInstanceOf[(Boolean, String)]
|
||||
}
|
||||
|
||||
def delete(recordsToDelete: util.Map[TopicPartition, RecordsToDelete]): (Boolean, String) = {
|
||||
withAdminClientAndCatchError(admin => {
|
||||
admin.deleteRecords(recordsToDelete, withTimeoutMs(new DeleteRecordsOptions())).all().get()
|
||||
(true, "")
|
||||
}, e => {
|
||||
log.error("delete message error.", e)
|
||||
(false, "delete error :" + e.getMessage)
|
||||
}).asInstanceOf[(Boolean, String)]
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
package kafka.console
|
||||
|
||||
import com.xuxd.kafka.console.config.KafkaConfig
|
||||
import com.xuxd.kafka.console.config.{ContextConfigHolder, KafkaConfig}
|
||||
import kafka.admin.ReassignPartitionsCommand
|
||||
import org.apache.kafka.clients.admin.{ElectLeadersOptions, ListPartitionReassignmentsOptions, PartitionReassignment}
|
||||
import org.apache.kafka.clients.consumer.KafkaConsumer
|
||||
@@ -34,6 +34,7 @@ class OperationConsole(config: KafkaConfig, topicConsole: TopicConsole,
|
||||
throw new UnsupportedOperationException("exist consumer client.")
|
||||
}
|
||||
}
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
val thatGroupDescriptionList = thatAdmin.describeConsumerGroups(searchGroupIds).all().get(timeoutMs, TimeUnit.MILLISECONDS).values()
|
||||
if (groupDescriptionList.isEmpty) {
|
||||
throw new IllegalArgumentException("that consumer group info is null.")
|
||||
@@ -101,6 +102,7 @@ class OperationConsole(config: KafkaConfig, topicConsole: TopicConsole,
|
||||
thatMinOffset: util.Map[TopicPartition, Long]): (Boolean, String) = {
|
||||
val thatAdmin = createAdminClient(props)
|
||||
try {
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
val searchGroupIds = Collections.singleton(groupId)
|
||||
val groupDescriptionList = consumerConsole.getConsumerGroupList(searchGroupIds)
|
||||
if (groupDescriptionList.isEmpty) {
|
||||
@@ -178,6 +180,7 @@ class OperationConsole(config: KafkaConfig, topicConsole: TopicConsole,
|
||||
val thatConsumer = new KafkaConsumer(props, new ByteArrayDeserializer, new ByteArrayDeserializer)
|
||||
|
||||
try {
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
val thisTopicPartitions = consumerConsole.listSubscribeTopics(groupId).get(topic).asScala.sortBy(_.partition())
|
||||
val thatTopicPartitionMap = thatAdmin.listConsumerGroupOffsets(
|
||||
groupId
|
||||
@@ -239,8 +242,8 @@ class OperationConsole(config: KafkaConfig, topicConsole: TopicConsole,
|
||||
withAdminClientAndCatchError(admin => {
|
||||
admin.listPartitionReassignments(withTimeoutMs(new ListPartitionReassignmentsOptions)).reassignments().get()
|
||||
}, e => {
|
||||
Collections.emptyMap()
|
||||
log.error("listPartitionReassignments error.", e)
|
||||
Collections.emptyMap()
|
||||
}).asInstanceOf[util.Map[TopicPartition, PartitionReassignment]]
|
||||
}
|
||||
|
||||
@@ -253,4 +256,20 @@ class OperationConsole(config: KafkaConfig, topicConsole: TopicConsole,
|
||||
throw e
|
||||
}).asInstanceOf[util.Map[TopicPartition, Throwable]]
|
||||
}
|
||||
|
||||
def proposedAssignments(reassignmentJson: String,
|
||||
brokerListString: String): util.Map[TopicPartition, util.List[Int]] = {
|
||||
withAdminClientAndCatchError(admin => {
|
||||
val map = ReassignPartitionsCommand.generateAssignment(admin, reassignmentJson, brokerListString, true)._1
|
||||
val res = new util.HashMap[TopicPartition, util.List[Int]]()
|
||||
for (tp <- map.keys) {
|
||||
res.put(tp, map(tp).asJava)
|
||||
// res.put(tp, map.getOrElse(tp, Seq.empty).asJava)
|
||||
}
|
||||
res
|
||||
}, e => {
|
||||
log.error("proposedAssignments error.", e)
|
||||
throw e
|
||||
})
|
||||
}.asInstanceOf[util.Map[TopicPartition, util.List[Int]]]
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
package kafka.console
|
||||
|
||||
import com.xuxd.kafka.console.config.KafkaConfig
|
||||
import com.xuxd.kafka.console.config.{ContextConfigHolder, KafkaConfig}
|
||||
import kafka.admin.ReassignPartitionsCommand._
|
||||
import kafka.utils.Json
|
||||
import org.apache.kafka.clients.admin._
|
||||
@@ -28,6 +28,7 @@ class TopicConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaConfig
|
||||
* @return all topic name set.
|
||||
*/
|
||||
def getTopicNameList(internal: Boolean = true): Set[String] = {
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
withAdminClientAndCatchError(admin => admin.listTopics(new ListTopicsOptions().listInternal(internal)).names()
|
||||
.get(timeoutMs, TimeUnit.MILLISECONDS),
|
||||
e => {
|
||||
@@ -42,6 +43,7 @@ class TopicConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaConfig
|
||||
* @return internal topic name set.
|
||||
*/
|
||||
def getInternalTopicNameList(): Set[String] = {
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
withAdminClientAndCatchError(admin => admin.listTopics(new ListTopicsOptions().listInternal(true)).listings()
|
||||
.get(timeoutMs, TimeUnit.MILLISECONDS).asScala.filter(_.isInternal).map(_.name()).toSet[String].asJava,
|
||||
e => {
|
||||
@@ -64,16 +66,17 @@ class TopicConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaConfig
|
||||
/**
|
||||
* delete topic by topic name.
|
||||
*
|
||||
* @param topic topic name.
|
||||
* @param topics topic name list.
|
||||
* @return result or : fail message.
|
||||
*/
|
||||
def deleteTopic(topic: String): (Boolean, String) = {
|
||||
def deleteTopics(topics: util.Collection[String]): (Boolean, String) = {
|
||||
withAdminClientAndCatchError(admin => {
|
||||
admin.deleteTopics(Collections.singleton(topic), new DeleteTopicsOptions().retryOnQuotaViolation(false)).all().get(timeoutMs, TimeUnit.MILLISECONDS)
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
admin.deleteTopics(topics, new DeleteTopicsOptions().retryOnQuotaViolation(false)).all().get(timeoutMs, TimeUnit.MILLISECONDS)
|
||||
(true, "")
|
||||
},
|
||||
e => {
|
||||
log.error("delete topic error, topic: " + topic, e)
|
||||
log.error("delete topic error, topic: " + topics, e)
|
||||
(false, e.getMessage)
|
||||
}).asInstanceOf[(Boolean, String)]
|
||||
}
|
||||
@@ -103,6 +106,7 @@ class TopicConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaConfig
|
||||
*/
|
||||
def createTopic(topic: NewTopic): (Boolean, String) = {
|
||||
withAdminClientAndCatchError(admin => {
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
val createResult = admin.createTopics(Collections.singleton(topic), new CreateTopicsOptions().retryOnQuotaViolation(false))
|
||||
createResult.all().get(timeoutMs, TimeUnit.MILLISECONDS)
|
||||
(true, "")
|
||||
@@ -117,6 +121,7 @@ class TopicConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaConfig
|
||||
*/
|
||||
def createPartitions(newPartitions: util.Map[String, NewPartitions]): (Boolean, String) = {
|
||||
withAdminClientAndCatchError(admin => {
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
admin.createPartitions(newPartitions,
|
||||
new CreatePartitionsOptions().retryOnQuotaViolation(false)).all().get(timeoutMs, TimeUnit.MILLISECONDS)
|
||||
(true, "")
|
||||
@@ -241,6 +246,7 @@ class TopicConsole(config: KafkaConfig) extends KafkaConsole(config: KafkaConfig
|
||||
.asScala.map(info => new TopicPartition(topic, info.partition())).toSeq
|
||||
case None => throw new IllegalArgumentException("topic is not exist.")
|
||||
}
|
||||
val timeoutMs = ContextConfigHolder.CONTEXT_CONFIG.get().getRequestTimeoutMs()
|
||||
val offsetMap = KafkaConsole.getLogTimestampOffsets(admin, partitions, timestamp, timeoutMs)
|
||||
offsetMap.map(tuple2 => (tuple2._1, tuple2._2.offset())).toMap.asJava
|
||||
}, e => {
|
||||
|
||||
160
ui/package-lock.json
generated
@@ -1820,63 +1820,6 @@
|
||||
"integrity": "sha1-/q7SVZc9LndVW4PbwIhRpsY1IPo=",
|
||||
"dev": true
|
||||
},
|
||||
"ansi-styles": {
|
||||
"version": "4.3.0",
|
||||
"resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz",
|
||||
"integrity": "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==",
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"requires": {
|
||||
"color-convert": "^2.0.1"
|
||||
}
|
||||
},
|
||||
"chalk": {
|
||||
"version": "4.1.2",
|
||||
"resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.2.tgz",
|
||||
"integrity": "sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==",
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"requires": {
|
||||
"ansi-styles": "^4.1.0",
|
||||
"supports-color": "^7.1.0"
|
||||
}
|
||||
},
|
||||
"color-convert": {
|
||||
"version": "2.0.1",
|
||||
"resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz",
|
||||
"integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==",
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"requires": {
|
||||
"color-name": "~1.1.4"
|
||||
}
|
||||
},
|
||||
"color-name": {
|
||||
"version": "1.1.4",
|
||||
"resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz",
|
||||
"integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==",
|
||||
"dev": true,
|
||||
"optional": true
|
||||
},
|
||||
"has-flag": {
|
||||
"version": "4.0.0",
|
||||
"resolved": "https://registry.npmjs.org/has-flag/-/has-flag-4.0.0.tgz",
|
||||
"integrity": "sha512-EykJT/Q1KjTWctppgIAgfSO0tKVuZUjhgMr17kqTumMl6Afv3EISleU7qZUzoXDFTAHTDC4NOoG/ZxU3EvlMPQ==",
|
||||
"dev": true,
|
||||
"optional": true
|
||||
},
|
||||
"loader-utils": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/loader-utils/-/loader-utils-2.0.0.tgz",
|
||||
"integrity": "sha512-rP4F0h2RaWSvPEkD7BLDFQnvSf+nK+wr3ESUjNTyAGobqrijmW92zc+SO6d4p4B1wh7+B/Jg1mkQe5NYUEHtHQ==",
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"requires": {
|
||||
"big.js": "^5.2.2",
|
||||
"emojis-list": "^3.0.0",
|
||||
"json5": "^2.1.2"
|
||||
}
|
||||
},
|
||||
"ssri": {
|
||||
"version": "8.0.1",
|
||||
"resolved": "https://registry.nlark.com/ssri/download/ssri-8.0.1.tgz",
|
||||
@@ -1885,28 +1828,6 @@
|
||||
"requires": {
|
||||
"minipass": "^3.1.1"
|
||||
}
|
||||
},
|
||||
"supports-color": {
|
||||
"version": "7.2.0",
|
||||
"resolved": "https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz",
|
||||
"integrity": "sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==",
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"requires": {
|
||||
"has-flag": "^4.0.0"
|
||||
}
|
||||
},
|
||||
"vue-loader-v16": {
|
||||
"version": "npm:vue-loader@16.5.0",
|
||||
"resolved": "https://registry.npmjs.org/vue-loader/-/vue-loader-16.5.0.tgz",
|
||||
"integrity": "sha512-WXh+7AgFxGTgb5QAkQtFeUcHNIEq3PGVQ8WskY5ZiFbWBkOwcCPRs4w/2tVyTbh2q6TVRlO3xfvIukUtjsu62A==",
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"requires": {
|
||||
"chalk": "^4.1.0",
|
||||
"hash-sum": "^2.0.0",
|
||||
"loader-utils": "^2.0.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -12176,6 +12097,87 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"vue-loader-v16": {
|
||||
"version": "npm:vue-loader@16.8.3",
|
||||
"resolved": "https://registry.npmjs.org/vue-loader/-/vue-loader-16.8.3.tgz",
|
||||
"integrity": "sha512-7vKN45IxsKxe5GcVCbc2qFU5aWzyiLrYJyUuMz4BQLKctCj/fmCa0w6fGiiQ2cLFetNcek1ppGJQDCup0c1hpA==",
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"requires": {
|
||||
"chalk": "^4.1.0",
|
||||
"hash-sum": "^2.0.0",
|
||||
"loader-utils": "^2.0.0"
|
||||
},
|
||||
"dependencies": {
|
||||
"ansi-styles": {
|
||||
"version": "4.3.0",
|
||||
"resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz",
|
||||
"integrity": "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==",
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"requires": {
|
||||
"color-convert": "^2.0.1"
|
||||
}
|
||||
},
|
||||
"chalk": {
|
||||
"version": "4.1.2",
|
||||
"resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.2.tgz",
|
||||
"integrity": "sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==",
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"requires": {
|
||||
"ansi-styles": "^4.1.0",
|
||||
"supports-color": "^7.1.0"
|
||||
}
|
||||
},
|
||||
"color-convert": {
|
||||
"version": "2.0.1",
|
||||
"resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz",
|
||||
"integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==",
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"requires": {
|
||||
"color-name": "~1.1.4"
|
||||
}
|
||||
},
|
||||
"color-name": {
|
||||
"version": "1.1.4",
|
||||
"resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz",
|
||||
"integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==",
|
||||
"dev": true,
|
||||
"optional": true
|
||||
},
|
||||
"has-flag": {
|
||||
"version": "4.0.0",
|
||||
"resolved": "https://registry.npmjs.org/has-flag/-/has-flag-4.0.0.tgz",
|
||||
"integrity": "sha512-EykJT/Q1KjTWctppgIAgfSO0tKVuZUjhgMr17kqTumMl6Afv3EISleU7qZUzoXDFTAHTDC4NOoG/ZxU3EvlMPQ==",
|
||||
"dev": true,
|
||||
"optional": true
|
||||
},
|
||||
"loader-utils": {
|
||||
"version": "2.0.2",
|
||||
"resolved": "https://registry.npmjs.org/loader-utils/-/loader-utils-2.0.2.tgz",
|
||||
"integrity": "sha512-TM57VeHptv569d/GKh6TAYdzKblwDNiumOdkFnejjD0XwTH87K90w3O7AiJRqdQoXygvi1VQTJTLGhJl7WqA7A==",
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"requires": {
|
||||
"big.js": "^5.2.2",
|
||||
"emojis-list": "^3.0.0",
|
||||
"json5": "^2.1.2"
|
||||
}
|
||||
},
|
||||
"supports-color": {
|
||||
"version": "7.2.0",
|
||||
"resolved": "https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz",
|
||||
"integrity": "sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==",
|
||||
"dev": true,
|
||||
"optional": true,
|
||||
"requires": {
|
||||
"has-flag": "^4.0.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"vue-ref": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npm.taobao.org/vue-ref/download/vue-ref-2.0.0.tgz",
|
||||
|
||||
|
Before Width: | Height: | Size: 4.2 KiB After Width: | Height: | Size: 5.4 KiB |
BIN
ui/public/vue.ico
Normal file
|
After Width: | Height: | Size: 4.2 KiB |
@@ -11,19 +11,24 @@
|
||||
><router-link to="/group-page" class="pad-l-r">消费组</router-link>
|
||||
<span>|</span
|
||||
><router-link to="/message-page" class="pad-l-r">消息</router-link>
|
||||
<span v-show="config.enableAcl">|</span
|
||||
><router-link to="/acl-page" class="pad-l-r" v-show="config.enableAcl"
|
||||
<span v-show="enableSasl">|</span
|
||||
><router-link to="/acl-page" class="pad-l-r" v-show="enableSasl"
|
||||
>Acl</router-link
|
||||
>
|
||||
<span>|</span
|
||||
><router-link to="/op-page" class="pad-l-r">运维</router-link>
|
||||
<span class="right">集群:{{ clusterName }}</span>
|
||||
</div>
|
||||
<router-view class="content" />
|
||||
</div>
|
||||
</template>
|
||||
<script>
|
||||
import { KafkaConfigApi } from "@/utils/api";
|
||||
import { KafkaClusterApi } from "@/utils/api";
|
||||
import request from "@/utils/request";
|
||||
import { mapMutations, mapState } from "vuex";
|
||||
import { getClusterInfo } from "@/utils/local-cache";
|
||||
import notification from "ant-design-vue/lib/notification";
|
||||
import { CLUSTER } from "@/store/mutation-types";
|
||||
|
||||
export default {
|
||||
data() {
|
||||
@@ -32,12 +37,35 @@ export default {
|
||||
};
|
||||
},
|
||||
created() {
|
||||
request({
|
||||
url: KafkaConfigApi.getConfig.url,
|
||||
method: KafkaConfigApi.getConfig.method,
|
||||
}).then((res) => {
|
||||
this.config = res.data;
|
||||
});
|
||||
const clusterInfo = getClusterInfo();
|
||||
if (!clusterInfo) {
|
||||
request({
|
||||
url: KafkaClusterApi.peekClusterInfo.url,
|
||||
method: KafkaClusterApi.peekClusterInfo.method,
|
||||
}).then((res) => {
|
||||
if (res.code == 0) {
|
||||
this.switchCluster(res.data);
|
||||
} else {
|
||||
notification.error({
|
||||
message: "error",
|
||||
description: res.msg,
|
||||
});
|
||||
}
|
||||
});
|
||||
} else {
|
||||
this.switchCluster(clusterInfo);
|
||||
}
|
||||
},
|
||||
computed: {
|
||||
...mapState({
|
||||
clusterName: (state) => state.clusterInfo.clusterName,
|
||||
enableSasl: (state) => state.clusterInfo.enableSasl,
|
||||
}),
|
||||
},
|
||||
methods: {
|
||||
...mapMutations({
|
||||
switchCluster: CLUSTER.SWITCH,
|
||||
}),
|
||||
},
|
||||
};
|
||||
</script>
|
||||
@@ -90,4 +118,10 @@ export default {
|
||||
top: 1%;
|
||||
position: absolute;
|
||||
}
|
||||
.right {
|
||||
float: right;
|
||||
right: 1%;
|
||||
top: 2%;
|
||||
position: absolute;
|
||||
}
|
||||
</style>
|
||||
|
||||
@@ -1,11 +1,34 @@
|
||||
import Vue from "vue";
|
||||
import Vuex from "vuex";
|
||||
import { CLUSTER } from "@/store/mutation-types";
|
||||
import { setClusterInfo } from "@/utils/local-cache";
|
||||
|
||||
Vue.use(Vuex);
|
||||
|
||||
export default new Vuex.Store({
|
||||
state: {},
|
||||
mutations: {},
|
||||
state: {
|
||||
clusterInfo: {
|
||||
id: undefined,
|
||||
clusterName: undefined,
|
||||
enableSasl: false,
|
||||
},
|
||||
},
|
||||
mutations: {
|
||||
[CLUSTER.SWITCH](state, clusterInfo) {
|
||||
state.clusterInfo.id = clusterInfo.id;
|
||||
state.clusterInfo.clusterName = clusterInfo.clusterName;
|
||||
let enableSasl = false;
|
||||
for (let p in clusterInfo.properties) {
|
||||
if (enableSasl) {
|
||||
break;
|
||||
}
|
||||
enableSasl =
|
||||
clusterInfo.properties[p].indexOf("security.protocol=SASL") != -1;
|
||||
}
|
||||
state.clusterInfo.enableSasl = enableSasl;
|
||||
setClusterInfo(clusterInfo);
|
||||
},
|
||||
},
|
||||
actions: {},
|
||||
modules: {},
|
||||
});
|
||||
|
||||
3
ui/src/store/mutation-types.js
Normal file
@@ -0,0 +1,3 @@
|
||||
export const CLUSTER = {
|
||||
SWITCH: "switchCluster",
|
||||
};
|
||||
@@ -51,7 +51,7 @@ export const KafkaAclApi = {
|
||||
|
||||
export const KafkaConfigApi = {
|
||||
getConfig: {
|
||||
url: "/config",
|
||||
url: "/config/console",
|
||||
method: "get",
|
||||
},
|
||||
getTopicConfig: {
|
||||
@@ -183,6 +183,30 @@ export const KafkaClusterApi = {
|
||||
url: "/cluster",
|
||||
method: "get",
|
||||
},
|
||||
getClusterInfoList: {
|
||||
url: "/cluster/info",
|
||||
method: "get",
|
||||
},
|
||||
addClusterInfo: {
|
||||
url: "/cluster/info",
|
||||
method: "post",
|
||||
},
|
||||
deleteClusterInfo: {
|
||||
url: "/cluster/info",
|
||||
method: "delete",
|
||||
},
|
||||
updateClusterInfo: {
|
||||
url: "/cluster/info",
|
||||
method: "put",
|
||||
},
|
||||
peekClusterInfo: {
|
||||
url: "/cluster/info/peek",
|
||||
method: "get",
|
||||
},
|
||||
getBrokerApiVersionInfo: {
|
||||
url: "/cluster/info/api/version",
|
||||
method: "get",
|
||||
},
|
||||
};
|
||||
|
||||
export const KafkaOpApi = {
|
||||
@@ -222,6 +246,10 @@ export const KafkaOpApi = {
|
||||
url: "/op/replication/reassignments",
|
||||
method: "delete",
|
||||
},
|
||||
proposedAssignment: {
|
||||
url: "/op/replication/reassignments/proposed",
|
||||
method: "post",
|
||||
},
|
||||
};
|
||||
export const KafkaMessageApi = {
|
||||
searchByTime: {
|
||||
@@ -248,4 +276,8 @@ export const KafkaMessageApi = {
|
||||
url: "/message/resend",
|
||||
method: "post",
|
||||
},
|
||||
delete: {
|
||||
url: "/message",
|
||||
method: "delete",
|
||||
},
|
||||
};
|
||||
|
||||
@@ -1,3 +1,7 @@
|
||||
export const ConstantEvent = {
|
||||
updateUserDialogData: "updateUserDialogData",
|
||||
};
|
||||
|
||||
export const Cache = {
|
||||
clusterInfo: "clusterInfo",
|
||||
};
|
||||
|
||||
10
ui/src/utils/local-cache.js
Normal file
@@ -0,0 +1,10 @@
|
||||
import { Cache } from "@/utils/constants";
|
||||
|
||||
export function setClusterInfo(clusterInfo) {
|
||||
localStorage.setItem(Cache.clusterInfo, JSON.stringify(clusterInfo));
|
||||
}
|
||||
|
||||
export function getClusterInfo() {
|
||||
const str = localStorage.getItem(Cache.clusterInfo);
|
||||
return str ? JSON.parse(str) : undefined;
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
import axios from "axios";
|
||||
import notification from "ant-design-vue/es/notification";
|
||||
import { VueAxios } from "./axios";
|
||||
import { getClusterInfo } from "@/utils/local-cache";
|
||||
|
||||
// 创建 axios 实例
|
||||
const request = axios.create({
|
||||
@@ -22,10 +23,14 @@ const errorHandler = (error) => {
|
||||
};
|
||||
|
||||
// request interceptor
|
||||
// request.interceptors.request.use(config => {
|
||||
//
|
||||
// return config
|
||||
// }, errorHandler)
|
||||
request.interceptors.request.use((config) => {
|
||||
const clusterInfo = getClusterInfo();
|
||||
if (clusterInfo) {
|
||||
config.headers["X-Cluster-Info-Id"] = clusterInfo.id;
|
||||
// config.headers["X-Cluster-Info-Name"] = encodeURIComponent(clusterInfo.clusterName);
|
||||
}
|
||||
return config;
|
||||
}, errorHandler);
|
||||
|
||||
// response interceptor
|
||||
request.interceptors.response.use((response) => {
|
||||
|
||||
@@ -1,32 +1,128 @@
|
||||
<template>
|
||||
<div class="home">
|
||||
<a-card title="kafka console 配置" style="width: 100%">
|
||||
<!-- <a slot="extra" href="#">more</a>-->
|
||||
<a-card title="控制台默认配置" class="card-style">
|
||||
<p v-for="(v, k) in config" :key="k">{{ k }}={{ v }}</p>
|
||||
</a-card>
|
||||
<p></p>
|
||||
<hr />
|
||||
<h3>kafka API 版本兼容性</h3>
|
||||
<a-spin :spinning="apiVersionInfoLoading">
|
||||
<a-table
|
||||
:columns="columns"
|
||||
:data-source="brokerApiVersionInfo"
|
||||
bordered
|
||||
row-key="brokerId"
|
||||
>
|
||||
<div slot="operation" slot-scope="record">
|
||||
<a-button
|
||||
size="small"
|
||||
href="javascript:;"
|
||||
class="operation-btn"
|
||||
@click="openApiVersionInfoDialog(record)"
|
||||
>详情
|
||||
</a-button>
|
||||
</div>
|
||||
</a-table>
|
||||
</a-spin>
|
||||
<VersionInfo
|
||||
:version-info="apiVersionInfo"
|
||||
:visible="showApiVersionInfoDialog"
|
||||
@closeApiVersionInfoDialog="closeApiVersionInfoDialog"
|
||||
>
|
||||
</VersionInfo>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script>
|
||||
// @ is an alias to /src
|
||||
import request from "@/utils/request";
|
||||
import { KafkaConfigApi } from "@/utils/api";
|
||||
import { KafkaConfigApi, KafkaClusterApi } from "@/utils/api";
|
||||
import notification from "ant-design-vue/lib/notification";
|
||||
import VersionInfo from "@/views/home/VersionInfo";
|
||||
export default {
|
||||
name: "Home",
|
||||
components: {},
|
||||
components: { VersionInfo },
|
||||
data() {
|
||||
return {
|
||||
config: {},
|
||||
columns,
|
||||
brokerApiVersionInfo: [],
|
||||
showApiVersionInfoDialog: false,
|
||||
apiVersionInfo: [],
|
||||
apiVersionInfoLoading: false,
|
||||
};
|
||||
},
|
||||
methods: {
|
||||
openApiVersionInfoDialog(record) {
|
||||
this.apiVersionInfo = record.versionInfo;
|
||||
this.showApiVersionInfoDialog = true;
|
||||
},
|
||||
closeApiVersionInfoDialog() {
|
||||
this.showApiVersionInfoDialog = false;
|
||||
},
|
||||
},
|
||||
|
||||
created() {
|
||||
request({
|
||||
url: KafkaConfigApi.getConfig.url,
|
||||
method: KafkaConfigApi.getConfig.method,
|
||||
}).then((res) => {
|
||||
this.config = res.data;
|
||||
if (res.code == 0) {
|
||||
this.config = res.data;
|
||||
} else {
|
||||
notification.error({
|
||||
message: "error",
|
||||
description: res.msg,
|
||||
});
|
||||
}
|
||||
});
|
||||
this.apiVersionInfoLoading = true;
|
||||
request({
|
||||
url: KafkaClusterApi.getBrokerApiVersionInfo.url,
|
||||
method: KafkaClusterApi.getBrokerApiVersionInfo.method,
|
||||
}).then((res) => {
|
||||
this.apiVersionInfoLoading = false;
|
||||
if (res.code == 0) {
|
||||
this.brokerApiVersionInfo = res.data;
|
||||
} else {
|
||||
notification.error({
|
||||
message: "error",
|
||||
description: res.msg,
|
||||
});
|
||||
}
|
||||
});
|
||||
},
|
||||
};
|
||||
const columns = [
|
||||
{
|
||||
title: "id",
|
||||
dataIndex: "brokerId",
|
||||
key: "brokerId",
|
||||
},
|
||||
{
|
||||
title: "地址",
|
||||
dataIndex: "host",
|
||||
key: "host",
|
||||
},
|
||||
{
|
||||
title: "支持的api数量",
|
||||
dataIndex: "supportNums",
|
||||
key: "supportNums",
|
||||
},
|
||||
{
|
||||
title: "不支持的api数量",
|
||||
dataIndex: "unSupportNums",
|
||||
key: "unSupportNums",
|
||||
},
|
||||
{
|
||||
title: "操作",
|
||||
key: "operation",
|
||||
scopedSlots: { customRender: "operation" },
|
||||
},
|
||||
];
|
||||
</script>
|
||||
<style scoped>
|
||||
.card-style {
|
||||
width: 100%;
|
||||
}
|
||||
</style>
|
||||
|
||||
@@ -232,11 +232,13 @@ export default {
|
||||
}
|
||||
},
|
||||
onDeleteUser(row) {
|
||||
this.loading = true;
|
||||
request({
|
||||
url: KafkaAclApi.deleteKafkaUser.url,
|
||||
method: KafkaAclApi.deleteKafkaUser.method,
|
||||
data: { username: row.username },
|
||||
}).then((res) => {
|
||||
this.loading = false;
|
||||
this.getAclList();
|
||||
if (res.code == 0) {
|
||||
this.$message.success(res.msg);
|
||||
|
||||
@@ -8,20 +8,22 @@
|
||||
:footer="null"
|
||||
@cancel="handleCancel"
|
||||
>
|
||||
<a-form :form="form" :label-col="{ span: 5 }" :wrapper-col="{ span: 12 }">
|
||||
<a-form-item label="用户名">
|
||||
<span>{{ user.username }}</span>
|
||||
</a-form-item>
|
||||
<a-form-item label="密码">
|
||||
<span>{{ user.password }}</span>
|
||||
</a-form-item>
|
||||
<a-form-item label="凭证信息">
|
||||
<span>{{ user.credentialInfos }}</span>
|
||||
</a-form-item>
|
||||
<a-form-item label="数据一致性说明">
|
||||
<strong>{{ user.consistencyDescription }}</strong>
|
||||
</a-form-item>
|
||||
</a-form>
|
||||
<a-spin :spinning="loading">
|
||||
<a-form :form="form" :label-col="{ span: 5 }" :wrapper-col="{ span: 12 }">
|
||||
<a-form-item label="用户名">
|
||||
<span>{{ user.username }}</span>
|
||||
</a-form-item>
|
||||
<a-form-item label="密码">
|
||||
<span>{{ user.password }}</span>
|
||||
</a-form-item>
|
||||
<a-form-item label="凭证信息">
|
||||
<span>{{ user.credentialInfos }}</span>
|
||||
</a-form-item>
|
||||
<a-form-item label="数据一致性说明">
|
||||
<strong>{{ user.consistencyDescription }}</strong>
|
||||
</a-form-item>
|
||||
</a-form>
|
||||
</a-spin>
|
||||
</a-modal>
|
||||
</template>
|
||||
|
||||
@@ -47,6 +49,7 @@ export default {
|
||||
show: this.visible,
|
||||
form: this.$form.createForm(this, { name: "UserDetailForm" }),
|
||||
user: {},
|
||||
loading: false,
|
||||
};
|
||||
},
|
||||
watch: {
|
||||
@@ -63,11 +66,13 @@ export default {
|
||||
},
|
||||
getUserDetail() {
|
||||
const api = KafkaAclApi.getKafkaUserDetail;
|
||||
this.loading = true;
|
||||
request({
|
||||
url: api.url,
|
||||
method: api.method,
|
||||
params: { username: this.username },
|
||||
}).then((res) => {
|
||||
this.loading = false;
|
||||
if (res.code != 0) {
|
||||
this.$message.error(res.msg);
|
||||
} else {
|
||||
|
||||
@@ -45,6 +45,7 @@
|
||||
import request from "@/utils/request";
|
||||
import { KafkaClusterApi } from "@/utils/api";
|
||||
import BrokerConfig from "@/views/cluster/BrokerConfig";
|
||||
import notification from "ant-design-vue/lib/notification";
|
||||
|
||||
export default {
|
||||
name: "Topic",
|
||||
@@ -68,8 +69,15 @@ export default {
|
||||
method: KafkaClusterApi.getClusterInfo.method,
|
||||
}).then((res) => {
|
||||
this.loading = false;
|
||||
this.data = res.data.nodes;
|
||||
this.clusterId = res.data.clusterId;
|
||||
if (res.code == 0) {
|
||||
this.data = res.data.nodes;
|
||||
this.clusterId = res.data.clusterId;
|
||||
} else {
|
||||
notification.error({
|
||||
message: "error",
|
||||
description: res.msg,
|
||||
});
|
||||
}
|
||||
});
|
||||
},
|
||||
openBrokerConfigDialog(record, isLoggerConfig) {
|
||||
|
||||
@@ -196,7 +196,14 @@ export default {
|
||||
data: this.queryParam,
|
||||
}).then((res) => {
|
||||
this.loading = false;
|
||||
this.data = res.data.list;
|
||||
if (res.code == 0) {
|
||||
this.data = res.data.list;
|
||||
} else {
|
||||
notification.error({
|
||||
message: "error",
|
||||
description: res.msg,
|
||||
});
|
||||
}
|
||||
});
|
||||
},
|
||||
deleteGroup(group) {
|
||||
|
||||
61
ui/src/views/home/VersionInfo.vue
Normal file
@@ -0,0 +1,61 @@
|
||||
<template>
|
||||
<a-modal
|
||||
title="API版本信息"
|
||||
:visible="show"
|
||||
:width="600"
|
||||
:mask="false"
|
||||
:destroyOnClose="true"
|
||||
:footer="null"
|
||||
:maskClosable="false"
|
||||
@cancel="handleCancel"
|
||||
>
|
||||
<div>
|
||||
<h3>格式说明</h3>
|
||||
<p>请求类型(1):0 to n(2) [usage: v](3)</p>
|
||||
<ol>
|
||||
<li>表示客户端发出的请求类型</li>
|
||||
<li>该请求在broker中支持的版本号区间</li>
|
||||
<li>
|
||||
表示当前控制台的kafka客户端使用的是v版本,如果是UNSUPPORTED,说明broker版本太老,无法处理控制台的这些请求,可能影响相关功能的使用
|
||||
</li>
|
||||
</ol>
|
||||
|
||||
<hr />
|
||||
<ol>
|
||||
<li v-for="info in versionInfo" v-bind:key="info">{{ info }}</li>
|
||||
</ol>
|
||||
</div>
|
||||
</a-modal>
|
||||
</template>
|
||||
|
||||
<script>
|
||||
export default {
|
||||
name: "APIVersionInfo",
|
||||
props: {
|
||||
versionInfo: {
|
||||
type: Array,
|
||||
},
|
||||
visible: {
|
||||
type: Boolean,
|
||||
default: false,
|
||||
},
|
||||
},
|
||||
data() {
|
||||
return {
|
||||
show: false,
|
||||
};
|
||||
},
|
||||
watch: {
|
||||
visible(v) {
|
||||
this.show = v;
|
||||
},
|
||||
},
|
||||
methods: {
|
||||
handleCancel() {
|
||||
this.$emit("closeApiVersionInfoDialog", {});
|
||||
},
|
||||
},
|
||||
};
|
||||
</script>
|
||||
|
||||
<style scoped></style>
|
||||
196
ui/src/views/message/DeleteMessage.vue
Normal file
@@ -0,0 +1,196 @@
|
||||
<template>
|
||||
<div class="tab-content">
|
||||
<a-spin :spinning="loading">
|
||||
<div>
|
||||
<h4 class="hint-content">
|
||||
注意:以下删除,将删除该分区比该偏移位点小的所有消息(不包含该位点)
|
||||
</h4>
|
||||
<hr />
|
||||
</div>
|
||||
<div id="search-offset-form-advanced-search">
|
||||
<a-form
|
||||
class="ant-advanced-search-form"
|
||||
:form="form"
|
||||
@submit="handleSearch"
|
||||
>
|
||||
<a-row :gutter="24">
|
||||
<a-col :span="9">
|
||||
<a-form-item label="topic">
|
||||
<a-select
|
||||
class="topic-select"
|
||||
@change="handleTopicChange"
|
||||
show-search
|
||||
option-filter-prop="children"
|
||||
v-decorator="[
|
||||
'topic',
|
||||
{
|
||||
rules: [{ required: true, message: '请选择一个topic!' }],
|
||||
},
|
||||
]"
|
||||
placeholder="请选择一个topic"
|
||||
>
|
||||
<a-select-option v-for="v in topicList" :key="v" :value="v">
|
||||
{{ v }}
|
||||
</a-select-option>
|
||||
</a-select>
|
||||
</a-form-item>
|
||||
</a-col>
|
||||
<a-col :span="6">
|
||||
<a-form-item label="分区">
|
||||
<a-select
|
||||
class="type-select"
|
||||
show-search
|
||||
option-filter-prop="children"
|
||||
v-model="selectPartition"
|
||||
placeholder="请选择一个分区"
|
||||
>
|
||||
<a-select-option v-for="v in partitions" :key="v" :value="v">
|
||||
<span v-if="v == -1">全部</span> <span v-else>{{ v }}</span>
|
||||
</a-select-option>
|
||||
</a-select>
|
||||
</a-form-item>
|
||||
</a-col>
|
||||
<a-col :span="7">
|
||||
<a-form-item label="偏移">
|
||||
<a-input
|
||||
v-decorator="[
|
||||
'offset',
|
||||
{
|
||||
rules: [{ required: true, message: '请输入消息偏移!' }],
|
||||
},
|
||||
]"
|
||||
placeholder="消息偏移"
|
||||
/>
|
||||
</a-form-item>
|
||||
</a-col>
|
||||
<a-col :span="2" :style="{ textAlign: 'right' }">
|
||||
<a-form-item>
|
||||
<a-button type="primary" html-type="submit"> 执行删除</a-button>
|
||||
</a-form-item>
|
||||
</a-col>
|
||||
</a-row>
|
||||
</a-form>
|
||||
</div>
|
||||
</a-spin>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script>
|
||||
import request from "@/utils/request";
|
||||
import { KafkaMessageApi, KafkaTopicApi } from "@/utils/api";
|
||||
import notification from "ant-design-vue/lib/notification";
|
||||
|
||||
export default {
|
||||
name: "DeleteMessage",
|
||||
props: {
|
||||
topicList: {
|
||||
type: Array,
|
||||
},
|
||||
},
|
||||
data() {
|
||||
return {
|
||||
loading: false,
|
||||
form: this.$form.createForm(this, { name: "message_search_offset" }),
|
||||
partitions: [],
|
||||
selectPartition: undefined,
|
||||
};
|
||||
},
|
||||
methods: {
|
||||
handleSearch(e) {
|
||||
e.preventDefault();
|
||||
this.form.validateFields((err, values) => {
|
||||
if (!err) {
|
||||
const data = Object.assign({}, values, {
|
||||
partition: this.selectPartition,
|
||||
});
|
||||
this.loading = true;
|
||||
request({
|
||||
url: KafkaMessageApi.delete.url,
|
||||
method: KafkaMessageApi.delete.method,
|
||||
data: [data],
|
||||
}).then((res) => {
|
||||
this.loading = false;
|
||||
if (res.code == 0) {
|
||||
this.$message.success(res.msg);
|
||||
} else {
|
||||
notification.error({
|
||||
message: "error",
|
||||
description: res.msg,
|
||||
});
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
},
|
||||
getPartitionInfo(topic) {
|
||||
this.loading = true;
|
||||
request({
|
||||
url: KafkaTopicApi.getPartitionInfo.url + "?topic=" + topic,
|
||||
method: KafkaTopicApi.getPartitionInfo.method,
|
||||
}).then((res) => {
|
||||
this.loading = false;
|
||||
if (res.code != 0) {
|
||||
notification.error({
|
||||
message: "error",
|
||||
description: res.msg,
|
||||
});
|
||||
} else {
|
||||
this.partitions = res.data.map((v) => v.partition);
|
||||
}
|
||||
});
|
||||
},
|
||||
handleTopicChange(topic) {
|
||||
this.selectPartition =
|
||||
this.partitions.length > 0 ? this.partitions[0] : 0;
|
||||
this.getPartitionInfo(topic);
|
||||
},
|
||||
},
|
||||
};
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.tab-content {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
}
|
||||
|
||||
.ant-advanced-search-form {
|
||||
padding: 24px;
|
||||
background: #fbfbfb;
|
||||
border: 1px solid #d9d9d9;
|
||||
border-radius: 6px;
|
||||
}
|
||||
|
||||
.ant-advanced-search-form .ant-form-item {
|
||||
display: flex;
|
||||
}
|
||||
|
||||
.ant-advanced-search-form .ant-form-item-control-wrapper {
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
#components-form-topic-advanced-search .ant-form {
|
||||
max-width: none;
|
||||
margin-bottom: 1%;
|
||||
}
|
||||
|
||||
#search-offset-form-advanced-search .search-result-list {
|
||||
margin-top: 16px;
|
||||
border: 1px dashed #e9e9e9;
|
||||
border-radius: 6px;
|
||||
background-color: #fafafa;
|
||||
min-height: 200px;
|
||||
text-align: center;
|
||||
padding-top: 80px;
|
||||
}
|
||||
.topic-select {
|
||||
width: 400px !important;
|
||||
}
|
||||
.type-select {
|
||||
width: 200px !important;
|
||||
}
|
||||
|
||||
.hint-content {
|
||||
color: red;
|
||||
}
|
||||
</style>
|
||||
@@ -11,6 +11,9 @@
|
||||
<a-tab-pane key="3" tab="在线发送">
|
||||
<SendMessage :topic-list="topicList"></SendMessage>
|
||||
</a-tab-pane>
|
||||
<a-tab-pane key="4" tab="在线删除">
|
||||
<DeleteMessage :topic-list="topicList"></DeleteMessage>
|
||||
</a-tab-pane>
|
||||
</a-tabs>
|
||||
</a-spin>
|
||||
</div>
|
||||
@@ -23,9 +26,10 @@ import request from "@/utils/request";
|
||||
import { KafkaTopicApi } from "@/utils/api";
|
||||
import notification from "ant-design-vue/lib/notification";
|
||||
import SendMessage from "@/views/message/SendMessage";
|
||||
import DeleteMessage from "./DeleteMessage";
|
||||
export default {
|
||||
name: "Message",
|
||||
components: { SearchByTime, SearchByOffset, SendMessage },
|
||||
components: { DeleteMessage, SearchByTime, SearchByOffset, SendMessage },
|
||||
data() {
|
||||
return {
|
||||
loading: false,
|
||||
|
||||
@@ -203,7 +203,7 @@ export default {
|
||||
this.$emit("closeDetailDialog", { refresh: false });
|
||||
},
|
||||
formatTime(time) {
|
||||
return moment(time).format("YYYY-MM-DD HH:mm:ss:SSS");
|
||||
return time == -1 ? -1 : moment(time).format("YYYY-MM-DD HH:mm:ss:SSS");
|
||||
},
|
||||
keyDeserializerChange() {
|
||||
this.getMessageDetail();
|
||||
|
||||