Merge pull request #909 from Miracle778/fix-zhihu-specified-id-cdp

fix: 修复知乎 specified_id 和已有 Chrome CDP 连接问题

.env.example

@@ -0,0 +1,46 @@
+# MySQL Configuration
+MYSQL_DB_PWD=123456
+MYSQL_DB_USER=root
+MYSQL_DB_HOST=localhost
+MYSQL_DB_PORT=3306
+MYSQL_DB_NAME=media_crawler
+
+# Redis Configuration
+REDIS_DB_HOST=127.0.0.1
+REDIS_DB_PWD=123456
+REDIS_DB_PORT=6379
+REDIS_DB_NUM=0
+
+# MongoDB Configuration
+MONGODB_HOST=localhost
+MONGODB_PORT=27017
+MONGODB_USER=
+MONGODB_PWD=
+MONGODB_DB_NAME=media_crawler
+
+# PostgreSQL Configuration
+POSTGRES_DB_PWD=123456
+POSTGRES_DB_USER=postgres
+POSTGRES_DB_HOST=localhost
+POSTGRES_DB_PORT=5432
+POSTGRES_DB_NAME=media_crawler
+
+# Proxy Configuration (Wandou HTTP)
+# your_wandou_http_app_key
+WANDOU_APP_KEY=
+
+# Proxy Configuration (Kuaidaili)
+# your_kuaidaili_secret_id
+KDL_SECERT_ID=
+# your_kuaidaili_signature
+KDL_SIGNATURE=
+# your_kuaidaili_username
+KDL_USER_NAME=
+# your_kuaidaili_password
+KDL_USER_PWD=
+
+# Proxy Configuration (Jisu HTTP)
+# Get JiSu HTTP IP extraction key value
+jisu_key=
+# Get JiSu HTTP IP extraction encryption signature
+jisu_crypto=

.github/CODEOWNERS

@@ -0,0 +1,18 @@
+# 默认：仓库所有文件都需要 @NanmiCoder 审核
+*                                   @NanmiCoder
+
+
+.github/workflows/**               @NanmiCoder
+
+
+requirements.txt                   @NanmiCoder
+pyproject.toml                     @NanmiCoder
+Pipfile                            @NanmiCoder
+package.json                       @NanmiCoder
+package-lock.json                  @NanmiCoder
+pnpm-lock.yaml                     @NanmiCoder
+
+
+Dockerfile                         @NanmiCoder
+docker/**                          @NanmiCoder
+scripts/deploy/**                  @NanmiCoder

.github/FUNDING.yml

@@ -0,0 +1 @@
+github: NanmiCoder

.gitignore

@@ -178,4 +178,5 @@ docs/.vitepress/cache
 agent_zone
 debug_tools
 
-database/*.db
+database/*.db
+.omx/

.pre-commit-config.yaml

@@ -8,7 +8,7 @@ repos:
       # Python file header copyright check
       - id: check-file-headers
         name: Check Python file headers
-        entry: python tools/file_header_manager.py --check
+        entry: python3 tools/file_header_manager.py --check
         language: system
         types: [python]
         pass_filenames: true
@@ -17,7 +17,7 @@ repos:
       # Auto-fix Python file headers
       - id: add-file-headers
         name: Add copyright headers to Python files
-        entry: python tools/file_header_manager.py
+        entry: python3 tools/file_header_manager.py
         language: system
         types: [python]
         pass_filenames: true

README.md

@@ -1,19 +1,5 @@
 # 🔥 MediaCrawler - 自媒体平台爬虫 🕷️
 
-<div align="center" markdown="1">
-   <sup>Special thanks to:</sup>
-   <br>
-   <br>
-   <a href="https://go.warp.dev/MediaCrawler">
-      <img alt="Warp sponsorship" width="400" src="https://github.com/warpdotdev/brand-assets/blob/main/Github/Sponsor/Warp-Github-LG-02.png?raw=true">
-   </a>
-
-### [Warp is built for coding with multiple AI agents](https://go.warp.dev/MediaCrawler)
-
-
-</div>
-<hr>
-
 <div align="center">
 
 <a href="https://trendshift.io/repositories/8291" target="_blank">
@@ -53,6 +39,7 @@
 - **无需JS逆向**：利用保留登录态的浏览器上下文环境，通过 JS 表达式获取签名参数
 - **优势特点**：无需逆向复杂的加密算法，大幅降低技术门槛
 
+
 ## ✨ 功能特性
 | 平台   | 关键词搜索 | 指定帖子ID爬取 | 二级评论 | 指定创作者主页 | 登录态缓存 | IP代理池 | 生成评论词云图 |
 | ------ | ---------- | -------------- | -------- | -------------- | ---------- | -------- | -------------- |
@@ -66,13 +53,14 @@
 
 
 
-### 🚀 MediaCrawlerPro 重磅发布！
+<strong>MediaCrawlerPro 重磅发布！开源不易，欢迎订阅支持</strong>
 
 > 专注于学习成熟项目的架构设计，不仅仅是爬虫技术，Pro 版本的代码设计思路同样值得深入学习！
 
 [MediaCrawlerPro](https://github.com/MediaCrawlerPro) 相较于开源版本的核心优势：
 
 #### 🎯 核心功能升级
+- ✅ **自媒体内容拆解Agent**（新增功能）
 - ✅ **断点续爬功能**（重点特性）
 - ✅ **多账号 + IP代理池支持**（重点特性）
 - ✅ **去除 Playwright 依赖**，使用更简单
@@ -86,14 +74,16 @@
 #### 🎁 额外功能
 - ✅ **自媒体视频下载器桌面端**（适合学习全栈开发）
 - ✅ **多平台首页信息流推荐**（HomeFeed）
-- [ ] **基于自媒体平台的AI Agent正在开发中 🚀🚀**
+- ✅ **AI Agent Skill 支持**（[OpenClaw](https://openclaw.ai/) 🦞 / Claude Code / Cursor 一键安装，让 Agent 自动爬取数据）
+- [ ] **基于评论分析AI Agent正在开发中 🚀🚀**
 
 点击查看：[MediaCrawlerPro 项目主页](https://github.com/MediaCrawlerPro) 更多介绍
 
 
+
 ## 🚀 快速开始
 
-> 💡 **开源不易，如果这个项目对您有帮助，请给个 ⭐ Star 支持一下！**
+> 💡 **如果这个项目对您有帮助，请给个 ⭐ Star 支持一下！**
 
 ## 📋 前置依赖
 
@@ -122,22 +112,33 @@ cd MediaCrawler
 uv sync
 ```
 
-### 🌐 浏览器驱动安装
+### 🌐 浏览器驱动安装（可选）
+
+> 如果使用默认的 CDP 模式（连接已有 Chrome 浏览器），**无需安装浏览器驱动**。仅在使用标准 Playwright 模式时需要安装。
 
 ```shell
-# 安装浏览器驱动
+# 仅在标准 Playwright 模式下需要安装浏览器驱动
 uv run playwright install
 ```
 
-> **💡 提示**：MediaCrawler 目前已经支持使用 playwright 连接你本地的 Chrome 浏览器了，一些因为 Webdriver 导致的问题迎刃而解了。
+### 🌍 Chrome 浏览器配置（推荐）
+
+项目默认使用 CDP 模式连接用户已有的 Chrome 浏览器，可以复用浏览器已有的登录状态、Cookie、扩展等，**大幅降低平台风控检测风险**。
+
+使用前需要：
+
+1. **安装最新版 Chrome 浏览器**（版本 >= 144），[下载地址](https://www.google.com/chrome/)
+2. **开启远程调试功能**：在 Chrome 地址栏输入 `chrome://inspect/#remote-debugging`，勾选 **"Allow remote debugging for this browser instance"**
+3. 页面显示 `Server running at: 127.0.0.1:9222` 表示已就绪
+
+> 💡 **提示**：运行爬虫后，Chrome 浏览器会弹出确认对话框，点击"接受"即可。程序会等待用户确认，60秒内操作完成即可。
 >
-> 目前开放了 `xhs` 和 `dy` 这两个使用 CDP 的方式连接本地浏览器，如有需要，查看 `config/base_config.py` 中的配置项。
+> 如果不想使用 CDP 模式，可以在 `config/base_config.py` 中设置 `ENABLE_CDP_MODE = False` 切换为标准 Playwright 模式。
 
 ## 🚀 运行爬虫程序
 
 ```shell
-# 项目默认是没有开启评论爬取模式，如需评论请在 config/base_config.py 中的 ENABLE_GET_COMMENTS 变量修改
-# 一些其他支持项，也可以在 config/base_config.py 查看功能，写的有中文注释
+# 在 config/base_config.py 查看配置项目功能，写的有中文注释
 
 # 从配置文件中读取关键词搜索相关的帖子并爬取帖子信息与评论
 uv run main.py --platform xhs --lt qrcode --type search
@@ -151,6 +152,35 @@ uv run main.py --platform xhs --lt qrcode --type detail
 uv run main.py --help
 ```
 
+<details>
+<summary>🖥️ <strong>WebUI 可视化操作界面</strong></summary>
+
+MediaCrawler 提供了基于 Web 的可视化操作界面，无需命令行也能轻松使用爬虫功能。
+
+#### 启动 WebUI 服务
+
+```shell
+# 启动 API 服务器（默认端口 8080）
+uv run uvicorn api.main:app --port 8080 --reload
+
+# 或者使用模块方式启动
+uv run python -m api.main
+```
+
+启动成功后，访问 `http://localhost:8080` 即可打开 WebUI 界面。
+
+#### WebUI 功能特性
+
+- 可视化配置爬虫参数（平台、登录方式、爬取类型等）
+- 实时查看爬虫运行状态和日志
+- 数据预览和导出
+
+#### 界面预览
+
+<img src="docs/static/images/img_8.png" alt="WebUI 界面预览">
+
+</details>
+
 <details>
 <summary>🔗 <strong>使用 Python 原生 venv 管理环境（不推荐）</strong></summary>
 
@@ -209,87 +239,78 @@ python main.py --help
 
 ## 💾 数据保存
 
-支持多种数据存储方式：
-- **CSV 文件**：支持保存到 CSV 中（`data/` 目录下）
-- **JSON 文件**：支持保存到 JSON 中（`data/` 目录下）
-- **数据库存储**
-  - 使用参数 `--init_db` 进行数据库初始化（使用`--init_db`时不需要携带其他optional）
-  - **SQLite 数据库**：轻量级数据库，无需服务器，适合个人使用（推荐）
-    1. 初始化：`--init_db sqlite`
-    2. 数据存储：`--save_data_option sqlite`
-  - **MySQL 数据库**：支持关系型数据库 MySQL 中保存（需要提前创建数据库）
-    1. 初始化：`--init_db mysql`
-    2. 数据存储：`--save_data_option db`（db 参数为兼容历史更新保留）
+MediaCrawler 支持多种数据存储方式，包括 CSV、JSON、JSONL、Excel、SQLite 和 MySQL 数据库。
+
+📖 **详细使用说明请查看：[数据存储指南](docs/data_storage_guide.md)**
 
 
-### 使用示例：
-```shell
-# 初始化 SQLite 数据库（使用'--init_db'时不需要携带其他optional）
-uv run main.py --init_db sqlite
-# 使用 SQLite 存储数据（推荐个人用户使用）
-uv run main.py --platform xhs --lt qrcode --type search --save_data_option sqlite
-```
-```shell
-# 初始化 MySQL 数据库
-uv run main.py --init_db mysql
-# 使用 MySQL 存储数据（为适配历史更新，db参数进行沿用）
-uv run main.py --platform xhs --lt qrcode --type search --save_data_option db
-```
+[🚀 MediaCrawlerPro 重磅发布 🚀！更多的功能，更好的架构设计！开源不易，欢迎订阅支持！](https://github.com/MediaCrawlerPro)
 
 
-[🚀 MediaCrawlerPro 重磅发布 🚀！更多的功能，更好的架构设计！](https://github.com/MediaCrawlerPro)
-
-
-### 💬 交流群组
+## 💬 交流群组
 - **微信交流群**：[点击加入](https://nanmicoder.github.io/MediaCrawler/%E5%BE%AE%E4%BF%A1%E4%BA%A4%E6%B5%81%E7%BE%A4.html)
+- **B站账号**：[关注我](https://space.bilibili.com/434377496)，分享AI与爬虫技术知识
 
-### 📚 其他
-- **常见问题**：[MediaCrawler 完整文档](https://nanmicoder.github.io/MediaCrawler/)
-- **爬虫入门教程**：[CrawlerTutorial 免费教程](https://github.com/NanmiCoder/CrawlerTutorial)
-- **新闻爬虫开源项目**：[NewsCrawlerCollection](https://github.com/NanmiCoder/NewsCrawlerCollection)
----
 
-### 💰 赞助商展示
+## 💰 赞助商展示
 
-<a href="https://h.wandouip.com">
-<img src="docs/static/images/img_8.jpg">
+<a href="https://tikhub.io/?utm_source=github.com/NanmiCoder/MediaCrawler&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad">
+<img width="500" src="docs/static/images/tikhub_banner_zh.png">
 <br>
-豌豆HTTP自营千万级IP资源池，IP纯净度≥99.8%，每日保持IP高频更新，快速响应，稳定连接,满足多种业务场景，支持按需定制，注册免费提取10000ip。
+TikHub.io 提供 900+ 高稳定性数据接口，覆盖 TK、DY、XHS、Y2B、Ins、X 等 14+ 海内外主流平台，支持用户、内容、商品、评论等多维度公开数据 API，并配套 4000 万+ 已清洗结构化数据集，使用邀请码 <code>cfzyejV9</code> 注册并充值，即可额外获得 $2 赠送额度。
 </a>
+<br>
+<br>
+
+<a href="https://www.atlascloud.ai/?utm_source=github&utm_medium=link&utm_campaign=mei%27da%27c%27rmeidacrawler">
+<img width="500" alt="Atlas Cloud" src="docs/static/images/atlas_cloud_logo_black.png#gh-light-mode-only">
+<img width="500" alt="Atlas Cloud" src="docs/static/images/atlas_cloud_logo_white.png#gh-dark-mode-only">
+</a>
+<br>
+<a href="https://www.atlascloud.ai/?utm_source=github&utm_medium=link&utm_campaign=mei%27da%27c%27rmeidacrawler">Atlas Cloud</a> 是一个全模态 AI 推理平台，让开发者通过统一的 AI API 访问视频生成、图像生成和 LLM API，无需分别维护多个厂商集成，即可调用 300+ 精选模型。Atlas Cloud 最新推出 <a href="https://www.atlascloud.ai/console/coding-plan">coding plan 优惠</a>，为开发者提供更具性价比的 API 访问预算。
 
 ---
 
-<p align="center">
-  <a href="https://tikhub.io/?utm_source=github.com/NanmiCoder/MediaCrawler&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad">
-    <img style="border-radius:20px" width="500" alt="TikHub IO_Banner zh" src="docs/static/images/tikhub_banner_zh.png">
-  </a>
-</p>
-
-[TikHub](https://tikhub.io/?utm_source=github.com/NanmiCoder/MediaCrawler&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad) 提供超过 **700 个端点**，可用于从 **14+ 个社交媒体平台** 获取与分析数据 —— 包括视频、用户、评论、商店、商品与趋势等，一站式完成所有数据访问与分析。
-
-通过每日签到，可以获取免费额度。可以使用我的注册链接：[https://user.tikhub.io/users/signup?referral_code=cfzyejV9](https://user.tikhub.io/users/signup?referral_code=cfzyejV9&utm_source=github.com/NanmiCoder/MediaCrawler&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad) 或使用邀请码：`cfzyejV9`，注册并充值即可获得 **$2 免费额度**。
-
-[TikHub](https://tikhub.io/?utm_source=github.com/NanmiCoder/MediaCrawler&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad) 提供以下服务：
-
-- 🚀 丰富的社交媒体数据接口（TikTok、Douyin、XHS、YouTube、Instagram等）
-- 💎 每日签到免费领取额度
-- ⚡ 高成功率与高并发支持
-- 🌐 官网：[https://tikhub.io/](https://tikhub.io/?utm_source=github.com/NanmiCoder/MediaCrawler&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad)
-- 💻 GitHub地址：[https://github.com/TikHubIO/](https://github.com/TikHubIO/)
-
-
-
-
-### 🤝 成为赞助者
+## 🤝 成为赞助者
 
 成为赞助者，可以将您的产品展示在这里，每天获得大量曝光！
 
 **联系方式**：
 - 微信：`relakkes`
 - 邮箱：`relakkes@gmail.com`
+---
+
+## ☕ 请作者喝杯咖啡
+
+如果这个项目对您有帮助，欢迎打赏支持，您的每一份支持都是我持续更新的动力 ❤️
+
+<table>
+<tr>
+<td align="center" width="33%">
+<img src="docs/static/images/wechat_pay.jpeg" width="250" alt="微信赞赏"><br>
+<b>微信赞赏</b>
+</td>
+<td align="center" width="33%">
+<img src="docs/static/images/zfb_pay.png" width="250" alt="支付宝"><br>
+<b>支付宝</b>
+</td>
+<td align="center" width="33%">
+<a href="https://buymeacoffee.com/relakkes" target="_blank">
+<img src="docs/static/images/bmc_button.png" width="250" alt="Buy Me a Coffee">
+</a><br>
+<b>Buy Me a Coffee</b>
+</td>
+</tr>
+</table>
 
 ---
 
+## 📚 其他
+- **常见问题**：[MediaCrawler 完整文档](https://nanmicoder.github.io/MediaCrawler/)
+- **爬虫入门教程**：[CrawlerTutorial 免费教程](https://github.com/NanmiCoder/CrawlerTutorial)
+- **新闻爬虫开源项目**：[NewsCrawlerCollection](https://github.com/NanmiCoder/NewsCrawlerCollection)
+
+
 ## ⭐ Star 趋势图
 
 如果这个项目对您有帮助，请给个 ⭐ Star 支持一下，让更多的人看到 MediaCrawler！
@@ -297,7 +318,6 @@ uv run main.py --platform xhs --lt qrcode --type search --save_data_option db
 [![Star History Chart](https://api.star-history.com/svg?repos=NanmiCoder/MediaCrawler&type=Date)](https://star-history.com/#NanmiCoder/MediaCrawler&Date)
 
 
-
 ## 📚 参考
 
 - **小红书签名仓库**：[Cloxl 的 xhs 签名仓库](https://github.com/Cloxl/xhshow)

README_en.md

@@ -1,16 +1,3 @@
-<div align="center" markdown="1">
-   <sup>Special thanks to:</sup>
-   <br>
-   <br>
-   <a href="https://go.warp.dev/MediaCrawler">
-      <img alt="Warp sponsorship" width="400" src="https://github.com/warpdotdev/brand-assets/blob/main/Github/Sponsor/Warp-Github-LG-02.png?raw=true">
-   </a>
-
-### [Warp is built for coding with multiple AI agents](https://go.warp.dev/MediaCrawler)
-
-
-</div>
-<hr>
 # 🔥 MediaCrawler - Social Media Platform Crawler 🕷️
 
 <div align="center">
@@ -60,16 +47,14 @@ A powerful **multi-platform social media data collection tool** that supports cr
 | Zhihu   | ✅          | ✅              | ✅        | ✅              | ✅          | ✅        | ✅              |
 
 
-<details id="pro-version">
-<summary>🔗 <strong>🚀 MediaCrawlerPro Major Release! More features, better architectural design!</strong></summary>
-
-### 🚀 MediaCrawlerPro Major Release!
+<strong>MediaCrawlerPro Major Release! Open source is not easy, welcome to subscribe and support!</strong>
 
 > Focus on learning mature project architectural design, not just crawling technology. The code design philosophy of the Pro version is equally worth in-depth study!
 
 [MediaCrawlerPro](https://github.com/MediaCrawlerPro) core advantages over the open-source version:
 
 #### 🎯 Core Feature Upgrades
+- ✅ **Content Deconstruction Agent** (New feature)
 - ✅ **Resume crawling functionality** (Key feature)
 - ✅ **Multi-account + IP proxy pool support** (Key feature)
 - ✅ **Remove Playwright dependency**, easier to use
@@ -83,10 +68,9 @@ A powerful **multi-platform social media data collection tool** that supports cr
 #### 🎁 Additional Features
 - ✅ **Social media video downloader desktop app** (suitable for learning full-stack development)
 - ✅ **Multi-platform homepage feed recommendations** (HomeFeed)
-- [ ] **AI Agent based on social media platforms is under development 🚀🚀**
+- [ ] **AI Agent based on comment analysis is under development 🚀🚀**
 
 Click to view: [MediaCrawlerPro Project Homepage](https://github.com/MediaCrawlerPro) for more information
-</details>
 
 ## 🚀 Quick Start
 
@@ -148,6 +132,37 @@ uv run main.py --platform xhs --lt qrcode --type detail
 uv run main.py --help
 ```
 
+## WebUI Support
+
+<details>
+<summary>🖥️ <strong>WebUI Visual Operation Interface</strong></summary>
+
+MediaCrawler provides a web-based visual operation interface, allowing you to easily use crawler features without command line.
+
+#### Start WebUI Service
+
+```shell
+# Start API server (default port 8080)
+uv run uvicorn api.main:app --port 8080 --reload
+
+# Or start using module method
+uv run python -m api.main
+```
+
+After successful startup, visit `http://localhost:8080` to open the WebUI interface.
+
+#### WebUI Features
+
+- Visualize crawler parameter configuration (platform, login method, crawling type, etc.)
+- Real-time view of crawler running status and logs
+- Data preview and export
+
+#### Interface Preview
+
+<img src="docs/static/images/img_8.png" alt="WebUI Interface Preview">
+
+</details>
+
 <details>
 <summary>🔗 <strong>Using Python native venv environment management (Not recommended)</strong></summary>
 
@@ -206,76 +221,37 @@ python main.py --help
 
 ## 💾 Data Storage
 
-Supports multiple data storage methods:
-- **CSV Files**: Supports saving to CSV (under `data/` directory)
-- **JSON Files**: Supports saving to JSON (under `data/` directory)
-- **Database Storage**
-  - Use the `--init_db` parameter for database initialization (when using `--init_db`, no other optional arguments are needed)
-  - **SQLite Database**: Lightweight database, no server required, suitable for personal use (recommended)
-    1. Initialization: `--init_db sqlite`
-    2. Data Storage: `--save_data_option sqlite`
-  - **MySQL Database**: Supports saving to relational database MySQL (database needs to be created in advance)
-    1. Initialization: `--init_db mysql`
-    2. Data Storage: `--save_data_option db` (the db parameter is retained for compatibility with historical updates)
+MediaCrawler supports multiple data storage methods, including CSV, JSON, JSONL, Excel, SQLite, and MySQL databases.
 
-
-### Usage Examples:
-```shell
-# Initialize SQLite database (when using '--init_db', no other optional arguments are needed)
-uv run main.py --init_db sqlite
-# Use SQLite to store data (recommended for personal users)
-uv run main.py --platform xhs --lt qrcode --type search --save_data_option sqlite
-```
-```shell
-# Initialize MySQL database
-uv run main.py --init_db mysql
-# Use MySQL to store data (the db parameter is retained for compatibility with historical updates)
-uv run main.py --platform xhs --lt qrcode --type search --save_data_option db
-```
+📖 **For detailed usage instructions, please see: [Data Storage Guide](docs/data_storage_guide.md)**
 
 ---
 
 [🚀 MediaCrawlerPro Major Release 🚀! More features, better architectural design!](https://github.com/MediaCrawlerPro)
 
-## 🤝 Community & Support
-
 ### 💬 Discussion Groups
 - **WeChat Discussion Group**: [Click to join](https://nanmicoder.github.io/MediaCrawler/%E5%BE%AE%E4%BF%A1%E4%BA%A4%E6%B5%81%E7%BE%A4.html)
+- **Bilibili Account**: [Follow me](https://space.bilibili.com/434377496), sharing AI and crawler technology knowledge
 
-### 📚 Documentation & Tutorials
-- **Online Documentation**: [MediaCrawler Complete Documentation](https://nanmicoder.github.io/MediaCrawler/)
-- **Crawler Tutorial**: [CrawlerTutorial Free Tutorial](https://github.com/NanmiCoder/CrawlerTutorial)
-
-
-# Other common questions can be viewed in the online documentation
->
-> The online documentation includes usage methods, common questions, joining project discussion groups, etc.
-> [MediaCrawler Online Documentation](https://nanmicoder.github.io/MediaCrawler/)
->
-
-# Author's Knowledge Services
-> If you want to quickly get started and learn the usage of this project, source code architectural design, learn programming technology, or want to understand the source code design of MediaCrawlerPro, you can check out my paid knowledge column.
-
-[Author's Paid Knowledge Column Introduction](https://nanmicoder.github.io/MediaCrawler/%E7%9F%A5%E8%AF%86%E4%BB%98%E8%B4%B9%E4%BB%8B%E7%BB%8D.html)
-
-
----
-
-## ⭐ Star Trend Chart
-
-If this project helps you, please give a ⭐ Star to support and let more people see MediaCrawler!
-
-[![Star History Chart](https://api.star-history.com/svg?repos=NanmiCoder/MediaCrawler&type=Date)](https://star-history.com/#NanmiCoder/MediaCrawler&Date)
 
 ### 💰 Sponsor Display
 
-<a href="https://www.swiftproxy.net/?ref=nanmi">
-<img src="docs/static/images/img_5.png">
+<a href="https://tikhub.io/?utm_source=github.com/NanmiCoder/MediaCrawler&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad">
+<img width="500" src="docs/static/images/tikhub_banner_zh.png">
 <br>
-**Swiftproxy** - 90M+ global high-quality pure residential IPs, register to get free 500MB test traffic, dynamic traffic never expires!
-> Exclusive discount code: **GHB5** Get 10% off instantly!
+TikHub.io provides 900+ highly stable data interfaces, covering 14+ mainstream domestic and international platforms including TK, DY, XHS, Y2B, Ins, X, etc. Supports multi-dimensional public data APIs for users, content, products, comments, etc., with 40M+ cleaned structured datasets. Use invitation code <code>cfzyejV9</code> to register and recharge, and get an additional $2 bonus.
 </a>
+<br>
+<br>
 
+<a href="https://www.atlascloud.ai/?utm_source=github&utm_medium=link&utm_campaign=mei%27da%27c%27rmeidacrawler">
+<img width="500" alt="Atlas Cloud" src="docs/static/images/atlas_cloud_logo_black.png#gh-light-mode-only">
+<img width="500" alt="Atlas Cloud" src="docs/static/images/atlas_cloud_logo_white.png#gh-dark-mode-only">
+</a>
+<br>
+<a href="https://www.atlascloud.ai/?utm_source=github&utm_medium=link&utm_campaign=mei%27da%27c%27rmeidacrawler">Atlas Cloud</a> is a full-modal AI inference platform that gives developers a single AI API to access video generation, image generation, and LLM APIs. Instead of managing multiple vendor integrations, you connect once and get unified access to 300+ curated models across all modalities. Check out Atlas Cloud's new <a href="https://www.atlascloud.ai/console/coding-plan">coding plan promotion</a> for more budget-friendly API access.
+
+---
 
 ### 🤝 Become a Sponsor
 
@@ -284,10 +260,24 @@ Become a sponsor and showcase your product here, getting massive exposure daily!
 **Contact Information**:
 - WeChat: `relakkes`
 - Email: `relakkes@gmail.com`
+---
+
+### 📚 Other
+- **FAQ**: [MediaCrawler Complete Documentation](https://nanmicoder.github.io/MediaCrawler/)
+- **Crawler Beginner Tutorial**: [CrawlerTutorial Free Tutorial](https://github.com/NanmiCoder/CrawlerTutorial)
+- **News Crawler Open Source Project**: [NewsCrawlerCollection](https://github.com/NanmiCoder/NewsCrawlerCollection)
+
+
+## ⭐ Star Trend Chart
+
+If this project helps you, please give a ⭐ Star to support and let more people see MediaCrawler!
+
+[![Star History Chart](https://api.star-history.com/svg?repos=NanmiCoder/MediaCrawler&type=Date)](https://star-history.com/#NanmiCoder/MediaCrawler&Date)
 
 
 ## 📚 References
 
+- **Xiaohongshu Signature Repository**: [Cloxl's xhs signature repository](https://github.com/Cloxl/xhshow)
 - **Xiaohongshu Client**: [ReaJason's xhs repository](https://github.com/ReaJason/xhs)
 - **SMS Forwarding**: [SmsForwarder reference repository](https://github.com/pppscn/SmsForwarder)
 - **Intranet Penetration Tool**: [ngrok official documentation](https://ngrok.com/docs/)

README_es.md

@@ -1,17 +1,3 @@
-<div align="center" markdown="1">
-   <sup>Special thanks to:</sup>
-   <br>
-   <br>
-   <a href="https://go.warp.dev/MediaCrawler">
-      <img alt="Warp sponsorship" width="400" src="https://github.com/warpdotdev/brand-assets/blob/main/Github/Sponsor/Warp-Github-LG-02.png?raw=true">
-   </a>
-
-### [Warp is built for coding with multiple AI agents](https://go.warp.dev/MediaCrawler)
-
-
-</div>
-<hr>
-
 # 🔥 MediaCrawler - Rastreador de Plataformas de Redes Sociales 🕷️
 
 <div align="center">
@@ -61,16 +47,14 @@ Una poderosa **herramienta de recolección de datos de redes sociales multiplata
 | Zhihu   | ✅          | ✅              | ✅        | ✅              | ✅          | ✅        | ✅              |
 
 
-<details id="pro-version">
-<summary>🔗 <strong>🚀 ¡Lanzamiento Mayor de MediaCrawlerPro! ¡Más características, mejor diseño arquitectónico!</strong></summary>
-
-### 🚀 ¡Lanzamiento Mayor de MediaCrawlerPro!
+<strong>¡Lanzamiento Mayor de MediaCrawlerPro! ¡El código abierto no es fácil, bienvenido a suscribirse y apoyar!</strong>
 
 > Enfócate en aprender el diseño arquitectónico de proyectos maduros, no solo tecnología de rastreo. ¡La filosofía de diseño de código de la versión Pro también vale la pena estudiar en profundidad!
 
 [MediaCrawlerPro](https://github.com/MediaCrawlerPro) ventajas principales sobre la versión de código abierto:
 
 #### 🎯 Actualizaciones de Características Principales
+- ✅ **Agente de Deconstrucción de Contenido** (Nueva función)
 - ✅ **Funcionalidad de reanudación de rastreo** (Característica clave)
 - ✅ **Soporte de múltiples cuentas + pool de proxy IP** (Característica clave)
 - ✅ **Eliminar dependencia de Playwright**, más fácil de usar
@@ -84,10 +68,9 @@ Una poderosa **herramienta de recolección de datos de redes sociales multiplata
 #### 🎁 Características Adicionales
 - ✅ **Aplicación de escritorio descargadora de videos de redes sociales** (adecuada para aprender desarrollo full-stack)
 - ✅ **Recomendaciones de feed de página de inicio multiplataforma** (HomeFeed)
-- [ ] **Agente AI basado en plataformas de redes sociales está en desarrollo 🚀🚀**
+- [ ] **Agente AI basado en análisis de comentarios está en desarrollo 🚀🚀**
 
 Haga clic para ver: [Página de Inicio del Proyecto MediaCrawlerPro](https://github.com/MediaCrawlerPro) para más información
-</details>
 
 ## 🚀 Inicio Rápido
 
@@ -149,6 +132,37 @@ uv run main.py --platform xhs --lt qrcode --type detail
 uv run main.py --help
 ```
 
+## Soporte WebUI
+
+<details>
+<summary>🖥️ <strong>Interfaz de Operación Visual WebUI</strong></summary>
+
+MediaCrawler proporciona una interfaz de operación visual basada en web, permitiéndole usar fácilmente las funciones del rastreador sin línea de comandos.
+
+#### Iniciar Servicio WebUI
+
+```shell
+# Iniciar servidor API (puerto predeterminado 8080)
+uv run uvicorn api.main:app --port 8080 --reload
+
+# O iniciar usando método de módulo
+uv run python -m api.main
+```
+
+Después de iniciar exitosamente, visite `http://localhost:8080` para abrir la interfaz WebUI.
+
+#### Características de WebUI
+
+- Configuración visual de parámetros del rastreador (plataforma, método de login, tipo de rastreo, etc.)
+- Vista en tiempo real del estado de ejecución del rastreador y logs
+- Vista previa y exportación de datos
+
+#### Vista Previa de la Interfaz
+
+<img src="docs/static/images/img_8.png" alt="Vista Previa de Interfaz WebUI">
+
+</details>
+
 <details>
 <summary>🔗 <strong>Usando gestión de entorno venv nativo de Python (No recomendado)</strong></summary>
 
@@ -207,76 +221,29 @@ python main.py --help
 
 ## 💾 Almacenamiento de Datos
 
-Soporta múltiples métodos de almacenamiento de datos:
-- **Archivos CSV**: Soporta guardar en CSV (bajo el directorio `data/`)
-- **Archivos JSON**: Soporta guardar en JSON (bajo el directorio `data/`)
-- **Almacenamiento en Base de Datos**
-  - Use el parámetro `--init_db` para la inicialización de la base de datos (cuando use `--init_db`, no se necesitan otros argumentos opcionales)
-  - **Base de Datos SQLite**: Base de datos ligera, no requiere servidor, adecuada para uso personal (recomendado)
-    1. Inicialización: `--init_db sqlite`
-    2. Almacenamiento de Datos: `--save_data_option sqlite`
-  - **Base de Datos MySQL**: Soporta guardar en la base de datos relacional MySQL (la base de datos debe crearse con anticipación)
-    1. Inicialización: `--init_db mysql`
-    2. Almacenamiento de Datos: `--save_data_option db` (el parámetro db se mantiene por compatibilidad con actualizaciones históricas)
+MediaCrawler soporta múltiples métodos de almacenamiento de datos, incluyendo CSV, JSON, JSONL, Excel, SQLite y bases de datos MySQL.
 
+📖 **Para instrucciones de uso detalladas, por favor vea: [Guía de Almacenamiento de Datos](docs/data_storage_guide.md)**
 
-### Ejemplos de Uso:
-```shell
-# Inicializar la base de datos SQLite (cuando use '--init_db', no se necesitan otros argumentos opcionales)
-uv run main.py --init_db sqlite
-# Usar SQLite para almacenar datos (recomendado para usuarios personales)
-uv run main.py --platform xhs --lt qrcode --type search --save_data_option sqlite
-```
-```shell
-# Inicializar la base de datos MySQL
-uv run main.py --init_db mysql
-# Usar MySQL para almacenar datos (el parámetro db se mantiene por compatibilidad con actualizaciones históricas)
-uv run main.py --platform xhs --lt qrcode --type search --save_data_option db
-```
-
----
 
 [🚀 ¡Lanzamiento Mayor de MediaCrawlerPro 🚀! ¡Más características, mejor diseño arquitectónico!](https://github.com/MediaCrawlerPro)
 
-## 🤝 Comunidad y Soporte
 
 ### 💬 Grupos de Discusión
 - **Grupo de Discusión WeChat**: [Haga clic para unirse](https://nanmicoder.github.io/MediaCrawler/%E5%BE%AE%E4%BF%A1%E4%BA%A4%E6%B5%81%E7%BE%A4.html)
+- **Cuenta de Bilibili**: [Sígueme](https://space.bilibili.com/434377496), compartiendo conocimientos de tecnología de IA y rastreo
 
-### 📚 Documentación y Tutoriales
-- **Documentación en Línea**: [Documentación Completa de MediaCrawler](https://nanmicoder.github.io/MediaCrawler/)
-- **Tutorial de Rastreador**: [Tutorial Gratuito CrawlerTutorial](https://github.com/NanmiCoder/CrawlerTutorial)
-
-
-# Otras preguntas comunes pueden verse en la documentación en línea
->
-> La documentación en línea incluye métodos de uso, preguntas comunes, unirse a grupos de discusión del proyecto, etc.
-> [Documentación en Línea de MediaCrawler](https://nanmicoder.github.io/MediaCrawler/)
->
-
-# Servicios de Conocimiento del Autor
-> Si quiere comenzar rápidamente y aprender el uso de este proyecto, diseño arquitectónico del código fuente, aprender tecnología de programación, o quiere entender el diseño del código fuente de MediaCrawlerPro, puede revisar mi columna de conocimiento pagado.
-
-[Introducción de la Columna de Conocimiento Pagado del Autor](https://nanmicoder.github.io/MediaCrawler/%E7%9F%A5%E8%AF%86%E4%BB%98%E8%B4%B9%E4%BB%8B%E7%BB%8D.html)
-
-
----
-
-## ⭐ Gráfico de Tendencia de Estrellas
-
-¡Si este proyecto te ayuda, por favor da una ⭐ Estrella para apoyar y que más personas vean MediaCrawler!
-
-[![Star History Chart](https://api.star-history.com/svg?repos=NanmiCoder/MediaCrawler&type=Date)](https://star-history.com/#NanmiCoder/MediaCrawler&Date)
 
 ### 💰 Exhibición de Patrocinadores
 
-<a href="https://www.swiftproxy.net/?ref=nanmi">
-<img src="docs/static/images/img_5.png">
+<a href="https://tikhub.io/?utm_source=github.com/NanmiCoder/MediaCrawler&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad">
+<img width="500" src="docs/static/images/tikhub_banner_zh.png">
 <br>
-**Swiftproxy** - ¡90M+ IPs residenciales puras de alta calidad globales, regístrese para obtener 500MB de tráfico de prueba gratuito, el tráfico dinámico nunca expira!
-> Código de descuento exclusivo: **GHB5** ¡Obtenga 10% de descuento instantáneamente!
+TikHub.io proporciona 900+ interfaces de datos altamente estables, cubriendo 14+ plataformas principales nacionales e internacionales incluyendo TK, DY, XHS, Y2B, Ins, X, etc. Soporta APIs de datos públicos multidimensionales para usuarios, contenido, productos, comentarios, etc., con 40M+ conjuntos de datos estructurados limpios. Use el código de invitación <code>cfzyejV9</code> para registrarse y recargar, y obtenga $2 adicionales de bonificación.
 </a>
 
+---
+
 ### 🤝 Conviértase en Patrocinador
 
 ¡Conviértase en patrocinador y muestre su producto aquí, obteniendo exposición masiva diariamente!
@@ -284,10 +251,24 @@ uv run main.py --platform xhs --lt qrcode --type search --save_data_option db
 **Información de Contacto**:
 - WeChat: `relakkes`
 - Email: `relakkes@gmail.com`
+---
+
+### 📚 Otros
+- **Preguntas Frecuentes**: [Documentación Completa de MediaCrawler](https://nanmicoder.github.io/MediaCrawler/)
+- **Tutorial de Rastreador para Principiantes**: [Tutorial Gratuito CrawlerTutorial](https://github.com/NanmiCoder/CrawlerTutorial)
+- **Proyecto de Código Abierto de Rastreador de Noticias**: [NewsCrawlerCollection](https://github.com/NanmiCoder/NewsCrawlerCollection)
+
+
+## ⭐ Gráfico de Tendencia de Estrellas
+
+¡Si este proyecto te ayuda, por favor da una ⭐ Estrella para apoyar y que más personas vean MediaCrawler!
+
+[![Star History Chart](https://api.star-history.com/svg?repos=NanmiCoder/MediaCrawler&type=Date)](https://star-history.com/#NanmiCoder/MediaCrawler&Date)
 
 
 ## 📚 Referencias
 
+- **Repositorio de Firma Xiaohongshu**: [Repositorio de firma xhs de Cloxl](https://github.com/Cloxl/xhshow)
 - **Cliente Xiaohongshu**: [Repositorio xhs de ReaJason](https://github.com/ReaJason/xhs)
 - **Reenvío de SMS**: [Repositorio de referencia SmsForwarder](https://github.com/pppscn/SmsForwarder)
 - **Herramienta de Penetración de Intranet**: [Documentación oficial de ngrok](https://ngrok.com/docs/)

api/__init__.py

@@ -0,0 +1,19 @@
+# -*- coding: utf-8 -*-
+# Copyright (c) 2025 relakkes@gmail.com
+#
+# This file is part of MediaCrawler project.
+# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/__init__.py
+# GitHub: https://github.com/NanmiCoder
+# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
+#
+# 声明：本代码仅供学习和研究目的使用。使用者应遵守以下原则：
+# 1. 不得用于任何商业用途。
+# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
+# 3. 不得进行大规模爬取或对平台造成运营干扰。
+# 4. 应合理控制请求频率，避免给目标平台带来不必要的负担。
+# 5. 不得用于任何非法或不当的用途。
+#
+# 详细许可条款请参阅项目根目录下的LICENSE文件。
+# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
+
+# WebUI API Module for MediaCrawler

api/main.py

@@ -0,0 +1,200 @@
+# -*- coding: utf-8 -*-
+# Copyright (c) 2025 relakkes@gmail.com
+#
+# This file is part of MediaCrawler project.
+# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/main.py
+# GitHub: https://github.com/NanmiCoder
+# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
+#
+# 声明：本代码仅供学习和研究目的使用。使用者应遵守以下原则：
+# 1. 不得用于任何商业用途。
+# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
+# 3. 不得进行大规模爬取或对平台造成运营干扰。
+# 4. 应合理控制请求频率，避免给目标平台带来不必要的负担。
+# 5. 不得用于任何非法或不当的用途。
+#
+# 详细许可条款请参阅项目根目录下的LICENSE文件。
+# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
+
+"""
+MediaCrawler WebUI API Server
+Start command: uvicorn api.main:app --port 8080 --reload
+Or: python -m api.main
+"""
+import asyncio
+import os
+import sys
+import subprocess
+import uvicorn
+from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.staticfiles import StaticFiles
+from fastapi.responses import FileResponse
+
+from .routers import crawler_router, data_router, websocket_router
+
+app = FastAPI(
+    title="MediaCrawler WebUI API",
+    description="API for controlling MediaCrawler from WebUI",
+    version="1.0.0"
+)
+
+# Get webui static files directory
+WEBUI_DIR = os.path.join(os.path.dirname(__file__), "webui")
+
+# CORS configuration - allow frontend dev server access
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=[
+        "http://localhost:5173",  # Vite dev server
+        "http://localhost:3000",  # Backup port
+        "http://127.0.0.1:5173",
+        "http://127.0.0.1:3000",
+    ],
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+# Register routers
+app.include_router(crawler_router, prefix="/api")
+app.include_router(data_router, prefix="/api")
+app.include_router(websocket_router, prefix="/api")
+
+
+@app.get("/")
+async def serve_frontend():
+    """Return frontend page"""
+    index_path = os.path.join(WEBUI_DIR, "index.html")
+    if os.path.exists(index_path):
+        return FileResponse(index_path)
+    return {
+        "message": "MediaCrawler WebUI API",
+        "version": "1.0.0",
+        "docs": "/docs",
+        "note": "WebUI not found, please build it first: cd webui && npm run build"
+    }
+
+
+@app.get("/api/health")
+async def health_check():
+    return {"status": "ok"}
+
+
+@app.get("/api/env/check")
+async def check_environment():
+    """Check if MediaCrawler environment is configured correctly"""
+    try:
+        # Run uv run main.py --help command to check environment
+        if sys.platform == "win32":
+            loop = asyncio.get_running_loop()
+            process = await loop.run_in_executor(
+                None,
+                lambda: subprocess.run(
+                    ["uv", "run", "main.py", "--help"],
+                    capture_output=True,
+                    timeout=30.0,
+                    cwd="."
+                )
+            )
+            stdout, stderr = process.stdout, process.stderr  # bytes
+        else:
+            process = await asyncio.create_subprocess_exec(
+                "uv", "run", "main.py", "--help",
+                stdout=subprocess.PIPE,
+                stderr=subprocess.PIPE,
+                cwd="."  # Project root directory
+            )
+            stdout, stderr = await asyncio.wait_for(
+                process.communicate(),
+                timeout=30.0  # 30 seconds timeout
+            )
+        if process.returncode == 0:
+            return {
+                "success": True,
+                "message": "MediaCrawler environment configured correctly",
+                "output": stdout.decode("utf-8", errors="ignore")[:500]  # Truncate to first 500 characters
+            }
+        else:
+            error_msg = stderr.decode("utf-8", errors="ignore") or stdout.decode("utf-8", errors="ignore")
+            return {
+                "success": False,
+                "message": "Environment check failed",
+                "error": error_msg[:500]
+            }
+    except asyncio.TimeoutError:
+        return {
+            "success": False,
+            "message": "Environment check timeout",
+            "error": "Command execution exceeded 30 seconds"
+        }
+    except FileNotFoundError:
+        return {
+            "success": False,
+            "message": "uv command not found",
+            "error": "Please ensure uv is installed and configured in system PATH"
+        }
+    except Exception as e:
+        return {
+            "success": False,
+            "message": "Environment check error",
+            "error": f"{type(e).__name__}: {str(e) or 'Unknown'}"
+        }
+
+
+@app.get("/api/config/platforms")
+async def get_platforms():
+    """Get list of supported platforms"""
+    return {
+        "platforms": [
+            {"value": "xhs", "label": "Xiaohongshu", "icon": "book-open"},
+            {"value": "dy", "label": "Douyin", "icon": "music"},
+            {"value": "ks", "label": "Kuaishou", "icon": "video"},
+            {"value": "bili", "label": "Bilibili", "icon": "tv"},
+            {"value": "wb", "label": "Weibo", "icon": "message-circle"},
+            {"value": "tieba", "label": "Baidu Tieba", "icon": "messages-square"},
+            {"value": "zhihu", "label": "Zhihu", "icon": "help-circle"},
+        ]
+    }
+
+
+@app.get("/api/config/options")
+async def get_config_options():
+    """Get all configuration options"""
+    return {
+        "login_types": [
+            {"value": "qrcode", "label": "QR Code Login"},
+            {"value": "cookie", "label": "Cookie Login"},
+        ],
+        "crawler_types": [
+            {"value": "search", "label": "Search Mode"},
+            {"value": "detail", "label": "Detail Mode"},
+            {"value": "creator", "label": "Creator Mode"},
+        ],
+        "save_options": [
+            {"value": "jsonl", "label": "JSONL File"},
+            {"value": "json", "label": "JSON File"},
+            {"value": "csv", "label": "CSV File"},
+            {"value": "excel", "label": "Excel File"},
+            {"value": "sqlite", "label": "SQLite Database"},
+            {"value": "db", "label": "MySQL Database"},
+            {"value": "mongodb", "label": "MongoDB Database"},
+        ],
+    }
+
+
+# Mount static resources - must be placed after all routes
+if os.path.exists(WEBUI_DIR):
+    assets_dir = os.path.join(WEBUI_DIR, "assets")
+    if os.path.exists(assets_dir):
+        app.mount("/assets", StaticFiles(directory=assets_dir), name="assets")
+    # Mount logos directory
+    logos_dir = os.path.join(WEBUI_DIR, "logos")
+    if os.path.exists(logos_dir):
+        app.mount("/logos", StaticFiles(directory=logos_dir), name="logos")
+    # Mount other static files (e.g., vite.svg)
+    app.mount("/static", StaticFiles(directory=WEBUI_DIR), name="webui-static")
+
+
+if __name__ == "__main__":
+    uvicorn.run(app, host="0.0.0.0", port=8080)

api/routers/__init__.py

@@ -0,0 +1,23 @@
+# -*- coding: utf-8 -*-
+# Copyright (c) 2025 relakkes@gmail.com
+#
+# This file is part of MediaCrawler project.
+# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/__init__.py
+# GitHub: https://github.com/NanmiCoder
+# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
+#
+# 声明：本代码仅供学习和研究目的使用。使用者应遵守以下原则：
+# 1. 不得用于任何商业用途。
+# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
+# 3. 不得进行大规模爬取或对平台造成运营干扰。
+# 4. 应合理控制请求频率，避免给目标平台带来不必要的负担。
+# 5. 不得用于任何非法或不当的用途。
+#
+# 详细许可条款请参阅项目根目录下的LICENSE文件。
+# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
+
+from .crawler import router as crawler_router
+from .data import router as data_router
+from .websocket import router as websocket_router
+
+__all__ = ["crawler_router", "data_router", "websocket_router"]

api/routers/crawler.py

@@ -0,0 +1,63 @@
+# -*- coding: utf-8 -*-
+# Copyright (c) 2025 relakkes@gmail.com
+#
+# This file is part of MediaCrawler project.
+# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/crawler.py
+# GitHub: https://github.com/NanmiCoder
+# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
+#
+# 声明：本代码仅供学习和研究目的使用。使用者应遵守以下原则：
+# 1. 不得用于任何商业用途。
+# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
+# 3. 不得进行大规模爬取或对平台造成运营干扰。
+# 4. 应合理控制请求频率，避免给目标平台带来不必要的负担。
+# 5. 不得用于任何非法或不当的用途。
+#
+# 详细许可条款请参阅项目根目录下的LICENSE文件。
+# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
+
+from fastapi import APIRouter, HTTPException
+
+from ..schemas import CrawlerStartRequest, CrawlerStatusResponse
+from ..services import crawler_manager
+
+router = APIRouter(prefix="/crawler", tags=["crawler"])
+
+
+@router.post("/start")
+async def start_crawler(request: CrawlerStartRequest):
+    """Start crawler task"""
+    success = await crawler_manager.start(request)
+    if not success:
+        # Handle concurrent/duplicate requests: if process is already running, return 400 instead of 500
+        if crawler_manager.process and crawler_manager.process.poll() is None:
+            raise HTTPException(status_code=400, detail="Crawler is already running")
+        raise HTTPException(status_code=500, detail="Failed to start crawler")
+
+    return {"status": "ok", "message": "Crawler started successfully"}
+
+
+@router.post("/stop")
+async def stop_crawler():
+    """Stop crawler task"""
+    success = await crawler_manager.stop()
+    if not success:
+        # Handle concurrent/duplicate requests: if process already exited/doesn't exist, return 400 instead of 500
+        if not crawler_manager.process or crawler_manager.process.poll() is not None:
+            raise HTTPException(status_code=400, detail="No crawler is running")
+        raise HTTPException(status_code=500, detail="Failed to stop crawler")
+
+    return {"status": "ok", "message": "Crawler stopped successfully"}
+
+
+@router.get("/status", response_model=CrawlerStatusResponse)
+async def get_crawler_status():
+    """Get crawler status"""
+    return crawler_manager.get_status()
+
+
+@router.get("/logs")
+async def get_logs(limit: int = 100):
+    """Get recent logs"""
+    logs = crawler_manager.logs[-limit:] if limit > 0 else crawler_manager.logs
+    return {"logs": [log.model_dump() for log in logs]}

api/routers/data.py

@@ -0,0 +1,230 @@
+# -*- coding: utf-8 -*-
+# Copyright (c) 2025 relakkes@gmail.com
+#
+# This file is part of MediaCrawler project.
+# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/data.py
+# GitHub: https://github.com/NanmiCoder
+# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
+#
+# 声明：本代码仅供学习和研究目的使用。使用者应遵守以下原则：
+# 1. 不得用于任何商业用途。
+# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
+# 3. 不得进行大规模爬取或对平台造成运营干扰。
+# 4. 应合理控制请求频率，避免给目标平台带来不必要的负担。
+# 5. 不得用于任何非法或不当的用途。
+#
+# 详细许可条款请参阅项目根目录下的LICENSE文件。
+# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
+
+import os
+import json
+from pathlib import Path
+from typing import Optional
+
+from fastapi import APIRouter, HTTPException
+from fastapi.responses import FileResponse
+
+router = APIRouter(prefix="/data", tags=["data"])
+
+# Data directory
+DATA_DIR = Path(__file__).parent.parent.parent / "data"
+
+
+def get_file_info(file_path: Path) -> dict:
+    """Get file information"""
+    stat = file_path.stat()
+    record_count = None
+
+    # Try to get record count
+    try:
+        if file_path.suffix == ".json":
+            with open(file_path, "r", encoding="utf-8") as f:
+                data = json.load(f)
+                if isinstance(data, list):
+                    record_count = len(data)
+        elif file_path.suffix == ".csv":
+            with open(file_path, "r", encoding="utf-8") as f:
+                record_count = sum(1 for _ in f) - 1  # Subtract header row
+    except Exception:
+        pass
+
+    return {
+        "name": file_path.name,
+        "path": str(file_path.relative_to(DATA_DIR)),
+        "size": stat.st_size,
+        "modified_at": stat.st_mtime,
+        "record_count": record_count,
+        "type": file_path.suffix[1:] if file_path.suffix else "unknown"
+    }
+
+
+@router.get("/files")
+async def list_data_files(platform: Optional[str] = None, file_type: Optional[str] = None):
+    """Get data file list"""
+    if not DATA_DIR.exists():
+        return {"files": []}
+
+    files = []
+    supported_extensions = {".json", ".csv", ".xlsx", ".xls"}
+
+    for root, dirs, filenames in os.walk(DATA_DIR):
+        root_path = Path(root)
+        for filename in filenames:
+            file_path = root_path / filename
+            if file_path.suffix.lower() not in supported_extensions:
+                continue
+
+            # Platform filter
+            if platform:
+                rel_path = str(file_path.relative_to(DATA_DIR))
+                if platform.lower() not in rel_path.lower():
+                    continue
+
+            # Type filter
+            if file_type and file_path.suffix[1:].lower() != file_type.lower():
+                continue
+
+            try:
+                files.append(get_file_info(file_path))
+            except Exception:
+                continue
+
+    # Sort by modification time (newest first)
+    files.sort(key=lambda x: x["modified_at"], reverse=True)
+
+    return {"files": files}
+
+
+@router.get("/files/{file_path:path}")
+async def get_file_content(file_path: str, preview: bool = True, limit: int = 100):
+    """Get file content or preview"""
+    full_path = DATA_DIR / file_path
+
+    if not full_path.exists():
+        raise HTTPException(status_code=404, detail="File not found")
+
+    if not full_path.is_file():
+        raise HTTPException(status_code=400, detail="Not a file")
+
+    # Security check: ensure within DATA_DIR
+    try:
+        full_path.resolve().relative_to(DATA_DIR.resolve())
+    except ValueError:
+        raise HTTPException(status_code=403, detail="Access denied")
+
+    if preview:
+        # Return preview data
+        try:
+            if full_path.suffix == ".json":
+                with open(full_path, "r", encoding="utf-8") as f:
+                    data = json.load(f)
+                    if isinstance(data, list):
+                        return {"data": data[:limit], "total": len(data)}
+                    return {"data": data, "total": 1}
+            elif full_path.suffix == ".csv":
+                import csv
+                with open(full_path, "r", encoding="utf-8") as f:
+                    reader = csv.DictReader(f)
+                    rows = []
+                    for i, row in enumerate(reader):
+                        if i >= limit:
+                            break
+                        rows.append(row)
+                    # Re-read to get total count
+                    f.seek(0)
+                    total = sum(1 for _ in f) - 1
+                    return {"data": rows, "total": total}
+            elif full_path.suffix.lower() in (".xlsx", ".xls"):
+                import pandas as pd
+                # Read first limit rows
+                df = pd.read_excel(full_path, nrows=limit)
+                # Get total row count (only read first column to save memory)
+                df_count = pd.read_excel(full_path, usecols=[0])
+                total = len(df_count)
+                # Convert to list of dictionaries, handle NaN values
+                rows = df.where(pd.notnull(df), None).to_dict(orient='records')
+                return {
+                    "data": rows,
+                    "total": total,
+                    "columns": list(df.columns)
+                }
+            else:
+                raise HTTPException(status_code=400, detail="Unsupported file type for preview")
+        except json.JSONDecodeError:
+            raise HTTPException(status_code=400, detail="Invalid JSON file")
+        except Exception as e:
+            raise HTTPException(status_code=500, detail=str(e))
+    else:
+        # Return file download
+        return FileResponse(
+            path=full_path,
+            filename=full_path.name,
+            media_type="application/octet-stream"
+        )
+
+
+@router.get("/download/{file_path:path}")
+async def download_file(file_path: str):
+    """Download file"""
+    full_path = DATA_DIR / file_path
+
+    if not full_path.exists():
+        raise HTTPException(status_code=404, detail="File not found")
+
+    if not full_path.is_file():
+        raise HTTPException(status_code=400, detail="Not a file")
+
+    # Security check
+    try:
+        full_path.resolve().relative_to(DATA_DIR.resolve())
+    except ValueError:
+        raise HTTPException(status_code=403, detail="Access denied")
+
+    return FileResponse(
+        path=full_path,
+        filename=full_path.name,
+        media_type="application/octet-stream"
+    )
+
+
+@router.get("/stats")
+async def get_data_stats():
+    """Get data statistics"""
+    if not DATA_DIR.exists():
+        return {"total_files": 0, "total_size": 0, "by_platform": {}, "by_type": {}}
+
+    stats = {
+        "total_files": 0,
+        "total_size": 0,
+        "by_platform": {},
+        "by_type": {}
+    }
+
+    supported_extensions = {".json", ".csv", ".xlsx", ".xls"}
+
+    for root, dirs, filenames in os.walk(DATA_DIR):
+        root_path = Path(root)
+        for filename in filenames:
+            file_path = root_path / filename
+            if file_path.suffix.lower() not in supported_extensions:
+                continue
+
+            try:
+                stat = file_path.stat()
+                stats["total_files"] += 1
+                stats["total_size"] += stat.st_size
+
+                # Statistics by type
+                file_type = file_path.suffix[1:].lower()
+                stats["by_type"][file_type] = stats["by_type"].get(file_type, 0) + 1
+
+                # Statistics by platform (inferred from path)
+                rel_path = str(file_path.relative_to(DATA_DIR))
+                for platform in ["xhs", "dy", "ks", "bili", "wb", "tieba", "zhihu"]:
+                    if platform in rel_path.lower():
+                        stats["by_platform"][platform] = stats["by_platform"].get(platform, 0) + 1
+                        break
+            except Exception:
+                continue
+
+    return stats

api/routers/websocket.py

@@ -0,0 +1,151 @@
+# -*- coding: utf-8 -*-
+# Copyright (c) 2025 relakkes@gmail.com
+#
+# This file is part of MediaCrawler project.
+# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/websocket.py
+# GitHub: https://github.com/NanmiCoder
+# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
+#
+# 声明：本代码仅供学习和研究目的使用。使用者应遵守以下原则：
+# 1. 不得用于任何商业用途。
+# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
+# 3. 不得进行大规模爬取或对平台造成运营干扰。
+# 4. 应合理控制请求频率，避免给目标平台带来不必要的负担。
+# 5. 不得用于任何非法或不当的用途。
+#
+# 详细许可条款请参阅项目根目录下的LICENSE文件。
+# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
+
+import asyncio
+from typing import Set, Optional
+
+from fastapi import APIRouter, WebSocket, WebSocketDisconnect
+
+from ..services import crawler_manager
+
+router = APIRouter(tags=["websocket"])
+
+
+class ConnectionManager:
+    """WebSocket connection manager"""
+
+    def __init__(self):
+        self.active_connections: Set[WebSocket] = set()
+
+    async def connect(self, websocket: WebSocket):
+        await websocket.accept()
+        self.active_connections.add(websocket)
+
+    def disconnect(self, websocket: WebSocket):
+        self.active_connections.discard(websocket)
+
+    async def broadcast(self, message: dict):
+        """Broadcast message to all connections"""
+        if not self.active_connections:
+            return
+
+        disconnected = []
+        for connection in list(self.active_connections):
+            try:
+                await connection.send_json(message)
+            except Exception:
+                disconnected.append(connection)
+
+        # Clean up disconnected connections
+        for conn in disconnected:
+            self.disconnect(conn)
+
+
+manager = ConnectionManager()
+
+
+async def log_broadcaster():
+    """Background task: read logs from queue and broadcast"""
+    queue = crawler_manager.get_log_queue()
+    while True:
+        try:
+            # Get log entry from queue
+            entry = await queue.get()
+            # Broadcast to all WebSocket connections
+            await manager.broadcast(entry.model_dump())
+        except asyncio.CancelledError:
+            break
+        except Exception as e:
+            print(f"Log broadcaster error: {e}")
+            await asyncio.sleep(0.1)
+
+
+# Global broadcast task
+_broadcaster_task: Optional[asyncio.Task] = None
+
+
+def start_broadcaster():
+    """Start broadcast task"""
+    global _broadcaster_task
+    if _broadcaster_task is None or _broadcaster_task.done():
+        _broadcaster_task = asyncio.create_task(log_broadcaster())
+
+
+@router.websocket("/ws/logs")
+async def websocket_logs(websocket: WebSocket):
+    """WebSocket log stream"""
+    print("[WS] New connection attempt")
+
+    try:
+        # Ensure broadcast task is running
+        start_broadcaster()
+
+        await manager.connect(websocket)
+        print(f"[WS] Connected, active connections: {len(manager.active_connections)}")
+
+        # Send existing logs
+        for log in crawler_manager.logs:
+            try:
+                await websocket.send_json(log.model_dump())
+            except Exception as e:
+                print(f"[WS] Error sending existing log: {e}")
+                break
+
+        print(f"[WS] Sent {len(crawler_manager.logs)} existing logs, entering main loop")
+
+        while True:
+            # Keep connection alive, receive heartbeat or any message
+            try:
+                data = await asyncio.wait_for(
+                    websocket.receive_text(),
+                    timeout=30.0
+                )
+                if data == "ping":
+                    await websocket.send_text("pong")
+            except asyncio.TimeoutError:
+                # Send ping to keep connection alive
+                try:
+                    await websocket.send_text("ping")
+                except Exception as e:
+                    print(f"[WS] Error sending ping: {e}")
+                    break
+
+    except WebSocketDisconnect:
+        print("[WS] Client disconnected")
+    except Exception as e:
+        print(f"[WS] Error: {type(e).__name__}: {e}")
+    finally:
+        manager.disconnect(websocket)
+        print(f"[WS] Cleanup done, active connections: {len(manager.active_connections)}")
+
+
+@router.websocket("/ws/status")
+async def websocket_status(websocket: WebSocket):
+    """WebSocket status stream"""
+    await websocket.accept()
+
+    try:
+        while True:
+            # Send status every second
+            status = crawler_manager.get_status()
+            await websocket.send_json(status)
+            await asyncio.sleep(1)
+    except WebSocketDisconnect:
+        pass
+    except Exception:
+        pass

api/schemas/__init__.py

@@ -0,0 +1,37 @@
+# -*- coding: utf-8 -*-
+# Copyright (c) 2025 relakkes@gmail.com
+#
+# This file is part of MediaCrawler project.
+# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/schemas/__init__.py
+# GitHub: https://github.com/NanmiCoder
+# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
+#
+# 声明：本代码仅供学习和研究目的使用。使用者应遵守以下原则：
+# 1. 不得用于任何商业用途。
+# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
+# 3. 不得进行大规模爬取或对平台造成运营干扰。
+# 4. 应合理控制请求频率，避免给目标平台带来不必要的负担。
+# 5. 不得用于任何非法或不当的用途。
+#
+# 详细许可条款请参阅项目根目录下的LICENSE文件。
+# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
+
+from .crawler import (
+    PlatformEnum,
+    LoginTypeEnum,
+    CrawlerTypeEnum,
+    SaveDataOptionEnum,
+    CrawlerStartRequest,
+    CrawlerStatusResponse,
+    LogEntry,
+)
+
+__all__ = [
+    "PlatformEnum",
+    "LoginTypeEnum",
+    "CrawlerTypeEnum",
+    "SaveDataOptionEnum",
+    "CrawlerStartRequest",
+    "CrawlerStatusResponse",
+    "LogEntry",
+]

api/schemas/crawler.py

@@ -0,0 +1,104 @@
+# -*- coding: utf-8 -*-
+# Copyright (c) 2025 relakkes@gmail.com
+#
+# This file is part of MediaCrawler project.
+# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/schemas/crawler.py
+# GitHub: https://github.com/NanmiCoder
+# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
+#
+# 声明：本代码仅供学习和研究目的使用。使用者应遵守以下原则：
+# 1. 不得用于任何商业用途。
+# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
+# 3. 不得进行大规模爬取或对平台造成运营干扰。
+# 4. 应合理控制请求频率，避免给目标平台带来不必要的负担。
+# 5. 不得用于任何非法或不当的用途。
+#
+# 详细许可条款请参阅项目根目录下的LICENSE文件。
+# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
+
+from enum import Enum
+from typing import Optional, Literal
+from pydantic import BaseModel, Field
+
+
+MAX_API_LIMIT_COUNT = 10000
+
+
+class PlatformEnum(str, Enum):
+    """Supported media platforms"""
+    XHS = "xhs"
+    DOUYIN = "dy"
+    KUAISHOU = "ks"
+    BILIBILI = "bili"
+    WEIBO = "wb"
+    TIEBA = "tieba"
+    ZHIHU = "zhihu"
+
+
+class LoginTypeEnum(str, Enum):
+    """Login method"""
+    QRCODE = "qrcode"
+    PHONE = "phone"
+    COOKIE = "cookie"
+
+
+class CrawlerTypeEnum(str, Enum):
+    """Crawler type"""
+    SEARCH = "search"
+    DETAIL = "detail"
+    CREATOR = "creator"
+
+
+class SaveDataOptionEnum(str, Enum):
+    """Data save option"""
+    CSV = "csv"
+    DB = "db"
+    JSON = "json"
+    JSONL = "jsonl"
+    SQLITE = "sqlite"
+    MONGODB = "mongodb"
+    EXCEL = "excel"
+
+
+class CrawlerStartRequest(BaseModel):
+    """Crawler start request"""
+    platform: PlatformEnum
+    login_type: LoginTypeEnum = LoginTypeEnum.QRCODE
+    crawler_type: CrawlerTypeEnum = CrawlerTypeEnum.SEARCH
+    keywords: str = ""  # Keywords for search mode
+    specified_ids: str = ""  # Post/video ID list for detail mode, comma-separated
+    creator_ids: str = ""  # Creator ID list for creator mode, comma-separated
+    start_page: int = 1
+    enable_comments: bool = True
+    enable_sub_comments: bool = False
+    save_option: SaveDataOptionEnum = SaveDataOptionEnum.JSONL
+    cookies: str = ""
+    headless: bool = False
+    max_notes_count: Optional[int] = Field(default=None, ge=1, le=MAX_API_LIMIT_COUNT)
+    max_comments_count: Optional[int] = Field(default=None, ge=1, le=MAX_API_LIMIT_COUNT)
+
+
+class CrawlerStatusResponse(BaseModel):
+    """Crawler status response"""
+    status: Literal["idle", "running", "stopping", "error"]
+    platform: Optional[str] = None
+    crawler_type: Optional[str] = None
+    started_at: Optional[str] = None
+    error_message: Optional[str] = None
+
+
+class LogEntry(BaseModel):
+    """Log entry"""
+    id: int
+    timestamp: str
+    level: Literal["info", "warning", "error", "success", "debug"]
+    message: str
+
+
+class DataFileInfo(BaseModel):
+    """Data file information"""
+    name: str
+    path: str
+    size: int
+    modified_at: str
+    record_count: Optional[int] = None

api/services/__init__.py

@@ -0,0 +1,21 @@
+# -*- coding: utf-8 -*-
+# Copyright (c) 2025 relakkes@gmail.com
+#
+# This file is part of MediaCrawler project.
+# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/services/__init__.py
+# GitHub: https://github.com/NanmiCoder
+# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
+#
+# 声明：本代码仅供学习和研究目的使用。使用者应遵守以下原则：
+# 1. 不得用于任何商业用途。
+# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
+# 3. 不得进行大规模爬取或对平台造成运营干扰。
+# 4. 应合理控制请求频率，避免给目标平台带来不必要的负担。
+# 5. 不得用于任何非法或不当的用途。
+#
+# 详细许可条款请参阅项目根目录下的LICENSE文件。
+# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
+
+from .crawler_manager import CrawlerManager, crawler_manager
+
+__all__ = ["CrawlerManager", "crawler_manager"]

api/services/crawler_manager.py

@@ -0,0 +1,288 @@
+# -*- coding: utf-8 -*-
+# Copyright (c) 2025 relakkes@gmail.com
+#
+# This file is part of MediaCrawler project.
+# Repository: https://github.com/NanmiCoder/MediaCrawler/blob/main/api/services/crawler_manager.py
+# GitHub: https://github.com/NanmiCoder
+# Licensed under NON-COMMERCIAL LEARNING LICENSE 1.1
+#
+# 声明：本代码仅供学习和研究目的使用。使用者应遵守以下原则：
+# 1. 不得用于任何商业用途。
+# 2. 使用时应遵守目标平台的使用条款和robots.txt规则。
+# 3. 不得进行大规模爬取或对平台造成运营干扰。
+# 4. 应合理控制请求频率，避免给目标平台带来不必要的负担。
+# 5. 不得用于任何非法或不当的用途。
+#
+# 详细许可条款请参阅项目根目录下的LICENSE文件。
+# 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
+
+import asyncio
+import subprocess
+import signal
+import os
+from typing import Optional, List
+from datetime import datetime
+from pathlib import Path
+
+from ..schemas import CrawlerStartRequest, LogEntry
+
+
+class CrawlerManager:
+    """Crawler process manager"""
+
+    def __init__(self):
+        self._lock = asyncio.Lock()
+        self.process: Optional[subprocess.Popen] = None
+        self.status = "idle"
+        self.started_at: Optional[datetime] = None
+        self.current_config: Optional[CrawlerStartRequest] = None
+        self._log_id = 0
+        self._logs: List[LogEntry] = []
+        self._read_task: Optional[asyncio.Task] = None
+        # Project root directory
+        self._project_root = Path(__file__).parent.parent.parent
+        # Log queue - for pushing to WebSocket
+        self._log_queue: Optional[asyncio.Queue] = None
+
+    @property
+    def logs(self) -> List[LogEntry]:
+        return self._logs
+
+    def get_log_queue(self) -> asyncio.Queue:
+        """Get or create log queue"""
+        if self._log_queue is None:
+            self._log_queue = asyncio.Queue()
+        return self._log_queue
+
+    def _create_log_entry(self, message: str, level: str = "info") -> LogEntry:
+        """Create log entry"""
+        self._log_id += 1
+        entry = LogEntry(
+            id=self._log_id,
+            timestamp=datetime.now().strftime("%H:%M:%S"),
+            level=level,
+            message=message
+        )
+        self._logs.append(entry)
+        # Keep last 500 logs
+        if len(self._logs) > 500:
+            self._logs = self._logs[-500:]
+        return entry
+
+    async def _push_log(self, entry: LogEntry):
+        """Push log to queue"""
+        if self._log_queue is not None:
+            try:
+                self._log_queue.put_nowait(entry)
+            except asyncio.QueueFull:
+                pass
+
+    def _parse_log_level(self, line: str) -> str:
+        """Parse log level"""
+        line_upper = line.upper()
+        if "ERROR" in line_upper or "FAILED" in line_upper:
+            return "error"
+        elif "WARNING" in line_upper or "WARN" in line_upper:
+            return "warning"
+        elif "SUCCESS" in line_upper or "完成" in line or "成功" in line:
+            return "success"
+        elif "DEBUG" in line_upper:
+            return "debug"
+        return "info"
+
+    async def start(self, config: CrawlerStartRequest) -> bool:
+        """Start crawler process"""
+        async with self._lock:
+            if self.process and self.process.poll() is None:
+                return False
+
+            # Clear old logs
+            self._logs = []
+            self._log_id = 0
+
+            # Clear pending queue (don't replace object to avoid WebSocket broadcast coroutine holding old queue reference)
+            if self._log_queue is None:
+                self._log_queue = asyncio.Queue()
+            else:
+                try:
+                    while True:
+                        self._log_queue.get_nowait()
+                except asyncio.QueueEmpty:
+                    pass
+
+            # Build command line arguments
+            cmd = self._build_command(config)
+
+            # Log start information
+            entry = self._create_log_entry(f"Starting crawler: {' '.join(cmd)}", "info")
+            await self._push_log(entry)
+
+            try:
+                # Start subprocess
+                self.process = subprocess.Popen(
+                    cmd,
+                    stdout=subprocess.PIPE,
+                    stderr=subprocess.STDOUT,
+                    text=True,
+                    encoding='utf-8',
+                    bufsize=1,
+                    cwd=str(self._project_root),
+                    env={**os.environ, "PYTHONUNBUFFERED": "1"}
+                )
+
+                self.status = "running"
+                self.started_at = datetime.now()
+                self.current_config = config
+
+                entry = self._create_log_entry(
+                    f"Crawler started on platform: {config.platform.value}, type: {config.crawler_type.value}",
+                    "success"
+                )
+                await self._push_log(entry)
+
+                # Start log reading task
+                self._read_task = asyncio.create_task(self._read_output())
+
+                return True
+            except Exception as e:
+                self.status = "error"
+                entry = self._create_log_entry(f"Failed to start crawler: {str(e)}", "error")
+                await self._push_log(entry)
+                return False
+
+    async def stop(self) -> bool:
+        """Stop crawler process"""
+        async with self._lock:
+            if not self.process or self.process.poll() is not None:
+                return False
+
+            self.status = "stopping"
+            entry = self._create_log_entry("Sending SIGTERM to crawler process...", "warning")
+            await self._push_log(entry)
+
+            try:
+                self.process.send_signal(signal.SIGTERM)
+
+                # Wait for graceful exit (up to 15 seconds)
+                for _ in range(30):
+                    if self.process.poll() is not None:
+                        break
+                    await asyncio.sleep(0.5)
+
+                # If still not exited, force kill
+                if self.process.poll() is None:
+                    entry = self._create_log_entry("Process not responding, sending SIGKILL...", "warning")
+                    await self._push_log(entry)
+                    self.process.kill()
+
+                entry = self._create_log_entry("Crawler process terminated", "info")
+                await self._push_log(entry)
+
+            except Exception as e:
+                entry = self._create_log_entry(f"Error stopping crawler: {str(e)}", "error")
+                await self._push_log(entry)
+
+            self.status = "idle"
+            self.current_config = None
+
+            # Cancel log reading task
+            if self._read_task:
+                self._read_task.cancel()
+                self._read_task = None
+
+            return True
+
+    def get_status(self) -> dict:
+        """Get current status"""
+        return {
+            "status": self.status,
+            "platform": self.current_config.platform.value if self.current_config else None,
+            "crawler_type": self.current_config.crawler_type.value if self.current_config else None,
+            "started_at": self.started_at.isoformat() if self.started_at else None,
+            "error_message": None
+        }
+
+    def _build_command(self, config: CrawlerStartRequest) -> list:
+        """Build main.py command line arguments"""
+        cmd = ["uv", "run", "python", "main.py"]
+
+        cmd.extend(["--platform", config.platform.value])
+        cmd.extend(["--lt", config.login_type.value])
+        cmd.extend(["--type", config.crawler_type.value])
+        cmd.extend(["--save_data_option", config.save_option.value])
+
+        # Pass different arguments based on crawler type
+        if config.crawler_type.value == "search" and config.keywords:
+            cmd.extend(["--keywords", config.keywords])
+        elif config.crawler_type.value == "detail" and config.specified_ids:
+            cmd.extend(["--specified_id", config.specified_ids])
+        elif config.crawler_type.value == "creator" and config.creator_ids:
+            cmd.extend(["--creator_id", config.creator_ids])
+
+        if config.start_page != 1:
+            cmd.extend(["--start", str(config.start_page)])
+
+        cmd.extend(["--get_comment", "true" if config.enable_comments else "false"])
+        cmd.extend(["--get_sub_comment", "true" if config.enable_sub_comments else "false"])
+
+        if config.max_notes_count is not None:
+            cmd.extend(["--crawler_max_notes_count", str(config.max_notes_count)])
+
+        if config.max_comments_count is not None:
+            cmd.extend(["--max_comments_count_singlenotes", str(config.max_comments_count)])
+
+        if config.cookies:
+            cmd.extend(["--cookies", config.cookies])
+
+        cmd.extend(["--headless", "true" if config.headless else "false"])
+
+        return cmd
+
+    async def _read_output(self):
+        """Asynchronously read process output"""
+        loop = asyncio.get_event_loop()
+
+        try:
+            while self.process and self.process.poll() is None:
+                # Read a line in thread pool
+                line = await loop.run_in_executor(
+                    None, self.process.stdout.readline
+                )
+                if line:
+                    line = line.strip()
+                    if line:
+                        level = self._parse_log_level(line)
+                        entry = self._create_log_entry(line, level)
+                        await self._push_log(entry)
+
+            # Read remaining output
+            if self.process and self.process.stdout:
+                remaining = await loop.run_in_executor(
+                    None, self.process.stdout.read
+                )
+                if remaining:
+                    for line in remaining.strip().split('\n'):
+                        if line.strip():
+                            level = self._parse_log_level(line)
+                            entry = self._create_log_entry(line.strip(), level)
+                            await self._push_log(entry)
+
+            # Process ended
+            if self.status == "running":
+                exit_code = self.process.returncode if self.process else -1
+                if exit_code == 0:
+                    entry = self._create_log_entry("Crawler completed successfully", "success")
+                else:
+                    entry = self._create_log_entry(f"Crawler exited with code: {exit_code}", "warning")
+                await self._push_log(entry)
+                self.status = "idle"
+
+        except asyncio.CancelledError:
+            pass
+        except Exception as e:
+            entry = self._create_log_entry(f"Error reading output: {str(e)}", "error")
+            await self._push_log(entry)
+
+
+# Global singleton
+crawler_manager = CrawlerManager()

api/webui/index.html

@@ -0,0 +1,17 @@
+<!doctype html>
+<html lang="zh-CN">
+  <head>
+    <meta charset="UTF-8" />
+    <link rel="icon" type="image/svg+xml" href="/vite.svg" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <title>MediaCrawler - Command Center</title>
+    <link rel="preconnect" href="https://fonts.googleapis.com">
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
+    <link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet">
+    <script type="module" crossorigin src="/assets/index-DvClRayq.js"></script>
+    <link rel="stylesheet" crossorigin href="/assets/index-OiBmsgXF.css">
+  </head>
+  <body>
+    <div id="root"></div>
+  </body>
+</html>

api/webui/vite.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100"><circle cx="50" cy="50" r="40" fill="#de283b"/></svg>

base/base_crawler.py

@@ -53,14 +53,14 @@ class AbstractCrawler(ABC):
 
     async def launch_browser_with_cdp(self, playwright: Playwright, playwright_proxy: Optional[Dict], user_agent: Optional[str], headless: bool = True) -> BrowserContext:
         """
-        使用CDP模式启动浏览器（可选实现）
-        :param playwright: playwright实例
-        :param playwright_proxy: playwright代理配置
-        :param user_agent: 用户代理
-        :param headless: 无头模式
-        :return: 浏览器上下文
+        Launch browser using CDP mode (optional implementation)
+        :param playwright: playwright instance
+        :param playwright_proxy: playwright proxy configuration
+        :param user_agent: user agent
+        :param headless: headless mode
+        :return: browser context
         """
-        # 默认实现：回退到标准模式
+        # Default implementation: fallback to standard mode
         return await self.launch_browser(playwright.chromium, playwright_proxy, user_agent, headless)
 
 

cache/abs_cache.py

@@ -20,9 +20,9 @@
 
 # -*- coding: utf-8 -*-
 # @Author  : relakkes@gmail.com
-# @Name    : 程序员阿江-Relakkes
+# @Name    : Programmer AJiang-Relakkes
 # @Time    : 2024/6/2 11:06
-# @Desc    : 抽象类
+# @Desc    : Abstract class
 
 from abc import ABC, abstractmethod
 from typing import Any, List, Optional
@@ -33,9 +33,9 @@ class AbstractCache(ABC):
     @abstractmethod
     def get(self, key: str) -> Optional[Any]:
         """
-        从缓存中获取键的值。
-        这是一个抽象方法。子类必须实现这个方法。
-        :param key: 键
+        Get the value of a key from the cache.
+        This is an abstract method. Subclasses must implement this method.
+        :param key: The key
         :return:
         """
         raise NotImplementedError
@@ -43,11 +43,11 @@ class AbstractCache(ABC):
     @abstractmethod
     def set(self, key: str, value: Any, expire_time: int) -> None:
         """
-        将键的值设置到缓存中。
-        这是一个抽象方法。子类必须实现这个方法。
-        :param key: 键
-        :param value: 值
-        :param expire_time: 过期时间
+        Set the value of a key in the cache.
+        This is an abstract method. Subclasses must implement this method.
+        :param key: The key
+        :param value: The value
+        :param expire_time: Expiration time
         :return:
         """
         raise NotImplementedError
@@ -55,8 +55,8 @@ class AbstractCache(ABC):
     @abstractmethod
     def keys(self, pattern: str) -> List[str]:
         """
-        获取所有符合pattern的key
-        :param pattern: 匹配模式
+        Get all keys matching the pattern
+        :param pattern: Matching pattern
         :return:
         """
         raise NotImplementedError

cache/cache_factory.py

@@ -20,23 +20,23 @@
 
 # -*- coding: utf-8 -*-
 # @Author  : relakkes@gmail.com
-# @Name    : 程序员阿江-Relakkes
+# @Name    : Programmer AJiang-Relakkes
 # @Time    : 2024/6/2 11:23
 # @Desc    :
 
 
 class CacheFactory:
     """
-    缓存工厂类
+    Cache factory class
     """
 
     @staticmethod
     def create_cache(cache_type: str, *args, **kwargs):
         """
-        创建缓存对象
-        :param cache_type: 缓存类型
-        :param args: 参数
-        :param kwargs: 关键字参数
+        Create cache object
+        :param cache_type: Cache type
+        :param args: Arguments
+        :param kwargs: Keyword arguments
         :return:
         """
         if cache_type == 'memory':

cache/local_cache.py

@@ -20,9 +20,9 @@
 
 # -*- coding: utf-8 -*-
 # @Author  : relakkes@gmail.com
-# @Name    : 程序员阿江-Relakkes
+# @Name    : Programmer AJiang-Relakkes
 # @Time    : 2024/6/2 11:05
-# @Desc    : 本地缓存
+# @Desc    : Local cache
 
 import asyncio
 import time
@@ -35,19 +35,19 @@ class ExpiringLocalCache(AbstractCache):
 
     def __init__(self, cron_interval: int = 10):
         """
-        初始化本地缓存
-        :param cron_interval: 定时清楚cache的时间间隔
+        Initialize local cache
+        :param cron_interval: Time interval for scheduled cache cleanup
         :return:
         """
         self._cron_interval = cron_interval
         self._cache_container: Dict[str, Tuple[Any, float]] = {}
         self._cron_task: Optional[asyncio.Task] = None
-        # 开启定时清理任务
+        # Start scheduled cleanup task
         self._schedule_clear()
 
     def __del__(self):
         """
-        析构函数，清理定时任务
+        Destructor function, cleanup scheduled task
         :return:
         """
         if self._cron_task is not None:
@@ -55,7 +55,7 @@ class ExpiringLocalCache(AbstractCache):
 
     def get(self, key: str) -> Optional[Any]:
         """
-        从缓存中获取键的值
+        Get the value of a key from the cache
         :param key:
         :return:
         """
@@ -63,7 +63,7 @@ class ExpiringLocalCache(AbstractCache):
         if value is None:
             return None
 
-        # 如果键已过期，则删除键并返回None
+        # If the key has expired, delete it and return None
         if expire_time < time.time():
             del self._cache_container[key]
             return None
@@ -72,7 +72,7 @@ class ExpiringLocalCache(AbstractCache):
 
     def set(self, key: str, value: Any, expire_time: int) -> None:
         """
-        将键的值设置到缓存中
+        Set the value of a key in the cache
         :param key:
         :param value:
         :param expire_time:
@@ -82,14 +82,14 @@ class ExpiringLocalCache(AbstractCache):
 
     def keys(self, pattern: str) -> List[str]:
         """
-        获取所有符合pattern的key
-        :param pattern: 匹配模式
+        Get all keys matching the pattern
+        :param pattern: Matching pattern
         :return:
         """
         if pattern == '*':
             return list(self._cache_container.keys())
 
-        # 本地缓存通配符暂时将*替换为空
+        # For local cache wildcard, temporarily replace * with empty string
         if '*' in pattern:
             pattern = pattern.replace('*', '')
 
@@ -97,7 +97,7 @@ class ExpiringLocalCache(AbstractCache):
 
     def _schedule_clear(self):
         """
-        开启定时清理任务,
+        Start scheduled cleanup task
         :return:
         """
 
@@ -111,7 +111,7 @@ class ExpiringLocalCache(AbstractCache):
 
     def _clear(self):
         """
-        根据过期时间清理缓存
+        Clean up cache based on expiration time
         :return:
         """
         for key, (value, expire_time) in self._cache_container.items():
@@ -120,7 +120,7 @@ class ExpiringLocalCache(AbstractCache):
 
     async def _start_clear_cron(self):
         """
-        开启定时清理任务
+        Start scheduled cleanup task
         :return:
         """
         while True:
@@ -130,7 +130,7 @@ class ExpiringLocalCache(AbstractCache):
 
 if __name__ == '__main__':
     cache = ExpiringLocalCache(cron_interval=2)
-    cache.set('name', '程序员阿江-Relakkes', 3)
+    cache.set('name', 'Programmer AJiang-Relakkes', 3)
     print(cache.get('key'))
     print(cache.keys("*"))
     time.sleep(4)

cache/redis_cache.py

@@ -20,14 +20,15 @@
 
 # -*- coding: utf-8 -*-
 # @Author  : relakkes@gmail.com
-# @Name    : 程序员阿江-Relakkes
+# @Name    : Programmer AJiang-Relakkes
 # @Time    : 2024/5/29 22:57
-# @Desc    : RedisCache实现
+# @Desc    : RedisCache implementation
 import pickle
 import time
 from typing import Any, List
 
 from redis import Redis
+from redis.exceptions import ResponseError
 
 from cache.abs_cache import AbstractCache
 from config import db_config
@@ -36,13 +37,13 @@ from config import db_config
 class RedisCache(AbstractCache):
 
     def __init__(self) -> None:
-        # 连接redis, 返回redis客户端
+        # Connect to redis, return redis client
         self._redis_client = self._connet_redis()
 
     @staticmethod
     def _connet_redis() -> Redis:
         """
-        连接redis, 返回redis客户端, 这里按需配置redis连接信息
+        Connect to redis, return redis client, configure redis connection information as needed
         :return:
         """
         return Redis(
@@ -54,7 +55,7 @@ class RedisCache(AbstractCache):
 
     def get(self, key: str) -> Any:
         """
-        从缓存中获取键的值, 并且反序列化
+        Get the value of a key from the cache and deserialize it
         :param key:
         :return:
         """
@@ -65,7 +66,7 @@ class RedisCache(AbstractCache):
 
     def set(self, key: str, value: Any, expire_time: int) -> None:
         """
-        将键的值设置到缓存中, 并且序列化
+        Set the value of a key in the cache and serialize it
         :param key:
         :param value:
         :param expire_time:
@@ -75,15 +76,32 @@ class RedisCache(AbstractCache):
 
     def keys(self, pattern: str) -> List[str]:
         """
-        获取所有符合pattern的key
+        Get all keys matching the pattern
+        First try KEYS command, if not supported fallback to SCAN
         """
-        return [key.decode() for key in self._redis_client.keys(pattern)]
+        try:
+            # Try KEYS command first (faster for standard Redis)
+            return [key.decode() if isinstance(key, bytes) else key for key in self._redis_client.keys(pattern)]
+        except ResponseError as e:
+            # If KEYS is not supported (e.g., Redis Cluster or cloud Redis), use SCAN
+            if "unknown command" in str(e).lower() or "keys" in str(e).lower():
+                keys_list: List[str] = []
+                cursor = 0
+                while True:
+                    cursor, keys = self._redis_client.scan(cursor=cursor, match=pattern, count=100)
+                    keys_list.extend([key.decode() if isinstance(key, bytes) else key for key in keys])
+                    if cursor == 0:
+                        break
+                return keys_list
+            else:
+                # Re-raise if it's a different error
+                raise
 
 
 if __name__ == '__main__':
     redis_cache = RedisCache()
     # basic usage
-    redis_cache.set("name", "程序员阿江-Relakkes", 1)
+    redis_cache.set("name", "Programmer AJiang-Relakkes", 1)
     print(redis_cache.get("name"))  # Relakkes
     print(redis_cache.keys("*"))  # ['name']
     time.sleep(2)

cmd_arg/arg.py

@@ -22,6 +22,7 @@ from __future__ import annotations
 
 
 import sys
+import re
 from enum import Enum
 from types import SimpleNamespace
 from typing import Iterable, Optional, Sequence, Type, TypeVar
@@ -37,7 +38,7 @@ EnumT = TypeVar("EnumT", bound=Enum)
 
 
 class PlatformEnum(str, Enum):
-    """支持的媒体平台枚举"""
+    """Supported media platform enumeration"""
 
     XHS = "xhs"
     DOUYIN = "dy"
@@ -49,7 +50,7 @@ class PlatformEnum(str, Enum):
 
 
 class LoginTypeEnum(str, Enum):
-    """登录方式枚举"""
+    """Login type enumeration"""
 
     QRCODE = "qrcode"
     PHONE = "phone"
@@ -57,7 +58,7 @@ class LoginTypeEnum(str, Enum):
 
 
 class CrawlerTypeEnum(str, Enum):
-    """爬虫类型枚举"""
+    """Crawler type enumeration"""
 
     SEARCH = "search"
     DETAIL = "detail"
@@ -65,19 +66,24 @@ class CrawlerTypeEnum(str, Enum):
 
 
 class SaveDataOptionEnum(str, Enum):
-    """数据保存方式枚举"""
+    """Data save option enumeration"""
 
     CSV = "csv"
     DB = "db"
     JSON = "json"
+    JSONL = "jsonl"
     SQLITE = "sqlite"
+    MONGODB = "mongodb"
+    EXCEL = "excel"
+    POSTGRES = "postgres"
 
 
 class InitDbOptionEnum(str, Enum):
-    """数据库初始化选项"""
+    """Database initialization option"""
 
     SQLITE = "sqlite"
     MYSQL = "mysql"
+    POSTGRES = "postgres"
 
 
 def _to_bool(value: bool | str) -> bool:
@@ -100,7 +106,7 @@ def _coerce_enum(
         return enum_cls(value)
     except ValueError:
         typer.secho(
-            f"⚠️ 配置值 '{value}' 不在 {enum_cls.__name__} 支持的范围内，已回退到默认值 '{default.value}'.",
+            f"⚠️ Config value '{value}' is not within the supported range of {enum_cls.__name__}, falling back to default value '{default.value}'.",
             fg=typer.colors.YELLOW,
         )
         return default
@@ -130,8 +136,23 @@ def _inject_init_db_default(args: Sequence[str]) -> list[str]:
     return normalized
 
 
+def _normalize_tieba_note_id(value: str) -> str:
+    """Accept a raw Tieba thread id or a /p/<id> URL."""
+    value = value.strip()
+    match = re.search(r"/p/(\d+)", value)
+    return match.group(1) if match else value
+
+
+def _normalize_tieba_creator_url(value: str) -> str:
+    """Accept a Tieba creator homepage URL or a portrait id."""
+    value = value.strip()
+    if value.startswith("http://") or value.startswith("https://"):
+        return value
+    return f"https://tieba.baidu.com/home/main?id={value}"
+
+
 async def parse_cmd(argv: Optional[Sequence[str]] = None):
-    """使用 Typer 解析命令行参数。"""
+    """Parse command line arguments using Typer."""
 
     app = typer.Typer(add_completion=False)
 
@@ -141,48 +162,48 @@ async def parse_cmd(argv: Optional[Sequence[str]] = None):
             PlatformEnum,
             typer.Option(
                 "--platform",
-                help="媒体平台选择 (xhs=小红书 | dy=抖音 | ks=快手 | bili=哔哩哔哩 | wb=微博 | tieba=百度贴吧 | zhihu=知乎)",
-                rich_help_panel="基础配置",
+                help="Media platform selection (xhs=XiaoHongShu | dy=Douyin | ks=Kuaishou | bili=Bilibili | wb=Weibo | tieba=Baidu Tieba | zhihu=Zhihu)",
+                rich_help_panel="Basic Configuration",
             ),
         ] = _coerce_enum(PlatformEnum, config.PLATFORM, PlatformEnum.XHS),
         lt: Annotated[
             LoginTypeEnum,
             typer.Option(
                 "--lt",
-                help="登录方式 (qrcode=二维码 | phone=手机号 | cookie=Cookie)",
-                rich_help_panel="账号配置",
+                help="Login type (qrcode=QR Code | phone=Phone | cookie=Cookie)",
+                rich_help_panel="Account Configuration",
             ),
         ] = _coerce_enum(LoginTypeEnum, config.LOGIN_TYPE, LoginTypeEnum.QRCODE),
         crawler_type: Annotated[
             CrawlerTypeEnum,
             typer.Option(
                 "--type",
-                help="爬取类型 (search=搜索 | detail=详情 | creator=创作者)",
-                rich_help_panel="基础配置",
+                help="Crawler type (search=Search | detail=Detail | creator=Creator)",
+                rich_help_panel="Basic Configuration",
             ),
         ] = _coerce_enum(CrawlerTypeEnum, config.CRAWLER_TYPE, CrawlerTypeEnum.SEARCH),
         start: Annotated[
             int,
             typer.Option(
                 "--start",
-                help="起始页码",
-                rich_help_panel="基础配置",
+                help="Starting page number",
+                rich_help_panel="Basic Configuration",
             ),
         ] = config.START_PAGE,
         keywords: Annotated[
             str,
             typer.Option(
                 "--keywords",
-                help="请输入关键词，多个关键词用逗号分隔",
-                rich_help_panel="基础配置",
+                help="Enter keywords, multiple keywords separated by commas",
+                rich_help_panel="Basic Configuration",
             ),
         ] = config.KEYWORDS,
         get_comment: Annotated[
             str,
             typer.Option(
                 "--get_comment",
-                help="是否爬取一级评论，支持 yes/true/t/y/1 或 no/false/f/n/0",
-                rich_help_panel="评论配置",
+                help="Whether to crawl first-level comments, supports yes/true/t/y/1 or no/false/f/n/0",
+                rich_help_panel="Comment Configuration",
                 show_default=True,
             ),
         ] = str(config.ENABLE_GET_COMMENTS),
@@ -190,44 +211,140 @@ async def parse_cmd(argv: Optional[Sequence[str]] = None):
             str,
             typer.Option(
                 "--get_sub_comment",
-                help="是否爬取二级评论，支持 yes/true/t/y/1 或 no/false/f/n/0",
-                rich_help_panel="评论配置",
+                help="Whether to crawl second-level comments, supports yes/true/t/y/1 or no/false/f/n/0",
+                rich_help_panel="Comment Configuration",
                 show_default=True,
             ),
         ] = str(config.ENABLE_GET_SUB_COMMENTS),
+        headless: Annotated[
+            str,
+            typer.Option(
+                "--headless",
+                help="Whether to enable headless mode (applies to both Playwright and CDP), supports yes/true/t/y/1 or no/false/f/n/0",
+                rich_help_panel="Runtime Configuration",
+                show_default=True,
+            ),
+        ] = str(config.HEADLESS),
         save_data_option: Annotated[
             SaveDataOptionEnum,
             typer.Option(
                 "--save_data_option",
-                help="数据保存方式 (csv=CSV文件 | db=MySQL数据库 | json=JSON文件 | sqlite=SQLite数据库)",
-                rich_help_panel="存储配置",
+                help="Data save option (csv=CSV file | db=MySQL database | json=JSON file | jsonl=JSONL file | sqlite=SQLite database | mongodb=MongoDB database | excel=Excel file | postgres=PostgreSQL database)",
+                rich_help_panel="Storage Configuration",
             ),
         ] = _coerce_enum(
-            SaveDataOptionEnum, config.SAVE_DATA_OPTION, SaveDataOptionEnum.JSON
+            SaveDataOptionEnum, config.SAVE_DATA_OPTION, SaveDataOptionEnum.JSONL
         ),
         init_db: Annotated[
             Optional[InitDbOptionEnum],
             typer.Option(
                 "--init_db",
-                help="初始化数据库表结构 (sqlite | mysql)",
-                rich_help_panel="存储配置",
+                help="Initialize database table structure (sqlite | mysql | postgres)",
+                rich_help_panel="Storage Configuration",
             ),
         ] = None,
         cookies: Annotated[
             str,
             typer.Option(
                 "--cookies",
-                help="Cookie 登录方式使用的 Cookie 值",
-                rich_help_panel="账号配置",
+                help="Cookie value used for Cookie login method",
+                rich_help_panel="Account Configuration",
             ),
         ] = config.COOKIES,
+        specified_id: Annotated[
+            str,
+            typer.Option(
+                "--specified_id",
+                help="Post/video ID list in detail mode, multiple IDs separated by commas (supports full URL or ID)",
+                rich_help_panel="Basic Configuration",
+            ),
+        ] = "",
+        creator_id: Annotated[
+            str,
+            typer.Option(
+                "--creator_id",
+                help="Creator ID list in creator mode, multiple IDs separated by commas (supports full URL or ID)",
+                rich_help_panel="Basic Configuration",
+            ),
+        ] = "",
+        max_comments_count_singlenotes: Annotated[
+            int,
+            typer.Option(
+                "--max_comments_count_singlenotes",
+                help="Maximum number of first-level comments to crawl per post/video",
+                rich_help_panel="Comment Configuration",
+            ),
+        ] = config.CRAWLER_MAX_COMMENTS_COUNT_SINGLENOTES,
+        crawler_max_notes_count: Annotated[
+            int,
+            typer.Option(
+                "--crawler_max_notes_count",
+                help="Maximum number of videos/posts to crawl",
+                rich_help_panel="Basic Configuration",
+            ),
+        ] = config.CRAWLER_MAX_NOTES_COUNT,
+        max_concurrency_num: Annotated[
+            int,
+            typer.Option(
+                "--max_concurrency_num",
+                help="Maximum number of concurrent crawlers",
+                rich_help_panel="Performance Configuration",
+            ),
+        ] = config.MAX_CONCURRENCY_NUM,
+        save_data_path: Annotated[
+            str,
+            typer.Option(
+                "--save_data_path",
+                help="Data save path, default is empty and will save to data folder",
+                rich_help_panel="Storage Configuration",
+            ),
+        ] = config.SAVE_DATA_PATH,
+        enable_ip_proxy: Annotated[
+            str,
+            typer.Option(
+                "--enable_ip_proxy",
+                help="Whether to enable IP proxy, supports yes/true/t/y/1 or no/false/f/n/0",
+                rich_help_panel="Proxy Configuration",
+                show_default=True,
+            ),
+        ] = str(config.ENABLE_IP_PROXY),
+        ip_proxy_pool_count: Annotated[
+            int,
+            typer.Option(
+                "--ip_proxy_pool_count",
+                help="IP proxy pool count",
+                rich_help_panel="Proxy Configuration",
+            ),
+        ] = config.IP_PROXY_POOL_COUNT,
+        ip_proxy_provider_name: Annotated[
+            str,
+            typer.Option(
+                "--ip_proxy_provider_name",
+                help="IP proxy provider name (kuaidaili | wandouhttp | static)",
+                rich_help_panel="Proxy Configuration",
+            ),
+        ] = config.IP_PROXY_PROVIDER_NAME,
+        static_proxy_url: Annotated[
+            str,
+            typer.Option(
+                "--static_proxy_url",
+                help="Static proxy URL, for example http://user:password@host:port",
+                rich_help_panel="Proxy Configuration",
+            ),
+        ] = config.STATIC_PROXY_URL,
     ) -> SimpleNamespace:
         """MediaCrawler 命令行入口"""
 
         enable_comment = _to_bool(get_comment)
         enable_sub_comment = _to_bool(get_sub_comment)
+        enable_headless = _to_bool(headless)
+        enable_ip_proxy_value = _to_bool(enable_ip_proxy)
         init_db_value = init_db.value if init_db else None
 
+        # Parse specified_id and creator_id into lists
+        specified_id_list = [id.strip() for id in specified_id.split(",") if id.strip()] if specified_id else []
+        creator_id_list = [id.strip() for id in creator_id.split(",") if id.strip()] if creator_id else []
+
         # override global config
         config.PLATFORM = platform.value
         config.LOGIN_TYPE = lt.value
@@ -236,8 +353,53 @@ async def parse_cmd(argv: Optional[Sequence[str]] = None):
         config.KEYWORDS = keywords
         config.ENABLE_GET_COMMENTS = enable_comment
         config.ENABLE_GET_SUB_COMMENTS = enable_sub_comment
+        config.HEADLESS = enable_headless
+        config.CDP_HEADLESS = enable_headless
         config.SAVE_DATA_OPTION = save_data_option.value
         config.COOKIES = cookies
+        config.CRAWLER_MAX_COMMENTS_COUNT_SINGLENOTES = max_comments_count_singlenotes
+        config.CRAWLER_MAX_NOTES_COUNT = crawler_max_notes_count
+        config.MAX_CONCURRENCY_NUM = max_concurrency_num
+        config.SAVE_DATA_PATH = save_data_path
+        config.ENABLE_IP_PROXY = enable_ip_proxy_value
+        config.IP_PROXY_POOL_COUNT = ip_proxy_pool_count
+        config.IP_PROXY_PROVIDER_NAME = ip_proxy_provider_name
+        config.STATIC_PROXY_URL = static_proxy_url
+
+        # Set platform-specific ID lists for detail/creator mode
+        if specified_id_list:
+            if platform == PlatformEnum.XHS:
+                config.XHS_SPECIFIED_NOTE_URL_LIST = specified_id_list
+            elif platform == PlatformEnum.BILIBILI:
+                config.BILI_SPECIFIED_ID_LIST = specified_id_list
+            elif platform == PlatformEnum.DOUYIN:
+                config.DY_SPECIFIED_ID_LIST = specified_id_list
+            elif platform == PlatformEnum.WEIBO:
+                config.WEIBO_SPECIFIED_ID_LIST = specified_id_list
+            elif platform == PlatformEnum.KUAISHOU:
+                config.KS_SPECIFIED_ID_LIST = specified_id_list
+            elif platform == PlatformEnum.TIEBA:
+                config.TIEBA_SPECIFIED_ID_LIST = [
+                    _normalize_tieba_note_id(item) for item in specified_id_list
+                ]
+            elif platform == PlatformEnum.ZHIHU:
+                config.ZHIHU_SPECIFIED_ID_LIST = specified_id_list
+
+        if creator_id_list:
+            if platform == PlatformEnum.XHS:
+                config.XHS_CREATOR_ID_LIST = creator_id_list
+            elif platform == PlatformEnum.BILIBILI:
+                config.BILI_CREATOR_ID_LIST = creator_id_list
+            elif platform == PlatformEnum.DOUYIN:
+                config.DY_CREATOR_ID_LIST = creator_id_list
+            elif platform == PlatformEnum.WEIBO:
+                config.WEIBO_CREATOR_ID_LIST = creator_id_list
+            elif platform == PlatformEnum.KUAISHOU:
+                config.KS_CREATOR_ID_LIST = creator_id_list
+            elif platform == PlatformEnum.TIEBA:
+                config.TIEBA_CREATOR_URL_LIST = [
+                    _normalize_tieba_creator_url(item) for item in creator_id_list
+                ]
 
         return SimpleNamespace(
             platform=config.PLATFORM,
@@ -247,9 +409,12 @@ async def parse_cmd(argv: Optional[Sequence[str]] = None):
             keywords=config.KEYWORDS,
             get_comment=config.ENABLE_GET_COMMENTS,
             get_sub_comment=config.ENABLE_GET_SUB_COMMENTS,
+            headless=config.HEADLESS,
             save_data_option=config.SAVE_DATA_OPTION,
             init_db=init_db_value,
             cookies=config.COOKIES,
+            specified_id=specified_id,
+            creator_id=creator_id,
         )
 
     command = typer.main.get_command(app)

config/base_config.py

@@ -17,106 +17,129 @@
 # 详细许可条款请参阅项目根目录下的LICENSE文件。
 # 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
 
-# 基础配置
-PLATFORM = "xhs"  # 平台，xhs | dy | ks | bili | wb | tieba | zhihu
-KEYWORDS = "编程副业,编程兼职"  # 关键词搜索配置，以英文逗号分隔
+# Basic configuration
+PLATFORM = "xhs"  # Platform, xhs | dy | ks | bili | wb | tieba | zhihu
+
+# 是否使用海外版小红书 (rednote.com)
+# 开启后 API 走 webapi.rednote.com，cookie 域使用 .rednote.com
+XHS_INTERNATIONAL = False
+
+KEYWORDS = "编程副业,编程兼职"  # Keyword search configuration, separated by English commas
 LOGIN_TYPE = "qrcode"  # qrcode or phone or cookie
 COOKIES = ""
 CRAWLER_TYPE = (
-    "search"  # 爬取类型，search(关键词搜索) | detail(帖子详情)| creator(创作者主页数据)
+    "search"  # Crawling type, search (keyword search) | detail (post details) | creator (creator homepage data)
 )
-# 是否开启 IP 代理
+# Whether to enable IP proxy
 ENABLE_IP_PROXY = False
 
-# 代理IP池数量
+# Number of proxy IP pools
 IP_PROXY_POOL_COUNT = 2
 
-# 代理IP提供商名称
-IP_PROXY_PROVIDER_NAME = "kuaidaili"  # kuaidaili | wandouhttp
+# Proxy IP provider name
+IP_PROXY_PROVIDER_NAME = "kuaidaili"  # kuaidaili | wandouhttp | static
 
-# 设置为True不会打开浏览器（无头浏览器）
-# 设置False会打开一个浏览器
-# 小红书如果一直扫码登录不通过，打开浏览器手动过一下滑动验证码
-# 抖音如果一直提示失败，打开浏览器看下是否扫码登录之后出现了手机号验证，如果出现了手动过一下再试。
+# Static proxy configuration (used when IP_PROXY_PROVIDER_NAME is set to "static")
+# Format: "http://your_home_domain:port" or "http://user:password@your_home_domain:port"
+STATIC_PROXY_URL = ""
+
+# Setting to True will not open the browser (headless browser)
+# Setting False will open a browser
+# If Xiaohongshu keeps scanning the code to log in but fails, open the browser and manually pass the sliding verification code.
+# If Douyin keeps prompting failure, open the browser and see if mobile phone number verification appears after scanning the QR code to log in. If it does, manually go through it and try again.
 HEADLESS = False
 
-# 是否保存登录状态
+# Whether to save login status
 SAVE_LOGIN_STATE = True
 
 # ==================== CDP (Chrome DevTools Protocol) 配置 ====================
-# 是否启用CDP模式 - 使用用户现有的Chrome/Edge浏览器进行爬取，提供更好的反检测能力
-# 启用后将自动检测并启动用户的Chrome/Edge浏览器，通过CDP协议进行控制
-# 这种方式使用真实的浏览器环境，包括用户的扩展、Cookie和设置，大大降低被检测的风险
+# 是否启用 CDP 模式 - 使用用户本地的 Chrome/Edge 浏览器进行爬取，具有更好的反检测能力
+# 开启后，会自动检测并启动用户的 Chrome/Edge 浏览器，通过 CDP 协议进行控制
+# 该方式使用真实浏览器环境，包括用户的扩展、Cookie 和设置，大幅降低被风控检测的风险
 ENABLE_CDP_MODE = True
 
-# CDP调试端口，用于与浏览器通信
+# CDP 调试端口，用于与浏览器通信
 # 如果端口被占用，系统会自动尝试下一个可用端口
 CDP_DEBUG_PORT = 9222
 
 # 自定义浏览器路径（可选）
-# 如果为空，系统会自动检测Chrome/Edge的安装路径
-# Windows示例: "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe"
-# macOS示例: "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
+# 如果为空，系统会自动检测 Chrome/Edge 的安装路径
+# Windows 示例: "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe"
+# macOS 示例: "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
 CUSTOM_BROWSER_PATH = ""
 
-# CDP模式下是否启用无头模式
-# 注意：即使设置为True，某些反检测功能在无头模式下可能效果不佳
+# 是否在 CDP 模式下启用无头模式
+# 注意：即使设置为 True，某些反检测功能在无头模式下可能无法正常工作
 CDP_HEADLESS = False
 
 # 浏览器启动超时时间（秒）
 BROWSER_LAUNCH_TIMEOUT = 60
 
-# 是否在程序结束时自动关闭浏览器
-# 设置为False可以保持浏览器运行，便于调试
+# 是否连接用户已打开的浏览器，而不是启动新的浏览器
+# 开启后，程序会连接一个已经启用了远程调试的浏览器
+# 用户需要在 Chrome 中开启远程调试：chrome://inspect/#remote-debugging
+# 或者使用命令行参数启动 Chrome：--remote-debugging-port=9222
+# 这种方式反检测效果最好，因为直接使用用户真实浏览器的所有 Cookie、扩展和浏览历史
+CDP_CONNECT_EXISTING = True
+
+# 程序结束时是否自动关闭浏览器
+# 设置为 False 可以保持浏览器运行，方便调试
 AUTO_CLOSE_BROWSER = True
 
-# 数据保存类型选项配置,支持四种类型：csv、db、json、sqlite, 最好保存到DB，有排重的功能。
-SAVE_DATA_OPTION = "json"  # csv or db or json or sqlite
+# Data saving type option configuration, supports: csv, db, json, jsonl, sqlite, excel, postgres. It is best to save to DB, with deduplication function.
+SAVE_DATA_OPTION = "jsonl"  # csv or db or json or jsonl or sqlite or excel or postgres
 
-# 用户浏览器缓存的浏览器文件配置
+# Data saving path, if not specified by default, it will be saved to the data folder.
+SAVE_DATA_PATH = ""
+
+# Browser file configuration cached by the user's browser
 USER_DATA_DIR = "%s_user_data_dir"  # %s will be replaced by platform name
 
-# 爬取开始页数 默认从第一页开始
+# The number of pages to start crawling starts from the first page by default
 START_PAGE = 1
 
-# 爬取视频/帖子的数量控制
+# Control the number of crawled videos/posts
 CRAWLER_MAX_NOTES_COUNT = 15
 
-# 并发爬虫数量控制
+# Controlling the number of concurrent crawlers
 MAX_CONCURRENCY_NUM = 1
 
-# 是否开启爬媒体模式（包含图片或视频资源），默认不开启爬媒体
+# Whether to enable crawling media mode (including image or video resources), crawling media is not enabled by default
 ENABLE_GET_MEIDAS = False
 
-# 是否开启爬评论模式, 默认开启爬评论
+# Whether to enable comment crawling mode. Comment crawling is enabled by default.
 ENABLE_GET_COMMENTS = True
 
-# 爬取一级评论的数量控制(单视频/帖子)
+# Control the number of crawled first-level comments (single video/post)
 CRAWLER_MAX_COMMENTS_COUNT_SINGLENOTES = 10
 
-# 是否开启爬二级评论模式, 默认不开启爬二级评论
-# 老版本项目使用了 db, 则需参考 schema/tables.sql line 287 增加表字段
+# Whether to enable the mode of crawling second-level comments. By default, crawling of second-level comments is not enabled.
+# If the old version of the project uses db, you need to refer to schema/tables.sql line 287 to add table fields.
 ENABLE_GET_SUB_COMMENTS = False
 
-# 词云相关
-# 是否开启生成评论词云图
+# word cloud related
+# Whether to enable generating comment word clouds
 ENABLE_GET_WORDCLOUD = False
-# 自定义词语及其分组
-# 添加规则：xx:yy 其中xx为自定义添加的词组，yy为将xx该词组分到的组名。
+# Custom words and their groups
+# Add rule: xx:yy where xx is a custom-added phrase, and yy is the group name to which the phrase xx is assigned.
 CUSTOM_WORDS = {
-    "零几": "年份",  # 将“零几”识别为一个整体
-    "高频词": "专业术语",  # 示例自定义词
+    "零几": "年份",  # Recognize "zero points" as a whole
+    "高频词": "专业术语",  # Example custom words
 }
 
-# 停用(禁用)词文件路径
+# Deactivate (disabled) word file path
 STOP_WORDS_FILE = "./docs/hit_stopwords.txt"
 
-# 中文字体文件路径
+# Chinese font file path
 FONT_PATH = "./docs/STZHONGS.TTF"
 
-# 爬取间隔时间
+# Crawl interval
 CRAWLER_MAX_SLEEP_SEC = 2
 
+# 是否禁用 SSL 证书验证。仅在使用企业代理、Burp Suite、mitmproxy 等会注入自签名证书的中间人代理时设为 True。
+# 警告：禁用 SSL 验证将使所有流量暴露于中间人攻击风险，请勿在生产环境中开启。
+DISABLE_SSL_VERIFY = False
+
 from .bilibili_config import *
 from .xhs_config import *
 from .dy_config import *

config/bilibili_config.py

@@ -16,15 +16,15 @@
 #
 # 详细许可条款请参阅项目根目录下的LICENSE文件。
 # 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
-# bilili 平台配置
+# bilili platform configuration
 
-# 每天爬取视频/帖子的数量控制
+# Control the number of videos/posts crawled per day
 MAX_NOTES_PER_DAY = 1
 
-# 指定B站视频URL列表 (支持完整URL或BV号)
-# 示例:
-# - 完整URL: "https://www.bilibili.com/video/BV1dwuKzmE26/?spm_id_from=333.1387.homepage.video_card.click"
-# - BV号: "BV1d54y1g7db"
+# Specify Bilibili video URL list (supports complete URL or BV number)
+# Example:
+# - Full URL: "https://www.bilibili.com/video/BV1dwuKzmE26/?spm_id_from=333.1387.homepage.video_card.click"
+# - BV number: "BV1d54y1g7db"
 BILI_SPECIFIED_ID_LIST = [
     "https://www.bilibili.com/video/BV1dwuKzmE26/?spm_id_from=333.1387.homepage.video_card.click",
     "BV1Sz4y1U77N",
@@ -32,9 +32,9 @@ BILI_SPECIFIED_ID_LIST = [
     # ........................
 ]
 
-# 指定B站创作者URL列表 (支持完整URL或UID)
-# 示例:
-# - 完整URL: "https://space.bilibili.com/434377496?spm_id_from=333.1007.0.0"
+# Specify the URL list of Bilibili creators (supports full URL or UID)
+# Example:
+# - Full URL: "https://space.bilibili.com/434377496?spm_id_from=333.1007.0.0"
 # - UID: "20813884"
 BILI_CREATOR_ID_LIST = [
     "https://space.bilibili.com/434377496?spm_id_from=333.1007.0.0",
@@ -42,26 +42,26 @@ BILI_CREATOR_ID_LIST = [
     # ........................
 ]
 
-# 指定时间范围
+# Specify time range
 START_DAY = "2024-01-01"
 END_DAY = "2024-01-01"
 
-# 搜索模式
+# Search mode
 BILI_SEARCH_MODE = "normal"
 
-# 视频清晰度（qn）配置，常见取值：
-# 16=360p, 32=480p, 64=720p, 80=1080p, 112=1080p高码率, 116=1080p60, 120=4K
-# 注意：更高清晰度需要账号/视频本身支持
+# Video definition (qn) configuration, common values:
+# 16=360p, 32=480p, 64=720p, 80=1080p, 112=1080p high bit rate, 116=1080p60, 120=4K
+# Note: Higher definition requires account/video support
 BILI_QN = 80
 
-# 是否爬取用户信息
+# Whether to crawl user information
 CREATOR_MODE = True
 
-# 开始爬取用户信息页码
+# Start crawling user information page number
 START_CONTACTS_PAGE = 1
 
-# 单个视频/帖子最大爬取评论数
+# Maximum number of crawled comments for a single video/post
 CRAWLER_MAX_CONTACTS_COUNT_SINGLENOTES = 100
 
-# 单个视频/帖子最大爬取动态数
+# Maximum number of crawled dynamics for a single video/post
 CRAWLER_MAX_DYNAMICS_COUNT_SINGLENOTES = 50

config/db_config.py

@@ -37,7 +37,7 @@ mysql_db_config = {
 
 
 # redis config
-REDIS_DB_HOST = "127.0.0.1"  # your redis host
+REDIS_DB_HOST = os.getenv("REDIS_DB_HOST", "127.0.0.1")  # your redis host
 REDIS_DB_PWD = os.getenv("REDIS_DB_PWD", "123456")  # your redis password
 REDIS_DB_PORT = os.getenv("REDIS_DB_PORT", 6379)  # your redis port
 REDIS_DB_NUM = os.getenv("REDIS_DB_NUM", 0)  # your redis db num
@@ -67,3 +67,18 @@ mongodb_config = {
     "password": MONGODB_PWD,
     "db_name": MONGODB_DB_NAME,
 }
+
+# postgres config
+POSTGRES_DB_PWD = os.getenv("POSTGRES_DB_PWD", "123456")
+POSTGRES_DB_USER = os.getenv("POSTGRES_DB_USER", "postgres")
+POSTGRES_DB_HOST = os.getenv("POSTGRES_DB_HOST", "localhost")
+POSTGRES_DB_PORT = os.getenv("POSTGRES_DB_PORT", 5432)
+POSTGRES_DB_NAME = os.getenv("POSTGRES_DB_NAME", "media_crawler")
+
+postgres_db_config = {
+    "user": POSTGRES_DB_USER,
+    "password": POSTGRES_DB_PWD,
+    "host": POSTGRES_DB_HOST,
+    "port": POSTGRES_DB_PORT,
+    "db_name": POSTGRES_DB_NAME,
+}

config/dy_config.py

@@ -17,16 +17,16 @@
 # 详细许可条款请参阅项目根目录下的LICENSE文件。
 # 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
 
-# 抖音平台配置
+# Douyin platform configuration
 PUBLISH_TIME_TYPE = 0
 
-# 指定DY视频URL列表 (支持多种格式)
-# 支持格式:
-# 1. 完整视频URL: "https://www.douyin.com/video/7525538910311632128"
-# 2. 带modal_id的URL: "https://www.douyin.com/user/xxx?modal_id=7525538910311632128"
-# 3. 搜索页带modal_id: "https://www.douyin.com/root/search/python?modal_id=7525538910311632128"
-# 4. 短链接: "https://v.douyin.com/drIPtQ_WPWY/"
-# 5. 纯视频ID: "7280854932641664319"
+# Specify DY video URL list (supports multiple formats)
+# Supported formats:
+# 1. Full video URL: "https://www.douyin.com/video/7525538910311632128"
+# 2. URL with modal_id: "https://www.douyin.com/user/xxx?modal_id=7525538910311632128"
+# 3. The search page has modal_id: "https://www.douyin.com/root/search/python?modal_id=7525538910311632128"
+# 4. Short link: "https://v.douyin.com/drIPtQ_WPWY/"
+# 5. Pure video ID: "7280854932641664319"
 DY_SPECIFIED_ID_LIST = [
     "https://www.douyin.com/video/7525538910311632128",
     "https://v.douyin.com/drIPtQ_WPWY/",
@@ -35,9 +35,9 @@ DY_SPECIFIED_ID_LIST = [
     # ........................
 ]
 
-# 指定DY创作者URL列表 (支持完整URL或sec_user_id)
-# 支持格式:
-# 1. 完整创作者主页URL: "https://www.douyin.com/user/MS4wLjABAAAATJPY7LAlaa5X-c8uNdWkvz0jUGgpw4eeXIwu_8BhvqE?from_tab_name=main"
+# Specify DY creator URL list (supports full URL or sec_user_id)
+# Supported formats:
+# 1. Complete creator homepage URL: "https://www.douyin.com/user/MS4wLjABAAAATJPY7LAlaa5X-c8uNdWkvz0jUGgpw4eeXIwu_8BhvqE?from_tab_name=main"
 # 2. sec_user_id: "MS4wLjABAAAATJPY7LAlaa5X-c8uNdWkvz0jUGgpw4eeXIwu_8BhvqE"
 DY_CREATOR_ID_LIST = [
     "https://www.douyin.com/user/MS4wLjABAAAATJPY7LAlaa5X-c8uNdWkvz0jUGgpw4eeXIwu_8BhvqE?from_tab_name=main",

config/ks_config.py

@@ -17,22 +17,22 @@
 # 详细许可条款请参阅项目根目录下的LICENSE文件。
 # 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
 
-# 快手平台配置
+# Kuaishou platform configuration
 
-# 指定快手视频URL列表 (支持完整URL或纯ID)
-# 支持格式:
-# 1. 完整视频URL: "https://www.kuaishou.com/short-video/3x3zxz4mjrsc8ke?authorId=3x84qugg4ch9zhs&streamSource=search"
-# 2. 纯视频ID: "3xf8enb8dbj6uig"
+# Specify Kuaishou video URL list (supports complete URL or pure ID)
+# Supported formats:
+# 1. Full video URL: "https://www.kuaishou.com/short-video/3x3zxz4mjrsc8ke?authorId=3x84qugg4ch9zhs&streamSource=search"
+# 2. Pure video ID: "3xf8enb8dbj6uig"
 KS_SPECIFIED_ID_LIST = [
     "https://www.kuaishou.com/short-video/3x3zxz4mjrsc8ke?authorId=3x84qugg4ch9zhs&streamSource=search&area=searchxxnull&searchKey=python",
     "3xf8enb8dbj6uig",
     # ........................
 ]
 
-# 指定快手创作者URL列表 (支持完整URL或纯ID)
-# 支持格式:
-# 1. 创作者主页URL: "https://www.kuaishou.com/profile/3x84qugg4ch9zhs"
-# 2. 纯user_id: "3x4sm73aye7jq7i"
+# Specify Kuaishou creator URL list (supports full URL or pure ID)
+# Supported formats:
+# 1. Creator homepage URL: "https://www.kuaishou.com/profile/3x84qugg4ch9zhs"
+# 2. Pure user_id: "3x4sm73aye7jq7i"
 KS_CREATOR_ID_LIST = [
     "https://www.kuaishou.com/profile/3x84qugg4ch9zhs",
     "3x4sm73aye7jq7i",

config/tieba_config.py

@@ -17,17 +17,17 @@
 # 详细许可条款请参阅项目根目录下的LICENSE文件。
 # 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
 
-# 贴吧平台配置
+# Tieba platform configuration
 
-# 指定贴吧ID列表
+# Specify Tieba ID list
 TIEBA_SPECIFIED_ID_LIST = []
 
-# 指定贴吧名称列表
+# Specify a list of Tieba names
 TIEBA_NAME_LIST = [
-    # "盗墓笔记"
+    # "Tomb Robbery Notes"
 ]
 
-# 指定贴吧用户URL列表
+# Specify Tieba user URL list
 TIEBA_CREATOR_URL_LIST = [
     "https://tieba.baidu.com/home/main/?id=tb.1.7f139e2e.6CyEwxu3VJruH_-QqpCi6g&fr=frs",
     # ........................

config/weibo_config.py

@@ -18,19 +18,23 @@
 # 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
 
 
-# 微博平台配置
+# Weibo platform configuration
 
-# 搜索类型，具体的枚举值在media_platform/weibo/field.py中
+# Search type, the specific enumeration value is in media_platform/weibo/field.py
 WEIBO_SEARCH_TYPE = "default"
 
-# 指定微博ID列表
+# Specify Weibo ID list
 WEIBO_SPECIFIED_ID_LIST = [
     "4982041758140155",
     # ........................
 ]
 
-# 指定微博用户ID列表
+# Specify Weibo user ID list
 WEIBO_CREATOR_ID_LIST = [
-    "5533390220",
+    "5756404150",
     # ........................
 ]
+
+# Whether to enable the function of crawling the full text of Weibo. It is enabled by default.
+# If turned on, it will increase the probability of being risk controlled, which is equivalent to a keyword search request that will traverse all posts and request the post details again.
+ENABLE_WEIBO_FULL_TEXT = True

config/xhs_config.py

@@ -18,18 +18,18 @@
 # 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
 
 
-# 小红书平台配置
+# Xiaohongshu platform configuration
 
-# 排序方式，具体的枚举值在media_platform/xhs/field.py中
+# Sorting method, the specific enumeration value is in media_platform/xhs/field.py
 SORT_TYPE = "popularity_descending"
 
-# 指定笔记URL列表, 必须要携带xsec_token参数
+# Specify the note URL list, which must carry the xsec_token parameter
 XHS_SPECIFIED_NOTE_URL_LIST = [
     "https://www.xiaohongshu.com/explore/64b95d01000000000c034587?xsec_token=AB0EFqJvINCkj6xOCKCQgfNNh8GdnBC_6XecG4QOddo3Q=&xsec_source=pc_cfeed"
     # ........................
 ]
 
-# 指定创作者URL列表，需要携带xsec_token和xsec_source参数
+# Specify the creator URL list, which needs to carry xsec_token and xsec_source parameters.
 
 XHS_CREATOR_ID_LIST = [
     "https://www.xiaohongshu.com/user/profile/5f58bd990000000001003753?xsec_token=ABYVg1evluJZZzpMX-VWzchxQ1qSNVW3r-jOEnKqMcgZw=&xsec_source=pc_search"

config/zhihu_config.py

@@ -18,17 +18,17 @@
 # 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
 
 
-# 知乎平台配置
+# Zhihu platform configuration
 
-# 指定知乎用户URL列表
+# Specify Zhihu user URL list
 ZHIHU_CREATOR_URL_LIST = [
     "https://www.zhihu.com/people/yd1234567",
     # ........................
 ]
 
-# 指定知乎ID列表
+# Specify Zhihu ID list
 ZHIHU_SPECIFIED_ID_LIST = [
-    "https://www.zhihu.com/question/826896610/answer/4885821440",  # 回答
-    "https://zhuanlan.zhihu.com/p/673461588",  # 文章
-    "https://www.zhihu.com/zvideo/1539542068422144000",  # 视频
+    "https://www.zhihu.com/question/826896610/answer/4885821440",  # answer
+    "https://zhuanlan.zhihu.com/p/673461588",  # article
+    "https://www.zhihu.com/zvideo/1539542068422144000",  # video
 ]

database/db.py

@@ -17,9 +17,9 @@
 # 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
 
 # persist-1<persist1@126.com>
-# 原因：将 db.py 改造为模块，移除直接执行入口，修复相对导入问题。
-# 副作用：无
-# 回滚策略：还原此文件。
+# Reason: Refactored db.py into a module, removed direct execution entry point, fixed relative import issues.
+# Side effects: None
+# Rollback strategy: Restore this file.
 import asyncio
 import sys
 from pathlib import Path

database/db_session.py

@@ -22,7 +22,7 @@ from sqlalchemy.orm import sessionmaker
 from contextlib import asynccontextmanager
 from .models import Base
 import config
-from config.db_config import mysql_db_config, sqlite_db_config
+from config.db_config import mysql_db_config, sqlite_db_config, postgres_db_config
 
 # Keep a cache of engines
 _engines = {}
@@ -36,6 +36,18 @@ async def create_database_if_not_exists(db_type: str):
         async with engine.connect() as conn:
             await conn.execute(text(f"CREATE DATABASE IF NOT EXISTS {mysql_db_config['db_name']}"))
         await engine.dispose()
+    elif db_type == "postgres":
+        # Connect to the default 'postgres' database
+        server_url = f"postgresql+asyncpg://{postgres_db_config['user']}:{postgres_db_config['password']}@{postgres_db_config['host']}:{postgres_db_config['port']}/postgres"
+        print(f"[init_db] Connecting to Postgres: host={postgres_db_config['host']}, port={postgres_db_config['port']}, user={postgres_db_config['user']}, dbname=postgres")
+        # Isolation level AUTOCOMMIT is required for CREATE DATABASE
+        engine = create_async_engine(server_url, echo=False, isolation_level="AUTOCOMMIT")
+        async with engine.connect() as conn:
+            # Check if database exists
+            result = await conn.execute(text(f"SELECT 1 FROM pg_database WHERE datname = '{postgres_db_config['db_name']}'"))
+            if not result.scalar():
+                await conn.execute(text(f"CREATE DATABASE {postgres_db_config['db_name']}"))
+        await engine.dispose()
 
 
 def get_async_engine(db_type: str = None):
@@ -45,13 +57,15 @@ def get_async_engine(db_type: str = None):
     if db_type in _engines:
         return _engines[db_type]
 
-    if db_type in ["json", "csv"]:
+    if db_type in ["json", "jsonl", "csv"]:
         return None
 
     if db_type == "sqlite":
         db_url = f"sqlite+aiosqlite:///{sqlite_db_config['db_path']}"
     elif db_type == "mysql" or db_type == "db":
         db_url = f"mysql+asyncmy://{mysql_db_config['user']}:{mysql_db_config['password']}@{mysql_db_config['host']}:{mysql_db_config['port']}/{mysql_db_config['db_name']}"
+    elif db_type == "postgres":
+        db_url = f"postgresql+asyncpg://{postgres_db_config['user']}:{postgres_db_config['password']}@{postgres_db_config['host']}:{postgres_db_config['port']}/{postgres_db_config['db_name']}"
     else:
         raise ValueError(f"Unsupported database type: {db_type}")
 

database/models.py

@@ -24,429 +24,429 @@ Base = declarative_base()
 
 class BilibiliVideo(Base):
     __tablename__ = 'bilibili_video'
-    id = Column(Integer, primary_key=True)
-    video_id = Column(BigInteger, nullable=False, index=True, unique=True)
-    video_url = Column(Text, nullable=False)
-    user_id = Column(BigInteger, index=True)
-    nickname = Column(Text)
-    avatar = Column(Text)
-    liked_count = Column(Integer)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
-    video_type = Column(Text)
-    title = Column(Text)
-    desc = Column(Text)
-    create_time = Column(BigInteger, index=True)
-    disliked_count = Column(Text)
-    video_play_count = Column(Text)
-    video_favorite_count = Column(Text)
-    video_share_count = Column(Text)
-    video_coin_count = Column(Text)
-    video_danmaku = Column(Text)
-    video_comment = Column(Text)
-    video_cover_url = Column(Text)
-    source_keyword = Column(Text, default='')
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    video_id = Column(BigInteger, nullable=False, index=True, unique=True, comment='视频ID')
+    video_url = Column(Text, nullable=False, comment='视频URL')
+    user_id = Column(BigInteger, index=True, comment='用户ID')
+    nickname = Column(Text, comment='用户昵称')
+    avatar = Column(Text, comment='用户头像')
+    liked_count = Column(Integer, comment='点赞数')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
+    video_type = Column(Text, comment='视频类型')
+    title = Column(Text, comment='视频标题')
+    desc = Column(Text, comment='视频描述')
+    create_time = Column(BigInteger, index=True, comment='创建时间戳')
+    disliked_count = Column(Text, comment='点踩数')
+    video_play_count = Column(Text, comment='播放数')
+    video_favorite_count = Column(Text, comment='收藏数')
+    video_share_count = Column(Text, comment='分享数')
+    video_coin_count = Column(Text, comment='硬币数')
+    video_danmaku = Column(Text, comment='弹幕数')
+    video_comment = Column(Text, comment='评论数')
+    video_cover_url = Column(Text, comment='视频封面URL')
+    source_keyword = Column(Text, default='', comment='来源关键词')
 
 class BilibiliVideoComment(Base):
     __tablename__ = 'bilibili_video_comment'
-    id = Column(Integer, primary_key=True)
-    user_id = Column(String(255))
-    nickname = Column(Text)
-    sex = Column(Text)
-    sign = Column(Text)
-    avatar = Column(Text)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
-    comment_id = Column(BigInteger, index=True)
-    video_id = Column(BigInteger, index=True)
-    content = Column(Text)
-    create_time = Column(BigInteger)
-    sub_comment_count = Column(Text)
-    parent_comment_id = Column(String(255))
-    like_count = Column(Text, default='0')
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    user_id = Column(String(255), comment='用户ID')
+    nickname = Column(Text, comment='用户昵称')
+    sex = Column(Text, comment='性别')
+    sign = Column(Text, comment='签名')
+    avatar = Column(Text, comment='头像')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
+    comment_id = Column(BigInteger, index=True, comment='评论ID')
+    video_id = Column(BigInteger, index=True, comment='视频ID')
+    content = Column(Text, comment='评论内容')
+    create_time = Column(BigInteger, comment='创建时间戳')
+    sub_comment_count = Column(Text, comment='子评论数')
+    parent_comment_id = Column(String(255), comment='父评论ID')
+    like_count = Column(Text, default='0', comment='点赞数')
 
 class BilibiliUpInfo(Base):
     __tablename__ = 'bilibili_up_info'
-    id = Column(Integer, primary_key=True)
-    user_id = Column(BigInteger, index=True)
-    nickname = Column(Text)
-    sex = Column(Text)
-    sign = Column(Text)
-    avatar = Column(Text)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
-    total_fans = Column(Integer)
-    total_liked = Column(Integer)
-    user_rank = Column(Integer)
-    is_official = Column(Integer)
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    user_id = Column(BigInteger, index=True, comment='用户ID')
+    nickname = Column(Text, comment='用户昵称')
+    sex = Column(Text, comment='性别')
+    sign = Column(Text, comment='签名')
+    avatar = Column(Text, comment='头像')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
+    total_fans = Column(Integer, comment='总粉丝数')
+    total_liked = Column(Integer, comment='总获赞数')
+    user_rank = Column(Integer, comment='用户等级')
+    is_official = Column(Integer, comment='是否官方认证')
 
 class BilibiliContactInfo(Base):
     __tablename__ = 'bilibili_contact_info'
-    id = Column(Integer, primary_key=True)
-    up_id = Column(BigInteger, index=True)
-    fan_id = Column(BigInteger, index=True)
-    up_name = Column(Text)
-    fan_name = Column(Text)
-    up_sign = Column(Text)
-    fan_sign = Column(Text)
-    up_avatar = Column(Text)
-    fan_avatar = Column(Text)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    up_id = Column(BigInteger, index=True, comment='UP主ID')
+    fan_id = Column(BigInteger, index=True, comment='粉丝ID')
+    up_name = Column(Text, comment='UP主名称')
+    fan_name = Column(Text, comment='粉丝名称')
+    up_sign = Column(Text, comment='UP主签名')
+    fan_sign = Column(Text, comment='粉丝签名')
+    up_avatar = Column(Text, comment='UP主头像')
+    fan_avatar = Column(Text, comment='粉丝头像')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
 
 class BilibiliUpDynamic(Base):
     __tablename__ = 'bilibili_up_dynamic'
-    id = Column(Integer, primary_key=True)
-    dynamic_id = Column(BigInteger, index=True)
-    user_id = Column(String(255))
-    user_name = Column(Text)
-    text = Column(Text)
-    type = Column(Text)
-    pub_ts = Column(BigInteger)
-    total_comments = Column(Integer)
-    total_forwards = Column(Integer)
-    total_liked = Column(Integer)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    dynamic_id = Column(BigInteger, index=True, comment='动态ID')
+    user_id = Column(String(255), comment='用户ID')
+    user_name = Column(Text, comment='用户名称')
+    text = Column(Text, comment='动态内容')
+    type = Column(Text, comment='动态类型')
+    pub_ts = Column(BigInteger, comment='发布时间戳')
+    total_comments = Column(Integer, comment='总评论数')
+    total_forwards = Column(Integer, comment='总转发数')
+    total_liked = Column(Integer, comment='总点赞数')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
 
 class DouyinAweme(Base):
     __tablename__ = 'douyin_aweme'
-    id = Column(Integer, primary_key=True)
-    user_id = Column(String(255))
-    sec_uid = Column(String(255))
-    short_user_id = Column(String(255))
-    user_unique_id = Column(String(255))
-    nickname = Column(Text)
-    avatar = Column(Text)
-    user_signature = Column(Text)
-    ip_location = Column(Text)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
-    aweme_id = Column(BigInteger, index=True)
-    aweme_type = Column(Text)
-    title = Column(Text)
-    desc = Column(Text)
-    create_time = Column(BigInteger, index=True)
-    liked_count = Column(Text)
-    comment_count = Column(Text)
-    share_count = Column(Text)
-    collected_count = Column(Text)
-    aweme_url = Column(Text)
-    cover_url = Column(Text)
-    video_download_url = Column(Text)
-    music_download_url = Column(Text)
-    note_download_url = Column(Text)
-    source_keyword = Column(Text, default='')
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    user_id = Column(String(255), comment='用户ID')
+    sec_uid = Column(String(255), comment='安全用户ID')
+    short_user_id = Column(String(255), comment='短用户ID')
+    user_unique_id = Column(String(255), comment='用户唯一ID')
+    nickname = Column(Text, comment='用户昵称')
+    avatar = Column(Text, comment='用户头像')
+    user_signature = Column(Text, comment='用户签名')
+    ip_location = Column(Text, comment='IP地址位置')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
+    aweme_id = Column(BigInteger, index=True, comment='作品ID')
+    aweme_type = Column(Text, comment='作品类型')
+    title = Column(Text, comment='作品标题')
+    desc = Column(Text, comment='作品描述')
+    create_time = Column(BigInteger, index=True, comment='创建时间戳')
+    liked_count = Column(Text, comment='点赞数')
+    comment_count = Column(Text, comment='评论数')
+    share_count = Column(Text, comment='分享数')
+    collected_count = Column(Text, comment='收藏数')
+    aweme_url = Column(Text, comment='作品URL')
+    cover_url = Column(Text, comment='封面URL')
+    video_download_url = Column(Text, comment='视频下载URL')
+    music_download_url = Column(Text, comment='音乐下载URL')
+    note_download_url = Column(Text, comment='笔记下载URL')
+    source_keyword = Column(Text, default='', comment='来源关键词')
 
 class DouyinAwemeComment(Base):
     __tablename__ = 'douyin_aweme_comment'
-    id = Column(Integer, primary_key=True)
-    user_id = Column(String(255))
-    sec_uid = Column(String(255))
-    short_user_id = Column(String(255))
-    user_unique_id = Column(String(255))
-    nickname = Column(Text)
-    avatar = Column(Text)
-    user_signature = Column(Text)
-    ip_location = Column(Text)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
-    comment_id = Column(BigInteger, index=True)
-    aweme_id = Column(BigInteger, index=True)
-    content = Column(Text)
-    create_time = Column(BigInteger)
-    sub_comment_count = Column(Text)
-    parent_comment_id = Column(String(255))
-    like_count = Column(Text, default='0')
-    pictures = Column(Text, default='')
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    user_id = Column(String(255), comment='用户ID')
+    sec_uid = Column(String(255), comment='安全用户ID')
+    short_user_id = Column(String(255), comment='短用户ID')
+    user_unique_id = Column(String(255), comment='用户唯一ID')
+    nickname = Column(Text, comment='用户昵称')
+    avatar = Column(Text, comment='用户头像')
+    user_signature = Column(Text, comment='用户签名')
+    ip_location = Column(Text, comment='IP地址位置')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
+    comment_id = Column(BigInteger, index=True, comment='评论ID')
+    aweme_id = Column(BigInteger, index=True, comment='作品ID')
+    content = Column(Text, comment='评论内容')
+    create_time = Column(BigInteger, comment='创建时间戳')
+    sub_comment_count = Column(Text, comment='子评论数')
+    parent_comment_id = Column(String(255), comment='父评论ID')
+    like_count = Column(Text, default='0', comment='点赞数')
+    pictures = Column(Text, default='', comment='图片')
 
 class DyCreator(Base):
     __tablename__ = 'dy_creator'
-    id = Column(Integer, primary_key=True)
-    user_id = Column(String(255))
-    nickname = Column(Text)
-    avatar = Column(Text)
-    ip_location = Column(Text)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
-    desc = Column(Text)
-    gender = Column(Text)
-    follows = Column(Text)
-    fans = Column(Text)
-    interaction = Column(Text)
-    videos_count = Column(String(255))
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    user_id = Column(String(255), comment='用户ID')
+    nickname = Column(Text, comment='用户昵称')
+    avatar = Column(Text, comment='用户头像')
+    ip_location = Column(Text, comment='IP地址位置')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
+    desc = Column(Text, comment='描述')
+    gender = Column(Text, comment='性别')
+    follows = Column(Text, comment='关注数')
+    fans = Column(Text, comment='粉丝数')
+    interaction = Column(Text, comment='互动数')
+    videos_count = Column(String(255), comment='视频数量')
 
 class KuaishouVideo(Base):
     __tablename__ = 'kuaishou_video'
-    id = Column(Integer, primary_key=True)
-    user_id = Column(String(64))
-    nickname = Column(Text)
-    avatar = Column(Text)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
-    video_id = Column(String(255), index=True)
-    video_type = Column(Text)
-    title = Column(Text)
-    desc = Column(Text)
-    create_time = Column(BigInteger, index=True)
-    liked_count = Column(Text)
-    viewd_count = Column(Text)
-    video_url = Column(Text)
-    video_cover_url = Column(Text)
-    video_play_url = Column(Text)
-    source_keyword = Column(Text, default='')
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    user_id = Column(String(64), comment='用户ID')
+    nickname = Column(Text, comment='用户昵称')
+    avatar = Column(Text, comment='用户头像')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
+    video_id = Column(String(255), index=True, comment='视频ID')
+    video_type = Column(Text, comment='视频类型')
+    title = Column(Text, comment='视频标题')
+    desc = Column(Text, comment='视频描述')
+    create_time = Column(BigInteger, index=True, comment='创建时间戳')
+    liked_count = Column(Text, comment='点赞数')
+    viewd_count = Column(Text, comment='观看数')
+    video_url = Column(Text, comment='视频URL')
+    video_cover_url = Column(Text, comment='视频封面URL')
+    video_play_url = Column(Text, comment='视频播放URL')
+    source_keyword = Column(Text, default='', comment='来源关键词')
 
 class KuaishouVideoComment(Base):
     __tablename__ = 'kuaishou_video_comment'
-    id = Column(Integer, primary_key=True)
-    user_id = Column(Text)
-    nickname = Column(Text)
-    avatar = Column(Text)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
-    comment_id = Column(BigInteger, index=True)
-    video_id = Column(String(255), index=True)
-    content = Column(Text)
-    create_time = Column(BigInteger)
-    sub_comment_count = Column(Text)
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    user_id = Column(Text, comment='用户ID')
+    nickname = Column(Text, comment='用户昵称')
+    avatar = Column(Text, comment='用户头像')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
+    comment_id = Column(BigInteger, index=True, comment='评论ID')
+    video_id = Column(String(255), index=True, comment='视频ID')
+    content = Column(Text, comment='评论内容')
+    create_time = Column(BigInteger, comment='创建时间戳')
+    sub_comment_count = Column(Text, comment='子评论数')
 
 class WeiboNote(Base):
     __tablename__ = 'weibo_note'
-    id = Column(Integer, primary_key=True)
-    user_id = Column(String(255))
-    nickname = Column(Text)
-    avatar = Column(Text)
-    gender = Column(Text)
-    profile_url = Column(Text)
-    ip_location = Column(Text, default='')
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
-    note_id = Column(BigInteger, index=True)
-    content = Column(Text)
-    create_time = Column(BigInteger, index=True)
-    create_date_time = Column(String(255), index=True)
-    liked_count = Column(Text)
-    comments_count = Column(Text)
-    shared_count = Column(Text)
-    note_url = Column(Text)
-    source_keyword = Column(Text, default='')
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    user_id = Column(String(255), comment='用户ID')
+    nickname = Column(Text, comment='用户昵称')
+    avatar = Column(Text, comment='用户头像')
+    gender = Column(Text, comment='性别')
+    profile_url = Column(Text, comment='个人主页URL')
+    ip_location = Column(Text, default='', comment='IP地址位置')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
+    note_id = Column(BigInteger, index=True, comment='笔记ID')
+    content = Column(Text, comment='笔记内容')
+    create_time = Column(BigInteger, index=True, comment='创建时间戳')
+    create_date_time = Column(String(255), index=True, comment='创建日期时间')
+    liked_count = Column(Text, comment='点赞数')
+    comments_count = Column(Text, comment='评论数')
+    shared_count = Column(Text, comment='分享数')
+    note_url = Column(Text, comment='笔记URL')
+    source_keyword = Column(Text, default='', comment='来源关键词')
 
 class WeiboNoteComment(Base):
     __tablename__ = 'weibo_note_comment'
-    id = Column(Integer, primary_key=True)
-    user_id = Column(String(255))
-    nickname = Column(Text)
-    avatar = Column(Text)
-    gender = Column(Text)
-    profile_url = Column(Text)
-    ip_location = Column(Text, default='')
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
-    comment_id = Column(BigInteger, index=True)
-    note_id = Column(BigInteger, index=True)
-    content = Column(Text)
-    create_time = Column(BigInteger)
-    create_date_time = Column(String(255), index=True)
-    comment_like_count = Column(Text)
-    sub_comment_count = Column(Text)
-    parent_comment_id = Column(String(255))
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    user_id = Column(String(255), comment='用户ID')
+    nickname = Column(Text, comment='用户昵称')
+    avatar = Column(Text, comment='用户头像')
+    gender = Column(Text, comment='性别')
+    profile_url = Column(Text, comment='个人主页URL')
+    ip_location = Column(Text, default='', comment='IP地址位置')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
+    comment_id = Column(BigInteger, index=True, comment='评论ID')
+    note_id = Column(BigInteger, index=True, comment='笔记ID')
+    content = Column(Text, comment='评论内容')
+    create_time = Column(BigInteger, comment='创建时间戳')
+    create_date_time = Column(String(255), index=True, comment='创建日期时间')
+    comment_like_count = Column(Text, comment='评论点赞数')
+    sub_comment_count = Column(Text, comment='子评论数')
+    parent_comment_id = Column(String(255), comment='父评论ID')
 
 class WeiboCreator(Base):
     __tablename__ = 'weibo_creator'
-    id = Column(Integer, primary_key=True)
-    user_id = Column(String(255))
-    nickname = Column(Text)
-    avatar = Column(Text)
-    ip_location = Column(Text)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
-    desc = Column(Text)
-    gender = Column(Text)
-    follows = Column(Text)
-    fans = Column(Text)
-    tag_list = Column(Text)
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    user_id = Column(String(255), comment='用户ID')
+    nickname = Column(Text, comment='用户昵称')
+    avatar = Column(Text, comment='用户头像')
+    ip_location = Column(Text, comment='IP地址位置')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
+    desc = Column(Text, comment='描述')
+    gender = Column(Text, comment='性别')
+    follows = Column(Text, comment='关注数')
+    fans = Column(Text, comment='粉丝数')
+    tag_list = Column(Text, comment='标签列表')
 
 class XhsCreator(Base):
     __tablename__ = 'xhs_creator'
-    id = Column(Integer, primary_key=True)
-    user_id = Column(String(255))
-    nickname = Column(Text)
-    avatar = Column(Text)
-    ip_location = Column(Text)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
-    desc = Column(Text)
-    gender = Column(Text)
-    follows = Column(Text)
-    fans = Column(Text)
-    interaction = Column(Text)
-    tag_list = Column(Text)
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    user_id = Column(String(255), comment='用户ID')
+    nickname = Column(Text, comment='用户昵称')
+    avatar = Column(Text, comment='用户头像')
+    ip_location = Column(Text, comment='IP地址位置')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
+    desc = Column(Text, comment='描述')
+    gender = Column(Text, comment='性别')
+    follows = Column(Text, comment='关注数')
+    fans = Column(Text, comment='粉丝数')
+    interaction = Column(Text, comment='互动数')
+    tag_list = Column(Text, comment='标签列表')
 
 class XhsNote(Base):
     __tablename__ = 'xhs_note'
-    id = Column(Integer, primary_key=True)
-    user_id = Column(String(255))
-    nickname = Column(Text)
-    avatar = Column(Text)
-    ip_location = Column(Text)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
-    note_id = Column(String(255), index=True)
-    type = Column(Text)
-    title = Column(Text)
-    desc = Column(Text)
-    video_url = Column(Text)
-    time = Column(BigInteger, index=True)
-    last_update_time = Column(BigInteger)
-    liked_count = Column(Text)
-    collected_count = Column(Text)
-    comment_count = Column(Text)
-    share_count = Column(Text)
-    image_list = Column(Text)
-    tag_list = Column(Text)
-    note_url = Column(Text)
-    source_keyword = Column(Text, default='')
-    xsec_token = Column(Text)
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    user_id = Column(String(255), comment='用户ID')
+    nickname = Column(Text, comment='用户昵称')
+    avatar = Column(Text, comment='用户头像')
+    ip_location = Column(Text, comment='IP地址位置')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
+    note_id = Column(String(255), index=True, comment='笔记ID')
+    type = Column(Text, comment='笔记类型')
+    title = Column(Text, comment='笔记标题')
+    desc = Column(Text, comment='笔记描述')
+    video_url = Column(Text, comment='视频URL')
+    time = Column(BigInteger, index=True, comment='时间戳')
+    last_update_time = Column(BigInteger, comment='最后更新时间戳')
+    liked_count = Column(Text, comment='点赞数')
+    collected_count = Column(Text, comment='收藏数')
+    comment_count = Column(Text, comment='评论数')
+    share_count = Column(Text, comment='分享数')
+    image_list = Column(Text, comment='图片列表')
+    tag_list = Column(Text, comment='标签列表')
+    note_url = Column(Text, comment='笔记URL')
+    source_keyword = Column(Text, default='', comment='来源关键词')
+    xsec_token = Column(Text, comment='Xsec Token')
 
 class XhsNoteComment(Base):
     __tablename__ = 'xhs_note_comment'
-    id = Column(Integer, primary_key=True)
-    user_id = Column(String(255))
-    nickname = Column(Text)
-    avatar = Column(Text)
-    ip_location = Column(Text)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
-    comment_id = Column(String(255), index=True)
-    create_time = Column(BigInteger, index=True)
-    note_id = Column(String(255))
-    content = Column(Text)
-    sub_comment_count = Column(Integer)
-    pictures = Column(Text)
-    parent_comment_id = Column(String(255))
-    like_count = Column(Text)
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    user_id = Column(String(255), comment='用户ID')
+    nickname = Column(Text, comment='用户昵称')
+    avatar = Column(Text, comment='用户头像')
+    ip_location = Column(Text, comment='IP地址位置')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
+    comment_id = Column(String(255), index=True, comment='评论ID')
+    create_time = Column(BigInteger, index=True, comment='创建时间戳')
+    note_id = Column(String(255), comment='笔记ID')
+    content = Column(Text, comment='评论内容')
+    sub_comment_count = Column(Integer, comment='子评论数')
+    pictures = Column(Text, comment='图片')
+    parent_comment_id = Column(String(255), comment='父评论ID')
+    like_count = Column(Text, comment='点赞数')
 
 class TiebaNote(Base):
     __tablename__ = 'tieba_note'
-    id = Column(Integer, primary_key=True)
-    note_id = Column(String(644), index=True)
-    title = Column(Text)
-    desc = Column(Text)
-    note_url = Column(Text)
-    publish_time = Column(String(255), index=True)
-    user_link = Column(Text, default='')
-    user_nickname = Column(Text, default='')
-    user_avatar = Column(Text, default='')
-    tieba_id = Column(String(255), default='')
-    tieba_name = Column(Text)
-    tieba_link = Column(Text)
-    total_replay_num = Column(Integer, default=0)
-    total_replay_page = Column(Integer, default=0)
-    ip_location = Column(Text, default='')
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
-    source_keyword = Column(Text, default='')
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    note_id = Column(String(644), index=True, comment='笔记ID')
+    title = Column(Text, comment='笔记标题')
+    desc = Column(Text, comment='笔记描述')
+    note_url = Column(Text, comment='笔记URL')
+    publish_time = Column(String(255), index=True, comment='发布时间')
+    user_link = Column(Text, default='', comment='用户链接')
+    user_nickname = Column(Text, default='', comment='用户昵称')
+    user_avatar = Column(Text, default='', comment='用户头像')
+    tieba_id = Column(String(255), default='', comment='贴吧ID')
+    tieba_name = Column(Text, comment='贴吧名称')
+    tieba_link = Column(Text, comment='贴吧链接')
+    total_replay_num = Column(Integer, default=0, comment='总回复数')
+    total_replay_page = Column(Integer, default=0, comment='总回复页数')
+    ip_location = Column(Text, default='', comment='IP地址位置')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
+    source_keyword = Column(Text, default='', comment='来源关键词')
 
 class TiebaComment(Base):
     __tablename__ = 'tieba_comment'
-    id = Column(Integer, primary_key=True)
-    comment_id = Column(String(255), index=True)
-    parent_comment_id = Column(String(255), default='')
-    content = Column(Text)
-    user_link = Column(Text, default='')
-    user_nickname = Column(Text, default='')
-    user_avatar = Column(Text, default='')
-    tieba_id = Column(String(255), default='')
-    tieba_name = Column(Text)
-    tieba_link = Column(Text)
-    publish_time = Column(String(255), index=True)
-    ip_location = Column(Text, default='')
-    sub_comment_count = Column(Integer, default=0)
-    note_id = Column(String(255), index=True)
-    note_url = Column(Text)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    comment_id = Column(String(255), index=True, comment='评论ID')
+    parent_comment_id = Column(String(255), default='', comment='父评论ID')
+    content = Column(Text, comment='评论内容')
+    user_link = Column(Text, default='', comment='用户链接')
+    user_nickname = Column(Text, default='', comment='用户昵称')
+    user_avatar = Column(Text, default='', comment='用户头像')
+    tieba_id = Column(String(255), default='', comment='贴吧ID')
+    tieba_name = Column(Text, comment='贴吧名称')
+    tieba_link = Column(Text, comment='贴吧链接')
+    publish_time = Column(String(255), index=True, comment='发布时间')
+    ip_location = Column(Text, default='', comment='IP地址位置')
+    sub_comment_count = Column(Integer, default=0, comment='子评论数')
+    note_id = Column(String(255), index=True, comment='笔记ID')
+    note_url = Column(Text, comment='笔记URL')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
 
 class TiebaCreator(Base):
     __tablename__ = 'tieba_creator'
-    id = Column(Integer, primary_key=True)
-    user_id = Column(String(64))
-    user_name = Column(Text)
-    nickname = Column(Text)
-    avatar = Column(Text)
-    ip_location = Column(Text)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
-    gender = Column(Text)
-    follows = Column(Text)
-    fans = Column(Text)
-    registration_duration = Column(Text)
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    user_id = Column(String(64), comment='用户ID')
+    user_name = Column(Text, comment='用户名')
+    nickname = Column(Text, comment='用户昵称')
+    avatar = Column(Text, comment='用户头像')
+    ip_location = Column(Text, comment='IP地址位置')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
+    gender = Column(Text, comment='性别')
+    follows = Column(Text, comment='关注数')
+    fans = Column(Text, comment='粉丝数')
+    registration_duration = Column(Text, comment='注册时长')
 
 class ZhihuContent(Base):
     __tablename__ = 'zhihu_content'
-    id = Column(Integer, primary_key=True)
-    content_id = Column(String(64), index=True)
-    content_type = Column(Text)
-    content_text = Column(Text)
-    content_url = Column(Text)
-    question_id = Column(String(255))
-    title = Column(Text)
-    desc = Column(Text)
-    created_time = Column(String(32), index=True)
-    updated_time = Column(Text)
-    voteup_count = Column(Integer, default=0)
-    comment_count = Column(Integer, default=0)
-    source_keyword = Column(Text)
-    user_id = Column(String(255))
-    user_link = Column(Text)
-    user_nickname = Column(Text)
-    user_avatar = Column(Text)
-    user_url_token = Column(Text)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    content_id = Column(String(64), index=True, comment='内容ID')
+    content_type = Column(Text, comment='内容类型')
+    content_text = Column(Text, comment='内容文本')
+    content_url = Column(Text, comment='内容URL')
+    question_id = Column(String(255), comment='问题ID')
+    title = Column(Text, comment='标题')
+    desc = Column(Text, comment='描述')
+    created_time = Column(String(32), index=True, comment='创建时间')
+    updated_time = Column(Text, comment='更新时间')
+    voteup_count = Column(Integer, default=0, comment='赞同数')
+    comment_count = Column(Integer, default=0, comment='评论数')
+    source_keyword = Column(Text, comment='来源关键词')
+    user_id = Column(String(255), comment='用户ID')
+    user_link = Column(Text, comment='用户链接')
+    user_nickname = Column(Text, comment='用户昵称')
+    user_avatar = Column(Text, comment='用户头像')
+    user_url_token = Column(Text, comment='用户URL Token')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
 
     # persist-1<persist1@126.com>
-    # 原因：修复 ORM 模型定义错误，确保与数据库表结构一致。
-    # 副作用：无
-    # 回滚策略：还原此行
+    # Reason: Fixed ORM model definition error, ensuring consistency with database table structure.
+    # Side effects: None
+    # Rollback strategy: Restore this line
 
 class ZhihuComment(Base):
     __tablename__ = 'zhihu_comment'
-    id = Column(Integer, primary_key=True)
-    comment_id = Column(String(64), index=True)
-    parent_comment_id = Column(String(64))
-    content = Column(Text)
-    publish_time = Column(String(32), index=True)
-    ip_location = Column(Text)
-    sub_comment_count = Column(Integer, default=0)
-    like_count = Column(Integer, default=0)
-    dislike_count = Column(Integer, default=0)
-    content_id = Column(String(64), index=True)
-    content_type = Column(Text)
-    user_id = Column(String(64))
-    user_link = Column(Text)
-    user_nickname = Column(Text)
-    user_avatar = Column(Text)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    comment_id = Column(String(64), index=True, comment='评论ID')
+    parent_comment_id = Column(String(64), comment='父评论ID')
+    content = Column(Text, comment='评论内容')
+    publish_time = Column(String(32), index=True, comment='发布时间')
+    ip_location = Column(Text, comment='IP地址位置')
+    sub_comment_count = Column(Integer, default=0, comment='子评论数')
+    like_count = Column(Integer, default=0, comment='点赞数')
+    dislike_count = Column(Integer, default=0, comment='点踩数')
+    content_id = Column(String(64), index=True, comment='内容ID')
+    content_type = Column(Text, comment='内容类型')
+    user_id = Column(String(64), comment='用户ID')
+    user_link = Column(Text, comment='用户链接')
+    user_nickname = Column(Text, comment='用户昵称')
+    user_avatar = Column(Text, comment='用户头像')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')
 
 class ZhihuCreator(Base):
     __tablename__ = 'zhihu_creator'
-    id = Column(Integer, primary_key=True)
-    user_id = Column(String(64), unique=True, index=True)
-    user_link = Column(Text)
-    user_nickname = Column(Text)
-    user_avatar = Column(Text)
-    url_token = Column(Text)
-    gender = Column(Text)
-    ip_location = Column(Text)
-    follows = Column(Integer, default=0)
-    fans = Column(Integer, default=0)
-    anwser_count = Column(Integer, default=0)
-    video_count = Column(Integer, default=0)
-    question_count = Column(Integer, default=0)
-    article_count = Column(Integer, default=0)
-    column_count = Column(Integer, default=0)
-    get_voteup_count = Column(Integer, default=0)
-    add_ts = Column(BigInteger)
-    last_modify_ts = Column(BigInteger)
+    id = Column(Integer, primary_key=True, comment='主键ID')
+    user_id = Column(String(64), unique=True, index=True, comment='用户ID')
+    user_link = Column(Text, comment='用户链接')
+    user_nickname = Column(Text, comment='用户昵称')
+    user_avatar = Column(Text, comment='用户头像')
+    url_token = Column(Text, comment='URL Token')
+    gender = Column(Text, comment='性别')
+    ip_location = Column(Text, comment='IP地址位置')
+    follows = Column(Integer, default=0, comment='关注数')
+    fans = Column(Integer, default=0, comment='粉丝数')
+    anwser_count = Column(Integer, default=0, comment='回答数')
+    video_count = Column(Integer, default=0, comment='视频数')
+    question_count = Column(Integer, default=0, comment='问题数')
+    article_count = Column(Integer, default=0, comment='文章数')
+    column_count = Column(Integer, default=0, comment='专栏数')
+    get_voteup_count = Column(Integer, default=0, comment='获赞数')
+    add_ts = Column(BigInteger, comment='添加时间戳')
+    last_modify_ts = Column(BigInteger, comment='最后修改时间戳')

database/mongodb_store_base.py

@@ -16,7 +16,7 @@
 # 详细许可条款请参阅项目根目录下的LICENSE文件。
 # 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
 
-"""MongoDB存储基类：提供连接管理和通用存储方法"""
+"""MongoDB storage base class: Provides connection management and common storage methods"""
 import asyncio
 from typing import Dict, List, Optional
 from motor.motor_asyncio import AsyncIOMotorClient, AsyncIOMotorDatabase, AsyncIOMotorCollection
@@ -25,7 +25,7 @@ from tools import utils
 
 
 class MongoDBConnection:
-    """MongoDB连接管理（单例模式）"""
+    """MongoDB connection management (singleton pattern)"""
     _instance = None
     _client: Optional[AsyncIOMotorClient] = None
     _db: Optional[AsyncIOMotorDatabase] = None
@@ -37,7 +37,7 @@ class MongoDBConnection:
         return cls._instance
 
     async def get_client(self) -> AsyncIOMotorClient:
-        """获取客户端"""
+        """Get client"""
         if self._client is None:
             async with self._lock:
                 if self._client is None:
@@ -45,7 +45,7 @@ class MongoDBConnection:
         return self._client
 
     async def get_db(self) -> AsyncIOMotorDatabase:
-        """获取数据库"""
+        """Get database"""
         if self._db is None:
             async with self._lock:
                 if self._db is None:
@@ -53,7 +53,7 @@ class MongoDBConnection:
         return self._db
 
     async def _connect(self):
-        """建立连接"""
+        """Establish connection"""
         try:
             mongo_config = db_config.mongodb_config
             host = mongo_config["host"]
@@ -62,14 +62,14 @@ class MongoDBConnection:
             password = mongo_config["password"]
             db_name = mongo_config["db_name"]
 
-            # 构建连接URL（有认证/无认证）
+            # Build connection URL (with/without authentication)
             if user and password:
                 connection_url = f"mongodb://{user}:{password}@{host}:{port}/"
             else:
                 connection_url = f"mongodb://{host}:{port}/"
 
             self._client = AsyncIOMotorClient(connection_url, serverSelectionTimeoutMS=5000)
-            await self._client.server_info()  # 测试连接
+            await self._client.server_info()  # Test connection
             self._db = self._client[db_name]
             utils.logger.info(f"[MongoDBConnection] Connected to {host}:{port}/{db_name}")
         except Exception as e:
@@ -77,7 +77,7 @@ class MongoDBConnection:
             raise
 
     async def close(self):
-        """关闭连接"""
+        """Close connection"""
         if self._client is not None:
             self._client.close()
             self._client = None
@@ -86,24 +86,24 @@ class MongoDBConnection:
 
 
 class MongoDBStoreBase:
-    """MongoDB存储基类：提供通用的CRUD操作"""
+    """MongoDB storage base class: Provides common CRUD operations"""
 
     def __init__(self, collection_prefix: str):
-        """初始化存储基类
+        """Initialize storage base class
         Args:
-            collection_prefix: 平台前缀（xhs/douyin/bilibili等）
+            collection_prefix: Platform prefix (xhs/douyin/bilibili, etc.)
         """
         self.collection_prefix = collection_prefix
         self._connection = MongoDBConnection()
 
     async def get_collection(self, collection_suffix: str) -> AsyncIOMotorCollection:
-        """获取集合：{prefix}_{suffix}"""
+        """Get collection: {prefix}_{suffix}"""
         db = await self._connection.get_db()
         collection_name = f"{self.collection_prefix}_{collection_suffix}"
         return db[collection_name]
 
     async def save_or_update(self, collection_suffix: str, query: Dict, data: Dict) -> bool:
-        """保存或更新数据（upsert）"""
+        """Save or update data (upsert)"""
         try:
             collection = await self.get_collection(collection_suffix)
             await collection.update_one(query, {"$set": data}, upsert=True)
@@ -113,7 +113,7 @@ class MongoDBStoreBase:
             return False
 
     async def find_one(self, collection_suffix: str, query: Dict) -> Optional[Dict]:
-        """查询单条数据"""
+        """Query a single record"""
         try:
             collection = await self.get_collection(collection_suffix)
             return await collection.find_one(query)
@@ -122,7 +122,7 @@ class MongoDBStoreBase:
             return None
 
     async def find_many(self, collection_suffix: str, query: Dict, limit: int = 0) -> List[Dict]:
-        """查询多条数据（limit=0表示不限制）"""
+        """Query multiple records (limit=0 means no limit)"""
         try:
             collection = await self.get_collection(collection_suffix)
             cursor = collection.find(query)
@@ -134,7 +134,7 @@ class MongoDBStoreBase:
             return []
 
     async def create_index(self, collection_suffix: str, keys: List[tuple], unique: bool = False):
-        """创建索引：keys=[("field", 1)]"""
+        """Create index: keys=[("field", 1)]"""
         try:
             collection = await self.get_collection(collection_suffix)
             await collection.create_index(keys, unique=unique)

docs/.vitepress/config.mjs

@@ -1,7 +1,8 @@
 import {defineConfig} from 'vitepress'
+import {withMermaid} from 'vitepress-plugin-mermaid'
 
 // https://vitepress.dev/reference/site-config
-export default defineConfig({
+export default withMermaid(defineConfig({
     title: "MediaCrawler自媒体爬虫",
     description: "小红书爬虫，抖音爬虫， 快手爬虫， B站爬虫， 微博爬虫，百度贴吧爬虫，知乎爬虫...。  ",
     lastUpdated: true,
@@ -43,6 +44,7 @@ export default defineConfig({
                 text: 'MediaCrawler使用文档',
                 items: [
                     {text: '基本使用', link: '/'},
+                    {text: '项目架构文档', link: '/项目架构文档'},
                     {text: '常见问题汇总', link: '/常见问题'},
                     {text: 'IP代理使用', link: '/代理使用'},
                     {text: '词云图使用', link: '/词云图使用配置'},
@@ -59,7 +61,6 @@ export default defineConfig({
                         text: 'MediaCrawler源码剖析课',
                         link: 'https://relakkes.feishu.cn/wiki/JUgBwdhIeiSbAwkFCLkciHdAnhh'
                     },
-                    {text: '开发者咨询服务', link: '/开发者咨询'},
                 ]
             },
             {
@@ -85,4 +86,4 @@ export default defineConfig({
             {icon: 'github', link: 'https://github.com/NanmiCoder/MediaCrawler'}
         ]
     }
-})
+}))

docs/.vitepress/theme/DynamicAds.vue

@@ -11,9 +11,9 @@ const fetchAds = async () => {
   return [
     {
       id: 1,
-      imageUrl: 'https://github.com/NanmiCoder/MediaCrawler/raw/main/docs/static/images/auto_test.png',
-      landingUrl: 'https://item.jd.com/10124939676219.html',
-      text: '给好朋友虫师新书站台推荐 - 基于Python的自动化测试框架设计'
+      imageUrl: 'https://github.com/NanmiCoder/MediaCrawler/raw/main/docs/static/images/MediaCrawlerPro.jpg',
+      landingUrl: 'https://github.com/MediaCrawlerPro',
+      text: '👏欢迎大家来订阅MediaCrawlerPro源代码'
     }
   ]
 }
@@ -63,7 +63,8 @@ onUnmounted(() => {
 }
 
 .ad-image {
-  max-width: 130px;
+  max-width: 100%;
+  width: 280px;
   height: auto;
   margin-bottom: 0.5rem;
 }

docs/.vitepress/theme/custom.css

@@ -6,4 +6,5 @@
 :root {
   --vp-sidebar-width: 285px;
   --vp-sidebar-bg-color: var(--vp-c-bg-alt);
-}
+  --vp-aside-width: 300px;
+}

docs/CDP模式使用指南.md

@@ -12,34 +12,63 @@ CDP（Chrome DevTools Protocol）模式是一种高级的反检测爬虫技术
 4. **扩展支持**: 可以利用用户安装的广告拦截器、代理扩展等工具
 5. **更自然的行为**: 浏览器行为模式更接近真实用户
 
+### 📌 两种 CDP 模式
+
+CDP模式支持两种使用方式：
+
+| 模式 | 说明 | 适用场景 |
+|------|------|----------|
+| **连接已有浏览器**（默认推荐） | 连接用户正在使用的 Chrome 浏览器，复用真实的 Cookie、扩展和浏览历史 | 反检测要求高，需要最大程度降低风控风险 |
+| **启动新浏览器** | 自动检测并启动一个新的 Chrome/Edge 浏览器实例 | 不需要复用浏览器状态的场景 |
+
 ## 快速开始
 
-### 1. 启用CDP模式
+### 方式一：连接已有浏览器（默认推荐）
 
-在 `config/base_config.py` 中设置：
+这是**默认且推荐**的方式，直接连接你正在使用的 Chrome 浏览器，反检测效果最好。
+
+#### 第一步：确保 Chrome 版本
+
+需要 Chrome **144 或更高版本**（2026年1月起的稳定版均支持）。在地址栏输入 `chrome://version` 查看当前版本。
+
+如果版本过低，请前往 [Chrome 官网](https://www.google.com/chrome/) 下载最新版。
+
+#### 第二步：开启远程调试
+
+1. 在 Chrome 地址栏输入：`chrome://inspect/#remote-debugging`
+2. 勾选 **"Allow remote debugging for this browser instance"**
+3. 页面会显示 `Server running at: 127.0.0.1:9222`，表示已就绪
+
+#### 第三步：运行爬虫
+
+```bash
+uv run main.py --platform xhs --lt qrcode --type search
+```
+
+运行后，Chrome 浏览器会**弹出确认对话框**，点击"接受"即可。程序会等待用户确认（默认60秒超时）。
+
+#### 配置说明
+
+`config/base_config.py` 中的默认配置：
 
 ```python
 # 启用CDP模式
 ENABLE_CDP_MODE = True
 
-# CDP调试端口（可选，默认9222）
+# 连接已有浏览器（默认开启）
+CDP_CONNECT_EXISTING = True
+
+# CDP调试端口（与 chrome://inspect 页面显示的端口一致）
 CDP_DEBUG_PORT = 9222
-
-# 是否在无头模式下运行（建议设为False以获得最佳反检测效果）
-CDP_HEADLESS = False
-
-# 程序结束时是否自动关闭浏览器
-AUTO_CLOSE_BROWSER = True
 ```
 
-### 2. 运行测试
+### 方式二：启动新浏览器
 
-```bash
-# 运行CDP功能测试
-python examples/cdp_example.py
+如果不想连接已有浏览器，可以让程序自动启动一个新的浏览器实例：
 
-# 运行小红书爬虫（CDP模式）
-python main.py
+```python
+ENABLE_CDP_MODE = True
+CDP_CONNECT_EXISTING = False  # 关闭连接已有浏览器，改为启动新浏览器
 ```
 
 ## 配置选项详解
@@ -48,7 +77,8 @@ python main.py
 
 | 配置项 | 类型 | 默认值 | 说明 |
 |--------|------|--------|------|
-| `ENABLE_CDP_MODE` | bool | False | 是否启用CDP模式 |
+| `ENABLE_CDP_MODE` | bool | True | 是否启用CDP模式 |
+| `CDP_CONNECT_EXISTING` | bool | True | 是否连接已有浏览器（推荐开启） |
 | `CDP_DEBUG_PORT` | int | 9222 | CDP调试端口 |
 | `CDP_HEADLESS` | bool | False | CDP模式下的无头模式 |
 | `AUTO_CLOSE_BROWSER` | bool | True | 程序结束时是否关闭浏览器 |
@@ -57,8 +87,8 @@ python main.py
 
 | 配置项 | 类型 | 默认值 | 说明 |
 |--------|------|--------|------|
-| `CUSTOM_BROWSER_PATH` | str | "" | 自定义浏览器路径 |
-| `BROWSER_LAUNCH_TIMEOUT` | int | 30 | 浏览器启动超时时间（秒） |
+| `CUSTOM_BROWSER_PATH` | str | "" | 自定义浏览器路径（仅启动新浏览器模式下有效） |
+| `BROWSER_LAUNCH_TIMEOUT` | int | 60 | 浏览器连接超时时间（秒） |
 
 ### 自定义浏览器路径
 
@@ -219,15 +249,25 @@ ps aux | grep chrome
 
 ## 技术原理
 
-CDP模式的工作原理：
+### 连接已有浏览器模式（推荐）
+
+1. **用户开启远程调试**: 在 `chrome://inspect/#remote-debugging` 中勾选启用
+2. **WebSocket连接**: 程序通过 `ws://localhost:9222/devtools/browser` 直接连接浏览器
+3. **用户确认**: Chrome 弹出确认对话框，用户点击接受后连接建立
+4. **Playwright集成**: 使用 `connectOverCDP` 方法接管浏览器控制
+5. **上下文复用**: 直接使用浏览器已有的上下文（包含用户的Cookie、登录状态等）
+
+> 💡 与传统CDP模式的区别：传统方式通过 `--remote-debugging-port` 启动新浏览器，使用 HTTP 接口 `/json/version` 获取 WebSocket URL。而连接已有浏览器方式直接通过 WebSocket 连接，Chrome 新版（136+）的远程调试不提供 HTTP 接口，需要用户在浏览器端确认授权。
+
+### 启动新浏览器模式
 
 1. **浏览器检测**: 自动扫描系统中的Chrome/Edge安装路径
 2. **进程启动**: 使用`--remote-debugging-port`参数启动浏览器
-3. **CDP连接**: 通过WebSocket连接到浏览器的调试接口
+3. **CDP连接**: 通过 HTTP 获取 WebSocket URL，再连接到浏览器的调试接口
 4. **Playwright集成**: 使用`connectOverCDP`方法接管浏览器控制
 5. **上下文管理**: 创建或复用浏览器上下文进行操作
 
-这种方式绕过了传统WebDriver的检测机制，提供了更加隐蔽的自动化能力。
+两种方式都绕过了传统WebDriver的检测机制，提供了更加隐蔽的自动化能力。连接已有浏览器模式的反检测效果更好，因为使用的是用户真实的浏览器环境。
 
 ## 更新日志
 

docs/data_storage_guide.md

@@ -0,0 +1,71 @@
+# 数据保存指南 / Data Storage Guide
+
+
+### 💾 数据保存
+
+MediaCrawler 支持多种数据存储方式，您可以根据需求选择最适合的方案：
+
+#### 存储方式
+
+- **CSV 文件**：支持保存到 CSV 中（`data/` 目录下）
+- **JSON 文件**：支持保存到 JSON 中（`data/` 目录下）
+- **JSONL 文件**：支持保存到 JSONL 中（`data/` 目录下）— 默认格式，每行一个 JSON 对象，追加写入性能好
+- **Excel 文件**：支持保存到格式化的 Excel 文件（`data/` 目录下）✨ 新功能
+  - 多工作表支持（内容、评论、创作者）
+  - 专业格式化（标题样式、自动列宽、边框）
+  - 易于分析和分享
+- **数据库存储**
+  - 使用参数 `--init_db` 进行数据库初始化（使用`--init_db`时不需要携带其他optional）
+  - **SQLite 数据库**：轻量级数据库，无需服务器，适合个人使用（推荐）
+    1. 初始化：`--init_db sqlite`
+    2. 数据存储：`--save_data_option sqlite`
+  - **MySQL 数据库**：支持关系型数据库 MySQL 中保存（需要提前创建数据库）
+    1. 初始化：`--init_db mysql`
+    2. 数据存储：`--save_data_option db`（db 参数为兼容历史更新保留）
+  - **PostgreSQL 数据库**：支持高级关系型数据库 PostgreSQL 中保存（推荐生产环境使用）
+    1. 初始化：`--init_db postgres`
+    2. 数据存储：`--save_data_option postgres`
+
+#### 使用示例
+
+```shell
+# 使用 Excel 存储数据（推荐用于数据分析）✨ 新功能
+uv run main.py --platform xhs --lt qrcode --type search --save_data_option excel
+
+# 初始化 SQLite 数据库
+uv run main.py --init_db sqlite
+# 使用 SQLite 存储数据
+uv run main.py --platform xhs --lt qrcode --type search --save_data_option sqlite
+```
+
+```shell
+# 初始化 MySQL 数据库
+uv run main.py --init_db mysql
+# 使用 MySQL 存储数据（为适配历史更新，db参数进行沿用）
+uv run main.py --platform xhs --lt qrcode --type search --save_data_option db
+```
+
+```shell
+# 初始化 PostgreSQL 数据库
+uv run main.py --init_db postgres
+# 使用 PostgreSQL 存储数据
+uv run main.py --platform xhs --lt qrcode --type search --save_data_option postgres
+```
+
+```shell
+# 使用 CSV 存储数据
+uv run main.py --platform xhs --lt qrcode --type search --save_data_option csv
+
+# 使用 JSON 存储数据
+uv run main.py --platform xhs --lt qrcode --type search --save_data_option json
+
+# 使用 JSONL 存储数据（默认格式，无需指定）
+uv run main.py --platform xhs --lt qrcode --type search --save_data_option jsonl
+```
+
+#### 详细文档
+
+- **Excel 导出详细指南**：查看 [Excel 导出指南](excel_export_guide.md)
+- **数据库配置**：参考 [常见问题](常见问题.md)
+
+---

docs/excel_export_guide.md

@@ -0,0 +1,244 @@
+# Excel Export Guide
+
+## Overview
+
+MediaCrawler now supports exporting crawled data to formatted Excel files (.xlsx) with professional styling and multiple sheets for contents, comments, and creators.
+
+## Features
+
+- **Multi-sheet workbooks**: Separate sheets for Contents, Comments, and Creators
+- **Professional formatting**: 
+  - Styled headers with blue background and white text
+  - Auto-adjusted column widths
+  - Cell borders and text wrapping
+  - Clean, readable layout
+- **Smart export**: Empty sheets are automatically removed
+- **Organized storage**: Files saved to `data/{platform}/` directory with timestamps
+
+## Installation
+
+Excel export requires the `openpyxl` library:
+
+```bash
+# Using uv (recommended)
+uv sync
+
+# Or using pip
+pip install openpyxl
+```
+
+## Usage
+
+### Basic Usage
+
+1. **Configure Excel export** in `config/base_config.py`:
+
+```python
+SAVE_DATA_OPTION = "excel"  # Change from jsonl/json/csv/db to excel
+```
+
+2. **Run the crawler**:
+
+```bash
+# Xiaohongshu example
+uv run main.py --platform xhs --lt qrcode --type search
+
+# Douyin example
+uv run main.py --platform dy --lt qrcode --type search
+
+# Bilibili example
+uv run main.py --platform bili --lt qrcode --type search
+```
+
+3. **Find your Excel file** in `data/{platform}/` directory:
+   - Filename format: `{platform}_{crawler_type}_{timestamp}.xlsx`
+   - Example: `xhs_search_20250128_143025.xlsx`
+
+### Command Line Examples
+
+```bash
+# Search by keywords and export to Excel
+uv run main.py --platform xhs --lt qrcode --type search --save_data_option excel
+
+# Crawl specific posts and export to Excel
+uv run main.py --platform xhs --lt qrcode --type detail --save_data_option excel
+
+# Crawl creator profile and export to Excel
+uv run main.py --platform xhs --lt qrcode --type creator --save_data_option excel
+```
+
+## Excel File Structure
+
+### Contents Sheet
+Contains post/video information:
+- `note_id`: Unique post identifier
+- `title`: Post title
+- `desc`: Post description
+- `user_id`: Author user ID
+- `nickname`: Author nickname
+- `liked_count`: Number of likes
+- `comment_count`: Number of comments
+- `share_count`: Number of shares
+- `ip_location`: IP location
+- `image_list`: Comma-separated image URLs
+- `tag_list`: Comma-separated tags
+- `note_url`: Direct link to post
+- And more platform-specific fields...
+
+### Comments Sheet
+Contains comment information:
+- `comment_id`: Unique comment identifier
+- `note_id`: Associated post ID
+- `content`: Comment text
+- `user_id`: Commenter user ID
+- `nickname`: Commenter nickname
+- `like_count`: Comment likes
+- `create_time`: Comment timestamp
+- `ip_location`: Commenter location
+- `sub_comment_count`: Number of replies
+- And more...
+
+### Creators Sheet
+Contains creator/author information:
+- `user_id`: Unique user identifier
+- `nickname`: Display name
+- `gender`: Gender
+- `avatar`: Profile picture URL
+- `desc`: Bio/description
+- `fans`: Follower count
+- `follows`: Following count
+- `interaction`: Total interactions
+- And more...
+
+## Advantages Over Other Formats
+
+### vs CSV
+- ✅ Multiple sheets in one file
+- ✅ Professional formatting
+- ✅ Better handling of special characters
+- ✅ Auto-adjusted column widths
+- ✅ No encoding issues
+
+### vs JSON
+- ✅ Human-readable tabular format
+- ✅ Easy to open in Excel/Google Sheets
+- ✅ Better for data analysis
+- ✅ Easier to share with non-technical users
+
+### vs Database
+- ✅ No database setup required
+- ✅ Portable single-file format
+- ✅ Easy to share and archive
+- ✅ Works offline
+
+## Tips & Best Practices
+
+1. **Large datasets**: For very large crawls (>10,000 rows), consider using database storage instead for better performance
+
+2. **Data analysis**: Excel files work great with:
+   - Microsoft Excel
+   - Google Sheets
+   - LibreOffice Calc
+   - Python pandas: `pd.read_excel('file.xlsx')`
+
+3. **Combining data**: You can merge multiple Excel files using:
+   ```python
+   import pandas as pd
+   df1 = pd.read_excel('file1.xlsx', sheet_name='Contents')
+   df2 = pd.read_excel('file2.xlsx', sheet_name='Contents')
+   combined = pd.concat([df1, df2])
+   combined.to_excel('combined.xlsx', index=False)
+   ```
+
+4. **File size**: Excel files are typically 2-3x larger than CSV but smaller than JSON
+
+## Troubleshooting
+
+### "openpyxl not installed" error
+
+```bash
+# Install openpyxl
+uv add openpyxl
+# or
+pip install openpyxl
+```
+
+### Excel file not created
+
+Check that:
+1. `SAVE_DATA_OPTION = "excel"` in config
+2. Crawler successfully collected data
+3. No errors in console output
+4. `data/{platform}/` directory exists
+
+### Empty Excel file
+
+This happens when:
+- No data was crawled (check keywords/IDs)
+- Login failed (check login status)
+- Platform blocked requests (check IP/rate limits)
+
+## Example Output
+
+After running a successful crawl, you'll see:
+
+```
+[ExcelStoreBase] Initialized Excel export to: data/xhs/xhs_search_20250128_143025.xlsx
+[ExcelStoreBase] Stored content to Excel: 7123456789
+[ExcelStoreBase] Stored comment to Excel: comment_123
+...
+[Main] Excel file saved successfully
+```
+
+Your Excel file will have:
+- Professional blue headers
+- Clean borders
+- Wrapped text for long content
+- Auto-sized columns
+- Separate organized sheets
+
+## Advanced Usage
+
+### Programmatic Access
+
+```python
+from store.excel_store_base import ExcelStoreBase
+
+# Create store
+store = ExcelStoreBase(platform="xhs", crawler_type="search")
+
+# Store data
+await store.store_content({
+    "note_id": "123",
+    "title": "Test Post",
+    "liked_count": 100
+})
+
+# Save to file
+store.flush()
+```
+
+### Custom Formatting
+
+You can extend `ExcelStoreBase` to customize formatting:
+
+```python
+from store.excel_store_base import ExcelStoreBase
+
+class CustomExcelStore(ExcelStoreBase):
+    def _apply_header_style(self, sheet, row_num=1):
+        # Custom header styling
+        super()._apply_header_style(sheet, row_num)
+        # Add your customizations here
+```
+
+## Support
+
+For issues or questions:
+- Check [常见问题](常见问题.md)
+- Open an issue on GitHub
+- Join the WeChat discussion group
+
+---
+
+**Note**: Excel export is designed for learning and research purposes. Please respect platform terms of service and rate limits.

docs/index.md

@@ -1,5 +1,9 @@
 # MediaCrawler使用方法
 
+## 项目文档
+
+- [项目架构文档](项目架构文档.md) - 系统架构、模块设计、数据流向（含 Mermaid 图表）
+
 ## 推荐：使用 uv 管理依赖
 
 ### 1. 前置依赖

docs/常见问题.md

@@ -10,7 +10,7 @@ A: windows电脑去网站下载`https://nodejs.org/en/blog/release/v16.8.0` Wind
 ## xhs登录出现滑块一直验证不通过问题
 
 Q: 小红书扫码登录成功后，浏览器一直在验证滑块，无法登录？<br>
-A: 这种情况一般是因为使用playwright浏览器驱动被识别出来的问题，可以尝试删除项目目录下的`brower_data`文件夹，重新走登录流程。<br>
+A: 小红书平台风控非常严格，**强烈建议使用 CDP 模式连接自己的真实浏览器**（默认配置），不要使用无痕浏览器或标准 Playwright 模式。连接真实浏览器可以复用已有的 Cookie、登录状态和浏览历史，大幅降低被风控检测的概率。如果仍然出现滑块问题，可以尝试删除项目目录下的`brower_data`文件夹，重新走登录流程。<br>
 
 ## 如何指定关键词
 Q: 可以指定关键词爬取吗？<br>
@@ -43,3 +43,21 @@ A: 打开 config/base_config.py 文件, 找到`ENABLE_GET_WORDCLOUD` 以及`ENAB
 ## 词云图添加禁用词和自定义词组
 Q: 如何给词云图添加禁用词和自定义词组？
 A: 打开 `docs/hit_stopwords.txt` 输入禁用词(注意一个词语一行)。打开 config/base_config.py 文件找到 `CUSTOM_WORDS `按格式添加自定义词组即可。<br>
+
+## CDP 连接已有浏览器相关问题
+
+Q: 运行爬虫后提示无法连接到浏览器，报错 `Cannot connect to existing browser on port 9222`？<br>
+A: 请检查以下几点：<br>
+1. 确保 Chrome 浏览器已经打开并正在运行<br>
+2. 在 Chrome 地址栏输入 `chrome://inspect/#remote-debugging`，确保已勾选 **"Allow remote debugging for this browser instance"**<br>
+3. 页面上应显示 `Server running at: 127.0.0.1:9222`，如果没有显示说明远程调试未成功开启<br>
+4. 确保 Chrome 版本 >= 144，低版本不支持此功能，在地址栏输入 `chrome://version` 查看版本号<br>
+
+Q: 运行爬虫后浏览器弹出了确认对话框，需要怎么操作？<br>
+A: 这是正常行为。Chrome 连接已有浏览器时会弹出确认对话框，点击"接受"即可。程序会等待用户确认，默认超时时间为60秒，在此期间点击确认即可。<br>
+
+Q: 不想连接已有浏览器，想让程序自动启动一个新的浏览器怎么办？<br>
+A: 在 `config/base_config.py` 中设置 `CDP_CONNECT_EXISTING = False`，程序会自动检测并启动一个新的 Chrome/Edge 浏览器实例。<br>
+
+Q: 为什么推荐连接已有浏览器而不是启动新浏览器？<br>
+A: 连接已有浏览器可以直接复用你浏览器中真实的 Cookie、登录状态、扩展插件和浏览历史，平台很难区分这是自动化操作还是真实用户行为，**大幅降低被平台风控检测的风险**。而启动新浏览器是一个"干净"的环境，更容易被平台识别为爬虫。<br>

docs/开发者咨询.md

@@ -1,17 +0,0 @@
-# 开发者咨询
-
-## 咨询价格
-
-提供200/小时的咨询服务，最低收费为1小时，帮你快速解决项目中遇到的问题
-
-##### 支持的提问类别
-- MediaCrawler项目源码解读、安装、部署、使用问题
-- 爬虫项目开发问题
-- Python、Golang、JavaScript等编程问题
-- JS逆向问题
-- 其他问题（职业规划、工作经验等）
-
-## 加我微信
-> 备注：咨服服务
-> 
-![微信二维码](static/images/relakkes_weichat.jpg)

docs/微信交流群.md

@@ -9,4 +9,4 @@
 > 
 > 如果图片展示不出来或过期，可以直接添加我的微信号：relakkes，并备注github，会有拉群小助手自动拉你进群
 
-![relakkes_wechat](static/images/relakkes_weichat.jpg)
+![relakkes_wechat](static/images/QIWEI.png)

docs/知识付费介绍.md

@@ -1,13 +1,4 @@
 # 知识付费介绍
-开源是一种无私奉献，从MediaCrawler开源到现在有一年多，它并没有带给我多少实质性的东西，就拿收入来说，赞助费、赞赏等等全部加起来还没有之前一个月的薪水。
-
-后面搞了MediaCrawler源码剖析课程之后，收入稍微好一点，但也是群里兄弟对我开源的支持，在此也非常感谢你们～
-
-但是我依然坚持持续开源，从开始的xhs、dy 2个平台支持，到现在已经有**7个平台**支持，每一次增加一个平台其实都会耗费很大的时间去写代码和调试代码。。。。
-
-在今天跟一个群里好朋友聊天，他说：开源开发者也要活下去。你不要不好意思做知识付费，你的劳动是有价值的。
-
-他点醒我了，因此我把我所提供的知识付费内容放在下面，有需要的朋友可以看看～
 
 ## MediaCrawlerPro项目源码订阅服务
 [mediacrawlerpro订阅文档说明](mediacrawlerpro订阅.md)

docs/词云图使用配置.md

@@ -1,14 +1,14 @@
 # 关于词云图相关操作
 
 ## 1.如何正确调用词云图
-> ps:目前只有保存格式为json文件时，才会生成词云图。其他存储方式添加词云图将在近期添加。
+> ps:保存格式为json或jsonl文件时，才会生成词云图。其他存储方式添加词云图将在近期添加。
 
 需要修改的配置项（./config/base_config.py）：
 
 ```python
-# 数据保存类型选项配置,支持三种类型：csv、db、json
-#此处需要为json格式保存，原因如上
-SAVE_DATA_OPTION = "json"  # csv or db or json
+# 数据保存类型选项配置,支持多种类型：csv、db、json、jsonl等
+#此处需要为json或jsonl格式保存，原因如上
+SAVE_DATA_OPTION = "jsonl"  # csv or db or json or jsonl
 ```
 
 ```python
@@ -54,4 +54,4 @@ FONT_PATH= "./docs/STZHONGS.TTF"
 
 ![image-20240627204928601](https://rosyrain.oss-cn-hangzhou.aliyuncs.com/img2/202406272049662.png)
 
-如图，在data文件下的`words文件夹`下，其中json为词频统计文件，png为词云图。原本的评论内容在`json文件夹`下。
+如图，在data文件下的`words文件夹`下，其中json为词频统计文件，png为词云图。原本的评论内容在`jsonl文件夹`（或`json文件夹`）下。

docs/项目架构文档.md

@@ -0,0 +1,884 @@
+# MediaCrawler 项目架构文档
+
+## 1. 项目概述
+
+### 1.1 项目简介
+
+MediaCrawler 是一个多平台自媒体爬虫框架，采用 Python 异步编程实现，支持爬取主流社交媒体平台的内容、评论和创作者信息。
+
+### 1.2 支持的平台
+
+| 平台 | 代号 | 主要功能 |
+|------|------|---------|
+| 小红书 | `xhs` | 笔记搜索、详情、创作者 |
+| 抖音 | `dy` | 视频搜索、详情、创作者 |
+| 快手 | `ks` | 视频搜索、详情、创作者 |
+| B站 | `bili` | 视频搜索、详情、UP主 |
+| 微博 | `wb` | 微博搜索、详情、博主 |
+| 百度贴吧 | `tieba` | 帖子搜索、详情 |
+| 知乎 | `zhihu` | 问答搜索、详情、答主 |
+
+### 1.3 核心功能特性
+
+- **多平台支持**：统一的爬虫接口，支持 7 大主流平台
+- **多种登录方式**：二维码、手机号、Cookie 三种登录方式
+- **多种存储方式**：CSV、JSON、JSONL、SQLite、MySQL、MongoDB、Excel
+- **反爬虫对策**：CDP 模式、代理 IP 池、请求签名
+- **异步高并发**：基于 asyncio 的异步架构，高效并发爬取
+- **词云生成**：自动生成评论词云图
+
+---
+
+## 2. 系统架构总览
+
+### 2.1 高层架构图
+
+```mermaid
+flowchart TB
+    subgraph Entry["入口层"]
+        main["main.py<br/>程序入口"]
+        cmdarg["cmd_arg<br/>命令行参数"]
+        config["config<br/>配置管理"]
+    end
+
+    subgraph Core["核心爬虫层"]
+        factory["CrawlerFactory<br/>爬虫工厂"]
+        base["AbstractCrawler<br/>爬虫基类"]
+
+        subgraph Platforms["平台实现"]
+            xhs["XiaoHongShuCrawler"]
+            dy["DouYinCrawler"]
+            ks["KuaishouCrawler"]
+            bili["BilibiliCrawler"]
+            wb["WeiboCrawler"]
+            tieba["TieBaCrawler"]
+            zhihu["ZhihuCrawler"]
+        end
+    end
+
+    subgraph Client["API客户端层"]
+        absClient["AbstractApiClient<br/>客户端基类"]
+        xhsClient["XiaoHongShuClient"]
+        dyClient["DouYinClient"]
+        ksClient["KuaiShouClient"]
+        biliClient["BilibiliClient"]
+        wbClient["WeiboClient"]
+        tiebaClient["BaiduTieBaClient"]
+        zhihuClient["ZhiHuClient"]
+    end
+
+    subgraph Storage["数据存储层"]
+        storeFactory["StoreFactory<br/>存储工厂"]
+        csv["CSV存储"]
+        json["JSON存储"]
+        sqlite["SQLite存储"]
+        mysql["MySQL存储"]
+        mongodb["MongoDB存储"]
+        excel["Excel存储"]
+    end
+
+    subgraph Infra["基础设施层"]
+        browser["浏览器管理<br/>Playwright/CDP"]
+        proxy["代理IP池"]
+        cache["缓存系统"]
+        login["登录管理"]
+    end
+
+    main --> factory
+    cmdarg --> main
+    config --> main
+    factory --> base
+    base --> Platforms
+    Platforms --> Client
+    Client --> Storage
+    Client --> Infra
+    Storage --> storeFactory
+    storeFactory --> csv & json & sqlite & mysql & mongodb & excel
+```
+
+### 2.2 数据流向图
+
+```mermaid
+flowchart LR
+    subgraph Input["输入"]
+        keywords["关键词/ID"]
+        config["配置参数"]
+    end
+
+    subgraph Process["处理流程"]
+        browser["启动浏览器"]
+        login["登录认证"]
+        search["搜索/爬取"]
+        parse["数据解析"]
+        comment["获取评论"]
+    end
+
+    subgraph Output["输出"]
+        content["内容数据"]
+        comments["评论数据"]
+        creator["创作者数据"]
+        media["媒体文件"]
+    end
+
+    subgraph Storage["存储"]
+        file["文件存储<br/>CSV/JSON/Excel"]
+        db["数据库<br/>SQLite/MySQL"]
+        nosql["NoSQL<br/>MongoDB"]
+    end
+
+    keywords --> browser
+    config --> browser
+    browser --> login
+    login --> search
+    search --> parse
+    parse --> comment
+    parse --> content
+    comment --> comments
+    parse --> creator
+    parse --> media
+    content & comments & creator --> file & db & nosql
+    media --> file
+```
+
+---
+
+## 3. 目录结构
+
+```
+MediaCrawler/
+├── main.py                 # 程序入口
+├── var.py                  # 全局上下文变量
+├── pyproject.toml          # 项目配置
+│
+├── base/                   # 基础抽象类
+│   └── base_crawler.py     # 爬虫、登录、存储、客户端基类
+│
+├── config/                 # 配置管理
+│   ├── base_config.py      # 核心配置
+│   ├── db_config.py        # 数据库配置
+│   └── {platform}_config.py # 平台特定配置
+│
+├── media_platform/         # 平台爬虫实现
+│   ├── xhs/                # 小红书
+│   ├── douyin/             # 抖音
+│   ├── kuaishou/           # 快手
+│   ├── bilibili/           # B站
+│   ├── weibo/              # 微博
+│   ├── tieba/              # 百度贴吧
+│   └── zhihu/              # 知乎
+│
+├── store/                  # 数据存储
+│   ├── excel_store_base.py # Excel存储基类
+│   └── {platform}/         # 各平台存储实现
+│
+├── database/               # 数据库层
+│   ├── models.py           # ORM模型定义
+│   ├── db_session.py       # 数据库会话管理
+│   └── mongodb_store_base.py # MongoDB基类
+│
+├── proxy/                  # 代理管理
+│   ├── proxy_ip_pool.py    # IP池管理
+│   ├── proxy_mixin.py      # 代理刷新混入
+│   └── providers/          # 代理提供商
+│
+├── cache/                  # 缓存系统
+│   ├── abs_cache.py        # 缓存抽象类
+│   ├── local_cache.py      # 本地缓存
+│   └── redis_cache.py      # Redis缓存
+│
+├── tools/                  # 工具模块
+│   ├── app_runner.py       # 应用运行管理
+│   ├── browser_launcher.py # 浏览器启动
+│   ├── cdp_browser.py      # CDP浏览器管理
+│   ├── crawler_util.py     # 爬虫工具
+│   └── async_file_writer.py # 异步文件写入
+│
+├── model/                  # 数据模型
+│   └── m_{platform}.py     # Pydantic模型
+│
+├── libs/                   # JS脚本库
+│   └── stealth.min.js      # 反检测脚本
+│
+└── cmd_arg/                # 命令行参数
+    └── arg.py              # 参数定义
+```
+
+---
+
+## 4. 核心模块详解
+
+### 4.1 爬虫基类体系
+
+```mermaid
+classDiagram
+    class AbstractCrawler {
+        <<abstract>>
+        +start()* 启动爬虫
+        +search()* 搜索功能
+        +launch_browser() 启动浏览器
+        +launch_browser_with_cdp() CDP模式启动
+    }
+
+    class AbstractLogin {
+        <<abstract>>
+        +begin()* 开始登录
+        +login_by_qrcode()* 二维码登录
+        +login_by_mobile()* 手机号登录
+        +login_by_cookies()* Cookie登录
+    }
+
+    class AbstractStore {
+        <<abstract>>
+        +store_content()* 存储内容
+        +store_comment()* 存储评论
+        +store_creator()* 存储创作者
+        +store_image()* 存储图片
+        +store_video()* 存储视频
+    }
+
+    class AbstractApiClient {
+        <<abstract>>
+        +request()* HTTP请求
+        +update_cookies()* 更新Cookies
+    }
+
+    class ProxyRefreshMixin {
+        +init_proxy_pool() 初始化代理池
+        +_refresh_proxy_if_expired() 刷新过期代理
+    }
+
+    class XiaoHongShuCrawler {
+        +xhs_client: XiaoHongShuClient
+        +start()
+        +search()
+        +get_specified_notes()
+        +get_creators_and_notes()
+    }
+
+    class XiaoHongShuClient {
+        +playwright_page: Page
+        +cookie_dict: Dict
+        +request()
+        +pong() 检查登录状态
+        +get_note_by_keyword()
+        +get_note_by_id()
+    }
+
+    AbstractCrawler <|-- XiaoHongShuCrawler
+    AbstractApiClient <|-- XiaoHongShuClient
+    ProxyRefreshMixin <|-- XiaoHongShuClient
+```
+
+### 4.2 爬虫生命周期
+
+```mermaid
+sequenceDiagram
+    participant Main as main.py
+    participant Factory as CrawlerFactory
+    participant Crawler as XiaoHongShuCrawler
+    participant Browser as Playwright/CDP
+    participant Login as XiaoHongShuLogin
+    participant Client as XiaoHongShuClient
+    participant Store as StoreFactory
+
+    Main->>Factory: create_crawler("xhs")
+    Factory-->>Main: crawler实例
+
+    Main->>Crawler: start()
+
+    alt 启用IP代理
+        Crawler->>Crawler: create_ip_pool()
+    end
+
+    alt CDP模式
+        Crawler->>Browser: launch_browser_with_cdp()
+    else 标准模式
+        Crawler->>Browser: launch_browser()
+    end
+    Browser-->>Crawler: browser_context
+
+    Crawler->>Crawler: create_xhs_client()
+    Crawler->>Client: pong() 检查登录状态
+
+    alt 未登录
+        Crawler->>Login: begin()
+        Login->>Login: login_by_qrcode/mobile/cookie
+        Login-->>Crawler: 登录成功
+    end
+
+    alt search模式
+        Crawler->>Client: get_note_by_keyword()
+        Client-->>Crawler: 搜索结果
+        loop 获取详情
+            Crawler->>Client: get_note_by_id()
+            Client-->>Crawler: 笔记详情
+        end
+    else detail模式
+        Crawler->>Client: get_note_by_id()
+    else creator模式
+        Crawler->>Client: get_creator_info()
+    end
+
+    Crawler->>Store: store_content/comment/creator
+    Store-->>Crawler: 存储完成
+
+    Main->>Crawler: cleanup()
+    Crawler->>Browser: close()
+```
+
+### 4.3 平台爬虫实现结构
+
+每个平台目录包含以下核心文件：
+
+```
+media_platform/{platform}/
+├── __init__.py         # 模块导出
+├── core.py             # 爬虫主实现类
+├── client.py           # API客户端
+├── login.py            # 登录实现
+├── field.py            # 字段/枚举定义
+├── exception.py        # 异常定义
+├── help.py             # 辅助函数
+└── {特殊实现}.py       # 平台特定逻辑
+```
+
+### 4.4 三种爬虫模式
+
+| 模式 | 配置值 | 功能描述 | 适用场景 |
+|------|--------|---------|---------|
+| 搜索模式 | `search` | 根据关键词搜索内容 | 批量获取特定主题内容 |
+| 详情模式 | `detail` | 获取指定ID的详情 | 精确获取已知内容 |
+| 创作者模式 | `creator` | 获取创作者所有内容 | 追踪特定博主/UP主 |
+
+---
+
+## 5. 数据存储层
+
+### 5.1 存储架构图
+
+```mermaid
+classDiagram
+    class AbstractStore {
+        <<abstract>>
+        +store_content()*
+        +store_comment()*
+        +store_creator()*
+    }
+
+    class StoreFactory {
+        +STORES: Dict
+        +create_store() AbstractStore
+    }
+
+    class CsvStoreImplement {
+        +async_file_writer: AsyncFileWriter
+        +store_content()
+        +store_comment()
+    }
+
+    class JsonStoreImplement {
+        +async_file_writer: AsyncFileWriter
+        +store_content()
+        +store_comment()
+    }
+
+    class DbStoreImplement {
+        +session: AsyncSession
+        +store_content()
+        +store_comment()
+    }
+
+    class SqliteStoreImplement {
+        +session: AsyncSession
+        +store_content()
+        +store_comment()
+    }
+
+    class MongoStoreImplement {
+        +mongo_base: MongoDBStoreBase
+        +store_content()
+        +store_comment()
+    }
+
+    class ExcelStoreImplement {
+        +excel_base: ExcelStoreBase
+        +store_content()
+        +store_comment()
+    }
+
+    AbstractStore <|-- CsvStoreImplement
+    AbstractStore <|-- JsonStoreImplement
+    AbstractStore <|-- DbStoreImplement
+    AbstractStore <|-- SqliteStoreImplement
+    AbstractStore <|-- MongoStoreImplement
+    AbstractStore <|-- ExcelStoreImplement
+    StoreFactory --> AbstractStore
+```
+
+### 5.2 存储工厂模式
+
+```python
+# 以抖音为例
+class DouyinStoreFactory:
+    STORES = {
+        "csv": DouyinCsvStoreImplement,
+        "db": DouyinDbStoreImplement,
+        "json": DouyinJsonStoreImplement,
+        "sqlite": DouyinSqliteStoreImplement,
+        "mongodb": DouyinMongoStoreImplement,
+        "excel": DouyinExcelStoreImplement,
+    }
+
+    @staticmethod
+    def create_store() -> AbstractStore:
+        store_class = DouyinStoreFactory.STORES.get(config.SAVE_DATA_OPTION)
+        return store_class()
+```
+
+### 5.3 存储方式对比
+
+| 存储方式 | 配置值 | 优点 | 适用场景 |
+|---------|--------|-----|---------|
+| CSV | `csv` | 简单、通用 | 小规模数据、快速查看 |
+| JSON | `json` | 结构完整、易解析 | API对接、数据交换 |
+| JSONL | `jsonl` | 追加写入、性能好 | 大规模数据、增量爬取（默认） |
+| SQLite | `sqlite` | 轻量、无需服务 | 本地开发、小型项目 |
+| MySQL | `db` | 性能好、支持并发 | 生产环境、大规模数据 |
+| MongoDB | `mongodb` | 灵活、易扩展 | 非结构化数据、快速迭代 |
+| Excel | `excel` | 可视化、易分享 | 报告、数据分析 |
+
+---
+
+## 6. 基础设施层
+
+### 6.1 代理系统架构
+
+```mermaid
+flowchart TB
+    subgraph Config["配置"]
+        enable["ENABLE_IP_PROXY"]
+        provider["IP_PROXY_PROVIDER"]
+        count["IP_PROXY_POOL_COUNT"]
+    end
+
+    subgraph Pool["代理池管理"]
+        pool["ProxyIpPool"]
+        load["load_proxies()"]
+        validate["_is_valid_proxy()"]
+        get["get_proxy()"]
+        refresh["get_or_refresh_proxy()"]
+    end
+
+    subgraph Providers["代理提供商"]
+        kuaidl["快代理<br/>KuaiDaiLiProxy"]
+        wandou["万代理<br/>WanDouHttpProxy"]
+        jishu["技术IP<br/>JiShuHttpProxy"]
+    end
+
+    subgraph Client["API客户端"]
+        mixin["ProxyRefreshMixin"]
+        request["request()"]
+    end
+
+    enable --> pool
+    provider --> Providers
+    count --> load
+    pool --> load
+    load --> validate
+    validate --> Providers
+    pool --> get
+    pool --> refresh
+    mixin --> refresh
+    mixin --> Client
+    request --> mixin
+```
+
+### 6.2 登录流程
+
+```mermaid
+flowchart TB
+    Start([开始登录]) --> CheckType{登录类型?}
+
+    CheckType -->|qrcode| QR[显示二维码]
+    QR --> WaitScan[等待扫描]
+    WaitScan --> CheckQR{扫描成功?}
+    CheckQR -->|是| SaveCookie[保存Cookie]
+    CheckQR -->|否| WaitScan
+
+    CheckType -->|phone| Phone[输入手机号]
+    Phone --> SendCode[发送验证码]
+    SendCode --> Slider{需要滑块?}
+    Slider -->|是| DoSlider[滑动验证]
+    DoSlider --> InputCode[输入验证码]
+    Slider -->|否| InputCode
+    InputCode --> Verify[验证登录]
+    Verify --> SaveCookie
+
+    CheckType -->|cookie| LoadCookie[加载已保存Cookie]
+    LoadCookie --> VerifyCookie{Cookie有效?}
+    VerifyCookie -->|是| SaveCookie
+    VerifyCookie -->|否| Fail[登录失败]
+
+    SaveCookie --> UpdateContext[更新浏览器上下文]
+    UpdateContext --> End([登录完成])
+```
+
+### 6.3 浏览器管理
+
+```mermaid
+flowchart LR
+    subgraph Mode["启动模式"]
+        standard["标准模式<br/>Playwright"]
+        cdp["CDP模式<br/>Chrome DevTools"]
+    end
+
+    subgraph Standard["标准模式流程"]
+        launch["chromium.launch()"]
+        context["new_context()"]
+        stealth["注入stealth.js"]
+    end
+
+    subgraph CDP["CDP模式流程"]
+        detect["检测浏览器路径"]
+        start["启动浏览器进程"]
+        connect["connect_over_cdp()"]
+        cdpContext["获取已有上下文"]
+    end
+
+    subgraph Features["特性"]
+        f1["用户数据持久化"]
+        f2["扩展和设置继承"]
+        f3["反检测能力增强"]
+    end
+
+    standard --> Standard
+    cdp --> CDP
+    CDP --> Features
+```
+
+### 6.4 缓存系统
+
+```mermaid
+classDiagram
+    class AbstractCache {
+        <<abstract>>
+        +get(key)* 获取缓存
+        +set(key, value, expire)* 设置缓存
+        +keys(pattern)* 获取所有键
+    }
+
+    class ExpiringLocalCache {
+        -_cache: Dict
+        -_expire_times: Dict
+        +get(key)
+        +set(key, value, expire_time)
+        +keys(pattern)
+        -_is_expired(key)
+    }
+
+    class RedisCache {
+        -_client: Redis
+        +get(key)
+        +set(key, value, expire_time)
+        +keys(pattern)
+    }
+
+    class CacheFactory {
+        +create_cache(type) AbstractCache
+    }
+
+    AbstractCache <|-- ExpiringLocalCache
+    AbstractCache <|-- RedisCache
+    CacheFactory --> AbstractCache
+```
+
+---
+
+## 7. 数据模型
+
+### 7.1 ORM模型关系
+
+```mermaid
+erDiagram
+    DouyinAweme {
+        int id PK
+        string aweme_id UK
+        string aweme_type
+        string title
+        string desc
+        int create_time
+        int liked_count
+        int collected_count
+        int comment_count
+        int share_count
+        string user_id FK
+        datetime add_ts
+        datetime last_modify_ts
+    }
+
+    DouyinAwemeComment {
+        int id PK
+        string comment_id UK
+        string aweme_id FK
+        string content
+        int create_time
+        int sub_comment_count
+        string user_id
+        datetime add_ts
+        datetime last_modify_ts
+    }
+
+    DyCreator {
+        int id PK
+        string user_id UK
+        string nickname
+        string avatar
+        string desc
+        int follower_count
+        int total_favorited
+        datetime add_ts
+        datetime last_modify_ts
+    }
+
+    DouyinAweme ||--o{ DouyinAwemeComment : "has"
+    DyCreator ||--o{ DouyinAweme : "creates"
+```
+
+### 7.2 各平台数据表
+
+| 平台 | 内容表 | 评论表 | 创作者表 |
+|------|--------|--------|---------|
+| 抖音 | DouyinAweme | DouyinAwemeComment | DyCreator |
+| 小红书 | XHSNote | XHSNoteComment | XHSCreator |
+| 快手 | KuaishouVideo | KuaishouVideoComment | KsCreator |
+| B站 | BilibiliVideo | BilibiliVideoComment | BilibiliUpInfo |
+| 微博 | WeiboNote | WeiboNoteComment | WeiboCreator |
+| 贴吧 | TiebaNote | TiebaNoteComment | - |
+| 知乎 | ZhihuContent | ZhihuContentComment | ZhihuCreator |
+
+---
+
+## 8. 配置系统
+
+### 8.1 核心配置项
+
+```python
+# config/base_config.py
+
+# 平台选择
+PLATFORM = "xhs"  # xhs, dy, ks, bili, wb, tieba, zhihu
+
+# 登录配置
+LOGIN_TYPE = "qrcode"  # qrcode, phone, cookie
+SAVE_LOGIN_STATE = True
+
+# 爬虫配置
+CRAWLER_TYPE = "search"  # search, detail, creator
+KEYWORDS = "编程副业,编程兼职"
+CRAWLER_MAX_NOTES_COUNT = 15
+MAX_CONCURRENCY_NUM = 1
+
+# 评论配置
+ENABLE_GET_COMMENTS = True
+ENABLE_GET_SUB_COMMENTS = False
+CRAWLER_MAX_COMMENTS_COUNT_SINGLENOTES = 10
+
+# 浏览器配置
+HEADLESS = False
+ENABLE_CDP_MODE = True
+CDP_DEBUG_PORT = 9222
+
+# 代理配置
+ENABLE_IP_PROXY = False
+IP_PROXY_PROVIDER = "kuaidaili"
+IP_PROXY_POOL_COUNT = 2
+
+# 存储配置
+SAVE_DATA_OPTION = "jsonl"  # csv, db, json, jsonl, sqlite, mongodb, excel, postgres
+```
+
+### 8.2 数据库配置
+
+```python
+# config/db_config.py
+
+# MySQL
+MYSQL_DB_HOST = "localhost"
+MYSQL_DB_PORT = 3306
+MYSQL_DB_NAME = "media_crawler"
+
+# Redis
+REDIS_DB_HOST = "127.0.0.1"
+REDIS_DB_PORT = 6379
+
+# MongoDB
+MONGODB_HOST = "localhost"
+MONGODB_PORT = 27017
+
+# SQLite
+SQLITE_DB_PATH = "database/sqlite_tables.db"
+```
+
+---
+
+## 9. 工具模块
+
+### 9.1 工具函数概览
+
+| 模块 | 文件 | 主要功能 |
+|------|------|---------|
+| 应用运行器 | `app_runner.py` | 信号处理、优雅退出、清理管理 |
+| 浏览器启动 | `browser_launcher.py` | 检测浏览器路径、启动浏览器进程 |
+| CDP管理 | `cdp_browser.py` | CDP连接、浏览器上下文管理 |
+| 爬虫工具 | `crawler_util.py` | 二维码识别、验证码处理、User-Agent |
+| 文件写入 | `async_file_writer.py` | 异步CSV/JSON写入、词云生成 |
+| 滑块验证 | `slider_util.py` | 滑动验证码破解 |
+| 时间工具 | `time_util.py` | 时间戳转换、日期处理 |
+
+### 9.2 应用运行管理
+
+```mermaid
+flowchart TB
+    Start([程序启动]) --> Run["run(app_main, app_cleanup)"]
+    Run --> Main["执行 app_main()"]
+    Main --> Running{运行中}
+
+    Running -->|正常完成| Cleanup1["执行 app_cleanup()"]
+    Running -->|SIGINT/SIGTERM| Signal["捕获信号"]
+
+    Signal --> First{第一次信号?}
+    First -->|是| Cleanup2["启动清理流程"]
+    First -->|否| Force["强制退出"]
+
+    Cleanup1 & Cleanup2 --> Cancel["取消其他任务"]
+    Cancel --> Wait["等待任务完成<br/>(超时15秒)"]
+    Wait --> End([程序退出])
+    Force --> End
+```
+
+---
+
+## 10. 模块依赖关系
+
+```mermaid
+flowchart TB
+    subgraph Entry["入口层"]
+        main["main.py"]
+        config["config/"]
+        cmdarg["cmd_arg/"]
+    end
+
+    subgraph Core["核心层"]
+        base["base/base_crawler.py"]
+        platforms["media_platform/*/"]
+    end
+
+    subgraph Client["客户端层"]
+        client["*/client.py"]
+        login["*/login.py"]
+    end
+
+    subgraph Storage["存储层"]
+        store["store/"]
+        database["database/"]
+    end
+
+    subgraph Infra["基础设施"]
+        proxy["proxy/"]
+        cache["cache/"]
+        tools["tools/"]
+    end
+
+    subgraph External["外部依赖"]
+        playwright["Playwright"]
+        httpx["httpx"]
+        sqlalchemy["SQLAlchemy"]
+        motor["Motor/MongoDB"]
+    end
+
+    main --> config
+    main --> cmdarg
+    main --> Core
+
+    Core --> base
+    platforms --> base
+    platforms --> Client
+
+    client --> proxy
+    client --> httpx
+    login --> tools
+
+    platforms --> Storage
+    Storage --> sqlalchemy
+    Storage --> motor
+
+    client --> playwright
+    tools --> playwright
+
+    proxy --> cache
+```
+
+---
+
+## 11. 扩展指南
+
+### 11.1 添加新平台
+
+1. 在 `media_platform/` 下创建新目录
+2. 实现以下核心文件：
+   - `core.py` - 继承 `AbstractCrawler`
+   - `client.py` - 继承 `AbstractApiClient` 和 `ProxyRefreshMixin`
+   - `login.py` - 继承 `AbstractLogin`
+   - `field.py` - 定义平台枚举
+3. 在 `store/` 下创建对应存储目录
+4. 在 `main.py` 的 `CrawlerFactory.CRAWLERS` 中注册
+
+### 11.2 添加新存储方式
+
+1. 在 `store/` 下创建新的存储实现类
+2. 继承 `AbstractStore` 基类
+3. 实现 `store_content`、`store_comment`、`store_creator` 方法
+4. 在各平台的 `StoreFactory.STORES` 中注册
+
+### 11.3 添加新代理提供商
+
+1. 在 `proxy/providers/` 下创建新的代理类
+2. 继承 `BaseProxy` 基类
+3. 实现 `get_proxy()` 方法
+4. 在配置中注册
+
+---
+
+## 12. 快速参考
+
+### 12.1 常用命令
+
+```bash
+# 启动爬虫
+python main.py
+
+# 指定平台
+python main.py --platform xhs
+
+# 指定登录方式
+python main.py --lt qrcode
+
+# 指定爬虫类型
+python main.py --type search
+```
+
+### 12.2 关键文件路径
+
+| 用途 | 文件路径 |
+|------|---------|
+| 程序入口 | `main.py` |
+| 核心配置 | `config/base_config.py` |
+| 数据库配置 | `config/db_config.py` |
+| 爬虫基类 | `base/base_crawler.py` |
+| ORM模型 | `database/models.py` |
+| 代理池 | `proxy/proxy_ip_pool.py` |
+| CDP浏览器 | `tools/cdp_browser.py` |
+
+---
+
+*文档生成时间: 2025-12-18*

main.py

@@ -17,11 +17,20 @@
 # 详细许可条款请参阅项目根目录下的LICENSE文件。
 # 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
 
+import sys
+import io
+
+# Force UTF-8 encoding for stdout/stderr to prevent encoding errors
+# when outputting Chinese characters in non-UTF-8 terminals
+if sys.stdout and hasattr(sys.stdout, 'buffer'):
+    if sys.stdout.encoding and sys.stdout.encoding.lower() != 'utf-8':
+        sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8', errors='replace')
+if sys.stderr and hasattr(sys.stderr, 'buffer'):
+    if sys.stderr.encoding and sys.stderr.encoding.lower() != 'utf-8':
+        sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding='utf-8', errors='replace')
 
 import asyncio
-import sys
-import signal
-from typing import Optional
+from typing import Optional, Type
 
 import cmd_arg
 import config
@@ -39,7 +48,7 @@ from var import crawler_type_var
 
 
 class CrawlerFactory:
-    CRAWLERS = {
+    CRAWLERS: dict[str, Type[AbstractCrawler]] = {
         "xhs": XiaoHongShuCrawler,
         "dy": DouYinCrawler,
         "ks": KuaishouCrawler,
@@ -53,106 +62,96 @@ class CrawlerFactory:
     def create_crawler(platform: str) -> AbstractCrawler:
         crawler_class = CrawlerFactory.CRAWLERS.get(platform)
         if not crawler_class:
-            raise ValueError(
-                "Invalid Media Platform Currently only supported xhs or dy or ks or bili ..."
-            )
+            supported = ", ".join(sorted(CrawlerFactory.CRAWLERS))
+            raise ValueError(f"Invalid media platform: {platform!r}. Supported: {supported}")
         return crawler_class()
 
 
 crawler: Optional[AbstractCrawler] = None
 
 
-# persist-1<persist1@126.com>
-# 原因：增加 --init_db 功能，用于数据库初始化。
-# 副作用：无
-# 回滚策略：还原此文件。
-async def main():
-    # Init crawler
+def _flush_excel_if_needed() -> None:
+    if config.SAVE_DATA_OPTION != "excel":
+        return
+
+    try:
+        from store.excel_store_base import ExcelStoreBase
+
+        ExcelStoreBase.flush_all()
+        print("[Main] Excel files saved successfully")
+    except Exception as e:
+        print(f"[Main] Error flushing Excel data: {e}")
+
+
+async def _generate_wordcloud_if_needed() -> None:
+    if config.SAVE_DATA_OPTION not in ("json", "jsonl") or not config.ENABLE_GET_WORDCLOUD:
+        return
+
+    try:
+        file_writer = AsyncFileWriter(
+            platform=config.PLATFORM,
+            crawler_type=crawler_type_var.get(),
+        )
+        await file_writer.generate_wordcloud_from_comments()
+    except Exception as e:
+        print(f"[Main] Error generating wordcloud: {e}")
+
+
+async def main() -> None:
     global crawler
 
-    # parse cmd
     args = await cmd_arg.parse_cmd()
-
-    # init db
     if args.init_db:
         await db.init_db(args.init_db)
         print(f"Database {args.init_db} initialized successfully.")
-        return  # Exit the main function cleanly
-
-
+        return
 
     crawler = CrawlerFactory.create_crawler(platform=config.PLATFORM)
     await crawler.start()
 
+    _flush_excel_if_needed()
+
     # Generate wordcloud after crawling is complete
     # Only for JSON save mode
-    if config.SAVE_DATA_OPTION == "json" and config.ENABLE_GET_WORDCLOUD:
-        try:
-            file_writer = AsyncFileWriter(
-                platform=config.PLATFORM,
-                crawler_type=crawler_type_var.get()
-            )
-            await file_writer.generate_wordcloud_from_comments()
-        except Exception as e:
-            print(f"Error generating wordcloud: {e}")
+    await _generate_wordcloud_if_needed()
 
 
-async def async_cleanup():
-    """异步清理函数，用于处理CDP浏览器等异步资源"""
+async def async_cleanup() -> None:
     global crawler
     if crawler:
-        # 检查并清理CDP浏览器
-        if hasattr(crawler, 'cdp_manager') and crawler.cdp_manager:
+        if getattr(crawler, "cdp_manager", None):
             try:
-                await crawler.cdp_manager.cleanup(force=True)  # 强制清理浏览器进程
+                await crawler.cdp_manager.cleanup(force=True)
             except Exception as e:
-                # 只在非预期错误时打印
                 error_msg = str(e).lower()
                 if "closed" not in error_msg and "disconnected" not in error_msg:
-                    print(f"[Main] 清理CDP浏览器时出错: {e}")
+                    print(f"[Main] Error cleaning up CDP browser: {e}")
 
-        # 检查并清理标准浏览器上下文（仅在非CDP模式下）
-        elif hasattr(crawler, 'browser_context') and crawler.browser_context:
+        elif getattr(crawler, "browser_context", None):
             try:
-                # 检查上下文是否仍然打开
-                if hasattr(crawler.browser_context, 'pages'):
-                    await crawler.browser_context.close()
+                await crawler.browser_context.close()
             except Exception as e:
-                # 只在非预期错误时打印
                 error_msg = str(e).lower()
                 if "closed" not in error_msg and "disconnected" not in error_msg:
-                    print(f"[Main] 关闭浏览器上下文时出错: {e}")
+                    print(f"[Main] Error closing browser context: {e}")
 
-    # 关闭数据库连接
-    if config.SAVE_DATA_OPTION in ["db", "sqlite"]:
+    if config.SAVE_DATA_OPTION in ("db", "sqlite"):
         await db.close()
 
-def cleanup():
-    """同步清理函数"""
-    try:
-        # 创建新的事件循环来执行异步清理
-        loop = asyncio.new_event_loop()
-        asyncio.set_event_loop(loop)
-        loop.run_until_complete(async_cleanup())
-        loop.close()
-    except Exception as e:
-        print(f"[Main] 清理时出错: {e}")
-
-
-def signal_handler(signum, _frame):
-    """信号处理器，处理Ctrl+C等中断信号"""
-    print(f"\n[Main] 收到中断信号 {signum}，正在清理资源...")
-    cleanup()
-    sys.exit(0)
-
 if __name__ == "__main__":
-    # 注册信号处理器
-    signal.signal(signal.SIGINT, signal_handler)   # Ctrl+C
-    signal.signal(signal.SIGTERM, signal_handler)  # 终止信号
+    from tools.app_runner import run
 
-    try:
-        asyncio.get_event_loop().run_until_complete(main())
-    except KeyboardInterrupt:
-        print("\n[Main] 收到键盘中断，正在清理资源...")
-    finally:
-        cleanup()
+    def _force_stop() -> None:
+        c = crawler
+        if not c:
+            return
+        cdp_manager = getattr(c, "cdp_manager", None)
+        launcher = getattr(cdp_manager, "launcher", None)
+        if not launcher:
+            return
+        try:
+            launcher.cleanup()
+        except Exception:
+            pass
+
+    run(main, async_cleanup, cleanup_timeout_seconds=15.0, on_first_interrupt=_force_stop)

media_platform/bilibili/client.py

@@ -20,7 +20,7 @@
 # -*- coding: utf-8 -*-
 # @Author  : relakkes@gmail.com
 # @Time    : 2023/12/2 18:44
-# @Desc    : bilibili 请求客户端
+# @Desc    : bilibili request client
 import asyncio
 import json
 import random
@@ -29,6 +29,7 @@ from urllib.parse import urlencode
 
 import httpx
 from playwright.async_api import BrowserContext, Page
+from tools.httpx_util import make_async_client
 
 import config
 from base.base_crawler import AbstractApiClient
@@ -47,7 +48,7 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
 
     def __init__(
         self,
-        timeout=60,  # 若开启爬取媒体选项，b 站的长视频需要更久的超时时间
+        timeout=60,  # For media crawling, Bilibili long videos need a longer timeout
         proxy=None,
         *,
         headers: Dict[str, str],
@@ -59,16 +60,17 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
         self.timeout = timeout
         self.headers = headers
         self._host = "https://api.bilibili.com"
+        self.cookie_urls = ["https://www.bilibili.com"]
         self.playwright_page = playwright_page
         self.cookie_dict = cookie_dict
-        # 初始化代理池（来自 ProxyRefreshMixin）
+        # Initialize proxy pool (from ProxyRefreshMixin)
         self.init_proxy_pool(proxy_ip_pool)
 
     async def request(self, method, url, **kwargs) -> Any:
-        # 每次请求前检测代理是否过期
+        # Check if proxy has expired before each request
         await self._refresh_proxy_if_expired()
 
-        async with httpx.AsyncClient(proxy=self.proxy) as client:
+        async with make_async_client(proxy=self.proxy) as client:
             response = await client.request(method, url, timeout=self.timeout, **kwargs)
         try:
             data: Dict = response.json()
@@ -82,8 +84,8 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
 
     async def pre_request_data(self, req_data: Dict) -> Dict:
         """
-        发送请求进行请求参数签名
-        需要从 localStorage 拿 wbi_img_urls 这参数，值如下：
+        Send request to sign request parameters
+        Need to get wbi_img_urls parameter from localStorage, value as follows:
         https://i0.hdslb.com/bfs/wbi/7cd084941338484aae1ad9425b84077c.png-https://i0.hdslb.com/bfs/wbi/4932caff0ff746eab6f01bf08b70ac45.png
         :param req_data:
         :return:
@@ -95,7 +97,7 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
 
     async def get_wbi_keys(self) -> Tuple[str, str]:
         """
-        获取最新的 img_key 和 sub_key
+        Get the latest img_key and sub_key
         :return:
         """
         local_storage = await self.playwright_page.evaluate("() => window.localStorage")
@@ -144,8 +146,11 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
             ping_flag = False
         return ping_flag
 
-    async def update_cookies(self, browser_context: BrowserContext):
-        cookie_str, cookie_dict = utils.convert_cookies(await browser_context.cookies())
+    async def update_cookies(self, browser_context: BrowserContext, urls: Optional[list[str]] = None):
+        cookie_str, cookie_dict = await utils.convert_browser_context_cookies(
+            browser_context,
+            urls=urls or self.cookie_urls,
+        )
         self.headers["Cookie"] = cookie_str
         self.cookie_dict = cookie_dict
 
@@ -160,12 +165,12 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
     ) -> Dict:
         """
         KuaiShou web search api
-        :param keyword: 搜索关键词
-        :param page: 分页参数具体第几页
-        :param page_size: 每一页参数的数量
-        :param order: 搜索结果排序，默认位综合排序
-        :param pubtime_begin_s: 发布时间开始时间戳
-        :param pubtime_end_s: 发布时间结束时间戳
+        :param keyword: Search keyword
+        :param page: Page number for pagination
+        :param page_size: Number of items per page
+        :param order: Sort order for search results, default is comprehensive sorting
+        :param pubtime_begin_s: Publish time start timestamp
+        :param pubtime_end_s: Publish time end timestamp
         :return:
         """
         uri = "/x/web-interface/wbi/search/type"
@@ -182,13 +187,13 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
 
     async def get_video_info(self, aid: Union[int, None] = None, bvid: Union[str, None] = None) -> Dict:
         """
-        Bilibli web video detail api, aid 和 bvid任选一个参数
-        :param aid: 稿件avid
-        :param bvid: 稿件bvid
+        Bilibli web video detail api, choose one parameter between aid and bvid
+        :param aid: Video aid
+        :param bvid: Video bvid
         :return:
         """
         if not aid and not bvid:
-            raise ValueError("请提供 aid 或 bvid 中的至少一个参数")
+            raise ValueError("Please provide at least one parameter: aid or bvid")
 
         uri = "/x/web-interface/view/detail"
         params = dict()
@@ -201,12 +206,12 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
     async def get_video_play_url(self, aid: int, cid: int) -> Dict:
         """
         Bilibli web video play url api
-        :param aid: 稿件avid
+        :param aid: Video aid
         :param cid: cid
         :return:
         """
         if not aid or not cid or aid <= 0 or cid <= 0:
-            raise ValueError("aid 和 cid 必须存在")
+            raise ValueError("aid and cid must exist")
         uri = "/x/player/wbi/playurl"
         qn_value = getattr(config, "BILI_QN", 80)
         params = {
@@ -222,7 +227,7 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
 
     async def get_video_media(self, url: str) -> Union[bytes, None]:
         # Follow CDN 302 redirects and treat any 2xx as success (some endpoints return 206)
-        async with httpx.AsyncClient(proxy=self.proxy, follow_redirects=True) as client:
+        async with make_async_client(proxy=self.proxy, follow_redirects=True) as client:
             try:
                 response = await client.request("GET", url, timeout=self.timeout, headers=self.headers)
                 response.raise_for_status()
@@ -233,7 +238,7 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
                 )
                 return None
             except httpx.HTTPError as exc:  # some wrong when call httpx.request method, such as connection error, client error, server error or response status code is not 2xx
-                utils.logger.error(f"[BilibiliClient.get_video_media] {exc.__class__.__name__} for {exc.request.url} - {exc}")  # 保留原始异常类型名称，以便开发者调试
+                utils.logger.error(f"[BilibiliClient.get_video_media] {exc.__class__.__name__} for {exc.request.url} - {exc}")  # Keep original exception type name for developer debugging
                 return None
 
     async def get_video_comments(
@@ -243,9 +248,9 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
         next: int = 0,
     ) -> Dict:
         """get video comments
-        :param video_id: 视频 ID
-        :param order_mode: 排序方式
-        :param next: 评论页选择
+        :param video_id: Video ID
+        :param order_mode: Sort order
+        :param next: Comment page selection
         :return:
         """
         uri = "/x/v2/reply/wbi/main"
@@ -266,7 +271,7 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
         :param crawl_interval:
         :param is_fetch_sub_comments:
         :param callback:
-        max_count: 一次笔记爬取的最大评论数量
+        max_count: Maximum number of comments to crawl per note
 
         :return:
         """
@@ -299,7 +304,7 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
 
             comment_list: List[Dict] = comments_res.get("replies", [])
 
-            # 检查 is_end 和 next 是否存在
+            # Check if is_end and next exist
             if "is_end" not in cursor_info or "next" not in cursor_info:
                 utils.logger.warning(f"[BilibiliClient.get_video_all_comments] 'is_end' or 'next' not in cursor for video_id: {video_id}. Assuming end of comments.")
                 is_end = True
@@ -317,7 +322,7 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
                         {await self.get_video_all_level_two_comments(video_id, comment_id, CommentOrderType.DEFAULT, 10, crawl_interval, callback)}
             if len(result) + len(comment_list) > max_count:
                 comment_list = comment_list[:max_count - len(result)]
-            if callback:  # 如果有回调函数，就执行回调函数
+            if callback:  # If there is a callback function, execute it
                 await callback(video_id, comment_list)
             await asyncio.sleep(crawl_interval)
             if not is_fetch_sub_comments:
@@ -336,10 +341,10 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
     ) -> Dict:
         """
         get video all level two comments for a level one comment
-        :param video_id: 视频 ID
-        :param level_one_comment_id: 一级评论 ID
+        :param video_id: Video ID
+        :param level_one_comment_id: Level one comment ID
         :param order_mode:
-        :param ps: 一页评论数
+        :param ps: Number of comments per page
         :param crawl_interval:
         :param callback:
         :return:
@@ -349,7 +354,7 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
         while True:
             result = await self.get_video_level_two_comments(video_id, level_one_comment_id, pn, ps, order_mode)
             comment_list: List[Dict] = result.get("replies", [])
-            if callback:  # 如果有回调函数，就执行回调函数
+            if callback:  # If there is a callback function, execute it
                 await callback(video_id, comment_list)
             await asyncio.sleep(crawl_interval)
             if (int(result["page"]["count"]) <= pn * ps):
@@ -366,9 +371,9 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
         order_mode: CommentOrderType,
     ) -> Dict:
         """get video level two comments
-        :param video_id: 视频 ID
-        :param level_one_comment_id: 一级评论 ID
-        :param order_mode: 排序方式
+        :param video_id: Video ID
+        :param level_one_comment_id: Level one comment ID
+        :param order_mode: Sort order
 
         :return:
         """
@@ -386,10 +391,10 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
 
     async def get_creator_videos(self, creator_id: str, pn: int, ps: int = 30, order_mode: SearchOrderType = SearchOrderType.LAST_PUBLISH) -> Dict:
         """get all videos for a creator
-        :param creator_id: 创作者 ID
-        :param pn: 页数
-        :param ps: 一页视频数
-        :param order_mode: 排序方式
+        :param creator_id: Creator ID
+        :param pn: Page number
+        :param ps: Number of videos per page
+        :param order_mode: Sort order
 
         :return:
         """
@@ -405,7 +410,7 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
     async def get_creator_info(self, creator_id: int) -> Dict:
         """
         get creator info
-        :param creator_id: 作者 ID
+        :param creator_id: Creator ID
         """
         uri = "/x/space/wbi/acc/info"
         post_data = {
@@ -421,9 +426,9 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
     ) -> Dict:
         """
         get creator fans
-        :param creator_id: 创作者 ID
-        :param pn: 开始页数
-        :param ps: 每页数量
+        :param creator_id: Creator ID
+        :param pn: Start page number
+        :param ps: Number of items per page
         :return:
         """
         uri = "/x/relation/fans"
@@ -443,9 +448,9 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
     ) -> Dict:
         """
         get creator followings
-        :param creator_id: 创作者 ID
-        :param pn: 开始页数
-        :param ps: 每页数量
+        :param creator_id: Creator ID
+        :param pn: Start page number
+        :param ps: Number of items per page
         :return:
         """
         uri = "/x/relation/followings"
@@ -460,8 +465,8 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
     async def get_creator_dynamics(self, creator_id: int, offset: str = ""):
         """
         get creator comments
-        :param creator_id: 创作者 ID
-        :param offset: 发送请求所需参数
+        :param creator_id: Creator ID
+        :param offset: Parameter required for sending request
         :return:
         """
         uri = "/x/polymer/web-dynamic/v1/feed/space"
@@ -485,9 +490,9 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
         :param creator_info:
         :param crawl_interval:
         :param callback:
-        :param max_count: 一个up主爬取的最大粉丝数量
+        :param max_count: Maximum number of fans to crawl for a creator
 
-        :return: up主粉丝数列表
+        :return: List of creator fans
         """
         creator_id = creator_info["id"]
         result = []
@@ -499,7 +504,7 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
             pn += 1
             if len(result) + len(fans_list) > max_count:
                 fans_list = fans_list[:max_count - len(result)]
-            if callback:  # 如果有回调函数，就执行回调函数
+            if callback:  # If there is a callback function, execute it
                 await callback(creator_info, fans_list)
             await asyncio.sleep(crawl_interval)
             if not fans_list:
@@ -519,9 +524,9 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
         :param creator_info:
         :param crawl_interval:
         :param callback:
-        :param max_count: 一个up主爬取的最大关注者数量
+        :param max_count: Maximum number of followings to crawl for a creator
 
-        :return: up主关注者列表
+        :return: List of creator followings
         """
         creator_id = creator_info["id"]
         result = []
@@ -533,7 +538,7 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
             pn += 1
             if len(result) + len(followings_list) > max_count:
                 followings_list = followings_list[:max_count - len(result)]
-            if callback:  # 如果有回调函数，就执行回调函数
+            if callback:  # If there is a callback function, execute it
                 await callback(creator_info, followings_list)
             await asyncio.sleep(crawl_interval)
             if not followings_list:
@@ -553,9 +558,9 @@ class BilibiliClient(AbstractApiClient, ProxyRefreshMixin):
         :param creator_info:
         :param crawl_interval:
         :param callback:
-        :param max_count: 一个up主爬取的最大动态数量
+        :param max_count: Maximum number of dynamics to crawl for a creator
 
-        :return: up主关注者列表
+        :return: List of creator dynamics
         """
         creator_id = creator_info["id"]
         result = []

media_platform/bilibili/core.py

@@ -20,7 +20,7 @@
 # -*- coding: utf-8 -*-
 # @Author  : relakkes@gmail.com
 # @Time    : 2023/12/2 18:44
-# @Desc    : B站爬虫
+# @Desc    : Bilibili Crawler
 
 import asyncio
 import os
@@ -62,9 +62,10 @@ class BilibiliCrawler(AbstractCrawler):
 
     def __init__(self):
         self.index_url = "https://www.bilibili.com"
+        self.cookie_urls = [self.index_url]
         self.user_agent = utils.get_user_agent()
         self.cdp_manager = None
-        self.ip_proxy_pool = None  # 代理IP池，用于代理自动刷新
+        self.ip_proxy_pool = None  # Proxy IP pool for automatic proxy refresh
 
     async def start(self):
         playwright_proxy_format, httpx_proxy_format = None, None
@@ -74,9 +75,9 @@ class BilibiliCrawler(AbstractCrawler):
             playwright_proxy_format, httpx_proxy_format = utils.format_proxy_info(ip_proxy_info)
 
         async with async_playwright() as playwright:
-            # 根据配置选择启动模式
+            # Choose launch mode based on configuration
             if config.ENABLE_CDP_MODE:
-                utils.logger.info("[BilibiliCrawler] 使用CDP模式启动浏览器")
+                utils.logger.info("[BilibiliCrawler] Launching browser using CDP mode")
                 self.browser_context = await self.launch_browser_with_cdp(
                     playwright,
                     playwright_proxy_format,
@@ -84,7 +85,7 @@ class BilibiliCrawler(AbstractCrawler):
                     headless=config.CDP_HEADLESS,
                 )
             else:
-                utils.logger.info("[BilibiliCrawler] 使用标准模式启动浏览器")
+                utils.logger.info("[BilibiliCrawler] Launching browser using standard mode")
                 # Launch a browser context.
                 chromium = playwright.chromium
                 self.browser_context = await self.launch_browser(chromium, None, self.user_agent, headless=config.HEADLESS)
@@ -105,7 +106,10 @@ class BilibiliCrawler(AbstractCrawler):
                     cookie_str=config.COOKIES,
                 )
                 await login_obj.begin()
-                await self.bili_client.update_cookies(browser_context=self.browser_context)
+                await self.bili_client.update_cookies(
+                    browser_context=self.browser_context,
+                    urls=self.cookie_urls,
+                )
 
             crawler_type_var.set(config.CRAWLER_TYPE)
             if config.CRAWLER_TYPE == "search":
@@ -149,31 +153,31 @@ class BilibiliCrawler(AbstractCrawler):
         end: str = config.END_DAY,
     ) -> Tuple[str, str]:
         """
-        获取 bilibili 作品发布日期起始时间戳 pubtime_begin_s 与发布日期结束时间戳 pubtime_end_s
+        Get bilibili publish start timestamp pubtime_begin_s and publish end timestamp pubtime_end_s
         ---
-        :param start: 发布日期起始时间，YYYY-MM-DD
-        :param end: 发布日期结束时间，YYYY-MM-DD
+        :param start: Publish date start time, YYYY-MM-DD
+        :param end: Publish date end time, YYYY-MM-DD
 
         Note
         ---
-        - 搜索的时间范围为 start 至 end，包含 start 和 end
-        - 若要搜索同一天的内容，为了包含 start 当天的搜索内容，则 pubtime_end_s 的值应该为 pubtime_begin_s 的值加上一天再减去一秒，即 start 当天的最后一秒
-            - 如仅搜索 2024-01-05 的内容，pubtime_begin_s = 1704384000，pubtime_end_s = 1704470399
-              转换为可读的 datetime 对象：pubtime_begin_s = datetime.datetime(2024, 1, 5, 0, 0)，pubtime_end_s = datetime.datetime(2024, 1, 5, 23, 59, 59)
-        - 若要搜索 start 至 end 的内容，为了包含 end 当天的搜索内容，则 pubtime_end_s 的值应该为 pubtime_end_s 的值加上一天再减去一秒，即 end 当天的最后一秒
-            - 如搜索 2024-01-05 - 2024-01-06 的内容，pubtime_begin_s = 1704384000，pubtime_end_s = 1704556799
-              转换为可读的 datetime 对象：pubtime_begin_s = datetime.datetime(2024, 1, 5, 0, 0)，pubtime_end_s = datetime.datetime(2024, 1, 6, 23, 59, 59)
+        - Search time range is from start to end, including both start and end
+        - To search content from the same day, to include search content from that day, pubtime_end_s should be pubtime_begin_s plus one day minus one second, i.e., the last second of start day
+            - For example, searching only 2024-01-05 content, pubtime_begin_s = 1704384000, pubtime_end_s = 1704470399
+              Converted to readable datetime objects: pubtime_begin_s = datetime.datetime(2024, 1, 5, 0, 0), pubtime_end_s = datetime.datetime(2024, 1, 5, 23, 59, 59)
+        - To search content from start to end, to include search content from end day, pubtime_end_s should be pubtime_end_s plus one day minus one second, i.e., the last second of end day
+            - For example, searching 2024-01-05 - 2024-01-06 content, pubtime_begin_s = 1704384000, pubtime_end_s = 1704556799
+              Converted to readable datetime objects: pubtime_begin_s = datetime.datetime(2024, 1, 5, 0, 0), pubtime_end_s = datetime.datetime(2024, 1, 6, 23, 59, 59)
         """
-        # 转换 start 与 end 为 datetime 对象
+        # Convert start and end to datetime objects
         start_day: datetime = datetime.strptime(start, "%Y-%m-%d")
         end_day: datetime = datetime.strptime(end, "%Y-%m-%d")
         if start_day > end_day:
             raise ValueError("Wrong time range, please check your start and end argument, to ensure that the start cannot exceed end")
-        elif start_day == end_day:  # 搜索同一天的内容
-            end_day = (start_day + timedelta(days=1) - timedelta(seconds=1))  # 则将 end_day 设置为 start_day + 1 day - 1 second
-        else:  # 搜索 start 至 end
-            end_day = (end_day + timedelta(days=1) - timedelta(seconds=1))  # 则将 end_day 设置为 end_day + 1 day - 1 second
-        # 将其重新转换为时间戳
+        elif start_day == end_day:  # Searching content from the same day
+            end_day = (start_day + timedelta(days=1) - timedelta(seconds=1))  # Set end_day to start_day + 1 day - 1 second
+        else:  # Searching from start to end
+            end_day = (end_day + timedelta(days=1) - timedelta(seconds=1))  # Set end_day to end_day + 1 day - 1 second
+        # Convert back to timestamps
         return str(int(start_day.timestamp())), str(int(end_day.timestamp()))
 
     async def search_by_keywords(self):
@@ -203,8 +207,8 @@ class BilibiliCrawler(AbstractCrawler):
                     page=page,
                     page_size=bili_limit_count,
                     order=SearchOrderType.DEFAULT,
-                    pubtime_begin_s=0,  # 作品发布日期起始时间戳
-                    pubtime_end_s=0,  # 作品发布日期结束日期时间戳
+                    pubtime_begin_s=0,  # Publish date start timestamp
+                    pubtime_end_s=0,  # Publish date end timestamp
                 )
                 video_list: List[Dict] = videos_res.get("result")
 
@@ -462,7 +466,10 @@ class BilibiliCrawler(AbstractCrawler):
         :return: bilibili client
         """
         utils.logger.info("[BilibiliCrawler.create_bilibili_client] Begin create bilibili API client ...")
-        cookie_str, cookie_dict = utils.convert_cookies(await self.browser_context.cookies())
+        cookie_str, cookie_dict = await utils.convert_browser_context_cookies(
+            self.browser_context,
+            urls=self.cookie_urls,
+        )
         bilibili_client_obj = BilibiliClient(
             proxy=httpx_proxy,
             headers={
@@ -474,7 +481,7 @@ class BilibiliCrawler(AbstractCrawler):
             },
             playwright_page=self.context_page,
             cookie_dict=cookie_dict,
-            proxy_ip_pool=self.ip_proxy_pool,  # 传递代理池用于自动刷新
+            proxy_ip_pool=self.ip_proxy_pool,  # Pass proxy pool for automatic refresh
         )
         return bilibili_client_obj
 
@@ -508,7 +515,7 @@ class BilibiliCrawler(AbstractCrawler):
                     "height": 1080
                 },
                 user_agent=user_agent,
-                channel="chrome",  # 使用系统的Chrome稳定版
+                channel="chrome",  # Use system's stable Chrome version
             )
             return browser_context
         else:
@@ -525,7 +532,7 @@ class BilibiliCrawler(AbstractCrawler):
         headless: bool = True,
     ) -> BrowserContext:
         """
-        使用CDP模式启动浏览器
+        Launch browser using CDP mode
         """
         try:
             self.cdp_manager = CDPBrowserManager()
@@ -536,22 +543,22 @@ class BilibiliCrawler(AbstractCrawler):
                 headless=headless,
             )
 
-            # 显示浏览器信息
+            # Display browser information
             browser_info = await self.cdp_manager.get_browser_info()
-            utils.logger.info(f"[BilibiliCrawler] CDP浏览器信息: {browser_info}")
+            utils.logger.info(f"[BilibiliCrawler] CDP browser info: {browser_info}")
 
             return browser_context
 
         except Exception as e:
-            utils.logger.error(f"[BilibiliCrawler] CDP模式启动失败，回退到标准模式: {e}")
-            # 回退到标准模式
+            utils.logger.error(f"[BilibiliCrawler] CDP mode launch failed, fallback to standard mode: {e}")
+            # Fallback to standard mode
             chromium = playwright.chromium
             return await self.launch_browser(chromium, playwright_proxy, user_agent, headless)
 
     async def close(self):
         """Close browser context"""
         try:
-            # 如果使用CDP模式，需要特殊处理
+            # If using CDP mode, special handling is required
             if self.cdp_manager:
                 await self.cdp_manager.cleanup()
                 self.cdp_manager = None

media_platform/bilibili/field.py

@@ -27,28 +27,28 @@ from enum import Enum
 
 
 class SearchOrderType(Enum):
-    # 综合排序
+    # Comprehensive sorting
     DEFAULT = ""
 
-    # 最多点击
+    # Most clicks
     MOST_CLICK = "click"
 
-    # 最新发布
+    # Latest published
     LAST_PUBLISH = "pubdate"
 
-    # 最多弹幕
+    # Most danmu (comments)
     MOST_DANMU = "dm"
 
-    # 最多收藏
+    # Most bookmarks
     MOST_MARK = "stow"
 
 
 class CommentOrderType(Enum):
-    # 仅按热度
+    # By popularity only
     DEFAULT = 0
 
-    # 按热度+按时间
+    # By popularity + time
     MIXED = 1
 
-    # 按时间
+    # By time
     TIME = 2

media_platform/bilibili/help.py

@@ -21,8 +21,8 @@
 # -*- coding: utf-8 -*-
 # @Author  : relakkes@gmail.com
 # @Time    : 2023/12/2 23:26
-# @Desc    : bilibili 请求参数签名
-# 逆向实现参考：https://socialsisteryi.github.io/bilibili-API-collect/docs/misc/sign/wbi.html#wbi%E7%AD%BE%E5%90%8D%E7%AE%97%E6%B3%95
+# @Desc    : bilibili request parameter signing
+# Reverse engineering implementation reference: https://socialsisteryi.github.io/bilibili-API-collect/docs/misc/sign/wbi.html#wbi%E7%AD%BE%E5%90%8D%E7%AE%97%E6%B3%95
 import re
 import urllib.parse
 from hashlib import md5
@@ -45,7 +45,7 @@ class BilibiliSign:
 
     def get_salt(self) -> str:
         """
-        获取加盐的 key
+        Get the salted key
         :return:
         """
         salt = ""
@@ -56,8 +56,8 @@ class BilibiliSign:
 
     def sign(self, req_data: Dict) -> Dict:
         """
-        请求参数中加上当前时间戳对请求参数中的key进行字典序排序
-        再将请求参数进行 url 编码集合 salt 进行 md5 就可以生成w_rid参数了
+        Add current timestamp to request parameters, sort keys in dictionary order,
+        then URL encode the parameters and combine with salt to generate md5 for w_rid parameter
         :param req_data:
         :return:
         """
@@ -65,35 +65,35 @@ class BilibiliSign:
         req_data.update({"wts": current_ts})
         req_data = dict(sorted(req_data.items()))
         req_data = {
-            # 过滤 value 中的 "!'()*" 字符
+            # Filter "!'()*" characters from values
             k: ''.join(filter(lambda ch: ch not in "!'()*", str(v)))
             for k, v
             in req_data.items()
         }
         query = urllib.parse.urlencode(req_data)
         salt = self.get_salt()
-        wbi_sign = md5((query + salt).encode()).hexdigest()  # 计算 w_rid
+        wbi_sign = md5((query + salt).encode()).hexdigest()  # Calculate w_rid
         req_data['w_rid'] = wbi_sign
         return req_data
 
 
 def parse_video_info_from_url(url: str) -> VideoUrlInfo:
     """
-    从B站视频URL中解析出视频ID
+    Parse video ID from Bilibili video URL
     Args:
-        url: B站视频链接
+        url: Bilibili video link
             - https://www.bilibili.com/video/BV1dwuKzmE26/?spm_id_from=333.1387.homepage.video_card.click
             - https://www.bilibili.com/video/BV1d54y1g7db
-            - BV1d54y1g7db (直接传入BV号)
+            - BV1d54y1g7db (directly pass BV number)
     Returns:
-        VideoUrlInfo: 包含视频ID的对象
+        VideoUrlInfo: Object containing video ID
     """
-    # 如果传入的已经是BV号,直接返回
+    # If the input is already a BV number, return directly
     if url.startswith("BV"):
         return VideoUrlInfo(video_id=url)
 
-    # 使用正则表达式提取BV号
-    # 匹配 /video/BV... 或 /video/av... 格式
+    # Use regex to extract BV number
+    # Match /video/BV... or /video/av... format
     bv_pattern = r'/video/(BV[a-zA-Z0-9]+)'
     match = re.search(bv_pattern, url)
 
@@ -101,26 +101,26 @@ def parse_video_info_from_url(url: str) -> VideoUrlInfo:
         video_id = match.group(1)
         return VideoUrlInfo(video_id=video_id)
 
-    raise ValueError(f"无法从URL中解析出视频ID: {url}")
+    raise ValueError(f"Unable to parse video ID from URL: {url}")
 
 
 def parse_creator_info_from_url(url: str) -> CreatorUrlInfo:
     """
-    从B站创作者空间URL中解析出创作者ID
+    Parse creator ID from Bilibili creator space URL
     Args:
-        url: B站创作者空间链接
+        url: Bilibili creator space link
             - https://space.bilibili.com/434377496?spm_id_from=333.1007.0.0
             - https://space.bilibili.com/20813884
-            - 434377496 (直接传入UID)
+            - 434377496 (directly pass UID)
     Returns:
-        CreatorUrlInfo: 包含创作者ID的对象
+        CreatorUrlInfo: Object containing creator ID
     """
-    # 如果传入的已经是纯数字ID,直接返回
+    # If the input is already a numeric ID, return directly
     if url.isdigit():
         return CreatorUrlInfo(creator_id=url)
 
-    # 使用正则表达式提取UID
-    # 匹配 /space.bilibili.com/数字 格式
+    # Use regex to extract UID
+    # Match /space.bilibili.com/number format
     uid_pattern = r'space\.bilibili\.com/(\d+)'
     match = re.search(uid_pattern, url)
 
@@ -128,20 +128,20 @@ def parse_creator_info_from_url(url: str) -> CreatorUrlInfo:
         creator_id = match.group(1)
         return CreatorUrlInfo(creator_id=creator_id)
 
-    raise ValueError(f"无法从URL中解析出创作者ID: {url}")
+    raise ValueError(f"Unable to parse creator ID from URL: {url}")
 
 
 if __name__ == '__main__':
-    # 测试视频URL解析
+    # Test video URL parsing
     video_url1 = "https://www.bilibili.com/video/BV1dwuKzmE26/?spm_id_from=333.1387.homepage.video_card.click"
     video_url2 = "BV1d54y1g7db"
-    print("视频URL解析测试:")
+    print("Video URL parsing test:")
     print(f"URL1: {video_url1} -> {parse_video_info_from_url(video_url1)}")
     print(f"URL2: {video_url2} -> {parse_video_info_from_url(video_url2)}")
 
-    # 测试创作者URL解析
+    # Test creator URL parsing
     creator_url1 = "https://space.bilibili.com/434377496?spm_id_from=333.1007.0.0"
     creator_url2 = "20813884"
-    print("\n创作者URL解析测试:")
+    print("\nCreator URL parsing test:")
     print(f"URL1: {creator_url1} -> {parse_creator_info_from_url(creator_url1)}")
     print(f"URL2: {creator_url2} -> {parse_creator_info_from_url(creator_url2)}")

media_platform/bilibili/login.py

@@ -21,7 +21,7 @@
 # -*- coding: utf-8 -*-
 # @Author  : relakkes@gmail.com
 # @Time    : 2023/12/2 18:44
-# @Desc    : bilibli登录实现类
+# @Desc    : bilibili login implementation class
 
 import asyncio
 import functools

media_platform/douyin/client.py

@@ -29,6 +29,7 @@ from playwright.async_api import BrowserContext
 from base.base_crawler import AbstractApiClient
 from proxy.proxy_mixin import ProxyRefreshMixin
 from tools import utils
+from tools.httpx_util import make_async_client
 from var import request_keyword_var
 
 if TYPE_CHECKING:
@@ -43,7 +44,7 @@ class DouYinClient(AbstractApiClient, ProxyRefreshMixin):
 
     def __init__(
         self,
-        timeout=60,  # 若开启爬取媒体选项，抖音的短视频需要更久的超时时间
+        timeout=60,  # If the crawl media option is turned on, Douyin’s short videos will require a longer timeout.
         proxy=None,
         *,
         headers: Dict,
@@ -55,9 +56,16 @@ class DouYinClient(AbstractApiClient, ProxyRefreshMixin):
         self.timeout = timeout
         self.headers = headers
         self._host = "https://www.douyin.com"
+        self.cookie_urls = [
+            "https://douyin.com",
+            self._host,
+            "https://creator.douyin.com",
+            "https://douhot.douyin.com",
+            "https://live.douyin.com",
+        ]
         self.playwright_page = playwright_page
         self.cookie_dict = cookie_dict
-        # 初始化代理池（来自 ProxyRefreshMixin）
+        # Initialize proxy pool (from ProxyRefreshMixin)
         self.init_proxy_pool(proxy_ip_pool)
 
     async def __process_req_params(
@@ -103,7 +111,7 @@ class DouYinClient(AbstractApiClient, ProxyRefreshMixin):
         params.update(common_params)
         query_string = urllib.parse.urlencode(params)
 
-        # 20240927 a-bogus更新（JS版本）
+        # 20240927 a-bogus update (JS version)
         post_data = {}
         if request_method == "POST":
             post_data = params
@@ -113,10 +121,10 @@ class DouYinClient(AbstractApiClient, ProxyRefreshMixin):
             params["a_bogus"] = a_bogus
 
     async def request(self, method, url, **kwargs):
-        # 每次请求前检测代理是否过期
+        # Check whether the proxy has expired before each request
         await self._refresh_proxy_if_expired()
 
-        async with httpx.AsyncClient(proxy=self.proxy) as client:
+        async with make_async_client(proxy=self.proxy) as client:
             response = await client.request(method, url, timeout=self.timeout, **kwargs)
         try:
             if response.text == "" or response.text == "blocked":
@@ -144,11 +152,17 @@ class DouYinClient(AbstractApiClient, ProxyRefreshMixin):
         if local_storage.get("HasUserLogin", "") == "1":
             return True
 
-        _, cookie_dict = utils.convert_cookies(await browser_context.cookies())
+        _, cookie_dict = await utils.convert_browser_context_cookies(
+            browser_context,
+            urls=self.cookie_urls,
+        )
         return cookie_dict.get("LOGIN_STATUS") == "1"
 
-    async def update_cookies(self, browser_context: BrowserContext):
-        cookie_str, cookie_dict = utils.convert_cookies(await browser_context.cookies())
+    async def update_cookies(self, browser_context: BrowserContext, urls: Optional[list[str]] = None):
+        cookie_str, cookie_dict = await utils.convert_browser_context_cookies(
+            browser_context,
+            urls=urls or self.cookie_urls,
+        )
         self.headers["Cookie"] = cookie_str
         self.cookie_dict = cookie_dict
 
@@ -266,13 +280,13 @@ class DouYinClient(AbstractApiClient, ProxyRefreshMixin):
             if len(result) + len(comments) > max_count:
                 comments = comments[:max_count - len(result)]
             result.extend(comments)
-            if callback:  # 如果有回调函数，就执行回调函数
+            if callback:  # If there is a callback function, execute the callback function
                 await callback(aweme_id, comments)
 
             await asyncio.sleep(crawl_interval)
             if not is_fetch_sub_comments:
                 continue
-            # 获取二级评论
+            # Get secondary reviews
             for comment in comments:
                 reply_comment_total = comment.get("reply_comment_total")
 
@@ -290,7 +304,7 @@ class DouYinClient(AbstractApiClient, ProxyRefreshMixin):
                         if not sub_comments:
                             continue
                         result.extend(sub_comments)
-                        if callback:  # 如果有回调函数，就执行回调函数
+                        if callback:  # If there is a callback function, execute the callback function
                             await callback(aweme_id, sub_comments)
                         await asyncio.sleep(crawl_interval)
         return result
@@ -312,8 +326,6 @@ class DouYinClient(AbstractApiClient, ProxyRefreshMixin):
             "max_cursor": max_cursor,
             "locate_query": "false",
             "publish_video_strategy_type": 2,
-            'verifyFp': 'verify_ma3hrt8n_q2q2HyYA_uLyO_4N6D_BLvX_E2LgoGmkA1BU',
-            'fp': 'verify_ma3hrt8n_q2q2HyYA_uLyO_4N6D_BLvX_E2LgoGmkA1BU'
         }
         return await self.get(uri, params)
 
@@ -333,7 +345,7 @@ class DouYinClient(AbstractApiClient, ProxyRefreshMixin):
         return result
 
     async def get_aweme_media(self, url: str) -> Union[bytes, None]:
-        async with httpx.AsyncClient(proxy=self.proxy) as client:
+        async with make_async_client(proxy=self.proxy) as client:
             try:
                 response = await client.request("GET", url, timeout=self.timeout, follow_redirects=True)
                 response.raise_for_status()
@@ -343,7 +355,7 @@ class DouYinClient(AbstractApiClient, ProxyRefreshMixin):
                 else:
                     return response.content
             except httpx.HTTPError as exc:  # some wrong when call httpx.request method, such as connection error, client error, server error or response status code is not 2xx
-                utils.logger.error(f"[DouYinClient.get_aweme_media] {exc.__class__.__name__} for {exc.request.url} - {exc}")  # 保留原始异常类型名称，以便开发者调试
+                utils.logger.error(f"[DouYinClient.get_aweme_media] {exc.__class__.__name__} for {exc.request.url} - {exc}")  # Keep the original exception type name for developers to debug
                 return None
 
     async def resolve_short_url(self, short_url: str) -> str:
@@ -354,12 +366,12 @@ class DouYinClient(AbstractApiClient, ProxyRefreshMixin):
         Returns:
             重定向后的完整URL
         """
-        async with httpx.AsyncClient(proxy=self.proxy, follow_redirects=False) as client:
+        async with make_async_client(proxy=self.proxy, follow_redirects=False) as client:
             try:
                 utils.logger.info(f"[DouYinClient.resolve_short_url] Resolving short URL: {short_url}")
                 response = await client.get(short_url, timeout=10)
 
-                # 短链接通常返回302重定向
+                # Short links usually return a 302 redirect
                 if response.status_code in [301, 302, 303, 307, 308]:
                     redirect_url = response.headers.get("Location", "")
                     utils.logger.info(f"[DouYinClient.resolve_short_url] Resolved to: {redirect_url}")

media_platform/douyin/core.py

@@ -54,8 +54,15 @@ class DouYinCrawler(AbstractCrawler):
 
     def __init__(self) -> None:
         self.index_url = "https://www.douyin.com"
+        self.cookie_urls = [
+            "https://douyin.com",
+            self.index_url,
+            "https://creator.douyin.com",
+            "https://douhot.douyin.com",
+            "https://live.douyin.com",
+        ]
         self.cdp_manager = None
-        self.ip_proxy_pool = None  # 代理IP池，用于代理自动刷新
+        self.ip_proxy_pool = None  # Proxy IP pool for automatic proxy refresh
 
     async def start(self) -> None:
         playwright_proxy_format, httpx_proxy_format = None, None
@@ -65,7 +72,7 @@ class DouYinCrawler(AbstractCrawler):
             playwright_proxy_format, httpx_proxy_format = utils.format_proxy_info(ip_proxy_info)
 
         async with async_playwright() as playwright:
-            # 根据配置选择启动模式
+            # Select startup mode based on configuration
             if config.ENABLE_CDP_MODE:
                 utils.logger.info("[DouYinCrawler] 使用CDP模式启动浏览器")
                 self.browser_context = await self.launch_browser_with_cdp(
@@ -100,7 +107,10 @@ class DouYinCrawler(AbstractCrawler):
                     cookie_str=config.COOKIES,
                 )
                 await login_obj.begin()
-                await self.dy_client.update_cookies(browser_context=self.browser_context)
+                await self.dy_client.update_cookies(
+                    browser_context=self.browser_context,
+                    urls=self.cookie_urls,
+                )
             crawler_type_var.set(config.CRAWLER_TYPE)
             if config.CRAWLER_TYPE == "search":
                 # Search for notes and retrieve their comment information.
@@ -151,19 +161,24 @@ class DouYinCrawler(AbstractCrawler):
                     utils.logger.error(f"[DouYinCrawler.search] search douyin keyword: {keyword} failed，账号也许被风控了。")
                     break
                 dy_search_id = posts_res.get("extra", {}).get("logid", "")
+                page_aweme_list = []
                 for post_item in posts_res.get("data"):
                     try:
                         aweme_info: Dict = (post_item.get("aweme_info") or post_item.get("aweme_mix_info", {}).get("mix_items")[0])
                     except TypeError:
                         continue
                     aweme_list.append(aweme_info.get("aweme_id", ""))
+                    page_aweme_list.append(aweme_info.get("aweme_id", ""))
                     await douyin_store.update_douyin_aweme(aweme_item=aweme_info)
                     await self.get_aweme_media(aweme_item=aweme_info)
+                
+                # Batch get note comments for the current page
+                await self.batch_get_note_comments(page_aweme_list)
+
                 # Sleep after each page navigation
                 await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)
                 utils.logger.info(f"[DouYinCrawler.search] Sleeping for {config.CRAWLER_MAX_SLEEP_SEC} seconds after page {page-1}")
             utils.logger.info(f"[DouYinCrawler.search] keyword:{keyword}, aweme_list:{aweme_list}")
-            await self.batch_get_note_comments(aweme_list)
 
     async def get_specified_awemes(self):
         """Get the information and comments of the specified post from URLs or IDs"""
@@ -173,12 +188,12 @@ class DouYinCrawler(AbstractCrawler):
             try:
                 video_info = parse_video_info_from_url(video_url)
 
-                # 处理短链接
+                # Handling short links
                 if video_info.url_type == "short":
                     utils.logger.info(f"[DouYinCrawler.get_specified_awemes] Resolving short link: {video_url}")
                     resolved_url = await self.dy_client.resolve_short_url(video_url)
                     if resolved_url:
-                        # 从解析后的URL中提取视频ID
+                        # Extract video ID from parsed URL
                         video_info = parse_video_info_from_url(resolved_url)
                         utils.logger.info(f"[DouYinCrawler.get_specified_awemes] Short link resolved to aweme ID: {video_info.aweme_id}")
                     else:
@@ -235,7 +250,7 @@ class DouYinCrawler(AbstractCrawler):
     async def get_comments(self, aweme_id: str, semaphore: asyncio.Semaphore) -> None:
         async with semaphore:
             try:
-                # 将关键词列表传递给 get_aweme_all_comments 方法
+                # Pass the list of keywords to the get_aweme_all_comments method
                 # Use fixed crawling interval
                 crawl_interval = config.CRAWLER_MAX_SLEEP_SEC
                 await self.dy_client.get_aweme_all_comments(
@@ -293,7 +308,10 @@ class DouYinCrawler(AbstractCrawler):
 
     async def create_douyin_client(self, httpx_proxy: Optional[str]) -> DouYinClient:
         """Create douyin client"""
-        cookie_str, cookie_dict = utils.convert_cookies(await self.browser_context.cookies())  # type: ignore
+        cookie_str, cookie_dict = await utils.convert_browser_context_cookies(
+            self.browser_context,
+            urls=self.cookie_urls,
+        )  # type: ignore
         douyin_client = DouYinClient(
             proxy=httpx_proxy,
             headers={
@@ -306,7 +324,7 @@ class DouYinCrawler(AbstractCrawler):
             },
             playwright_page=self.context_page,
             cookie_dict=cookie_dict,
-            proxy_ip_pool=self.ip_proxy_pool,  # 传递代理池用于自动刷新
+            proxy_ip_pool=self.ip_proxy_pool,  # Pass proxy pool for automatic refresh
         )
         return douyin_client
 
@@ -356,10 +374,10 @@ class DouYinCrawler(AbstractCrawler):
                 headless=headless,
             )
 
-            # 添加反检测脚本
+            # Add anti-detection script
             await self.cdp_manager.add_stealth_script()
 
-            # 显示浏览器信息
+            # Show browser information
             browser_info = await self.cdp_manager.get_browser_info()
             utils.logger.info(f"[DouYinCrawler] CDP浏览器信息: {browser_info}")
 
@@ -367,13 +385,13 @@ class DouYinCrawler(AbstractCrawler):
 
         except Exception as e:
             utils.logger.error(f"[DouYinCrawler] CDP模式启动失败，回退到标准模式: {e}")
-            # 回退到标准模式
+            # Fall back to standard mode
             chromium = playwright.chromium
             return await self.launch_browser(chromium, playwright_proxy, user_agent, headless)
 
     async def close(self) -> None:
         """Close browser context"""
-        # 如果使用CDP模式，需要特殊处理
+        # If you use CDP mode, special processing is required
         if self.cdp_manager:
             await self.cdp_manager.cleanup()
             self.cdp_manager = None
@@ -391,11 +409,11 @@ class DouYinCrawler(AbstractCrawler):
         if not config.ENABLE_GET_MEIDAS:
             utils.logger.info(f"[DouYinCrawler.get_aweme_media] Crawling image mode is not enabled")
             return
-        # 笔记 urls 列表，若为短视频类型则返回为空列表
+        # List of note urls. If it is a short video type, an empty list will be returned.
         note_download_url: List[str] = douyin_store._extract_note_image_list(aweme_item)
-        # 视频 url，永远存在，但为短视频类型时的文件其实是音频文件
+        # The video URL will always exist, but when it is a short video type, the file is actually an audio file.
         video_download_url: str = douyin_store._extract_video_download_url(aweme_item)
-        # TODO: 抖音并没采用音视频分离的策略，故音频可从原视频中分离，暂不提取
+        # TODO: Douyin does not adopt the audio and video separation strategy, so the audio can be separated from the original video and will not be extracted for the time being.
         if note_download_url:
             await self.get_aweme_images(aweme_item)
         else:
@@ -411,7 +429,7 @@ class DouYinCrawler(AbstractCrawler):
         if not config.ENABLE_GET_MEIDAS:
             return
         aweme_id = aweme_item.get("aweme_id")
-        # 笔记 urls 列表，若为短视频类型则返回为空列表
+        # List of note urls. If it is a short video type, an empty list will be returned.
         note_download_url: List[str] = douyin_store._extract_note_image_list(aweme_item)
 
         if not note_download_url:
@@ -439,7 +457,7 @@ class DouYinCrawler(AbstractCrawler):
             return
         aweme_id = aweme_item.get("aweme_id")
 
-        # 视频 url，永远存在，但为短视频类型时的文件其实是音频文件
+        # The video URL will always exist, but when it is a short video type, the file is actually an audio file.
         video_download_url: str = douyin_store._extract_video_download_url(aweme_item)
 
         if not video_download_url:

media_platform/douyin/field.py

@@ -23,21 +23,21 @@ from enum import Enum
 
 class SearchChannelType(Enum):
     """search channel type"""
-    GENERAL = "aweme_general"  # 综合
-    VIDEO = "aweme_video_web"  # 视频
-    USER = "aweme_user_web"  # 用户
-    LIVE = "aweme_live"  # 直播
+    GENERAL = "aweme_general"  # General
+    VIDEO = "aweme_video_web"  # Video
+    USER = "aweme_user_web"  # User
+    LIVE = "aweme_live"  # Live
 
 
 class SearchSortType(Enum):
     """search sort type"""
-    GENERAL = 0  # 综合排序
-    MOST_LIKE = 1  # 最多点赞
-    LATEST = 2  # 最新发布
+    GENERAL = 0  # Comprehensive sorting
+    MOST_LIKE = 1  # Most likes
+    LATEST = 2  # Latest published
 
 class PublishTimeType(Enum):
     """publish time type"""
-    UNLIMITED = 0  # 不限
-    ONE_DAY = 1  # 一天内
-    ONE_WEEK = 7  # 一周内
-    SIX_MONTH = 180  # 半年内
+    UNLIMITED = 0  # Unlimited
+    ONE_DAY = 1  # Within one day
+    ONE_WEEK = 7  # Within one week
+    SIX_MONTH = 180  # Within six months

media_platform/douyin/help.py

@@ -20,9 +20,9 @@
 
 # -*- coding: utf-8 -*-
 # @Author  : relakkes@gmail.com
-# @Name    : 程序员阿江-Relakkes
+# @Name: Programmer Ajiang-Relakkes
 # @Time    : 2024/6/10 02:24
-# @Desc    : 获取 a_bogus 参数, 学习交流使用，请勿用作商业用途，侵权联系作者删除
+# @Desc    : Get a_bogus parameter, for learning and communication only, do not use for commercial purposes, contact author to delete if infringement
 
 import random
 import re
@@ -38,7 +38,7 @@ douyin_sign_obj = execjs.compile(open('libs/douyin.js', encoding='utf-8-sig').re
 
 def get_web_id():
     """
-    生成随机的webid
+    Generate random webid
     Returns:
 
     """
@@ -60,13 +60,13 @@ def get_web_id():
 
 async def get_a_bogus(url: str, params: str, post_data: dict, user_agent: str, page: Page = None):
     """
-    获取 a_bogus 参数, 目前不支持post请求类型的签名
+    Get a_bogus parameter, currently does not support POST request type signature
     """
     return get_a_bogus_from_js(url, params, user_agent)
 
 def get_a_bogus_from_js(url: str, params: str, user_agent: str):
     """
-    通过js获取 a_bogus 参数
+    Get a_bogus parameter through js
     Args:
         url:
         params:
@@ -82,10 +82,10 @@ def get_a_bogus_from_js(url: str, params: str, user_agent: str):
 
 
 
-async def get_a_bogus_from_playright(params: str, post_data: dict, user_agent: str, page: Page):
+async def get_a_bogus_from_playwright(params: str, post_data: dict, user_agent: str, page: Page):
     """
-    通过playright获取 a_bogus 参数
-    playwright版本已失效
+    Get a_bogus parameter through playwright
+    playwright version is deprecated
     Returns:
 
     """
@@ -100,73 +100,73 @@ async def get_a_bogus_from_playright(params: str, post_data: dict, user_agent: s
 
 def parse_video_info_from_url(url: str) -> VideoUrlInfo:
     """
-    从抖音视频URL中解析出视频ID
-    支持以下格式:
-    1. 普通视频链接: https://www.douyin.com/video/7525082444551310602
-    2. 带modal_id参数的链接:
+    Parse video ID from Douyin video URL
+    Supports the following formats:
+    1. Normal video link: https://www.douyin.com/video/7525082444551310602
+    2. Link with modal_id parameter:
        - https://www.douyin.com/user/MS4wLjABAAAATJPY7LAlaa5X-c8uNdWkvz0jUGgpw4eeXIwu_8BhvqE?modal_id=7525082444551310602
        - https://www.douyin.com/root/search/python?modal_id=7471165520058862848
-    3. 短链接: https://v.douyin.com/iF12345ABC/ (需要client解析)
-    4. 纯ID: 7525082444551310602
+    3. Short link: https://v.douyin.com/iF12345ABC/ (requires client parsing)
+    4. Pure ID: 7525082444551310602
 
     Args:
-        url: 抖音视频链接或ID
+        url: Douyin video link or ID
     Returns:
-        VideoUrlInfo: 包含视频ID的对象
+        VideoUrlInfo: Object containing video ID
     """
-    # 如果是纯数字ID,直接返回
+    # If it's a pure numeric ID, return directly
     if url.isdigit():
         return VideoUrlInfo(aweme_id=url, url_type="normal")
 
-    # 检查是否是短链接 (v.douyin.com)
+    # Check if it's a short link (v.douyin.com)
     if "v.douyin.com" in url or url.startswith("http") and len(url) < 50 and "video" not in url:
-        return VideoUrlInfo(aweme_id="", url_type="short")  # 需要通过client解析
+        return VideoUrlInfo(aweme_id="", url_type="short")  # Requires client parsing
 
-    # 尝试从URL参数中提取modal_id
+    # Try to extract modal_id from URL parameters
     params = extract_url_params_to_dict(url)
     modal_id = params.get("modal_id")
     if modal_id:
         return VideoUrlInfo(aweme_id=modal_id, url_type="modal")
 
-    # 从标准视频URL中提取ID: /video/数字
+    # Extract ID from standard video URL: /video/number
     video_pattern = r'/video/(\d+)'
     match = re.search(video_pattern, url)
     if match:
         aweme_id = match.group(1)
         return VideoUrlInfo(aweme_id=aweme_id, url_type="normal")
 
-    raise ValueError(f"无法从URL中解析出视频ID: {url}")
+    raise ValueError(f"Unable to parse video ID from URL: {url}")
 
 
 def parse_creator_info_from_url(url: str) -> CreatorUrlInfo:
     """
-    从抖音创作者主页URL中解析出创作者ID (sec_user_id)
-    支持以下格式:
-    1. 创作者主页: https://www.douyin.com/user/MS4wLjABAAAATJPY7LAlaa5X-c8uNdWkvz0jUGgpw4eeXIwu_8BhvqE?from_tab_name=main
-    2. 纯ID: MS4wLjABAAAATJPY7LAlaa5X-c8uNdWkvz0jUGgpw4eeXIwu_8BhvqE
+    Parse creator ID (sec_user_id) from Douyin creator homepage URL
+    Supports the following formats:
+    1. Creator homepage: https://www.douyin.com/user/MS4wLjABAAAATJPY7LAlaa5X-c8uNdWkvz0jUGgpw4eeXIwu_8BhvqE?from_tab_name=main
+    2. Pure ID: MS4wLjABAAAATJPY7LAlaa5X-c8uNdWkvz0jUGgpw4eeXIwu_8BhvqE
 
     Args:
-        url: 抖音创作者主页链接或sec_user_id
+        url: Douyin creator homepage link or sec_user_id
     Returns:
-        CreatorUrlInfo: 包含创作者ID的对象
+        CreatorUrlInfo: Object containing creator ID
     """
-    # 如果是纯ID格式(通常以MS4wLjABAAAA开头),直接返回
+    # If it's a pure ID format (usually starts with MS4wLjABAAAA), return directly
     if url.startswith("MS4wLjABAAAA") or (not url.startswith("http") and "douyin.com" not in url):
         return CreatorUrlInfo(sec_user_id=url)
 
-    # 从创作者主页URL中提取sec_user_id: /user/xxx
+    # Extract sec_user_id from creator homepage URL: /user/xxx
     user_pattern = r'/user/([^/?]+)'
     match = re.search(user_pattern, url)
     if match:
         sec_user_id = match.group(1)
         return CreatorUrlInfo(sec_user_id=sec_user_id)
 
-    raise ValueError(f"无法从URL中解析出创作者ID: {url}")
+    raise ValueError(f"Unable to parse creator ID from URL: {url}")
 
 
 if __name__ == '__main__':
-    # 测试视频URL解析
-    print("=== 视频URL解析测试 ===")
+    # Test video URL parsing
+    print("=== Video URL Parsing Test ===")
     test_urls = [
         "https://www.douyin.com/video/7525082444551310602",
         "https://www.douyin.com/user/MS4wLjABAAAATJPY7LAlaa5X-c8uNdWkvz0jUGgpw4eeXIwu_8BhvqE?from_tab_name=main&modal_id=7525082444551310602",
@@ -177,13 +177,13 @@ if __name__ == '__main__':
         try:
             result = parse_video_info_from_url(url)
             print(f"✓ URL: {url[:80]}...")
-            print(f"  结果: {result}\n")
+            print(f"  Result: {result}\n")
         except Exception as e:
             print(f"✗ URL: {url}")
-            print(f"  错误: {e}\n")
+            print(f"  Error: {e}\n")
 
-    # 测试创作者URL解析
-    print("=== 创作者URL解析测试 ===")
+    # Test creator URL parsing
+    print("=== Creator URL Parsing Test ===")
     test_creator_urls = [
         "https://www.douyin.com/user/MS4wLjABAAAATJPY7LAlaa5X-c8uNdWkvz0jUGgpw4eeXIwu_8BhvqE?from_tab_name=main",
         "MS4wLjABAAAATJPY7LAlaa5X-c8uNdWkvz0jUGgpw4eeXIwu_8BhvqE",
@@ -192,7 +192,7 @@ if __name__ == '__main__':
         try:
             result = parse_creator_info_from_url(url)
             print(f"✓ URL: {url[:80]}...")
-            print(f"  结果: {result}\n")
+            print(f"  Result: {result}\n")
         except Exception as e:
             print(f"✗ URL: {url}")
-            print(f"  错误: {e}\n")
+            print(f"  Error: {e}\n")

media_platform/douyin/login.py

@@ -53,7 +53,7 @@ class DouYinLogin(AbstractLogin):
     async def begin(self):
         """
             Start login douyin website
-            滑块中间页面的验证准确率不太OK... 如果没有特俗要求，建议不开抖音登录，或者使用cookies登录
+            The verification accuracy of the slider verification is not very good... If there are no special requirements, it is recommended not to use Douyin login, or use cookie login
         """
 
         # popup login dialog
@@ -69,7 +69,7 @@ class DouYinLogin(AbstractLogin):
         else:
             raise ValueError("[DouYinLogin.begin] Invalid Login Type Currently only supported qrcode or phone or cookie ...")
 
-        # 如果页面重定向到滑动验证码页面，需要再次滑动滑块
+        # If the page redirects to the slider verification page, need to slide again
         await asyncio.sleep(6)
         current_page_title = await self.context_page.title()
         if "验证码中间页" in current_page_title:
@@ -147,10 +147,10 @@ class DouYinLogin(AbstractLogin):
         send_sms_code_btn = self.context_page.locator("xpath=//span[text() = '获取验证码']")
         await send_sms_code_btn.click()
 
-        # 检查是否有滑动验证码
+        # Check if there is slider verification
         await self.check_page_display_slider(move_step=10, slider_level="easy")
         cache_client = CacheFactory.create_cache(config.CACHE_TYPE_MEMORY)
-        max_get_sms_code_time = 60 * 2  # 最长获取验证码的时间为2分钟
+        max_get_sms_code_time = 60 * 2  # Maximum time to get verification code is 2 minutes
         while max_get_sms_code_time > 0:
             utils.logger.info(f"[DouYinLogin.login_by_mobile] get douyin sms code from redis remaining time {max_get_sms_code_time}s ...")
             await asyncio.sleep(1)
@@ -164,20 +164,20 @@ class DouYinLogin(AbstractLogin):
             await sms_code_input_ele.fill(value=sms_code_value.decode())
             await asyncio.sleep(0.5)
             submit_btn_ele = self.context_page.locator("xpath=//button[@class='web-login-button']")
-            await submit_btn_ele.click()  # 点击登录
-            # todo ... 应该还需要检查验证码的正确性有可能输入的验证码不正确
+            await submit_btn_ele.click()  # Click login
+            # todo ... should also check the correctness of the verification code, it may be incorrect
             break
 
     async def check_page_display_slider(self, move_step: int = 10, slider_level: str = "easy"):
         """
-        检查页面是否出现滑动验证码
+        Check if slider verification appears on the page
         :return:
         """
-        # 等待滑动验证码的出现
+        # Wait for slider verification to appear
         back_selector = "#captcha-verify-image"
         try:
             await self.context_page.wait_for_selector(selector=back_selector, state="visible", timeout=30 * 1000)
-        except PlaywrightTimeoutError:  # 没有滑动验证码，直接返回
+        except PlaywrightTimeoutError:  # No slider verification, return directly
             return
 
         gap_selector = 'xpath=//*[@id="captcha_container"]/div/div[2]/img[2]'
@@ -191,16 +191,16 @@ class DouYinLogin(AbstractLogin):
                 await self.move_slider(back_selector, gap_selector, move_step, slider_level)
                 await asyncio.sleep(1)
 
-                # 如果滑块滑动慢了，或者验证失败了，会提示操作过慢，这里点一下刷新按钮
+                # If the slider is too slow or verification failed, it will prompt "The operation is too slow", click the refresh button here
                 page_content = await self.context_page.content()
                 if "操作过慢" in page_content or "提示重新操作" in page_content:
                     utils.logger.info("[DouYinLogin.check_page_display_slider] slider verify failed, retry ...")
                     await self.context_page.click(selector="//a[contains(@class, 'secsdk_captcha_refresh')]")
                     continue
 
-                # 滑动成功后，等待滑块消失
+                # After successful sliding, wait for the slider to disappear
                 await self.context_page.wait_for_selector(selector=back_selector, state="hidden", timeout=1000)
-                # 如果滑块消失了，说明验证成功了，跳出循环，如果没有消失，说明验证失败了，上面这一行代码会抛出异常被捕获后继续循环滑动验证码
+                # If the slider disappears, it means the verification is successful, break the loop. If not, it means the verification failed, the above line will throw an exception and be caught to continue the loop
                 utils.logger.info("[DouYinLogin.check_page_display_slider] slider verify success ...")
                 slider_verify_success = True
             except Exception as e:
@@ -213,10 +213,10 @@ class DouYinLogin(AbstractLogin):
     async def move_slider(self, back_selector: str, gap_selector: str, move_step: int = 10, slider_level="easy"):
         """
         Move the slider to the right to complete the verification
-        :param back_selector: 滑动验证码背景图片的选择器
-        :param gap_selector:  滑动验证码的滑块选择器
-        :param move_step: 是控制单次移动速度的比例是1/10 默认是1 相当于 传入的这个距离不管多远0.1秒钟移动完 越大越慢
-        :param slider_level: 滑块难度 easy hard,分别对应手机验证码的滑块和验证码中间的滑块
+        :param back_selector: Selector for the slider verification background image
+        :param gap_selector:  Selector for the slider verification slider
+        :param move_step: Controls the ratio of single movement speed, default is 1, meaning the distance moves in 0.1 seconds no matter how far, larger value means slower
+        :param slider_level: Slider difficulty easy hard, corresponding to the slider for mobile verification code and the slider in the middle of verification code
         :return:
         """
 
@@ -234,31 +234,31 @@ class DouYinLogin(AbstractLogin):
         )
         gap_src = str(await gap_elements.get_property("src")) # type: ignore
 
-        # 识别滑块位置
+        # Identify slider position
         slide_app = utils.Slide(gap=gap_src, bg=slide_back)
         distance = slide_app.discern()
 
-        # 获取移动轨迹
+        # Get movement trajectory
         tracks = utils.get_tracks(distance, slider_level)
         new_1 = tracks[-1] - (sum(tracks) - distance)
         tracks.pop()
         tracks.append(new_1)
 
-        # 根据轨迹拖拽滑块到指定位置
+        # Drag slider to specified position according to trajectory
         element = await self.context_page.query_selector(gap_selector)
         bounding_box = await element.bounding_box() # type: ignore
 
         await self.context_page.mouse.move(bounding_box["x"] + bounding_box["width"] / 2, # type: ignore
                                            bounding_box["y"] + bounding_box["height"] / 2) # type: ignore
-        # 这里获取到x坐标中心点位置
+        # Get x coordinate center position
         x = bounding_box["x"] + bounding_box["width"] / 2 # type: ignore
-        # 模拟滑动操作
+        # Simulate sliding operation
         await element.hover() # type: ignore
         await self.context_page.mouse.down()
 
         for track in tracks:
-            # 循环鼠标按照轨迹移动
-            # steps 是控制单次移动速度的比例是1/10 默认是1 相当于 传入的这个距离不管多远0.1秒钟移动完 越大越慢
+            # Loop mouse movement according to trajectory
+            # steps controls the ratio of single movement speed, default is 1, meaning the distance moves in 0.1 seconds no matter how far, larger value means slower
             await self.context_page.mouse.move(x + track, 0, steps=move_step)
             x += track
         await self.context_page.mouse.up()

media_platform/kuaishou/client.py

@@ -31,6 +31,7 @@ import config
 from base.base_crawler import AbstractApiClient
 from proxy.proxy_mixin import ProxyRefreshMixin
 from tools import utils
+from tools.httpx_util import make_async_client
 
 if TYPE_CHECKING:
     from proxy.proxy_ip_pool import ProxyIpPool
@@ -54,17 +55,19 @@ class KuaiShouClient(AbstractApiClient, ProxyRefreshMixin):
         self.timeout = timeout
         self.headers = headers
         self._host = "https://www.kuaishou.com/graphql"
+        self._rest_host = "https://www.kuaishou.com"
+        self.cookie_urls = [self._rest_host]
         self.playwright_page = playwright_page
         self.cookie_dict = cookie_dict
         self.graphql = KuaiShouGraphQL()
-        # 初始化代理池（来自 ProxyRefreshMixin）
+        # Initialize proxy pool (from ProxyRefreshMixin)
         self.init_proxy_pool(proxy_ip_pool)
 
     async def request(self, method, url, **kwargs) -> Any:
-        # 每次请求前检测代理是否过期
+        # Check if proxy is expired before each request
         await self._refresh_proxy_if_expired()
 
-        async with httpx.AsyncClient(proxy=self.proxy) as client:
+        async with make_async_client(proxy=self.proxy) as client:
             response = await client.request(method, url, timeout=self.timeout, **kwargs)
         data: Dict = response.json()
         if data.get("errors"):
@@ -86,6 +89,29 @@ class KuaiShouClient(AbstractApiClient, ProxyRefreshMixin):
             method="POST", url=f"{self._host}{uri}", data=json_str, headers=self.headers
         )
 
+    async def request_rest_v2(self, uri: str, data: dict) -> Dict:
+        """
+        Make REST API V2 request (for comment endpoints)
+        :param uri: API endpoint path
+        :param data: request body
+        :return: response data
+        """
+        await self._refresh_proxy_if_expired()
+
+        json_str = json.dumps(data, separators=(",", ":"), ensure_ascii=False)
+        async with make_async_client(proxy=self.proxy) as client:
+            response = await client.request(
+                method="POST",
+                url=f"{self._rest_host}{uri}",
+                data=json_str,
+                timeout=self.timeout,
+                headers=self.headers,
+            )
+        result: Dict = response.json()
+        if result.get("result") != 1:
+            raise DataFetchError(f"REST API V2 error: {result}")
+        return result
+
     async def pong(self) -> bool:
         """get a note to check if login state is ok"""
         utils.logger.info("[KuaiShouClient.pong] Begin pong kuaishou...")
@@ -108,8 +134,11 @@ class KuaiShouClient(AbstractApiClient, ProxyRefreshMixin):
             ping_flag = False
         return ping_flag
 
-    async def update_cookies(self, browser_context: BrowserContext):
-        cookie_str, cookie_dict = utils.convert_cookies(await browser_context.cookies())
+    async def update_cookies(self, browser_context: BrowserContext, urls: Optional[list[str]] = None):
+        cookie_str, cookie_dict = await utils.convert_browser_context_cookies(
+            browser_context,
+            urls=urls or self.cookie_urls,
+        )
         self.headers["Cookie"] = cookie_str
         self.cookie_dict = cookie_dict
 
@@ -149,36 +178,32 @@ class KuaiShouClient(AbstractApiClient, ProxyRefreshMixin):
         return await self.post("", post_data)
 
     async def get_video_comments(self, photo_id: str, pcursor: str = "") -> Dict:
-        """get video comments
-        :param photo_id: photo id you want to fetch
-        :param pcursor: last you get pcursor, defaults to ""
-        :return:
+        """Get video first-level comments using REST API V2
+        :param photo_id: video id you want to fetch
+        :param pcursor: pagination cursor, defaults to ""
+        :return: dict with rootCommentsV2, pcursorV2, commentCountV2
         """
         post_data = {
-            "operationName": "commentListQuery",
-            "variables": {"photoId": photo_id, "pcursor": pcursor},
-            "query": self.graphql.get("comment_list"),
+            "photoId": photo_id,
+            "pcursor": pcursor,
         }
-        return await self.post("", post_data)
+        return await self.request_rest_v2("/rest/v/photo/comment/list", post_data)
 
     async def get_video_sub_comments(
-        self, photo_id: str, rootCommentId: str, pcursor: str = ""
+        self, photo_id: str, root_comment_id: int, pcursor: str = ""
     ) -> Dict:
-        """get video sub comments
-        :param photo_id: photo id you want to fetch
-        :param pcursor: last you get pcursor, defaults to ""
-        :return:
+        """Get video second-level comments using REST API V2
+        :param photo_id: video id you want to fetch
+        :param root_comment_id: parent comment id (must be int type)
+        :param pcursor: pagination cursor, defaults to ""
+        :return: dict with subCommentsV2, pcursorV2
         """
         post_data = {
-            "operationName": "visionSubCommentList",
-            "variables": {
-                "photoId": photo_id,
-                "pcursor": pcursor,
-                "rootCommentId": rootCommentId,
-            },
-            "query": self.graphql.get("vision_sub_comment_list"),
+            "photoId": photo_id,
+            "pcursor": pcursor,
+            "rootCommentId": root_comment_id,  # Must be int type for V2 API
         }
-        return await self.post("", post_data)
+        return await self.request_rest_v2("/rest/v/photo/comment/sublist", post_data)
 
     async def get_creator_profile(self, userId: str) -> Dict:
         post_data = {
@@ -204,12 +229,12 @@ class KuaiShouClient(AbstractApiClient, ProxyRefreshMixin):
         max_count: int = 10,
     ):
         """
-        get video all comments include sub comments
-        :param photo_id:
-        :param crawl_interval:
-        :param callback:
-        :param max_count:
-        :return:
+        Get video all comments including sub comments (V2 REST API)
+        :param photo_id: video id
+        :param crawl_interval: delay between requests (seconds)
+        :param callback: callback function for processing comments
+        :param max_count: max number of comments to fetch
+        :return: list of all comments
         """
 
         result = []
@@ -217,12 +242,12 @@ class KuaiShouClient(AbstractApiClient, ProxyRefreshMixin):
 
         while pcursor != "no_more" and len(result) < max_count:
             comments_res = await self.get_video_comments(photo_id, pcursor)
-            vision_commen_list = comments_res.get("visionCommentList", {})
-            pcursor = vision_commen_list.get("pcursor", "")
-            comments = vision_commen_list.get("rootComments", [])
+            # V2 API returns data at top level, not nested in visionCommentList
+            pcursor = comments_res.get("pcursorV2", "no_more")
+            comments = comments_res.get("rootCommentsV2", [])
             if len(result) + len(comments) > max_count:
                 comments = comments[: max_count - len(result)]
-            if callback:  # 如果有回调函数，就执行回调函数
+            if callback:  # If there is a callback function, execute the callback function
                 await callback(photo_id, comments)
             result.extend(comments)
             await asyncio.sleep(crawl_interval)
@@ -240,14 +265,14 @@ class KuaiShouClient(AbstractApiClient, ProxyRefreshMixin):
         callback: Optional[Callable] = None,
     ) -> List[Dict]:
         """
-        获取指定一级评论下的所有二级评论, 该方法会一直查找一级评论下的所有二级评论信息
+        Get all second-level comments under specified first-level comments (V2 REST API)
         Args:
-            comments: 评论列表
-            photo_id: 视频id
-            crawl_interval: 爬取一次评论的延迟单位（秒）
-            callback: 一次评论爬取结束后
+            comments: Comment list
+            photo_id: Video ID
+            crawl_interval: Delay unit for crawling comments once (seconds)
+            callback: Callback after one comment crawl ends
         Returns:
-
+            List of sub comments
         """
         if not config.ENABLE_GET_SUB_COMMENTS:
             utils.logger.info(
@@ -257,35 +282,36 @@ class KuaiShouClient(AbstractApiClient, ProxyRefreshMixin):
 
         result = []
         for comment in comments:
-            sub_comments = comment.get("subComments")
-            if sub_comments and callback:
-                await callback(photo_id, sub_comments)
-
-            sub_comment_pcursor = comment.get("subCommentsPcursor")
-            if sub_comment_pcursor == "no_more":
+            # V2 API uses hasSubComments (boolean) instead of subCommentsPcursor (string)
+            has_sub_comments = comment.get("hasSubComments", False)
+            if not has_sub_comments:
+                continue
+
+            # V2 API uses comment_id (int) instead of commentId (string)
+            root_comment_id = comment.get("comment_id")
+            if not root_comment_id:
                 continue
 
-            root_comment_id = comment.get("commentId")
             sub_comment_pcursor = ""
 
             while sub_comment_pcursor != "no_more":
                 comments_res = await self.get_video_sub_comments(
                     photo_id, root_comment_id, sub_comment_pcursor
                 )
-                vision_sub_comment_list = comments_res.get("visionSubCommentList", {})
-                sub_comment_pcursor = vision_sub_comment_list.get("pcursor", "no_more")
+                # V2 API returns data at top level
+                sub_comment_pcursor = comments_res.get("pcursorV2", "no_more")
+                sub_comments = comments_res.get("subCommentsV2", [])
 
-                comments = vision_sub_comment_list.get("subComments", {})
-                if callback:
-                    await callback(photo_id, comments)
+                if callback and sub_comments:
+                    await callback(photo_id, sub_comments)
                 await asyncio.sleep(crawl_interval)
-                result.extend(comments)
+                result.extend(sub_comments)
         return result
 
     async def get_creator_info(self, user_id: str) -> Dict:
         """
         eg: https://www.kuaishou.com/profile/3x4jtnbfter525a
-        快手用户主页
+        Kuaishou user homepage
         """
 
         visionProfile = await self.get_creator_profile(user_id)
@@ -298,11 +324,11 @@ class KuaiShouClient(AbstractApiClient, ProxyRefreshMixin):
         callback: Optional[Callable] = None,
     ) -> List[Dict]:
         """
-        获取指定用户下的所有发过的帖子，该方法会一直查找一个用户下的所有帖子信息
+        Get all posts published by the specified user, this method will continue to find all post information under a user
         Args:
-            user_id: 用户ID
-            crawl_interval: 爬取一次的延迟单位（秒）
-            callback: 一次分页爬取结束后的更新回调函数
+            user_id: User ID
+            crawl_interval: Delay unit for crawling once (seconds)
+            callback: Update callback function after one page crawl ends
         Returns:
 
         """

media_platform/kuaishou/core.py

@@ -56,9 +56,10 @@ class KuaishouCrawler(AbstractCrawler):
 
     def __init__(self):
         self.index_url = "https://www.kuaishou.com"
+        self.cookie_urls = [self.index_url]
         self.user_agent = utils.get_user_agent()
         self.cdp_manager = None
-        self.ip_proxy_pool = None  # 代理IP池，用于代理自动刷新
+        self.ip_proxy_pool = None  # Proxy IP pool, used for automatic proxy refresh
 
     async def start(self):
         playwright_proxy_format, httpx_proxy_format = None, None
@@ -72,9 +73,9 @@ class KuaishouCrawler(AbstractCrawler):
             )
 
         async with async_playwright() as playwright:
-            # 根据配置选择启动模式
+            # Select startup mode based on configuration
             if config.ENABLE_CDP_MODE:
-                utils.logger.info("[KuaishouCrawler] 使用CDP模式启动浏览器")
+                utils.logger.info("[KuaishouCrawler] Launching browser using CDP mode")
                 self.browser_context = await self.launch_browser_with_cdp(
                     playwright,
                     playwright_proxy_format,
@@ -82,7 +83,7 @@ class KuaishouCrawler(AbstractCrawler):
                     headless=config.CDP_HEADLESS,
                 )
             else:
-                utils.logger.info("[KuaishouCrawler] 使用标准模式启动浏览器")
+                utils.logger.info("[KuaishouCrawler] Launching browser using standard mode")
                 # Launch a browser context.
                 chromium = playwright.chromium
                 self.browser_context = await self.launch_browser(
@@ -107,7 +108,8 @@ class KuaishouCrawler(AbstractCrawler):
                 )
                 await login_obj.begin()
                 await self.ks_client.update_cookies(
-                    browser_context=self.browser_context
+                    browser_context=self.browser_context,
+                    urls=self.cookie_urls,
                 )
 
             crawler_type_var.set(config.CRAWLER_TYPE)
@@ -158,14 +160,14 @@ class KuaishouCrawler(AbstractCrawler):
                     utils.logger.error(
                         f"[KuaishouCrawler.search] search info by keyword:{keyword} not found data"
                     )
-                    continue
+                    break
 
                 vision_search_photo: Dict = videos_res.get("visionSearchPhoto")
                 if vision_search_photo.get("result") != 1:
                     utils.logger.error(
                         f"[KuaishouCrawler.search] search info by keyword:{keyword} not found data "
                     )
-                    continue
+                    break
                 search_session_id = vision_search_photo.get("searchSessionId", "")
                 for video_detail in vision_search_photo.get("feeds"):
                     video_id_list.append(video_detail.get("photo", {}).get("id"))
@@ -296,7 +298,8 @@ class KuaishouCrawler(AbstractCrawler):
                 time.sleep(20)
                 await self.context_page.goto(f"{self.index_url}?isHome=1")
                 await self.ks_client.update_cookies(
-                    browser_context=self.browser_context
+                    browser_context=self.browser_context,
+                    urls=self.cookie_urls,
                 )
 
     async def create_ks_client(self, httpx_proxy: Optional[str]) -> KuaiShouClient:
@@ -304,8 +307,9 @@ class KuaishouCrawler(AbstractCrawler):
         utils.logger.info(
             "[KuaishouCrawler.create_ks_client] Begin create kuaishou API client ..."
         )
-        cookie_str, cookie_dict = utils.convert_cookies(
-            await self.browser_context.cookies()
+        cookie_str, cookie_dict = await utils.convert_browser_context_cookies(
+            self.browser_context,
+            urls=self.cookie_urls,
         )
         ks_client_obj = KuaiShouClient(
             proxy=httpx_proxy,
@@ -318,7 +322,7 @@ class KuaishouCrawler(AbstractCrawler):
             },
             playwright_page=self.context_page,
             cookie_dict=cookie_dict,
-            proxy_ip_pool=self.ip_proxy_pool,  # 传递代理池用于自动刷新
+            proxy_ip_pool=self.ip_proxy_pool,  # Pass proxy pool for automatic refresh
         )
         return ks_client_obj
 
@@ -344,7 +348,7 @@ class KuaishouCrawler(AbstractCrawler):
                 proxy=playwright_proxy,  # type: ignore
                 viewport={"width": 1920, "height": 1080},
                 user_agent=user_agent,
-                channel="chrome",  # 使用系统的Chrome稳定版
+                channel="chrome",  # Use system's stable Chrome version
             )
             return browser_context
         else:
@@ -362,7 +366,7 @@ class KuaishouCrawler(AbstractCrawler):
         headless: bool = True,
     ) -> BrowserContext:
         """
-        使用CDP模式启动浏览器
+        Launch browser using CDP mode
         """
         try:
             self.cdp_manager = CDPBrowserManager()
@@ -373,17 +377,17 @@ class KuaishouCrawler(AbstractCrawler):
                 headless=headless,
             )
 
-            # 显示浏览器信息
+            # Display browser information
             browser_info = await self.cdp_manager.get_browser_info()
-            utils.logger.info(f"[KuaishouCrawler] CDP浏览器信息: {browser_info}")
+            utils.logger.info(f"[KuaishouCrawler] CDP browser info: {browser_info}")
 
             return browser_context
 
         except Exception as e:
             utils.logger.error(
-                f"[KuaishouCrawler] CDP模式启动失败，回退到标准模式: {e}"
+                f"[KuaishouCrawler] CDP mode launch failed, fallback to standard mode: {e}"
             )
-            # 回退到标准模式
+            # Fallback to standard mode
             chromium = playwright.chromium
             return await self.launch_browser(
                 chromium, playwright_proxy, user_agent, headless
@@ -438,7 +442,7 @@ class KuaishouCrawler(AbstractCrawler):
 
     async def close(self):
         """Close browser context"""
-        # 如果使用CDP模式，需要特殊处理
+        # If using CDP mode, need special handling
         if self.cdp_manager:
             await self.cdp_manager.cleanup()
             self.cdp_manager = None

media_platform/kuaishou/graphql.py

@@ -18,8 +18,8 @@
 # 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
 
 
-# 快手的数据传输是基于GraphQL实现的
-# 这个类负责获取一些GraphQL的schema
+# Kuaishou's data transmission is based on GraphQL
+# This class is responsible for obtaining some GraphQL schemas
 from typing import Dict
 
 

media_platform/kuaishou/help.py

@@ -26,59 +26,59 @@ from model.m_kuaishou import VideoUrlInfo, CreatorUrlInfo
 
 def parse_video_info_from_url(url: str) -> VideoUrlInfo:
     """
-    从快手视频URL中解析出视频ID
-    支持以下格式:
-    1. 完整视频URL: "https://www.kuaishou.com/short-video/3x3zxz4mjrsc8ke?authorId=3x84qugg4ch9zhs&streamSource=search"
-    2. 纯视频ID: "3x3zxz4mjrsc8ke"
+    Parse video ID from Kuaishou video URL
+    Supports the following formats:
+    1. Full video URL: "https://www.kuaishou.com/short-video/3x3zxz4mjrsc8ke?authorId=3x84qugg4ch9zhs&streamSource=search"
+    2. Pure video ID: "3x3zxz4mjrsc8ke"
 
     Args:
-        url: 快手视频链接或视频ID
+        url: Kuaishou video link or video ID
     Returns:
-        VideoUrlInfo: 包含视频ID的对象
+        VideoUrlInfo: Object containing video ID
     """
-    # 如果不包含http且不包含kuaishou.com，认为是纯ID
+    # If it doesn't contain http and doesn't contain kuaishou.com, consider it as pure ID
     if not url.startswith("http") and "kuaishou.com" not in url:
         return VideoUrlInfo(video_id=url, url_type="normal")
 
-    # 从标准视频URL中提取ID: /short-video/视频ID
+    # Extract ID from standard video URL: /short-video/video_ID
     video_pattern = r'/short-video/([a-zA-Z0-9_-]+)'
     match = re.search(video_pattern, url)
     if match:
         video_id = match.group(1)
         return VideoUrlInfo(video_id=video_id, url_type="normal")
 
-    raise ValueError(f"无法从URL中解析出视频ID: {url}")
+    raise ValueError(f"Unable to parse video ID from URL: {url}")
 
 
 def parse_creator_info_from_url(url: str) -> CreatorUrlInfo:
     """
-    从快手创作者主页URL中解析出创作者ID
-    支持以下格式:
-    1. 创作者主页: "https://www.kuaishou.com/profile/3x84qugg4ch9zhs"
-    2. 纯ID: "3x4sm73aye7jq7i"
+    Parse creator ID from Kuaishou creator homepage URL
+    Supports the following formats:
+    1. Creator homepage: "https://www.kuaishou.com/profile/3x84qugg4ch9zhs"
+    2. Pure ID: "3x4sm73aye7jq7i"
 
     Args:
-        url: 快手创作者主页链接或user_id
+        url: Kuaishou creator homepage link or user_id
     Returns:
-        CreatorUrlInfo: 包含创作者ID的对象
+        CreatorUrlInfo: Object containing creator ID
     """
-    # 如果不包含http且不包含kuaishou.com，认为是纯ID
+    # If it doesn't contain http and doesn't contain kuaishou.com, consider it as pure ID
     if not url.startswith("http") and "kuaishou.com" not in url:
         return CreatorUrlInfo(user_id=url)
 
-    # 从创作者主页URL中提取user_id: /profile/xxx
+    # Extract user_id from creator homepage URL: /profile/xxx
     user_pattern = r'/profile/([a-zA-Z0-9_-]+)'
     match = re.search(user_pattern, url)
     if match:
         user_id = match.group(1)
         return CreatorUrlInfo(user_id=user_id)
 
-    raise ValueError(f"无法从URL中解析出创作者ID: {url}")
+    raise ValueError(f"Unable to parse creator ID from URL: {url}")
 
 
 if __name__ == '__main__':
-    # 测试视频URL解析
-    print("=== 视频URL解析测试 ===")
+    # Test video URL parsing
+    print("=== Video URL Parsing Test ===")
     test_video_urls = [
         "https://www.kuaishou.com/short-video/3x3zxz4mjrsc8ke?authorId=3x84qugg4ch9zhs&streamSource=search&area=searchxxnull&searchKey=python",
         "3xf8enb8dbj6uig",
@@ -87,13 +87,13 @@ if __name__ == '__main__':
         try:
             result = parse_video_info_from_url(url)
             print(f"✓ URL: {url[:80]}...")
-            print(f"  结果: {result}\n")
+            print(f"  Result: {result}\n")
         except Exception as e:
             print(f"✗ URL: {url}")
-            print(f"  错误: {e}\n")
+            print(f"  Error: {e}\n")
 
-    # 测试创作者URL解析
-    print("=== 创作者URL解析测试 ===")
+    # Test creator URL parsing
+    print("=== Creator URL Parsing Test ===")
     test_creator_urls = [
         "https://www.kuaishou.com/profile/3x84qugg4ch9zhs",
         "3x4sm73aye7jq7i",
@@ -102,7 +102,7 @@ if __name__ == '__main__':
         try:
             result = parse_creator_info_from_url(url)
             print(f"✓ URL: {url[:80]}...")
-            print(f"  结果: {result}\n")
+            print(f"  Result: {result}\n")
         except Exception as e:
             print(f"✗ URL: {url}")
-            print(f"  错误: {e}\n")
+            print(f"  Error: {e}\n")

media_platform/tieba/client.py

@@ -18,9 +18,10 @@
 # 使用本代码即表示您同意遵守上述原则和LICENSE中的所有条款。
 
 import asyncio
+import hashlib
 import json
 from typing import Any, Callable, Dict, List, Optional, Union
-from urllib.parse import urlencode, quote
+from urllib.parse import urlencode, quote, parse_qs, unquote, urlparse
 
 import requests
 from playwright.async_api import BrowserContext, Page
@@ -35,6 +36,8 @@ from tools import utils
 from .field import SearchNoteType, SearchSortType
 from .help import TieBaExtractor
 
+PC_SIGN_SECRET = "36770b1f34c9bbf2e7d1a99d2b82fa9e"
+
 
 class BaiduTieBaClient(AbstractApiClient):
 
@@ -48,29 +51,152 @@ class BaiduTieBaClient(AbstractApiClient):
     ):
         self.ip_pool: Optional[ProxyIpPool] = ip_pool
         self.timeout = timeout
-        # 使用传入的headers(包含真实浏览器UA)或默认headers
+        # Use provided headers (including real browser UA) or default headers
         self.headers = headers or {
             "User-Agent": utils.get_user_agent(),
             "Cookie": "",
         }
         self._host = "https://tieba.baidu.com"
+        self.cookie_urls = [self._host]
         self._page_extractor = TieBaExtractor()
         self.default_ip_proxy = default_ip_proxy
-        self.playwright_page = playwright_page  # Playwright页面对象
+        self.playwright_page = playwright_page  # Playwright page object
+        self._pc_tbs = ""
+
+    @staticmethod
+    def _sign_pc_params(params: Dict[str, Any]) -> str:
+        sign_text = ""
+        for key in sorted(params):
+            if key in {"sign", "sig"} or params[key] is None:
+                continue
+            sign_text += f"{key}={params[key]}"
+        sign_text += PC_SIGN_SECRET
+        return hashlib.md5(sign_text.encode("utf-8")).hexdigest()
+
+    async def _ensure_tieba_origin(self) -> None:
+        if not self.playwright_page:
+            raise Exception("playwright_page is required for tieba PC API requests")
+        if not self.playwright_page.url.startswith(self._host):
+            await self.playwright_page.goto(self._host, wait_until="domcontentloaded")
+
+    async def _fetch_json_by_browser(
+        self,
+        uri: str,
+        method: str = "GET",
+        params: Optional[Dict[str, Any]] = None,
+        data: Optional[Dict[str, Any]] = None,
+        use_sign: bool = False,
+    ) -> Dict:
+        """
+        Fetch current Tieba PC JSON APIs from the browser context.
+        These APIs rely on logged-in browser cookies and Baidu's PC signing
+        convention, while Python requests can be blocked by local proxy/TLS.
+        """
+        await self._ensure_tieba_origin()
+        params = {k: v for k, v in (params or {}).items() if v is not None}
+        data = {k: v for k, v in (data or {}).items() if v is not None}
+        if use_sign:
+            sign_source = data if method.upper() == "POST" else params
+            sign_source.setdefault("subapp_type", "pc")
+            sign_source.setdefault("_client_type", "20")
+            sign_source["sign"] = self._sign_pc_params(sign_source)
+
+        url = f"{self._host}{uri}"
+        if params:
+            url = f"{url}?{urlencode(params)}"
+        body = urlencode(data) if data else ""
+        response = await self.playwright_page.evaluate(
+            """async ({ url, method, body }) => {
+                const headers = { "Accept": "application/json, text/plain, */*" };
+                const options = { method, credentials: "include", headers };
+                if (method === "POST") {
+                    headers["Content-Type"] = "application/x-www-form-urlencoded;charset=UTF-8";
+                    options.body = body;
+                }
+                const resp = await fetch(url, options);
+                const text = await resp.text();
+                return { status: resp.status, text };
+            }""",
+            {"url": url, "method": method.upper(), "body": body},
+        )
+        if response["status"] != 200:
+            raise Exception(f"Tieba PC API failed, status={response['status']}, url={url}")
+        try:
+            json_data = json.loads(response["text"])
+        except json.JSONDecodeError as exc:
+            raise Exception(f"Tieba PC API returned non-JSON, url={url}, body={response['text'][:500]}") from exc
+        error_code = json_data.get("error_code", json_data.get("no", 0))
+        if str(error_code) not in {"0", "None"}:
+            raise Exception(f"Tieba PC API error, url={url}, response={json_data}")
+        return json_data
+
+    async def _get_pc_tbs(self) -> str:
+        if self._pc_tbs:
+            return self._pc_tbs
+        sync_data = await self._fetch_json_by_browser(
+            "/c/s/pc/sync",
+            params={"subapp_type": "pc", "_client_type": "20"},
+            use_sign=True,
+        )
+        self._pc_tbs = (
+            sync_data.get("data", {})
+            .get("anti", {})
+            .get("tbs", "")
+        )
+        if not self._pc_tbs:
+            raise Exception(f"Can not get Tieba tbs from pc sync API: {sync_data}")
+        return self._pc_tbs
+
+    async def _get_pc_page_data(self, note_id: str, page: int = 1) -> Dict:
+        tbs = await self._get_pc_tbs()
+        return await self._fetch_json_by_browser(
+            "/c/f/pb/page_pc",
+            method="POST",
+            data={
+                "pn": page,
+                "lz": 0,
+                "r": 2,
+                "mark_type": 0,
+                "back": 0,
+                "fr": "",
+                "kz": note_id,
+                "session_request_times": 1,
+                "tbs": tbs,
+                "subapp_type": "pc",
+                "_client_type": "20",
+            },
+            use_sign=True,
+        )
+
+    @staticmethod
+    def _extract_creator_portrait(creator_url: str) -> str:
+        creator_url = (creator_url or "").strip()
+        if not creator_url:
+            return ""
+        if not creator_url.startswith(("http://", "https://")):
+            return creator_url.split("?")[0]
+        parsed = urlparse(creator_url)
+        query = parse_qs(parsed.query)
+        portrait = (
+            query.get("id", [""])[0]
+            or query.get("portrait", [""])[0]
+            or query.get("un", [""])[0]
+        )
+        return unquote(portrait).split("?")[0]
 
     def _sync_request(self, method, url, proxy=None, **kwargs):
         """
-        同步的requests请求方法
+        Synchronous requests method
         Args:
-            method: 请求方法
-            url: 请求的URL
-            proxy: 代理IP
-            **kwargs: 其他请求参数
+            method: Request method
+            url: Request URL
+            proxy: Proxy IP
+            **kwargs: Other request parameters
 
         Returns:
-            response对象
+            Response object
         """
-        # 构造代理字典
+        # Construct proxy dictionary
         proxies = None
         if proxy:
             proxies = {
@@ -78,7 +204,7 @@ class BaiduTieBaClient(AbstractApiClient):
                 "https": proxy,
             }
 
-        # 发送请求
+        # Send request
         response = requests.request(
             method=method,
             url=url,
@@ -91,7 +217,7 @@ class BaiduTieBaClient(AbstractApiClient):
 
     async def _refresh_proxy_if_expired(self) -> None:
         """
-        检测代理是否过期，如果过期则自动刷新
+        Check if proxy is expired and automatically refresh if necessary
         """
         if self.ip_pool is None:
             return
@@ -101,7 +227,7 @@ class BaiduTieBaClient(AbstractApiClient):
                 "[BaiduTieBaClient._refresh_proxy_if_expired] Proxy expired, refreshing..."
             )
             new_proxy = await self.ip_pool.get_or_refresh_proxy()
-            # 更新代理URL
+            # Update proxy URL
             _, self.default_ip_proxy = utils.format_proxy_info(new_proxy)
             utils.logger.info(
                 f"[BaiduTieBaClient._refresh_proxy_if_expired] New proxy: {new_proxy.ip}:{new_proxy.port}"
@@ -110,23 +236,23 @@ class BaiduTieBaClient(AbstractApiClient):
     @retry(stop=stop_after_attempt(3), wait=wait_fixed(1))
     async def request(self, method, url, return_ori_content=False, proxy=None, **kwargs) -> Union[str, Any]:
         """
-        封装requests的公共请求方法，对请求响应做一些处理
+        Common request method wrapper for requests, handles request responses
         Args:
-            method: 请求方法
-            url: 请求的URL
-            return_ori_content: 是否返回原始内容
-            proxy: 代理IP
-            **kwargs: 其他请求参数，例如请求头、请求体等
+            method: Request method
+            url: Request URL
+            return_ori_content: Whether to return original content
+            proxy: Proxy IP
+            **kwargs: Other request parameters, such as headers, request body, etc.
 
         Returns:
 
         """
-        # 每次请求前检测代理是否过期
+        # Check if proxy is expired before each request
         await self._refresh_proxy_if_expired()
 
         actual_proxy = proxy if proxy else self.default_ip_proxy
 
-        # 在线程池中执行同步的requests请求
+        # Execute synchronous requests in thread pool
         response = await asyncio.to_thread(
             self._sync_request,
             method,
@@ -151,11 +277,11 @@ class BaiduTieBaClient(AbstractApiClient):
 
     async def get(self, uri: str, params=None, return_ori_content=False, **kwargs) -> Any:
         """
-        GET请求，对请求头签名
+        GET request with header signing
         Args:
-            uri: 请求路由
-            params: 请求参数
-            return_ori_content: 是否返回原始内容
+            uri: Request route
+            params: Request parameters
+            return_ori_content: Whether to return original content
 
         Returns:
 
@@ -175,15 +301,15 @@ class BaiduTieBaClient(AbstractApiClient):
                 self.default_ip_proxy = proxy
                 return res
 
-            utils.logger.error(f"[BaiduTieBaClient.get] 达到了最大重试次数，IP已经被Block，请尝试更换新的IP代理: {e}")
-            raise Exception(f"[BaiduTieBaClient.get] 达到了最大重试次数，IP已经被Block，请尝试更换新的IP代理: {e}")
+            utils.logger.error(f"[BaiduTieBaClient.get] Reached maximum retry attempts, IP is blocked, please try a new IP proxy: {e}")
+            raise Exception(f"[BaiduTieBaClient.get] Reached maximum retry attempts, IP is blocked, please try a new IP proxy: {e}")
 
     async def post(self, uri: str, data: dict, **kwargs) -> Dict:
         """
-        POST请求，对请求头签名
+        POST request with header signing
         Args:
-            uri: 请求路由
-            data: 请求体参数
+            uri: Request route
+            data: Request body parameters
 
         Returns:
 
@@ -193,13 +319,13 @@ class BaiduTieBaClient(AbstractApiClient):
 
     async def pong(self, browser_context: BrowserContext = None) -> bool:
         """
-        用于检查登录态是否失效了
-        使用Cookie检测而非API调用,避免被检测
+        Check if login state is still valid
+        Uses Cookie detection instead of API calls to avoid detection
         Args:
-            browser_context: 浏览器上下文对象
+            browser_context: Browser context object
 
         Returns:
-            bool: True表示已登录,False表示未登录
+            bool: True if logged in, False if not logged in
         """
         utils.logger.info("[BaiduTieBaClient.pong] Begin to check tieba login state by cookies...")
 
@@ -208,13 +334,16 @@ class BaiduTieBaClient(AbstractApiClient):
             return False
 
         try:
-            # 从浏览器获取cookies并检查关键登录cookie
-            _, cookie_dict = utils.convert_cookies(await browser_context.cookies())
+            # Get cookies from browser and check key login cookies
+            _, cookie_dict = await utils.convert_browser_context_cookies(
+                browser_context,
+                urls=self.cookie_urls,
+            )
 
-            # 百度贴吧的登录标识: STOKEN 或 PTOKEN
+            # Baidu Tieba login identifiers: STOKEN or PTOKEN
             stoken = cookie_dict.get("STOKEN")
             ptoken = cookie_dict.get("PTOKEN")
-            bduss = cookie_dict.get("BDUSS")  # 百度通用登录cookie
+            bduss = cookie_dict.get("BDUSS")  # Baidu universal login cookie
 
             if stoken or ptoken or bduss:
                 utils.logger.info(f"[BaiduTieBaClient.pong] Login state verified by cookies (STOKEN: {bool(stoken)}, PTOKEN: {bool(ptoken)}, BDUSS: {bool(bduss)})")
@@ -227,16 +356,19 @@ class BaiduTieBaClient(AbstractApiClient):
             utils.logger.error(f"[BaiduTieBaClient.pong] Check login state failed: {e}, assume not logged in")
             return False
 
-    async def update_cookies(self, browser_context: BrowserContext):
+    async def update_cookies(self, browser_context: BrowserContext, urls: Optional[list[str]] = None):
         """
-        API客户端提供的更新cookies方法，一般情况下登录成功后会调用此方法
+        Update cookies method provided by API client, usually called after successful login
         Args:
-            browser_context: 浏览器上下文对象
+            browser_context: Browser context object
 
         Returns:
 
         """
-        cookie_str, cookie_dict = utils.convert_cookies(await browser_context.cookies())
+        cookie_str, cookie_dict = await utils.convert_browser_context_cookies(
+            browser_context,
+            urls=urls or self.cookie_urls,
+        )
         self.headers["Cookie"] = cookie_str
         utils.logger.info("[BaiduTieBaClient.update_cookies] Cookie has been updated")
 
@@ -249,13 +381,13 @@ class BaiduTieBaClient(AbstractApiClient):
         note_type: SearchNoteType = SearchNoteType.FIXED_THREAD,
     ) -> List[TiebaNote]:
         """
-        根据关键词搜索贴吧帖子 (使用Playwright访问页面,避免API检测)
+        Search Tieba posts by keyword (uses Playwright to access page, avoiding API detection)
         Args:
-            keyword: 关键词
-            page: 分页第几页
-            page_size: 每页大小
-            sort: 结果排序方式
-            note_type: 帖子类型（主题贴｜主题+回复混合模式）
+            keyword: Keyword
+            page: Page number
+            page_size: Page size
+            sort: Result sort method
+            note_type: Post type (main thread | main thread + reply mixed mode)
         Returns:
 
         """
@@ -263,76 +395,58 @@ class BaiduTieBaClient(AbstractApiClient):
             utils.logger.error("[BaiduTieBaClient.get_notes_by_keyword] playwright_page is None, cannot use browser mode")
             raise Exception("playwright_page is required for browser-based search")
 
-        # 构造搜索URL
-        # 示例: https://tieba.baidu.com/f/search/res?ie=utf-8&qw=编程
-        search_url = f"{self._host}/f/search/res"
         params = {
-            "ie": "utf-8",
-            "qw": keyword,
-            "rn": page_size,
+            "rn": max(page_size, 20),
+            "st": sort.value,
+            "word": keyword,
+            "needbrand": 1,
+            "sug_type": 2,
             "pn": page,
-            "sm": sort.value,
-            "only_thread": note_type.value,
+            "come_from": "search",
+            "subapp_type": "pc",
+            "_client_type": "20",
         }
-
-        # 拼接完整URL
-        full_url = f"{search_url}?{urlencode(params)}"
-        utils.logger.info(f"[BaiduTieBaClient.get_notes_by_keyword] 访问搜索页面: {full_url}")
+        utils.logger.info(
+            f"[BaiduTieBaClient.get_notes_by_keyword] Accessing search API: "
+            f"{self._host}/mo/q/search/multsearch?{urlencode(params)}"
+        )
 
         try:
-            # 使用Playwright访问搜索页面
-            await self.playwright_page.goto(full_url, wait_until="domcontentloaded")
-
-            # 等待页面加载,使用配置文件中的延时设置
-            await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)
-
-            # 获取页面HTML内容
-            page_content = await self.playwright_page.content()
-            utils.logger.info(f"[BaiduTieBaClient.get_notes_by_keyword] 成功获取搜索页面HTML,长度: {len(page_content)}")
-
-            # 提取搜索结果
-            notes = self._page_extractor.extract_search_note_list(page_content)
-            utils.logger.info(f"[BaiduTieBaClient.get_notes_by_keyword] 提取到 {len(notes)} 条帖子")
+            api_data = await self._fetch_json_by_browser(
+                "/mo/q/search/multsearch",
+                params=params,
+                use_sign=True,
+            )
+            notes = self._page_extractor.extract_search_note_list_from_api(api_data)[:page_size]
+            utils.logger.info(f"[BaiduTieBaClient.get_notes_by_keyword] Extracted {len(notes)} posts")
             return notes
 
         except Exception as e:
-            utils.logger.error(f"[BaiduTieBaClient.get_notes_by_keyword] 搜索失败: {e}")
+            utils.logger.error(f"[BaiduTieBaClient.get_notes_by_keyword] Search failed: {e}")
             raise
 
     async def get_note_by_id(self, note_id: str) -> TiebaNote:
         """
-        根据帖子ID获取帖子详情 (使用Playwright访问页面,避免API检测)
+        Get post details by post ID (uses Playwright to access page, avoiding API detection)
         Args:
-            note_id: 帖子ID
+            note_id: Post ID
 
         Returns:
-            TiebaNote: 帖子详情对象
+            TiebaNote: Post detail object
         """
         if not self.playwright_page:
             utils.logger.error("[BaiduTieBaClient.get_note_by_id] playwright_page is None, cannot use browser mode")
             raise Exception("playwright_page is required for browser-based note detail fetching")
 
-        # 构造帖子详情URL
-        note_url = f"{self._host}/p/{note_id}"
-        utils.logger.info(f"[BaiduTieBaClient.get_note_by_id] 访问帖子详情页面: {note_url}")
+        utils.logger.info(f"[BaiduTieBaClient.get_note_by_id] Accessing post detail API, note_id: {note_id}")
 
         try:
-            # 使用Playwright访问帖子详情页面
-            await self.playwright_page.goto(note_url, wait_until="domcontentloaded")
-
-            # 等待页面加载,使用配置文件中的延时设置
-            await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)
-
-            # 获取页面HTML内容
-            page_content = await self.playwright_page.content()
-            utils.logger.info(f"[BaiduTieBaClient.get_note_by_id] 成功获取帖子详情HTML,长度: {len(page_content)}")
-
-            # 提取帖子详情
-            note_detail = self._page_extractor.extract_note_detail(page_content)
+            api_data = await self._get_pc_page_data(note_id=note_id, page=1)
+            note_detail = self._page_extractor.extract_note_detail_from_api(api_data)
             return note_detail
 
         except Exception as e:
-            utils.logger.error(f"[BaiduTieBaClient.get_note_by_id] 获取帖子详情失败: {e}")
+            utils.logger.error(f"[BaiduTieBaClient.get_note_by_id] Failed to get post details: {e}")
             raise
 
     async def get_note_all_comments(
@@ -343,14 +457,14 @@ class BaiduTieBaClient(AbstractApiClient):
         max_count: int = 10,
     ) -> List[TiebaComment]:
         """
-        获取指定帖子下的所有一级评论 (使用Playwright访问页面,避免API检测)
+        Get all first-level comments for specified post (uses Playwright to access page, avoiding API detection)
         Args:
-            note_detail: 帖子详情对象
-            crawl_interval: 爬取一次笔记的延迟单位（秒）
-            callback: 一次笔记爬取结束后的回调函数
-            max_count: 一次帖子爬取的最大评论数量
+            note_detail: Post detail object
+            crawl_interval: Crawl delay interval in seconds
+            callback: Callback function after one post crawl completes
+            max_count: Maximum number of comments to crawl per post
         Returns:
-            List[TiebaComment]: 评论列表
+            List[TiebaComment]: Comment list
         """
         if not self.playwright_page:
             utils.logger.error("[BaiduTieBaClient.get_note_all_comments] playwright_page is None, cannot use browser mode")
@@ -360,30 +474,22 @@ class BaiduTieBaClient(AbstractApiClient):
         current_page = 1
 
         while note_detail.total_replay_page >= current_page and len(result) < max_count:
-            # 构造评论页URL
-            comment_url = f"{self._host}/p/{note_detail.note_id}?pn={current_page}"
-            utils.logger.info(f"[BaiduTieBaClient.get_note_all_comments] 访问评论页面: {comment_url}")
+            utils.logger.info(
+                f"[BaiduTieBaClient.get_note_all_comments] Accessing comment API, "
+                f"note_id: {note_detail.note_id}, page: {current_page}"
+            )
 
             try:
-                # 使用Playwright访问评论页面
-                await self.playwright_page.goto(comment_url, wait_until="domcontentloaded")
-
-                # 等待页面加载,使用配置文件中的延时设置
-                await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)
-
-                # 获取页面HTML内容
-                page_content = await self.playwright_page.content()
-
-                # 提取评论
-                comments = self._page_extractor.extract_tieba_note_parment_comments(
-                    page_content, note_id=note_detail.note_id
+                api_data = await self._get_pc_page_data(note_id=note_detail.note_id, page=current_page)
+                comments = self._page_extractor.extract_tieba_note_parent_comments_from_api(
+                    api_data, note_detail=note_detail
                 )
 
                 if not comments:
-                    utils.logger.info(f"[BaiduTieBaClient.get_note_all_comments] 第{current_page}页没有评论,停止爬取")
+                    utils.logger.info(f"[BaiduTieBaClient.get_note_all_comments] Page {current_page} has no comments, stopping crawl")
                     break
 
-                # 限制评论数量
+                # Limit comment count
                 if len(result) + len(comments) > max_count:
                     comments = comments[:max_count - len(result)]
 
@@ -392,7 +498,7 @@ class BaiduTieBaClient(AbstractApiClient):
 
                 result.extend(comments)
 
-                # 获取所有子评论
+                # Get all sub-comments
                 await self.get_comments_all_sub_comments(
                     comments, crawl_interval=crawl_interval, callback=callback
                 )
@@ -401,10 +507,10 @@ class BaiduTieBaClient(AbstractApiClient):
                 current_page += 1
 
             except Exception as e:
-                utils.logger.error(f"[BaiduTieBaClient.get_note_all_comments] 获取第{current_page}页评论失败: {e}")
+                utils.logger.error(f"[BaiduTieBaClient.get_note_all_comments] Failed to get page {current_page} comments: {e}")
                 break
 
-        utils.logger.info(f"[BaiduTieBaClient.get_note_all_comments] 共获取 {len(result)} 条一级评论")
+        utils.logger.info(f"[BaiduTieBaClient.get_note_all_comments] Total retrieved {len(result)} first-level comments")
         return result
 
     async def get_comments_all_sub_comments(
@@ -414,14 +520,14 @@ class BaiduTieBaClient(AbstractApiClient):
         callback: Optional[Callable] = None,
     ) -> List[TiebaComment]:
         """
-        获取指定评论下的所有子评论 (使用Playwright访问页面,避免API检测)
+        Get all sub-comments for specified comments (uses Playwright to access page, avoiding API detection)
         Args:
-            comments: 评论列表
-            crawl_interval: 爬取一次笔记的延迟单位（秒）
-            callback: 一次笔记爬取结束后的回调函数
+            comments: Comment list
+            crawl_interval: Crawl delay interval in seconds
+            callback: Callback function after one post crawl completes
 
         Returns:
-            List[TiebaComment]: 子评论列表
+            List[TiebaComment]: Sub-comment list
         """
         if not config.ENABLE_GET_SUB_COMMENTS:
             return []
@@ -440,7 +546,7 @@ class BaiduTieBaClient(AbstractApiClient):
             max_sub_page_num = parment_comment.sub_comment_count // 10 + 1
 
             while max_sub_page_num >= current_page:
-                # 构造子评论URL
+                # Construct sub-comment URL
                 sub_comment_url = (
                     f"{self._host}/p/comment?"
                     f"tid={parment_comment.note_id}&"
@@ -448,19 +554,19 @@ class BaiduTieBaClient(AbstractApiClient):
                     f"fid={parment_comment.tieba_id}&"
                     f"pn={current_page}"
                 )
-                utils.logger.info(f"[BaiduTieBaClient.get_comments_all_sub_comments] 访问子评论页面: {sub_comment_url}")
+                utils.logger.info(f"[BaiduTieBaClient.get_comments_all_sub_comments] Accessing sub-comment page: {sub_comment_url}")
 
                 try:
-                    # 使用Playwright访问子评论页面
+                    # Use Playwright to access sub-comment page
                     await self.playwright_page.goto(sub_comment_url, wait_until="domcontentloaded")
 
-                    # 等待页面加载,使用配置文件中的延时设置
+                    # Wait for page loading, using delay setting from config file
                     await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)
 
-                    # 获取页面HTML内容
+                    # Get page HTML content
                     page_content = await self.playwright_page.content()
 
-                    # 提取子评论
+                    # Extract sub-comments
                     sub_comments = self._page_extractor.extract_tieba_note_sub_comments(
                         page_content, parent_comment=parment_comment
                     )
@@ -468,7 +574,7 @@ class BaiduTieBaClient(AbstractApiClient):
                     if not sub_comments:
                         utils.logger.info(
                             f"[BaiduTieBaClient.get_comments_all_sub_comments] "
-                            f"评论{parment_comment.comment_id}第{current_page}页没有子评论,停止爬取"
+                            f"Comment {parment_comment.comment_id} page {current_page} has no sub-comments, stopping crawl"
                         )
                         break
 
@@ -482,125 +588,169 @@ class BaiduTieBaClient(AbstractApiClient):
                 except Exception as e:
                     utils.logger.error(
                         f"[BaiduTieBaClient.get_comments_all_sub_comments] "
-                        f"获取评论{parment_comment.comment_id}第{current_page}页子评论失败: {e}"
+                        f"Failed to get comment {parment_comment.comment_id} page {current_page} sub-comments: {e}"
                     )
                     break
 
-        utils.logger.info(f"[BaiduTieBaClient.get_comments_all_sub_comments] 共获取 {len(all_sub_comments)} 条子评论")
+        utils.logger.info(f"[BaiduTieBaClient.get_comments_all_sub_comments] Total retrieved {len(all_sub_comments)} sub-comments")
         return all_sub_comments
 
     async def get_notes_by_tieba_name(self, tieba_name: str, page_num: int) -> List[TiebaNote]:
         """
-        根据贴吧名称获取帖子列表 (使用Playwright访问页面,避免API检测)
+        Get post list by Tieba name from current PC forum JSON API.
         Args:
-            tieba_name: 贴吧名称
-            page_num: 分页页码
+            tieba_name: Tieba name
+            page_num: Page number
 
         Returns:
-            List[TiebaNote]: 帖子列表
+            List[TiebaNote]: Post list
         """
         if not self.playwright_page:
             utils.logger.error("[BaiduTieBaClient.get_notes_by_tieba_name] playwright_page is None, cannot use browser mode")
             raise Exception("playwright_page is required for browser-based tieba note fetching")
 
-        # 构造贴吧帖子列表URL
-        tieba_url = f"{self._host}/f?kw={quote(tieba_name)}&pn={page_num}"
-        utils.logger.info(f"[BaiduTieBaClient.get_notes_by_tieba_name] 访问贴吧页面: {tieba_url}")
+        page_size = 30
+        api_page = page_num // page_size + 1
+        tbs = await self._get_pc_tbs()
+        utils.logger.info(
+            f"[BaiduTieBaClient.get_notes_by_tieba_name] Accessing Tieba FRS API, "
+            f"tieba_name: {tieba_name}, page: {api_page}"
+        )
 
         try:
-            # 使用Playwright访问贴吧页面
-            await self.playwright_page.goto(tieba_url, wait_until="domcontentloaded")
-
-            # 等待页面加载,使用配置文件中的延时设置
-            await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)
-
-            # 获取页面HTML内容
-            page_content = await self.playwright_page.content()
-            utils.logger.info(f"[BaiduTieBaClient.get_notes_by_tieba_name] 成功获取贴吧页面HTML,长度: {len(page_content)}")
-
-            # 提取帖子列表
-            notes = self._page_extractor.extract_tieba_note_list(page_content)
-            utils.logger.info(f"[BaiduTieBaClient.get_notes_by_tieba_name] 提取到 {len(notes)} 条帖子")
+            api_data = await self._fetch_json_by_browser(
+                "/c/f/frs/page_pc",
+                method="POST",
+                data={
+                    "kw": quote(tieba_name),
+                    "pn": api_page,
+                    "sort_type": -1,
+                    "is_newfrs": 1,
+                    "is_newfeed": 1,
+                    "rn": page_size,
+                    "rn_need": 10,
+                    "tbs": tbs,
+                    "subapp_type": "pc",
+                    "_client_type": "20",
+                },
+                use_sign=True,
+            )
+            notes = self._page_extractor.extract_tieba_note_list_from_frs_api(api_data)[:page_size]
+            utils.logger.info(f"[BaiduTieBaClient.get_notes_by_tieba_name] Extracted {len(notes)} posts")
             return notes
 
         except Exception as e:
-            utils.logger.error(f"[BaiduTieBaClient.get_notes_by_tieba_name] 获取贴吧帖子列表失败: {e}")
+            utils.logger.error(f"[BaiduTieBaClient.get_notes_by_tieba_name] Failed to get Tieba post list: {e}")
             raise
 
-    async def get_creator_info_by_url(self, creator_url: str) -> str:
+    async def get_creator_info_by_url(self, creator_url: str) -> TiebaCreator:
         """
-        根据创作者URL获取创作者信息 (使用Playwright访问页面,避免API检测)
+        Get creator information by creator URL from current PC JSON API.
         Args:
-            creator_url: 创作者主页URL
+            creator_url: Creator homepage URL
 
         Returns:
-            str: 页面HTML内容
+            TiebaCreator: Creator information
         """
         if not self.playwright_page:
             utils.logger.error("[BaiduTieBaClient.get_creator_info_by_url] playwright_page is None, cannot use browser mode")
             raise Exception("playwright_page is required for browser-based creator info fetching")
 
-        utils.logger.info(f"[BaiduTieBaClient.get_creator_info_by_url] 访问创作者主页: {creator_url}")
+        portrait = self._extract_creator_portrait(creator_url)
+        if not portrait:
+            raise Exception(f"Can not extract Tieba creator portrait from url: {creator_url}")
+
+        utils.logger.info(
+            f"[BaiduTieBaClient.get_creator_info_by_url] Accessing creator info API, portrait: {portrait}"
+        )
 
         try:
-            # 使用Playwright访问创作者主页
-            await self.playwright_page.goto(creator_url, wait_until="domcontentloaded")
-
-            # 等待页面加载,使用配置文件中的延时设置
-            await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)
-
-            # 获取页面HTML内容
-            page_content = await self.playwright_page.content()
-            utils.logger.info(f"[BaiduTieBaClient.get_creator_info_by_url] 成功获取创作者主页HTML,长度: {len(page_content)}")
-
-            return page_content
+            api_data = await self._fetch_json_by_browser(
+                "/c/u/pc/homeSidebarRight",
+                params={
+                    "portrait": portrait,
+                    "un": "",
+                    "subapp_type": "pc",
+                    "_client_type": "20",
+                },
+                use_sign=True,
+            )
+            return self._page_extractor.extract_creator_info_from_api(api_data)
 
         except Exception as e:
-            utils.logger.error(f"[BaiduTieBaClient.get_creator_info_by_url] 获取创作者主页失败: {e}")
+            utils.logger.error(f"[BaiduTieBaClient.get_creator_info_by_url] Failed to get creator info: {e}")
             raise
 
+    async def get_notes_by_creator_portrait(
+        self, portrait: str, page_number: int, page_size: int = 20
+    ) -> Dict:
+        """
+        Get creator's thread feed by creator portrait from current PC JSON API.
+        """
+        if not self.playwright_page:
+            utils.logger.error("[BaiduTieBaClient.get_notes_by_creator_portrait] playwright_page is None, cannot use browser mode")
+            raise Exception("playwright_page is required for browser-based creator notes fetching")
+
+        utils.logger.info(
+            f"[BaiduTieBaClient.get_notes_by_creator_portrait] Accessing creator feed API, "
+            f"portrait: {portrait}, page: {page_number}"
+        )
+        return await self._fetch_json_by_browser(
+            "/c/u/feed/myThread",
+            params={
+                "pn": page_number,
+                "rn": page_size,
+                "portrait": portrait,
+                "type": 1,
+                "un": "",
+                "subapp_type": "pc",
+                "_client_type": "20",
+            },
+            use_sign=True,
+        )
+
     async def get_notes_by_creator(self, user_name: str, page_number: int) -> Dict:
         """
-        根据创作者获取创作者的帖子 (使用Playwright访问页面,避免API检测)
+        Get creator's posts by creator (uses Playwright to access page, avoiding API detection)
         Args:
-            user_name: 创作者用户名
-            page_number: 页码
+            user_name: Creator username
+            page_number: Page number
 
         Returns:
-            Dict: 包含帖子数据的字典
+            Dict: Dictionary containing post data
         """
         if not self.playwright_page:
             utils.logger.error("[BaiduTieBaClient.get_notes_by_creator] playwright_page is None, cannot use browser mode")
             raise Exception("playwright_page is required for browser-based creator notes fetching")
 
-        # 构造创作者帖子列表URL
+        # Construct creator post list URL
         creator_url = f"{self._host}/home/get/getthread?un={quote(user_name)}&pn={page_number}&id=utf-8&_={utils.get_current_timestamp()}"
-        utils.logger.info(f"[BaiduTieBaClient.get_notes_by_creator] 访问创作者帖子列表: {creator_url}")
+        utils.logger.info(f"[BaiduTieBaClient.get_notes_by_creator] Accessing creator post list: {creator_url}")
 
         try:
-            # 使用Playwright访问创作者帖子列表页面
+            # Use Playwright to access creator post list page
             await self.playwright_page.goto(creator_url, wait_until="domcontentloaded")
 
-            # 等待页面加载,使用配置文件中的延时设置
+            # Wait for page loading, using delay setting from config file
             await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)
 
-            # 获取页面内容(这个接口返回JSON)
+            # Get page content (this API returns JSON)
             page_content = await self.playwright_page.content()
 
-            # 提取JSON数据(页面会包含<pre>标签或直接是JSON)
+            # Extract JSON data (page will contain <pre> tag or is directly JSON)
             try:
-                # 尝试从页面中提取JSON
+                # Try to extract JSON from page
                 json_text = await self.playwright_page.evaluate("() => document.body.innerText")
                 result = json.loads(json_text)
-                utils.logger.info(f"[BaiduTieBaClient.get_notes_by_creator] 成功获取创作者帖子数据")
+                utils.logger.info(f"[BaiduTieBaClient.get_notes_by_creator] Successfully retrieved creator post data")
                 return result
             except json.JSONDecodeError as e:
-                utils.logger.error(f"[BaiduTieBaClient.get_notes_by_creator] JSON解析失败: {e}")
-                utils.logger.error(f"[BaiduTieBaClient.get_notes_by_creator] 页面内容: {page_content[:500]}")
+                utils.logger.error(f"[BaiduTieBaClient.get_notes_by_creator] JSON parsing failed: {e}")
+                utils.logger.error(f"[BaiduTieBaClient.get_notes_by_creator] Page content: {page_content[:500]}")
                 raise Exception(f"Failed to parse JSON from creator notes page: {e}")
 
         except Exception as e:
-            utils.logger.error(f"[BaiduTieBaClient.get_notes_by_creator] 获取创作者帖子列表失败: {e}")
+            utils.logger.error(f"[BaiduTieBaClient.get_notes_by_creator] Failed to get creator post list: {e}")
             raise
 
     async def get_all_notes_by_creator_user_name(
@@ -612,18 +762,18 @@ class BaiduTieBaClient(AbstractApiClient):
         creator_page_html_content: str = None,
     ) -> List[TiebaNote]:
         """
-        根据创作者用户名获取创作者所有帖子
+        Get all creator posts by creator username
         Args:
-            user_name: 创作者用户名
-            crawl_interval: 爬取一次笔记的延迟单位（秒）
-            callback: 一次笔记爬取结束后的回调函数，是一个awaitable类型的函数
-            max_note_count: 帖子最大获取数量，如果为0则获取所有
-            creator_page_html_content: 创作者主页HTML内容
+            user_name: Creator username
+            crawl_interval: Crawl delay interval in seconds
+            callback: Callback function after one post crawl completes, an awaitable function
+            max_note_count: Maximum number of posts to retrieve, if 0 then get all
+            creator_page_html_content: Creator homepage HTML content
 
         Returns:
 
         """
-        # 百度贴吧比较特殊一些，前10个帖子是直接展示在主页上的，要单独处理，通过API获取不到
+        # Baidu Tieba is special, the first 10 posts are directly displayed on the homepage and need special handling, cannot be obtained through API
         result: List[TiebaNote] = []
         if creator_page_html_content:
             thread_id_list = (self._page_extractor.extract_tieba_thread_id_list_from_creator_page(creator_page_html_content))
@@ -641,12 +791,12 @@ class BaiduTieBaClient(AbstractApiClient):
         while notes_has_more == 1 and (max_note_count == 0 or total_get_count < max_note_count):
             notes_res = await self.get_notes_by_creator(user_name, page_number)
             if not notes_res or notes_res.get("no") != 0:
-                utils.logger.error(f"[WeiboClient.get_notes_by_creator] got user_name:{user_name} notes failed, notes_res: {notes_res}")
+                utils.logger.error(f"[TieBaClient.get_notes_by_creator] got user_name:{user_name} notes failed, notes_res: {notes_res}")
                 break
             notes_data = notes_res.get("data")
             notes_has_more = notes_data.get("has_more")
             notes = notes_data["thread_list"]
-            utils.logger.info(f"[WeiboClient.get_all_notes_by_creator] got user_name:{user_name} notes len : {len(notes)}")
+            utils.logger.info(f"[TieBaClient.get_all_notes_by_creator] got user_name:{user_name} notes len : {len(notes)}")
 
             note_detail_task = [self.get_note_by_id(note['thread_id']) for note in notes]
             notes = await asyncio.gather(*note_detail_task)
@@ -657,3 +807,59 @@ class BaiduTieBaClient(AbstractApiClient):
             page_number += 1
             total_get_count += page_per_count
         return result
+
+    async def get_all_notes_by_creator_url(
+        self,
+        creator_url: str,
+        crawl_interval: float = 1.0,
+        callback: Optional[Callable] = None,
+        max_note_count: int = 0,
+    ) -> List[TiebaNote]:
+        """
+        Get all creator posts by current PC creator feed API.
+        """
+        portrait = self._extract_creator_portrait(creator_url)
+        if not portrait:
+            raise Exception(f"Can not extract Tieba creator portrait from url: {creator_url}")
+
+        result: List[TiebaNote] = []
+        page_number = 1
+        page_size = 20
+
+        while max_note_count == 0 or len(result) < max_note_count:
+            notes_res = await self.get_notes_by_creator_portrait(
+                portrait=portrait,
+                page_number=page_number,
+                page_size=page_size,
+            )
+            thread_id_list = self._page_extractor.extract_creator_thread_id_list_from_api(notes_res)
+            if not thread_id_list:
+                utils.logger.info(
+                    f"[BaiduTieBaClient.get_all_notes_by_creator_url] "
+                    f"Creator portrait:{portrait} page:{page_number} has no threads"
+                )
+                break
+
+            if max_note_count:
+                thread_id_list = thread_id_list[: max_note_count - len(result)]
+
+            utils.logger.info(
+                f"[BaiduTieBaClient.get_all_notes_by_creator_url] "
+                f"got portrait:{portrait} thread ids len: {len(thread_id_list)}"
+            )
+            note_detail_task = [self.get_note_by_id(thread_id) for thread_id in thread_id_list]
+            notes = await asyncio.gather(*note_detail_task)
+            notes = [note for note in notes if note]
+            if callback and notes:
+                await callback(notes)
+            result.extend(notes)
+
+            data = notes_res.get("data", {})
+            has_more = int(data.get("has_more") or 0)
+            if not has_more:
+                break
+
+            await asyncio.sleep(crawl_interval)
+            page_number += 1
+
+        return result

media_platform/tieba/core.py

@@ -54,6 +54,7 @@ class TieBaCrawler(AbstractCrawler):
 
     def __init__(self) -> None:
         self.index_url = "https://tieba.baidu.com"
+        self.cookie_urls = [self.index_url]
         self.user_agent = utils.get_user_agent()
         self._page_extractor = TieBaExtractor()
         self.cdp_manager = None
@@ -79,9 +80,9 @@ class TieBaCrawler(AbstractCrawler):
             )
 
         async with async_playwright() as playwright:
-            # 根据配置选择启动模式
+            # Choose startup mode based on configuration
             if config.ENABLE_CDP_MODE:
-                utils.logger.info("[BaiduTieBaCrawler] 使用CDP模式启动浏览器")
+                utils.logger.info("[BaiduTieBaCrawler] Launching browser in CDP mode")
                 self.browser_context = await self.launch_browser_with_cdp(
                     playwright,
                     playwright_proxy_format,
@@ -89,7 +90,7 @@ class TieBaCrawler(AbstractCrawler):
                     headless=config.CDP_HEADLESS,
                 )
             else:
-                utils.logger.info("[BaiduTieBaCrawler] 使用标准模式启动浏览器")
+                utils.logger.info("[BaiduTieBaCrawler] Launching browser in standard mode")
                 # Launch a browser context.
                 chromium = playwright.chromium
                 self.browser_context = await self.launch_browser(
@@ -99,12 +100,12 @@ class TieBaCrawler(AbstractCrawler):
                     headless=config.HEADLESS,
                 )
 
-            # 注入反检测脚本 - 针对百度的特殊检测
+            # Inject anti-detection scripts - for Baidu's special detection
             await self._inject_anti_detection_scripts()
 
             self.context_page = await self.browser_context.new_page()
 
-            # 先访问百度首页,再点击贴吧链接,避免触发安全验证
+            # First visit Baidu homepage, then click Tieba link to avoid triggering security verification
             await self._navigate_to_tieba_via_baidu()
 
             # Create a client to interact with the baidutieba website.
@@ -123,7 +124,10 @@ class TieBaCrawler(AbstractCrawler):
                     cookie_str=config.COOKIES,
                 )
                 await login_obj.begin()
-                await self.tieba_client.update_cookies(browser_context=self.browser_context)
+                await self.tieba_client.update_cookies(
+                    browser_context=self.browser_context,
+                    urls=self.cookie_urls,
+                )
 
             crawler_type_var.set(config.CRAWLER_TYPE)
             if config.CRAWLER_TYPE == "search":
@@ -209,7 +213,7 @@ class TieBaCrawler(AbstractCrawler):
         Returns:
 
         """
-        tieba_limit_count = 50
+        tieba_limit_count = 30
         if config.CRAWLER_MAX_NOTES_COUNT < tieba_limit_count:
             config.CRAWLER_MAX_NOTES_COUNT = tieba_limit_count
         for tieba_name in config.TIEBA_NAME_LIST:
@@ -241,7 +245,7 @@ class TieBaCrawler(AbstractCrawler):
                 page_number += tieba_limit_count
 
     async def get_specified_notes(
-        self, note_id_list: List[str] = config.TIEBA_SPECIFIED_ID_LIST
+        self, note_id_list: Optional[List[str]] = None
     ):
         """
         Get the information and comments of the specified post
@@ -251,6 +255,8 @@ class TieBaCrawler(AbstractCrawler):
         Returns:
 
         """
+        if note_id_list is None:
+            note_id_list = config.TIEBA_SPECIFIED_ID_LIST
         semaphore = asyncio.Semaphore(config.MAX_CONCURRENCY_NUM)
         task_list = [
             self.get_note_detail_async_task(note_id=note_id, semaphore=semaphore)
@@ -361,18 +367,15 @@ class TieBaCrawler(AbstractCrawler):
 
         """
         utils.logger.info(
-            "[WeiboCrawler.get_creators_and_notes] Begin get weibo creators"
+            "[TieBaCrawler.get_creators_and_notes] Begin get tieba creators"
         )
         for creator_url in config.TIEBA_CREATOR_URL_LIST:
-            creator_page_html_content = await self.tieba_client.get_creator_info_by_url(
+            creator_info: TiebaCreator = await self.tieba_client.get_creator_info_by_url(
                 creator_url=creator_url
             )
-            creator_info: TiebaCreator = self._page_extractor.extract_creator_info(
-                creator_page_html_content
-            )
             if creator_info:
                 utils.logger.info(
-                    f"[WeiboCrawler.get_creators_and_notes] creator info: {creator_info}"
+                    f"[TieBaCrawler.get_creators_and_notes] creator info: {creator_info}"
                 )
                 if not creator_info:
                     raise Exception("Get creator info error")
@@ -381,12 +384,11 @@ class TieBaCrawler(AbstractCrawler):
 
                 # Get all note information of the creator
                 all_notes_list = (
-                    await self.tieba_client.get_all_notes_by_creator_user_name(
-                        user_name=creator_info.user_name,
+                    await self.tieba_client.get_all_notes_by_creator_url(
+                        creator_url=creator_url,
                         crawl_interval=0,
                         callback=tieba_store.batch_update_tieba_notes,
                         max_note_count=config.CRAWLER_MAX_NOTES_COUNT,
-                        creator_page_html_content=creator_page_html_content,
                     )
                 )
 
@@ -394,34 +396,34 @@ class TieBaCrawler(AbstractCrawler):
 
             else:
                 utils.logger.error(
-                    f"[WeiboCrawler.get_creators_and_notes] get creator info error, creator_url:{creator_url}"
+                    f"[TieBaCrawler.get_creators_and_notes] get creator info error, creator_url:{creator_url}"
                 )
 
     async def _navigate_to_tieba_via_baidu(self):
         """
-        模拟真实用户访问路径:
-        1. 先访问百度首页 (https://www.baidu.com/)
-        2. 等待页面加载
-        3. 点击顶部导航栏的"贴吧"链接
-        4. 跳转到贴吧首页
+        Simulate real user access path:
+        1. First visit Baidu homepage (https://www.baidu.com/)
+        2. Wait for page to load
+        3. Click "Tieba" link in top navigation bar
+        4. Jump to Tieba homepage
 
-        这样做可以避免触发百度的安全验证
+        This avoids triggering Baidu's security verification
         """
-        utils.logger.info("[TieBaCrawler] 模拟真实用户访问路径...")
+        utils.logger.info("[TieBaCrawler] Simulating real user access path...")
 
         try:
-            # Step 1: 访问百度首页
-            utils.logger.info("[TieBaCrawler] Step 1: 访问百度首页 https://www.baidu.com/")
+            # Step 1: Visit Baidu homepage
+            utils.logger.info("[TieBaCrawler] Step 1: Visiting Baidu homepage https://www.baidu.com/")
             await self.context_page.goto("https://www.baidu.com/", wait_until="domcontentloaded")
 
-            # Step 2: 等待页面加载,使用配置文件中的延时设置
-            utils.logger.info(f"[TieBaCrawler] Step 2: 等待 {config.CRAWLER_MAX_SLEEP_SEC}秒 模拟用户浏览...")
+            # Step 2: Wait for page loading, using delay setting from config file
+            utils.logger.info(f"[TieBaCrawler] Step 2: Waiting {config.CRAWLER_MAX_SLEEP_SEC} seconds to simulate user browsing...")
             await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)
 
-            # Step 3: 查找并点击"贴吧"链接
-            utils.logger.info("[TieBaCrawler] Step 3: 查找并点击'贴吧'链接...")
+            # Step 3: Find and click "Tieba" link
+            utils.logger.info("[TieBaCrawler] Step 3: Finding and clicking 'Tieba' link...")
 
-            # 尝试多种选择器,确保能找到贴吧链接
+            # Try multiple selectors to ensure finding the Tieba link
             tieba_selectors = [
                 'a[href="http://tieba.baidu.com/"]',
                 'a[href="https://tieba.baidu.com/"]',
@@ -434,74 +436,74 @@ class TieBaCrawler(AbstractCrawler):
                 try:
                     tieba_link = await self.context_page.wait_for_selector(selector, timeout=5000)
                     if tieba_link:
-                        utils.logger.info(f"[TieBaCrawler] 找到贴吧链接 (selector: {selector})")
+                        utils.logger.info(f"[TieBaCrawler] Found Tieba link (selector: {selector})")
                         break
                 except Exception:
                     continue
 
             if not tieba_link:
-                utils.logger.warning("[TieBaCrawler] 未找到贴吧链接,直接访问贴吧首页")
+                utils.logger.warning("[TieBaCrawler] Tieba link not found, directly accessing Tieba homepage")
                 await self.context_page.goto(self.index_url, wait_until="domcontentloaded")
                 return
 
-            # Step 4: 点击贴吧链接 (检查是否会打开新标签页)
-            utils.logger.info("[TieBaCrawler] Step 4: 点击贴吧链接...")
+            # Step 4: Click Tieba link (check if it will open in a new tab)
+            utils.logger.info("[TieBaCrawler] Step 4: Clicking Tieba link...")
 
-            # 检查链接的target属性
+            # Check link's target attribute
             target_attr = await tieba_link.get_attribute("target")
-            utils.logger.info(f"[TieBaCrawler] 链接target属性: {target_attr}")
+            utils.logger.info(f"[TieBaCrawler] Link target attribute: {target_attr}")
 
             if target_attr == "_blank":
-                # 如果是新标签页,需要等待新页面并切换
-                utils.logger.info("[TieBaCrawler] 链接会在新标签页打开,等待新页面...")
+                # If it's a new tab, need to wait for new page and switch
+                utils.logger.info("[TieBaCrawler] Link will open in new tab, waiting for new page...")
 
                 async with self.browser_context.expect_page() as new_page_info:
                     await tieba_link.click()
 
-                # 获取新打开的页面
+                # Get newly opened page
                 new_page = await new_page_info.value
                 await new_page.wait_for_load_state("domcontentloaded")
 
-                # 关闭旧的百度首页
+                # Close old Baidu homepage
                 await self.context_page.close()
 
-                # 切换到新的贴吧页面
+                # Switch to new Tieba page
                 self.context_page = new_page
-                utils.logger.info("[TieBaCrawler] ✅ 已切换到新标签页 (贴吧页面)")
+                utils.logger.info("[TieBaCrawler] Successfully switched to new tab (Tieba page)")
             else:
-                # 如果是同一标签页跳转,正常等待导航
-                utils.logger.info("[TieBaCrawler] 链接在当前标签页跳转...")
+                # If it's same tab navigation, wait for navigation normally
+                utils.logger.info("[TieBaCrawler] Link navigates in current tab...")
                 async with self.context_page.expect_navigation(wait_until="domcontentloaded"):
                     await tieba_link.click()
 
-            # Step 5: 等待页面稳定,使用配置文件中的延时设置
-            utils.logger.info(f"[TieBaCrawler] Step 5: 页面加载完成,等待 {config.CRAWLER_MAX_SLEEP_SEC}秒...")
+            # Step 5: Wait for page to stabilize, using delay setting from config file
+            utils.logger.info(f"[TieBaCrawler] Step 5: Page loaded, waiting {config.CRAWLER_MAX_SLEEP_SEC} seconds...")
             await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)
 
             current_url = self.context_page.url
-            utils.logger.info(f"[TieBaCrawler] ✅ 成功通过百度首页进入贴吧! 当前URL: {current_url}")
+            utils.logger.info(f"[TieBaCrawler] Successfully entered Tieba via Baidu homepage! Current URL: {current_url}")
 
         except Exception as e:
-            utils.logger.error(f"[TieBaCrawler] 通过百度首页访问贴吧失败: {e}")
-            utils.logger.info("[TieBaCrawler] 回退:直接访问贴吧首页")
+            utils.logger.error(f"[TieBaCrawler] Failed to access Tieba via Baidu homepage: {e}")
+            utils.logger.info("[TieBaCrawler] Fallback: directly accessing Tieba homepage")
             await self.context_page.goto(self.index_url, wait_until="domcontentloaded")
 
     async def _inject_anti_detection_scripts(self):
         """
-        注入反检测JavaScript脚本
-        针对百度贴吧的特殊检测机制
+        Inject anti-detection JavaScript scripts
+        For Baidu Tieba's special detection mechanism
         """
         utils.logger.info("[TieBaCrawler] Injecting anti-detection scripts...")
 
-        # 轻量级反检测脚本,只覆盖关键检测点
+        # Lightweight anti-detection script, only covering key detection points
         anti_detection_js = """
-        // 覆盖 navigator.webdriver
+        // Override navigator.webdriver
         Object.defineProperty(navigator, 'webdriver', {
             get: () => undefined,
             configurable: true
         });
 
-        // 覆盖 window.navigator.chrome
+        // Override window.navigator.chrome
         if (!window.navigator.chrome) {
             window.navigator.chrome = {
                 runtime: {},
@@ -511,7 +513,7 @@ class TieBaCrawler(AbstractCrawler):
             };
         }
 
-        // 覆盖 Permissions API
+        // Override Permissions API
         const originalQuery = window.navigator.permissions.query;
         window.navigator.permissions.query = (parameters) => (
             parameters.name === 'notifications' ?
@@ -519,19 +521,19 @@ class TieBaCrawler(AbstractCrawler):
                 originalQuery(parameters)
         );
 
-        // 覆盖 plugins 长度(让它看起来有插件)
+        // Override plugins length (make it look like there are plugins)
         Object.defineProperty(navigator, 'plugins', {
             get: () => [1, 2, 3, 4, 5],
             configurable: true
         });
 
-        // 覆盖 languages
+        // Override languages
         Object.defineProperty(navigator, 'languages', {
             get: () => ['zh-CN', 'zh', 'en'],
             configurable: true
         });
 
-        // 移除 window.cdc_ 等 ChromeDriver 残留
+        // Remove window.cdc_ and other ChromeDriver remnants
         delete window.cdc_adoQpoasnfa76pfcZLmcfl_Array;
         delete window.cdc_adoQpoasnfa76pfcZLmcfl_Promise;
         delete window.cdc_adoQpoasnfa76pfcZLmcfl_Symbol;
@@ -548,21 +550,24 @@ class TieBaCrawler(AbstractCrawler):
         """
         Create tieba client with real browser User-Agent and complete headers
         Args:
-            httpx_proxy: HTTP代理
-            ip_pool: IP代理池
+            httpx_proxy: HTTP proxy
+            ip_pool: IP proxy pool
 
         Returns:
-            BaiduTieBaClient实例
+            BaiduTieBaClient instance
         """
         utils.logger.info("[TieBaCrawler.create_tieba_client] Begin create tieba API client...")
 
-        # 从真实浏览器提取User-Agent,避免被检测
+        # Extract User-Agent from real browser to avoid detection
         user_agent = await self.context_page.evaluate("() => navigator.userAgent")
         utils.logger.info(f"[TieBaCrawler.create_tieba_client] Extracted User-Agent from browser: {user_agent}")
 
-        cookie_str, cookie_dict = utils.convert_cookies(await self.browser_context.cookies())
+        cookie_str, cookie_dict = await utils.convert_browser_context_cookies(
+            self.browser_context,
+            urls=self.cookie_urls,
+        )
 
-        # 构建完整的浏览器请求头,模拟真实浏览器行为
+        # Build complete browser request headers, simulating real browser behavior
         tieba_client = BaiduTieBaClient(
             timeout=10,
             ip_pool=ip_pool,
@@ -572,7 +577,7 @@ class TieBaCrawler(AbstractCrawler):
                 "Accept-Language": "zh-CN,zh;q=0.9",
                 "Accept-Encoding": "gzip, deflate, br",
                 "Connection": "keep-alive",
-                "User-Agent": user_agent,  # 使用真实浏览器的UA
+                "User-Agent": user_agent,  # Use real browser UA
                 "Cookie": cookie_str,
                 "Host": "tieba.baidu.com",
                 "Referer": "https://tieba.baidu.com/",
@@ -585,7 +590,7 @@ class TieBaCrawler(AbstractCrawler):
                 "sec-ch-ua-mobile": "?0",
                 "sec-ch-ua-platform": '"macOS"',
             },
-            playwright_page=self.context_page,  # 传入playwright页面对象
+            playwright_page=self.context_page,  # Pass in playwright page object
         )
         return tieba_client
 
@@ -623,7 +628,7 @@ class TieBaCrawler(AbstractCrawler):
                 proxy=playwright_proxy,  # type: ignore
                 viewport={"width": 1920, "height": 1080},
                 user_agent=user_agent,
-                channel="chrome",  # 使用系统的Chrome稳定版
+                channel="chrome",  # Use system's stable Chrome version
             )
             return browser_context
         else:
@@ -641,7 +646,7 @@ class TieBaCrawler(AbstractCrawler):
         headless: bool = True,
     ) -> BrowserContext:
         """
-        使用CDP模式启动浏览器
+        Launch browser using CDP mode
         """
         try:
             self.cdp_manager = CDPBrowserManager()
@@ -652,15 +657,15 @@ class TieBaCrawler(AbstractCrawler):
                 headless=headless,
             )
 
-            # 显示浏览器信息
+            # Display browser information
             browser_info = await self.cdp_manager.get_browser_info()
-            utils.logger.info(f"[TieBaCrawler] CDP浏览器信息: {browser_info}")
+            utils.logger.info(f"[TieBaCrawler] CDP browser info: {browser_info}")
 
             return browser_context
 
         except Exception as e:
-            utils.logger.error(f"[TieBaCrawler] CDP模式启动失败，回退到标准模式: {e}")
-            # 回退到标准模式
+            utils.logger.error(f"[TieBaCrawler] CDP mode launch failed, falling back to standard mode: {e}")
+            # Fall back to standard mode
             chromium = playwright.chromium
             return await self.launch_browser(
                 chromium, playwright_proxy, user_agent, headless
@@ -672,7 +677,7 @@ class TieBaCrawler(AbstractCrawler):
         Returns:
 
         """
-        # 如果使用CDP模式，需要特殊处理
+        # If using CDP mode, need special handling
         if self.cdp_manager:
             await self.cdp_manager.cleanup()
             self.cdp_manager = None

media_platform/tieba/field.py

@@ -23,16 +23,16 @@ from enum import Enum
 
 class SearchSortType(Enum):
     """search sort type"""
-    # 按时间倒序
+    # Sort by time in descending order
     TIME_DESC = "1"
-    # 按时间顺序
+    # Sort by time in ascending order
     TIME_ASC = "0"
-    # 按相关性顺序
+    # Sort by relevance
     RELEVANCE_ORDER = "2"
 
 
 class SearchNoteType(Enum):
-    # 只看主题贴
+    # Only view main posts
     MAIN_THREAD = "1"
-    # 混合模式（帖子+回复）
+    # Mixed mode (posts + replies)
     FIXED_THREAD = "0"

media_platform/tieba/help.py

@@ -22,8 +22,8 @@
 import html
 import json
 import re
-from typing import Dict, List, Tuple
-from urllib.parse import parse_qs, unquote
+from typing import Any, Dict, List, Tuple
+from urllib.parse import parse_qs, quote, unquote, urljoin
 
 from parsel import Selector
 
@@ -39,172 +39,633 @@ class TieBaExtractor:
     def __init__(self):
         pass
 
+    @staticmethod
+    def _class_contains(class_name: str) -> str:
+        return f"contains(concat(' ', normalize-space(@class), ' '), ' {class_name} ')"
+
+    @staticmethod
+    def _normalize_text(text: str) -> str:
+        return re.sub(r"\s+", " ", text or "").strip()
+
+    @classmethod
+    def _selector_text(cls, selector: Selector, xpath: str) -> str:
+        node = selector.xpath(xpath)
+        if not node:
+            return ""
+        return cls._normalize_text(node[0].xpath("string(.)").get(default=""))
+
+    @staticmethod
+    def _absolute_url(url: str) -> str:
+        return urljoin(const.TIEBA_URL, (url or "").strip())
+
+    @staticmethod
+    def _extract_note_id_from_url(url: str) -> str:
+        note_id_match = re.search(r"/p/(\d+)", url or "")
+        return note_id_match.group(1) if note_id_match else ""
+
+    @staticmethod
+    def _text_to_int(text: str) -> int:
+        match = re.search(r"\d+", text or "")
+        return int(match.group(0)) if match else 0
+
+    @staticmethod
+    def _ensure_tieba_suffix(tieba_name: str) -> str:
+        tieba_name = (tieba_name or "").strip()
+        return tieba_name if not tieba_name or tieba_name.endswith("吧") else f"{tieba_name}吧"
+
+    @classmethod
+    def _tieba_link_from_name(cls, tieba_name: str) -> str:
+        if not tieba_name:
+            return const.TIEBA_URL
+        return f"{const.TIEBA_URL}/f?kw={quote(tieba_name.removesuffix('吧'))}"
+
+    @classmethod
+    def _extract_api_content_text(cls, content: Any) -> str:
+        if isinstance(content, str):
+            return cls._normalize_text(content)
+        if not isinstance(content, list):
+            return ""
+        text_list: List[str] = []
+        for item in content:
+            if not isinstance(item, dict):
+                continue
+            text = item.get("text") or item.get("c") or ""
+            if text:
+                text_list.append(str(text))
+        return cls._normalize_text("".join(text_list))
+
+    @staticmethod
+    def _api_user_map(api_data: Dict) -> Dict[str, Dict]:
+        return {str(user.get("id")): user for user in api_data.get("user_list", []) if user.get("id")}
+
+    @staticmethod
+    def _api_user_link(user: Dict) -> str:
+        portrait = (user or {}).get("portrait", "")
+        if not portrait:
+            return ""
+        return f"{const.TIEBA_URL}/home/main?id={quote(str(portrait))}"
+
+    @staticmethod
+    def _api_user_avatar(user: Dict) -> str:
+        image_data = (
+            (user or {})
+            .get("user_show_info", {})
+            .get("feed_head", {})
+            .get("image_data", {})
+        )
+        return image_data.get("img_url") or (
+            "https://gss0.bdstatic.com/6LZ1dD3d1sgCo2Kml5_Y_D3/sys/portrait/item/"
+            f"{user.get('portrait', '')}"
+            if user and user.get("portrait")
+            else ""
+        )
+
+    def extract_search_note_list_from_api(self, api_data: Dict) -> List[TiebaNote]:
+        """
+        Extract Tieba post list from current PC search JSON API.
+        """
+        result: List[TiebaNote] = []
+        cards = api_data.get("data", {}).get("card_list", [])
+        for card in cards:
+            if card.get("cardInfo") != "thread" and card.get("cardStyle") != "thread":
+                continue
+            item = card.get("data") or {}
+            note_id = str(item.get("tid") or "")
+            if not note_id:
+                continue
+            user = item.get("user") or {}
+            tieba_name = self._ensure_tieba_suffix(item.get("forum_name") or "")
+            tieba_note = TiebaNote(
+                note_id=note_id,
+                title=self._normalize_text(item.get("title") or ""),
+                desc=self._normalize_text(item.get("content") or ""),
+                note_url=f"{const.TIEBA_URL}/p/{note_id}",
+                publish_time=utils.get_time_str_from_unix_time(
+                    item.get("time") or item.get("create_time") or 0
+                ),
+                user_link="",
+                user_nickname=user.get("show_nickname") or user.get("user_name") or "",
+                user_avatar=user.get("portrait") or user.get("portraith") or "",
+                tieba_name=tieba_name,
+                tieba_link=self._tieba_link_from_name(tieba_name),
+                total_replay_num=item.get("post_num") or 0,
+            )
+            result.append(tieba_note)
+        return result
+
+    def extract_note_detail_from_api(self, api_data: Dict) -> TiebaNote:
+        """
+        Extract Tieba post detail from current PC page_pc JSON API.
+        """
+        thread = api_data.get("thread") or {}
+        first_floor = api_data.get("first_floor") or {}
+        forum = api_data.get("forum") or api_data.get("display_forum") or {}
+        page = api_data.get("page") or {}
+        user_map = self._api_user_map(api_data)
+        author = user_map.get(str(first_floor.get("author_id"))) or {}
+        note_id = str(thread.get("id") or thread.get("tid") or first_floor.get("tid") or "")
+        tieba_name = self._ensure_tieba_suffix(forum.get("name") or "")
+        note = TiebaNote(
+            note_id=note_id,
+            title=self._clean_title(thread.get("title") or first_floor.get("title") or "", tieba_name),
+            desc=self._extract_api_content_text(
+                first_floor.get("content")
+                or thread.get("origin_thread_info", {}).get("abstract")
+                or thread.get("origin_thread_info", {}).get("content")
+            ),
+            note_url=f"{const.TIEBA_URL}/p/{note_id}",
+            publish_time=utils.get_time_str_from_unix_time(
+                first_floor.get("time") or thread.get("create_time") or 0
+            ),
+            user_link=self._api_user_link(author),
+            user_nickname=author.get("name_show") or author.get("name") or "",
+            user_avatar=self._api_user_avatar(author),
+            tieba_name=tieba_name,
+            tieba_link=self._tieba_link_from_name(tieba_name),
+            total_replay_num=thread.get("reply_num") or 0,
+            total_replay_page=page.get("total_page") or 0,
+            ip_location=author.get("ip_address") or "",
+        )
+        return note
+
+    def extract_tieba_note_parent_comments_from_api(
+        self, api_data: Dict, note_detail: TiebaNote
+    ) -> List[TiebaComment]:
+        """
+        Extract first-level comments from current PC page_pc JSON API.
+        """
+        forum = api_data.get("forum") or api_data.get("display_forum") or {}
+        tieba_id = str(forum.get("id") or "")
+        tieba_name = note_detail.tieba_name or self._ensure_tieba_suffix(forum.get("name") or "")
+        tieba_link = note_detail.tieba_link or self._tieba_link_from_name(tieba_name)
+        user_map = self._api_user_map(api_data)
+        result: List[TiebaComment] = []
+        for item in api_data.get("post_list", []):
+            comment_id = str(item.get("id") or "")
+            if not comment_id:
+                continue
+            user = user_map.get(str(item.get("author_id"))) or {}
+            comment = TiebaComment(
+                comment_id=comment_id,
+                sub_comment_count=item.get("sub_post_number") or 0,
+                content=self._extract_api_content_text(item.get("content")),
+                note_url=note_detail.note_url,
+                user_link=self._api_user_link(user),
+                user_nickname=user.get("name_show") or user.get("name") or "",
+                user_avatar=self._api_user_avatar(user),
+                tieba_id=tieba_id,
+                tieba_name=tieba_name,
+                tieba_link=tieba_link,
+                ip_location=user.get("ip_address") or "",
+                publish_time=utils.get_time_str_from_unix_time(item.get("time") or 0),
+                note_id=note_detail.note_id,
+            )
+            result.append(comment)
+        return result
+
+    def extract_creator_info_from_api(self, api_data: Dict) -> TiebaCreator:
+        """
+        Extract Tieba creator information from current PC creator JSON API.
+        """
+        user = api_data.get("data", {}).get("user", {})
+        if not user:
+            raise ValueError(f"Creator API response does not contain user info: {api_data}")
+        gender_value = user.get("sex", user.get("gender", 0))
+        gender = "Unknown"
+        if gender_value == 1:
+            gender = "Male"
+        elif gender_value == 2:
+            gender = "Female"
+
+        return TiebaCreator(
+            user_id=str(user.get("id", "")),
+            user_name=str(user.get("name", "")),
+            nickname=str(user.get("name_show") or user.get("name") or ""),
+            avatar=self._api_user_avatar(user),
+            gender=gender,
+            ip_location=str(user.get("ip_address", "")),
+            follows=int(user.get("concern_num") or 0),
+            fans=int(user.get("fans_num") or 0),
+            registration_duration=str(user.get("tb_age", "")),
+        )
+
+    @staticmethod
+    def extract_creator_thread_id_list_from_api(api_data: Dict) -> List[str]:
+        """
+        Extract creator thread ids from current PC creator feed JSON API.
+        """
+        thread_ids: List[str] = []
+        for item in api_data.get("data", {}).get("list", []):
+            thread_info = item.get("thread_info") or {}
+            thread_id = thread_info.get("tid") or thread_info.get("id")
+            if thread_id:
+                thread_ids.append(str(thread_id))
+        return thread_ids
+
+    def extract_tieba_note_list_from_frs_api(self, api_data: Dict) -> List[TiebaNote]:
+        """
+        Extract Tieba thread ids from current PC forum page JSON API.
+
+        The by-forum command immediately fetches full details for every id, so
+        this list intentionally carries only stable routing fields.
+        """
+        forum = api_data.get("forum", {})
+        tieba_name = self._ensure_tieba_suffix(forum.get("name") or "")
+        tieba_link = self._tieba_link_from_name(tieba_name)
+        tids = [
+            tid.strip()
+            for tid in str(forum.get("tids") or "").split(",")
+            if tid.strip()
+        ]
+        return [
+            TiebaNote(
+                note_id=tid,
+                title="",
+                desc="",
+                note_url=f"{const.TIEBA_URL}/p/{tid}",
+                tieba_name=tieba_name,
+                tieba_link=tieba_link,
+            )
+            for tid in tids
+        ]
+
+    @staticmethod
+    def _decode_js_string(value: str) -> str:
+        if not value or value == "null":
+            return ""
+        try:
+            decoded_value = json.loads(f'"{value}"')
+            return decoded_value if isinstance(decoded_value, str) else str(decoded_value)
+        except Exception:
+            return value
+
+    @classmethod
+    def _extract_forum_info(cls, selector: Selector, page_content: str) -> Tuple[str, str]:
+        forum_xpath = f"//a[{cls._class_contains('card_title_fname')}]"
+        forum_link_selector = selector.xpath(forum_xpath)
+        tieba_name = cls._selector_text(selector, forum_xpath)
+        tieba_link = cls._absolute_url(forum_link_selector.xpath("./@href").get(default=""))
+
+        if not tieba_name:
+            patterns = [
+                r"PageData\.forum\s*=\s*\{.*?['\"]name['\"]\s*:\s*\"([^\"\\\\]*(?:\\\\.[^\"\\\\]*)*)\"",
+                r'"forum_name"\s*:\s*"([^"\\\\]*(?:\\\\.[^"\\\\]*)*)"',
+                r'"kw"\s*:\s*"([^"\\\\]*(?:\\\\.[^"\\\\]*)*)"',
+            ]
+            for pattern in patterns:
+                match = re.search(pattern, page_content, re.S)
+                if match:
+                    tieba_name = cls._decode_js_string(match.group(1))
+                    if tieba_name:
+                        break
+
+        if not tieba_name:
+            title = selector.xpath("//title/text()").get(default="")
+            match = re.search(r"(.+?)吧[-_]", title)
+            if match:
+                tieba_name = cls._normalize_text(match.group(1))
+
+        if not tieba_link and tieba_name:
+            tieba_link = f"{const.TIEBA_URL}/f?kw={quote(tieba_name.removesuffix('吧'))}"
+
+        return tieba_name, tieba_link or const.TIEBA_URL
+
+    @classmethod
+    def _clean_title(cls, title: str, tieba_name: str = "") -> str:
+        title = cls._normalize_text(title)
+        title = re.sub(r"_(?:百度贴吧|Baidu Tieba)$", "", title).strip()
+        for name in {tieba_name, tieba_name.removesuffix("吧")}:
+            if name:
+                title = title.replace(f"【{name}】", "").strip()
+        return title
+
     @staticmethod
     def extract_search_note_list(page_content: str) -> List[TiebaNote]:
         """
-        提取贴吧帖子列表，这里提取的关键词搜索结果页的数据，还缺少帖子的回复数和回复页等数据
+        Extract Tieba post list from keyword search result pages, still missing reply count and reply page data
         Args:
-            page_content: 页面内容的HTML字符串
+            page_content: HTML string of page content
 
         Returns:
-            包含帖子信息的字典列表
+            List of Tieba post objects
         """
-        xpath_selector = "//div[@class='s_post']"
-        post_list = Selector(text=page_content).xpath(xpath_selector)
+        extractor = TieBaExtractor()
+        selector = Selector(text=page_content)
+        post_list = selector.xpath(
+            f"//div[{extractor._class_contains('s_post')}]"
+        )
         result: List[TiebaNote] = []
         for post in post_list:
-            tieba_note = TiebaNote(note_id=post.xpath(".//span[@class='p_title']/a/@data-tid").get(default='').strip(),
-                                   title=post.xpath(".//span[@class='p_title']/a/text()").get(default='').strip(),
-                                   desc=post.xpath(".//div[@class='p_content']/text()").get(default='').strip(),
-                                   note_url=const.TIEBA_URL + post.xpath(".//span[@class='p_title']/a/@href").get(
-                                       default=''),
-                                   user_nickname=post.xpath(".//a[starts-with(@href, '/home/main')]/font/text()").get(
-                                       default='').strip(), user_link=const.TIEBA_URL + post.xpath(
-                    ".//a[starts-with(@href, '/home/main')]/@href").get(default=''),
-                                   tieba_name=post.xpath(".//a[@class='p_forum']/font/text()").get(default='').strip(),
-                                   tieba_link=const.TIEBA_URL + post.xpath(".//a[@class='p_forum']/@href").get(
-                                       default=''),
-                                   publish_time=post.xpath(".//font[@class='p_green p_date']/text()").get(
-                                       default='').strip(), )
+            title_link = post.xpath(".//*[contains(@class, 'p_title')]//a[1]")
+            note_url = extractor._absolute_url(title_link.xpath("./@href").get(default=""))
+            note_id = title_link.xpath("./@data-tid").get(default="").strip()
+            if not note_id:
+                note_id = extractor._extract_note_id_from_url(note_url)
+            user_selector = post.xpath(".//a[contains(@href, '/home/main')][1]")
+            forum_selector = post.xpath(f".//a[{extractor._class_contains('p_forum')}][1]")
+            tieba_note = TiebaNote(
+                note_id=note_id,
+                title=extractor._selector_text(post, ".//*[contains(@class, 'p_title')]//a[1]"),
+                desc=extractor._selector_text(
+                    post, f".//div[{extractor._class_contains('p_content')}]"
+                ),
+                note_url=note_url,
+                user_nickname=extractor._selector_text(
+                    post, ".//a[contains(@href, '/home/main')][1]"
+                ),
+                user_link=extractor._absolute_url(user_selector.xpath("./@href").get(default="")),
+                tieba_name=extractor._selector_text(
+                    post, f".//a[{extractor._class_contains('p_forum')}][1]"
+                ),
+                tieba_link=extractor._absolute_url(forum_selector.xpath("./@href").get(default="")),
+                publish_time=extractor._selector_text(
+                    post, ".//*[contains(@class, 'p_date')][1]"
+                ),
+            )
+            result.append(tieba_note)
+        if result:
+            return result
+
+        # Tieba search changed to a PC feed/card layout in 2026. The old
+        # s_post nodes disappeared, while each search result now lives in a
+        # threadcardclass card with overlay links to /p/<thread_id>.
+        post_list = selector.xpath(
+            f"//*[contains(concat(' ', normalize-space(@class), ' '), ' threadcardclass ') "
+            f"and .//a[contains(@href, '/p/')]]"
+        )
+        seen_note_ids = set()
+        for post in post_list:
+            title_link = post.xpath(
+                f".//a[{extractor._class_contains('action-link-bg')} and contains(@href, '/p/')][1]"
+                f"|.//a[contains(@href, '/p/')][1]"
+            )
+            note_url = extractor._absolute_url(title_link.xpath("./@href").get(default=""))
+            note_id = extractor._extract_note_id_from_url(note_url)
+            if not note_id or note_id in seen_note_ids:
+                continue
+            seen_note_ids.add(note_id)
+
+            tieba_name = extractor._selector_text(
+                post, f".//*[{extractor._class_contains('forum-name-text')}][1]"
+            )
+            tieba_link = ""
+            forum_link = post.xpath(".//a[contains(@href, '/f?')][1]/@href").get(default="")
+            if forum_link:
+                tieba_link = extractor._absolute_url(forum_link)
+            elif tieba_name:
+                tieba_keyword = tieba_name.removesuffix("吧")
+                tieba_link = f"{const.TIEBA_URL}/f?kw={quote(tieba_keyword)}"
+            else:
+                tieba_link = const.TIEBA_URL
+
+            publish_time = ""
+            top_title_text = extractor._selector_text(
+                post, f".//*[{extractor._class_contains('top-title')}][1]"
+            )
+            publish_match = re.search(r"发布于\s*([^\s]+)", top_title_text)
+            if publish_match:
+                publish_time = publish_match.group(1)
+
+            title = extractor._selector_text(
+                post, f".//*[{extractor._class_contains('title-wrap')}][1]"
+            )
+            desc = extractor._selector_text(
+                post, f".//*[{extractor._class_contains('abstract-wrap')}][1]"
+            )
+            if not title:
+                title = extractor._normalize_text(desc[:80])
+
+            user_nickname = extractor._selector_text(
+                post, f".//*[{extractor._class_contains('forum-attention')}][1]"
+            )
+            if not user_nickname and publish_time:
+                user_nickname = extractor._normalize_text(
+                    top_title_text.split("发布于", 1)[0]
+                )
+
+            comment_text = extractor._selector_text(
+                post, f".//a[{extractor._class_contains('comment-link-zone')}][1]"
+            )
+            tieba_note = TiebaNote(
+                note_id=note_id,
+                title=title,
+                desc=desc,
+                note_url=f"{const.TIEBA_URL}/p/{note_id}",
+                user_nickname=user_nickname,
+                user_link="",
+                tieba_name=tieba_name,
+                tieba_link=tieba_link,
+                publish_time=publish_time,
+                total_replay_num=extractor._text_to_int(comment_text),
+            )
             result.append(tieba_note)
         return result
 
     def extract_tieba_note_list(self, page_content: str) -> List[TiebaNote]:
         """
-        提取贴吧帖子列表
+        Extract Tieba post list from Tieba page
         Args:
-            page_content:
+            page_content: HTML string of page content
 
         Returns:
-
+            List of Tieba post objects
         """
         page_content = page_content.replace('<!--', "")
         content_selector = Selector(text=page_content)
-        xpath_selector = "//ul[@id='thread_list']/li"
+        xpath_selector = f"//ul[@id='thread_list']/li[{self._class_contains('j_thread_list')}]"
         post_list = content_selector.xpath(xpath_selector)
+        tieba_name, tieba_link = self._extract_forum_info(content_selector, page_content)
         result: List[TiebaNote] = []
         for post_selector in post_list:
             post_field_value: Dict = self.extract_data_field_value(post_selector)
             if not post_field_value:
                 continue
             note_id = str(post_field_value.get("id"))
-            tieba_note = TiebaNote(note_id=note_id,
-                                   title=post_selector.xpath(".//a[@class='j_th_tit ']/text()").get(default='').strip(),
-                                   desc=post_selector.xpath(
-                                       ".//div[@class='threadlist_abs threadlist_abs_onlyline ']/text()").get(
-                                       default='').strip(), note_url=const.TIEBA_URL + f"/p/{note_id}",
-                                   user_link=const.TIEBA_URL + post_selector.xpath(
-                                       ".//a[@class='frs-author-name j_user_card ']/@href").get(default='').strip(),
-                                   user_nickname=post_field_value.get("authoer_nickname") or post_field_value.get(
-                                       "author_name"),
-                                   tieba_name=content_selector.xpath("//a[@class='card_title_fname']/text()").get(
-                                       default='').strip(), tieba_link=const.TIEBA_URL + content_selector.xpath(
-                    "//a[@class='card_title_fname']/@href").get(default=''),
-                                   total_replay_num=post_field_value.get("reply_num", 0))
+            user_selector = post_selector.xpath(f".//a[{self._class_contains('frs-author-name')}][1]")
+            title = self._selector_text(post_selector, f".//a[{self._class_contains('j_th_tit')}][1]")
+            if not title:
+                title = self._selector_text(post_selector, f".//*[{self._class_contains('threadlist_title')}]//a[1]")
+            user_nickname = (
+                post_field_value.get("author_nickname")
+                or post_field_value.get("author_name")
+                or self._selector_text(
+                    post_selector, f".//a[{self._class_contains('frs-author-name')}][1]"
+                )
+            )
+            tieba_note = TiebaNote(
+                note_id=note_id,
+                title=title,
+                desc=self._selector_text(
+                    post_selector, f".//div[{self._class_contains('threadlist_abs')}]"
+                ),
+                note_url=const.TIEBA_URL + f"/p/{note_id}",
+                user_link=self._absolute_url(user_selector.xpath("./@href").get(default="")),
+                user_nickname=user_nickname,
+                tieba_name=tieba_name,
+                tieba_link=tieba_link,
+                total_replay_num=post_field_value.get("reply_num", 0),
+            )
             result.append(tieba_note)
         return result
 
     def extract_note_detail(self, page_content: str) -> TiebaNote:
         """
-        提取贴吧帖子详情
+        Extract Tieba post details from post detail page
         Args:
-            page_content:
+            page_content: HTML string of page content
 
         Returns:
-
+            Tieba post detail object
         """
         content_selector = Selector(text=page_content)
-        first_floor_selector = content_selector.xpath("//div[@class='p_postlist'][1]")
+        first_floor_selector = content_selector.xpath(
+            f"//div[{self._class_contains('l_post')} and {self._class_contains('j_l_post')}][1]"
+        )
         only_view_author_link = content_selector.xpath("//*[@id='lzonly_cntn']/@href").get(default='').strip()
         note_id = only_view_author_link.split("?")[0].split("/")[-1]
-        # 帖子回复数、回复页数
+        if not note_id:
+            note_id_match = re.search(r'"thread_id"\s*:\s*"?(\d+)"?', page_content)
+            note_id = note_id_match.group(1) if note_id_match else ""
+        # Post reply count and reply page count
         thread_num_infos = content_selector.xpath(
-            "//div[@id='thread_theme_5']//li[@class='l_reply_num']//span[@class='red']")
-        # IP地理位置、发表时间
-        other_info_content = content_selector.xpath(".//div[@class='post-tail-wrap']").get(default="").strip()
+            f"//div[@id='thread_theme_5']//li[{self._class_contains('l_reply_num')}]"
+            f"//span[{self._class_contains('red')}]"
+        )
+        # IP location and publish time
+        other_info_content = first_floor_selector.xpath(
+            f".//div[{self._class_contains('post-tail-wrap')}]"
+        ).get(default="").strip()
         ip_location, publish_time = self.extract_ip_and_pub_time(other_info_content)
-        note = TiebaNote(note_id=note_id, title=content_selector.xpath("//title/text()").get(default='').strip(),
-                         desc=content_selector.xpath("//meta[@name='description']/@content").get(default='').strip(),
-                         note_url=const.TIEBA_URL + f"/p/{note_id}",
-                         user_link=const.TIEBA_URL + first_floor_selector.xpath(
-                             ".//a[@class='p_author_face ']/@href").get(default='').strip(),
-                         user_nickname=first_floor_selector.xpath(
-                             ".//a[@class='p_author_name j_user_card']/text()").get(default='').strip(),
-                         user_avatar=first_floor_selector.xpath(".//a[@class='p_author_face ']/img/@src").get(
-                             default='').strip(),
-                         tieba_name=content_selector.xpath("//a[@class='card_title_fname']/text()").get(
-                             default='').strip(), tieba_link=const.TIEBA_URL + content_selector.xpath(
-                "//a[@class='card_title_fname']/@href").get(default=''), ip_location=ip_location,
-                         publish_time=publish_time,
-                         total_replay_num=thread_num_infos[0].xpath("./text()").get(default='').strip(),
-                         total_replay_page=thread_num_infos[1].xpath("./text()").get(default='').strip(), )
-        note.title = note.title.replace(f"【{note.tieba_name}】_百度贴吧", "")
+        tieba_name, tieba_link = self._extract_forum_info(content_selector, page_content)
+        first_floor_value = self.extract_data_field_value(first_floor_selector)
+        author_value = first_floor_value.get("author", {}) if first_floor_value else {}
+        author_link = first_floor_selector.xpath(
+            f".//a[{self._class_contains('p_author_face')} "
+            f"or {self._class_contains('p_author_name')}]/@href"
+        ).get(default="")
+        note = TiebaNote(
+            note_id=note_id,
+            title=content_selector.xpath("//title/text()").get(default="").strip(),
+            desc=content_selector.xpath("//meta[@name='description']/@content").get(default="").strip(),
+            note_url=const.TIEBA_URL + f"/p/{note_id}",
+            user_link=self._absolute_url(author_link),
+            user_nickname=(
+                self._selector_text(first_floor_selector, f".//a[{self._class_contains('p_author_name')}][1]")
+                or author_value.get("user_nickname")
+                or author_value.get("user_name", "")
+            ),
+            user_avatar=first_floor_selector.xpath(
+                f".//a[{self._class_contains('p_author_face')}]//img/@src"
+            ).get(default="").strip(),
+            tieba_name=tieba_name,
+            tieba_link=tieba_link,
+            ip_location=ip_location,
+            publish_time=publish_time,
+            total_replay_num=(
+                thread_num_infos[0].xpath("./text()").get(default="0").strip()
+                if len(thread_num_infos) > 0 else 0
+            ),
+            total_replay_page=(
+                thread_num_infos[1].xpath("./text()").get(default="0").strip()
+                if len(thread_num_infos) > 1 else 0
+            ),
+        )
+        note.title = self._clean_title(note.title, note.tieba_name)
         return note
 
     def extract_tieba_note_parment_comments(self, page_content: str, note_id: str) -> List[TiebaComment]:
         """
-        提取贴吧帖子一级评论
+        Extract Tieba post first-level comments from comment page
         Args:
-            page_content:
-            note_id:
+            page_content: HTML string of page content
+            note_id: Post ID
 
         Returns:
-
+            List of first-level comment objects
         """
-        xpath_selector = "//div[@class='l_post l_post_bright j_l_post clearfix  ']"
+        xpath_selector = f"//div[{self._class_contains('l_post')} and {self._class_contains('j_l_post')}]"
         comment_list = Selector(text=page_content).xpath(xpath_selector)
+        content_selector = Selector(text=page_content)
+        tieba_name, tieba_link = self._extract_forum_info(content_selector, page_content)
         result: List[TiebaComment] = []
         for comment_selector in comment_list:
             comment_field_value: Dict = self.extract_data_field_value(comment_selector)
-            if not comment_field_value:
+            comment_content_value = comment_field_value.get("content", {}) if comment_field_value else {}
+            if not comment_content_value:
                 continue
-            tieba_name = comment_selector.xpath("//a[@class='card_title_fname']/text()").get(default='').strip()
-            other_info_content = comment_selector.xpath(".//div[@class='post-tail-wrap']").get(default="").strip()
+            other_info_content = comment_selector.xpath(
+                f".//div[{self._class_contains('post-tail-wrap')}]"
+            ).get(default="").strip()
             ip_location, publish_time = self.extract_ip_and_pub_time(other_info_content)
-            tieba_comment = TiebaComment(comment_id=str(comment_field_value.get("content").get("post_id")),
-                                         sub_comment_count=comment_field_value.get("content").get("comment_num"),
-                                         content=utils.extract_text_from_html(
-                                             comment_field_value.get("content").get("content")),
-                                         note_url=const.TIEBA_URL + f"/p/{note_id}",
-                                         user_link=const.TIEBA_URL + comment_selector.xpath(
-                                             ".//a[@class='p_author_face ']/@href").get(default='').strip(),
-                                         user_nickname=comment_selector.xpath(
-                                             ".//a[@class='p_author_name j_user_card']/text()").get(default='').strip(),
-                                         user_avatar=comment_selector.xpath(
-                                             ".//a[@class='p_author_face ']/img/@src").get(default='').strip(),
-                                         tieba_id=str(comment_field_value.get("content").get("forum_id", "")),
-                                         tieba_name=tieba_name, tieba_link=f"https://tieba.baidu.com/f?kw={tieba_name}",
-                                         ip_location=ip_location, publish_time=publish_time, note_id=note_id, )
+            user_selector = comment_selector.xpath(f".//a[{self._class_contains('p_author_name')}][1]")
+            user_avatar = comment_selector.xpath(
+                f".//a[{self._class_contains('p_author_face')}]//img/@src"
+            ).get(default="").strip()
+            if not user_avatar and comment_field_value.get("author", {}).get("portrait"):
+                portrait = comment_field_value["author"]["portrait"]
+                user_avatar = (
+                    "https://gss0.bdstatic.com/6LZ1dD3d1sgCo2Kml5_Y_D3/sys/portrait/item/"
+                    f"{portrait}"
+                )
+            content_html = comment_content_value.get("content") or comment_selector.xpath(
+                f".//div[{self._class_contains('d_post_content')}]"
+            ).get(default="")
+            user_nickname = (
+                self._selector_text(comment_selector, f".//a[{self._class_contains('p_author_name')}][1]")
+                or comment_field_value.get("author", {}).get("user_nickname")
+                or comment_field_value.get("author", {}).get("user_name", "")
+            )
+            tieba_comment = TiebaComment(
+                comment_id=str(
+                    comment_content_value.get("post_id")
+                    or comment_selector.xpath("./@data-pid").get(default="")
+                ),
+                sub_comment_count=comment_content_value.get("comment_num") or 0,
+                content=utils.extract_text_from_html(content_html),
+                note_url=const.TIEBA_URL + f"/p/{note_id}",
+                user_link=self._absolute_url(user_selector.xpath("./@href").get(default="")),
+                user_nickname=user_nickname,
+                user_avatar=user_avatar,
+                tieba_id=str(comment_content_value.get("forum_id", "")),
+                tieba_name=tieba_name,
+                tieba_link=tieba_link,
+                ip_location=ip_location,
+                publish_time=publish_time,
+                note_id=note_id,
+            )
             result.append(tieba_comment)
         return result
 
     def extract_tieba_note_sub_comments(self, page_content: str, parent_comment: TiebaComment) -> List[TiebaComment]:
         """
-        提取贴吧帖子二级评论
+        Extract Tieba post second-level comments from sub-comment page
         Args:
-            page_content:
-            parent_comment:
+            page_content: HTML string of page content
+            parent_comment: Parent comment object
 
         Returns:
-
+            List of second-level comment objects
         """
         selector = Selector(page_content)
         comments = []
-        comment_ele_list = selector.xpath("//li[@class='lzl_single_post j_lzl_s_p first_no_border']")
-        comment_ele_list.extend(selector.xpath("//li[@class='lzl_single_post j_lzl_s_p ']"))
+        comment_ele_list = selector.xpath(
+            f"//li[{self._class_contains('lzl_single_post')} and {self._class_contains('j_lzl_s_p')}]"
+        )
         for comment_ele in comment_ele_list:
             comment_value = self.extract_data_field_value(comment_ele)
             if not comment_value:
                 continue
-            comment_user_a_selector = comment_ele.xpath("./a[@class='j_user_card lzl_p_p']")[0]
+            comment_user_a_selector = comment_ele.xpath(
+                f"./a[{self._class_contains('j_user_card')} and {self._class_contains('lzl_p_p')}][1]"
+            )
             content = utils.extract_text_from_html(
-                comment_ele.xpath(".//span[@class='lzl_content_main']").get(default=""))
+                comment_ele.xpath(f".//span[{self._class_contains('lzl_content_main')}]").get(default=""))
             comment = TiebaComment(
                 comment_id=str(comment_value.get("spid")), content=content,
-                user_link=comment_user_a_selector.xpath("./@href").get(default=""),
-                user_nickname=comment_value.get("showname"),
+                user_link=self._absolute_url(comment_user_a_selector.xpath("./@href").get(default="")),
+                user_nickname=str(comment_value.get("showname") or ""),
                 user_avatar=comment_user_a_selector.xpath("./img/@src").get(default=""),
-                publish_time=comment_ele.xpath(".//span[@class='lzl_time']/text()").get(default="").strip(),
+                publish_time=self._selector_text(comment_ele, f".//span[{self._class_contains('lzl_time')}]"),
                 parent_comment_id=parent_comment.comment_id,
                 note_id=parent_comment.note_id, note_url=parent_comment.note_url,
                 tieba_id=parent_comment.tieba_id, tieba_name=parent_comment.tieba_name,
@@ -215,12 +676,12 @@ class TieBaExtractor:
 
     def extract_creator_info(self, html_content: str) -> TiebaCreator:
         """
-        提取贴吧创作者信息
+        Extract Tieba creator information from creator homepage
         Args:
-            html_content:
+            html_content: HTML string of creator homepage
 
         Returns:
-
+            Tieba creator object
         """
         selector = Selector(text=html_content)
         user_link_selector = selector.xpath("//p[@class='space']/a")
@@ -251,12 +712,12 @@ class TieBaExtractor:
         html_content: str
     ) -> List[str]:
         """
-        提取贴吧创作者主页的帖子列表
+        Extract post ID list from Tieba creator's homepage
         Args:
-            html_content:
+            html_content: HTML string of creator homepage
 
         Returns:
-
+            List of post IDs
         """
         selector = Selector(text=html_content)
         thread_id_list = []
@@ -271,12 +732,12 @@ class TieBaExtractor:
 
     def extract_ip_and_pub_time(self, html_content: str) -> Tuple[str, str]:
         """
-        提取IP位置和发布时间
+        Extract IP location and publish time from HTML content
         Args:
-            html_content:
+            html_content: HTML string
 
         Returns:
-
+            Tuple of (IP location, publish time)
         """
         pattern_pub_time = re.compile(r'<span class="tail-info">(\d{4}-\d{2}-\d{2} \d{2}:\d{2})</span>')
         time_match = pattern_pub_time.search(html_content)
@@ -286,12 +747,12 @@ class TieBaExtractor:
     @staticmethod
     def extract_ip(html_content: str) -> str:
         """
-        提取IP
+        Extract IP location from HTML content
         Args:
-            html_content:
+            html_content: HTML string
 
         Returns:
-
+            IP location string
         """
         pattern_ip = re.compile(r'IP属地:(\S+)</span>')
         ip_match = pattern_ip.search(html_content)
@@ -301,28 +762,28 @@ class TieBaExtractor:
     @staticmethod
     def extract_gender(html_content: str) -> str:
         """
-        提取性别
+        Extract gender from HTML content
         Args:
-            html_content:
+            html_content: HTML string
 
         Returns:
-
+            Gender string ('Male', 'Female', or 'Unknown')
         """
         if GENDER_MALE in html_content:
-            return '男'
+            return 'Male'
         elif GENDER_FEMALE in html_content:
-            return '女'
-        return '未知'
+            return 'Female'
+        return 'Unknown'
 
     @staticmethod
     def extract_follow_and_fans(selectors: List[Selector]) -> Tuple[str, str]:
         """
-        提取关注数和粉丝数
+        Extract follow count and fan count from selectors
         Args:
-            selectors:
+            selectors: List of selector objects
 
         Returns:
-
+            Tuple of (follow count, fan count)
         """
         pattern = re.compile(r'<span class="concern_num">\(<a[^>]*>(\d+)</a>\)</span>')
         follow_match = pattern.findall(selectors[0].get())
@@ -334,9 +795,15 @@ class TieBaExtractor:
     @staticmethod
     def extract_registration_duration(html_content: str) -> str:
         """
-        "<span>吧龄:1.9年</span>"
-        Returns: 1.9年
+        Extract Tieba age from HTML content
+        Example: "<span>吧龄:1.9年</span>"
+        Returns: "1.9年"
 
+        Args:
+            html_content: HTML string
+
+        Returns:
+            Tieba age string
         """
         pattern = re.compile(r'<span>吧龄:(\S+)</span>')
         match = pattern.search(html_content)
@@ -345,22 +812,22 @@ class TieBaExtractor:
     @staticmethod
     def extract_data_field_value(selector: Selector) -> Dict:
         """
-        提取data-field的值
+        Extract data-field value from selector
         Args:
-            selector:
+            selector: Selector object
 
         Returns:
-
+            Dictionary containing data-field value
         """
         data_field_value = selector.xpath("./@data-field").get(default='').strip()
         if not data_field_value or data_field_value == "{}":
             return {}
         try:
-            # 先使用 html.unescape 处理转义字符 再json.loads 将 JSON 字符串转换为 Python 字典
+            # First use html.unescape to handle escape characters, then json.loads to convert JSON string to Python dictionary
             unescaped_json_str = html.unescape(data_field_value)
             data_field_dict_value = json.loads(unescaped_json_str)
         except Exception as ex:
-            print(f"extract_data_field_value，错误信息：{ex}, 尝试使用其他方式解析")
+            print(f"extract_data_field_value, error: {ex}, trying alternative parsing method")
             data_field_dict_value = {}
         return data_field_dict_value
 

media_platform/tieba/login.py

@@ -50,7 +50,7 @@ class BaiduTieBaLogin(AbstractLogin):
     @retry(stop=stop_after_attempt(600), wait=wait_fixed(1), retry=retry_if_result(lambda value: value is False))
     async def check_login_state(self) -> bool:
         """
-        轮训检查登录状态是否成功，成功返回True否则返回False
+        Poll to check if login status is successful, return True if successful, otherwise return False
 
         Returns:
 

media_platform/weibo/client.py

@@ -20,7 +20,7 @@
 # -*- coding: utf-8 -*-
 # @Author  : relakkes@gmail.com
 # @Time    : 2023/12/23 15:40
-# @Desc    : 微博爬虫 API 请求 client
+# @Desc    : Weibo crawler API request client
 
 import asyncio
 import copy
@@ -32,6 +32,7 @@ from urllib.parse import parse_qs, unquote, urlencode
 import httpx
 from httpx import Response
 from playwright.async_api import BrowserContext, Page
+from tools.httpx_util import make_async_client
 from tenacity import retry, stop_after_attempt, wait_fixed
 
 import config
@@ -49,7 +50,7 @@ class WeiboClient(ProxyRefreshMixin):
 
     def __init__(
         self,
-        timeout=60,  # 若开启爬取媒体选项，weibo 的图片需要更久的超时时间
+        timeout=60,  # If media crawling is enabled, Weibo images need a longer timeout
         proxy=None,
         *,
         headers: Dict[str, str],
@@ -61,19 +62,20 @@ class WeiboClient(ProxyRefreshMixin):
         self.timeout = timeout
         self.headers = headers
         self._host = "https://m.weibo.cn"
+        self.cookie_urls = [self._host]
         self.playwright_page = playwright_page
         self.cookie_dict = cookie_dict
         self._image_agent_host = "https://i1.wp.com/"
-        # 初始化代理池（来自 ProxyRefreshMixin）
+        # Initialize proxy pool (from ProxyRefreshMixin)
         self.init_proxy_pool(proxy_ip_pool)
 
     @retry(stop=stop_after_attempt(5), wait=wait_fixed(3))
     async def request(self, method, url, **kwargs) -> Union[Response, Dict]:
-        # 每次请求前检测代理是否过期
+        # Check if proxy is expired before each request
         await self._refresh_proxy_if_expired()
 
         enable_return_response = kwargs.pop("return_response", False)
-        async with httpx.AsyncClient(proxy=self.proxy) as client:
+        async with make_async_client(proxy=self.proxy) as client:
             response = await client.request(method, url, timeout=self.timeout, **kwargs)
 
         if enable_return_response:
@@ -82,7 +84,7 @@ class WeiboClient(ProxyRefreshMixin):
         try:
             data: Dict = response.json()
         except json.decoder.JSONDecodeError:
-            # issue: #771 搜索接口会报错432， 多次重试 + 更新 h5 cookies
+            # issue: #771 Search API returns error 432, retry multiple times + update h5 cookies
             utils.logger.error(f"[WeiboClient.request] request {method}:{url} err code: {response.status_code} res:{response.text}")
             await self.playwright_page.goto(self._host)
             await asyncio.sleep(2)
@@ -136,17 +138,16 @@ class WeiboClient(ProxyRefreshMixin):
         :param urls: Optional list of URLs to filter cookies (e.g., ["https://m.weibo.cn"])
                      If provided, only cookies for these URLs will be retrieved
         """
-        if urls:
-            cookies = await browser_context.cookies(urls=urls)
-            utils.logger.info(f"[WeiboClient.update_cookies] Updating cookies for specific URLs: {urls}")
-        else:
-            cookies = await browser_context.cookies()
-            utils.logger.info("[WeiboClient.update_cookies] Updating all cookies")
-
-        cookie_str, cookie_dict = utils.convert_cookies(cookies)
+        cookie_urls = urls or self.cookie_urls
+        cookie_str, cookie_dict = await utils.convert_browser_context_cookies(
+            browser_context,
+            urls=cookie_urls,
+        )
         self.headers["Cookie"] = cookie_str
         self.cookie_dict = cookie_dict
-        utils.logger.info(f"[WeiboClient.update_cookies] Cookie updated successfully, total: {len(cookie_dict)} cookies")
+        utils.logger.info(
+            f"[WeiboClient.update_cookies] Cookie updated successfully for {cookie_urls}, total: {len(cookie_dict)} cookies"
+        )
 
     async def get_note_by_keyword(
         self,
@@ -156,9 +157,9 @@ class WeiboClient(ProxyRefreshMixin):
     ) -> Dict:
         """
         search note by keyword
-        :param keyword: 微博搜搜的关键词
-        :param page: 分页参数 -当前页码
-        :param search_type: 搜索的类型，见 weibo/filed.py 中的枚举SearchType
+        :param keyword: Search keyword for Weibo
+        :param page: Pagination parameter - current page number
+        :param search_type: Search type, see SearchType enum in weibo/field.py
         :return:
         """
         uri = "/api/container/getIndex"
@@ -172,9 +173,9 @@ class WeiboClient(ProxyRefreshMixin):
 
     async def get_note_comments(self, mid_id: str, max_id: int, max_id_type: int = 0) -> Dict:
         """get notes comments
-        :param mid_id: 微博ID
-        :param max_id: 分页参数ID
-        :param max_id_type: 分页参数ID类型
+        :param mid_id: Weibo ID
+        :param max_id: Pagination parameter ID
+        :param max_id_type: Pagination parameter ID type
         :return:
         """
         uri = "/comments/hotflow"
@@ -218,7 +219,7 @@ class WeiboClient(ProxyRefreshMixin):
             is_end = max_id == 0
             if len(result) + len(comment_list) > max_count:
                 comment_list = comment_list[:max_count - len(result)]
-            if callback:  # 如果有回调函数，就执行回调函数
+            if callback:  # If callback function exists, execute it
                 await callback(note_id, comment_list)
             await asyncio.sleep(crawl_interval)
             result.extend(comment_list)
@@ -233,7 +234,7 @@ class WeiboClient(ProxyRefreshMixin):
         callback: Optional[Callable] = None,
     ) -> List[Dict]:
         """
-        获取评论的所有子评论
+        Get all sub-comments of comments
         Args:
             note_id:
             comment_list:
@@ -256,12 +257,12 @@ class WeiboClient(ProxyRefreshMixin):
 
     async def get_note_info_by_id(self, note_id: str) -> Dict:
         """
-        根据帖子ID获取详情
+        Get note details by note ID
         :param note_id:
         :return:
         """
         url = f"{self._host}/detail/{note_id}"
-        async with httpx.AsyncClient(proxy=self.proxy) as client:
+        async with make_async_client(proxy=self.proxy) as client:
             response = await client.request("GET", url, timeout=self.timeout, headers=self.headers)
             if response.status_code != 200:
                 raise DataFetchError(f"get weibo detail err: {response.text}")
@@ -273,25 +274,25 @@ class WeiboClient(ProxyRefreshMixin):
                 note_item = {"mblog": note_detail}
                 return note_item
             else:
-                utils.logger.info(f"[WeiboClient.get_note_info_by_id] 未找到$render_data的值")
+                utils.logger.info(f"[WeiboClient.get_note_info_by_id] $render_data value not found")
                 return dict()
 
     async def get_note_image(self, image_url: str) -> bytes:
-        image_url = image_url[8:]  # 去掉 https://
+        image_url = image_url[8:]  # Remove https://
         sub_url = image_url.split("/")
         image_url = ""
         for i in range(len(sub_url)):
             if i == 1:
-                image_url += "large/"  # 都获取高清大图
+                image_url += "large/"  # Get high-resolution images
             elif i == len(sub_url) - 1:
                 image_url += sub_url[i]
             else:
                 image_url += sub_url[i] + "/"
-        # 微博图床对外存在防盗链，所以需要代理访问
-        # 由于微博图片是通过 i1.wp.com 来访问的，所以需要拼接一下
+        # Weibo image hosting has anti-hotlinking, so proxy access is needed
+        # Since Weibo images are accessed through i1.wp.com, we need to concatenate the URL
         final_uri = (f"{self._image_agent_host}"
                      f"{image_url}")
-        async with httpx.AsyncClient(proxy=self.proxy) as client:
+        async with make_async_client(proxy=self.proxy) as client:
             try:
                 response = await client.request("GET", final_uri, timeout=self.timeout)
                 response.raise_for_status()
@@ -301,18 +302,18 @@ class WeiboClient(ProxyRefreshMixin):
                 else:
                     return response.content
             except httpx.HTTPError as exc:  # some wrong when call httpx.request method, such as connection error, client error, server error or response status code is not 2xx
-                utils.logger.error(f"[DouYinClient.get_aweme_media] {exc.__class__.__name__} for {exc.request.url} - {exc}")    # 保留原始异常类型名称，以便开发者调试
+                utils.logger.error(f"[DouYinClient.get_aweme_media] {exc.__class__.__name__} for {exc.request.url} - {exc}")    # Keep original exception type name for developer debugging
                 return None
 
     async def get_creator_container_info(self, creator_id: str) -> Dict:
         """
-        获取用户的容器ID, 容器信息代表着真实请求的API路径
-            fid_container_id：用户的微博详情API的容器ID
-            lfid_container_id：用户的微博列表API的容器ID
+        Get user's container ID, container information represents the real API request path
+            fid_container_id: Container ID for user's Weibo detail API
+            lfid_container_id: Container ID for user's Weibo list API
         Args:
-            creator_id:
+            creator_id: User ID
 
-        Returns: {
+        Returns: Dictionary with container IDs
 
         """
         response = await self.get(f"/u/{creator_id}", return_response=True)
@@ -324,7 +325,7 @@ class WeiboClient(ProxyRefreshMixin):
 
     async def get_creator_info_by_id(self, creator_id: str) -> Dict:
         """
-        根据用户ID获取用户详情
+        Get user details by user ID
         Args:
             creator_id:
 
@@ -349,11 +350,11 @@ class WeiboClient(ProxyRefreshMixin):
         since_id: str = "0",
     ) -> Dict:
         """
-        获取博主的笔记
+        Get creator's notes
         Args:
-            creator: 博主ID
-            container_id: 容器ID
-            since_id: 上一页最后一条笔记的ID
+            creator: Creator ID
+            container_id: Container ID
+            since_id: ID of the last note from previous page
         Returns:
 
         """
@@ -376,14 +377,14 @@ class WeiboClient(ProxyRefreshMixin):
         callback: Optional[Callable] = None,
     ) -> List[Dict]:
         """
-        获取指定用户下的所有发过的帖子，该方法会一直查找一个用户下的所有帖子信息
+        Get all posts published by a specified user, this method will continuously fetch all posts from a user
         Args:
-            creator_id:
-            container_id:
-            crawl_interval:
-            callback:
+            creator_id: Creator user ID
+            container_id: Container ID for the user
+            crawl_interval: Interval between requests in seconds
+            callback: Optional callback function to process notes
 
-        Returns:
+        Returns: List of all notes
 
         """
         result = []
@@ -393,7 +394,7 @@ class WeiboClient(ProxyRefreshMixin):
         while notes_has_more:
             notes_res = await self.get_notes_by_creator(creator_id, container_id, since_id)
             if not notes_res:
-                utils.logger.error(f"[WeiboClient.get_notes_by_creator] The current creator may have been banned by xhs, so they cannot access the data.")
+                utils.logger.error(f"[WeiboClient.get_notes_by_creator] The current creator may have been banned by Weibo, so they cannot access the data.")
                 break
             since_id = notes_res.get("cardlistInfo", {}).get("since_id", "0")
             if "cards" not in notes_res:

media_platform/weibo/core.py

@@ -20,7 +20,7 @@
 # -*- coding: utf-8 -*-
 # @Author  : relakkes@gmail.com
 # @Time    : 2023/12/23 15:41
-# @Desc    : 微博爬虫主流程代码
+# @Desc    : Weibo crawler main workflow code
 
 import asyncio
 import os
@@ -60,10 +60,11 @@ class WeiboCrawler(AbstractCrawler):
     def __init__(self):
         self.index_url = "https://www.weibo.com"
         self.mobile_index_url = "https://m.weibo.cn"
+        self.cookie_urls = [self.mobile_index_url]
         self.user_agent = utils.get_user_agent()
         self.mobile_user_agent = utils.get_mobile_user_agent()
         self.cdp_manager = None
-        self.ip_proxy_pool = None  # 代理IP池，用于代理自动刷新
+        self.ip_proxy_pool = None  # Proxy IP pool for automatic proxy refresh
 
     async def start(self):
         playwright_proxy_format, httpx_proxy_format = None, None
@@ -73,9 +74,9 @@ class WeiboCrawler(AbstractCrawler):
             playwright_proxy_format, httpx_proxy_format = utils.format_proxy_info(ip_proxy_info)
 
         async with async_playwright() as playwright:
-            # 根据配置选择启动模式
+            # Select launch mode based on configuration
             if config.ENABLE_CDP_MODE:
-                utils.logger.info("[WeiboCrawler] 使用CDP模式启动浏览器")
+                utils.logger.info("[WeiboCrawler] Launching browser with CDP mode")
                 self.browser_context = await self.launch_browser_with_cdp(
                     playwright,
                     playwright_proxy_format,
@@ -83,7 +84,7 @@ class WeiboCrawler(AbstractCrawler):
                     headless=config.CDP_HEADLESS,
                 )
             else:
-                utils.logger.info("[WeiboCrawler] 使用标准模式启动浏览器")
+                utils.logger.info("[WeiboCrawler] Launching browser with standard mode")
                 # Launch a browser context.
                 chromium = playwright.chromium
                 self.browser_context = await self.launch_browser(chromium, None, self.mobile_user_agent, headless=config.HEADLESS)
@@ -109,14 +110,14 @@ class WeiboCrawler(AbstractCrawler):
                 )
                 await login_obj.begin()
 
-                # 登录成功后重定向到手机端的网站，再更新手机端登录成功的cookie
+                # After successful login, redirect to mobile website and update mobile cookies
                 utils.logger.info("[WeiboCrawler.start] redirect weibo mobile homepage and update cookies on mobile platform")
                 await self.context_page.goto(self.mobile_index_url)
                 await asyncio.sleep(3)
-                # 只获取移动端的 cookies，避免 PC 端和移动端 cookies 混淆
+                # Only get mobile cookies to avoid confusion between PC and mobile cookies
                 await self.wb_client.update_cookies(
                     browser_context=self.browser_context,
-                    urls=[self.mobile_index_url]
+                    urls=self.cookie_urls,
                 )
 
             crawler_type_var.set(config.CRAWLER_TYPE)
@@ -170,6 +171,8 @@ class WeiboCrawler(AbstractCrawler):
                 search_res = await self.wb_client.get_note_by_keyword(keyword=keyword, page=page, search_type=search_type)
                 note_id_list: List[str] = []
                 note_list = filter_search_result_card(search_res.get("cards"))
+                # If full text fetching is enabled, batch get full text of posts
+                note_list = await self.batch_get_notes_full_text(note_list)
                 for note_item in note_list:
                     if note_item:
                         mblog: Dict = note_item.get("mblog")
@@ -276,11 +279,18 @@ class WeiboCrawler(AbstractCrawler):
             utils.logger.info(f"[WeiboCrawler.get_note_images] Crawling image mode is not enabled")
             return
 
-        pics: Dict = mblog.get("pics")
+        pics: List = mblog.get("pics")
         if not pics:
             return
         for pic in pics:
-            url = pic.get("url")
+            if isinstance(pic, str):
+                url = pic
+                pid = url.split("/")[-1].split(".")[0]
+            elif isinstance(pic, dict):
+                url = pic.get("url")
+                pid = pic.get("pid", "")
+            else:
+                continue
             if not url:
                 continue
             content = await self.wb_client.get_note_image(url)
@@ -288,7 +298,7 @@ class WeiboCrawler(AbstractCrawler):
             utils.logger.info(f"[WeiboCrawler.get_note_images] Sleeping for {config.CRAWLER_MAX_SLEEP_SEC} seconds after fetching image")
             if content != None:
                 extension_file_name = url.split(".")[-1]
-                await weibo_store.update_weibo_note_image(pic["pid"], content, extension_file_name)
+                await weibo_store.update_weibo_note_image(pid, content, extension_file_name)
 
     async def get_creators_and_notes(self) -> None:
         """
@@ -306,12 +316,18 @@ class WeiboCrawler(AbstractCrawler):
                     raise DataFetchError("Get creator info error")
                 await weibo_store.save_creator(user_id, user_info=createor_info)
 
+                # Create a wrapper callback to get full text before saving data
+                async def save_notes_with_full_text(note_list: List[Dict]):
+                    # If full text fetching is enabled, batch get full text first
+                    updated_note_list = await self.batch_get_notes_full_text(note_list)
+                    await weibo_store.batch_update_weibo_notes(updated_note_list)
+
                 # Get all note information of the creator
                 all_notes_list = await self.wb_client.get_all_notes_by_creator_id(
                     creator_id=user_id,
                     container_id=f"107603{user_id}",
                     crawl_interval=0,
-                    callback=weibo_store.batch_update_weibo_notes,
+                    callback=save_notes_with_full_text,
                 )
 
                 note_ids = [note_item.get("mblog", {}).get("id") for note_item in all_notes_list if note_item.get("mblog", {}).get("id")]
@@ -323,7 +339,10 @@ class WeiboCrawler(AbstractCrawler):
     async def create_weibo_client(self, httpx_proxy: Optional[str]) -> WeiboClient:
         """Create xhs client"""
         utils.logger.info("[WeiboCrawler.create_weibo_client] Begin create weibo API client ...")
-        cookie_str, cookie_dict = utils.convert_cookies(await self.browser_context.cookies(urls=[self.mobile_index_url]))
+        cookie_str, cookie_dict = await utils.convert_browser_context_cookies(
+            self.browser_context,
+            urls=self.cookie_urls,
+        )
         weibo_client_obj = WeiboClient(
             proxy=httpx_proxy,
             headers={
@@ -335,7 +354,7 @@ class WeiboCrawler(AbstractCrawler):
             },
             playwright_page=self.context_page,
             cookie_dict=cookie_dict,
-            proxy_ip_pool=self.ip_proxy_pool,  # 传递代理池用于自动刷新
+            proxy_ip_pool=self.ip_proxy_pool,  # Pass proxy pool for automatic refresh
         )
         return weibo_client_obj
 
@@ -360,7 +379,7 @@ class WeiboCrawler(AbstractCrawler):
                     "height": 1080
                 },
                 user_agent=user_agent,
-                channel="chrome",  # 使用系统的Chrome稳定版
+                channel="chrome",  # Use system's Chrome stable version
             )
             return browser_context
         else:
@@ -376,7 +395,7 @@ class WeiboCrawler(AbstractCrawler):
         headless: bool = True,
     ) -> BrowserContext:
         """
-        使用CDP模式启动浏览器
+        Launch browser with CDP mode
         """
         try:
             self.cdp_manager = CDPBrowserManager()
@@ -387,21 +406,76 @@ class WeiboCrawler(AbstractCrawler):
                 headless=headless,
             )
 
-            # 显示浏览器信息
+            # Display browser information
             browser_info = await self.cdp_manager.get_browser_info()
-            utils.logger.info(f"[WeiboCrawler] CDP浏览器信息: {browser_info}")
+            utils.logger.info(f"[WeiboCrawler] CDP browser info: {browser_info}")
 
             return browser_context
 
         except Exception as e:
-            utils.logger.error(f"[WeiboCrawler] CDP模式启动失败，回退到标准模式: {e}")
-            # 回退到标准模式
+            utils.logger.error(f"[WeiboCrawler] CDP mode startup failed, falling back to standard mode: {e}")
+            # Fallback to standard mode
             chromium = playwright.chromium
             return await self.launch_browser(chromium, playwright_proxy, user_agent, headless)
 
+    async def get_note_full_text(self, note_item: Dict) -> Dict:
+        """
+        Get full text content of a post
+        If the post content is truncated (isLongText=True), request the detail API to get complete content
+        :param note_item: Post data, contains mblog field
+        :return: Updated post data
+        """
+        if not config.ENABLE_WEIBO_FULL_TEXT:
+            return note_item
+
+        mblog = note_item.get("mblog", {})
+        if not mblog:
+            return note_item
+
+        # Check if it's a long text
+        is_long_text = mblog.get("isLongText", False)
+        if not is_long_text:
+            return note_item
+
+        note_id = mblog.get("id")
+        if not note_id:
+            return note_item
+
+        try:
+            utils.logger.info(f"[WeiboCrawler.get_note_full_text] Fetching full text for note: {note_id}")
+            full_note = await self.wb_client.get_note_info_by_id(note_id)
+            if full_note and full_note.get("mblog"):
+                # Replace original content with complete content
+                note_item["mblog"] = full_note["mblog"]
+                utils.logger.info(f"[WeiboCrawler.get_note_full_text] Successfully fetched full text for note: {note_id}")
+
+            # Sleep after request to avoid rate limiting
+            await asyncio.sleep(config.CRAWLER_MAX_SLEEP_SEC)
+        except DataFetchError as ex:
+            utils.logger.error(f"[WeiboCrawler.get_note_full_text] Failed to fetch full text for note {note_id}: {ex}")
+        except Exception as ex:
+            utils.logger.error(f"[WeiboCrawler.get_note_full_text] Unexpected error for note {note_id}: {ex}")
+
+        return note_item
+
+    async def batch_get_notes_full_text(self, note_list: List[Dict]) -> List[Dict]:
+        """
+        Batch get full text content of posts
+        :param note_list: List of posts
+        :return: Updated list of posts
+        """
+        if not config.ENABLE_WEIBO_FULL_TEXT:
+            return note_list
+
+        result = []
+        for note_item in note_list:
+            updated_note = await self.get_note_full_text(note_item)
+            result.append(updated_note)
+        return result
+
     async def close(self):
         """Close browser context"""
-        # 如果使用CDP模式，需要特殊处理
+        # Special handling if using CDP mode
         if self.cdp_manager:
             await self.cdp_manager.cleanup()
             self.cdp_manager = None

media_platform/weibo/field.py

@@ -26,14 +26,14 @@ from enum import Enum
 
 
 class SearchType(Enum):
-    # 综合
+    # Comprehensive
     DEFAULT = "1"
 
-    # 实时
+    # Real-time
     REAL_TIME = "61"
 
-    # 热门
+    # Popular
     POPULAR = "60"
 
-    # 视频
+    # Video
     VIDEO = "64"

media_platform/weibo/help.py

@@ -28,9 +28,9 @@ from typing import Dict, List
 
 def filter_search_result_card(card_list: List[Dict]) -> List[Dict]:
     """
-    过滤微博搜索的结果，只保留card_type为9类型的数据
-    :param card_list:
-    :return:
+    Filter Weibo search results, only keep data with card_type of 9
+    :param card_list: List of card items from search results
+    :return: Filtered list of note items
     """
     note_list: List[Dict] = []
     for card_item in card_list:

media_platform/weibo/login.py

@@ -21,7 +21,7 @@
 # -*- coding: utf-8 -*-
 # @Author  : relakkes@gmail.com
 # @Time    : 2023/12/23 15:42
-# @Desc    : 微博登录实现
+# @Desc    : Weibo login implementation
 
 import asyncio
 import functools

media_platform/xhs/client.py

@@ -20,11 +20,12 @@
 import asyncio
 import json
 from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, Union
-from urllib.parse import urlencode
+from urllib.parse import quote, urlencode
 
 import httpx
 from playwright.async_api import BrowserContext, Page
-from tenacity import retry, stop_after_attempt, wait_fixed
+from tenacity import retry, stop_after_attempt, wait_fixed, retry_if_not_exception_type
+from tools.httpx_util import make_async_client
 
 import config
 from base.base_crawler import AbstractApiClient
@@ -34,18 +35,18 @@ from tools import utils
 if TYPE_CHECKING:
     from proxy.proxy_ip_pool import ProxyIpPool
 
-from .exception import DataFetchError, IPBlockError
+from .exception import DataFetchError, IPBlockError, NoteNotFoundError
 from .field import SearchNoteType, SearchSortType
 from .help import get_search_id
 from .extractor import XiaoHongShuExtractor
-from .playwright_sign import sign_with_playwright
+from .playwright_sign import sign_with_xhshow
 
 
 class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
 
     def __init__(
         self,
-        timeout=60,  # 若开启爬取媒体选项，xhs 的长视频需要更久的超时时间
+        timeout=60,  # If media crawling is enabled, Xiaohongshu long videos need longer timeout
         proxy=None,
         *,
         headers: Dict[str, str],
@@ -56,45 +57,50 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
         self.proxy = proxy
         self.timeout = timeout
         self.headers = headers
-        self._host = "https://edith.xiaohongshu.com"
-        self._domain = "https://www.xiaohongshu.com"
-        self.IP_ERROR_STR = "网络连接异常，请检查网络设置或重启试试"
+        if config.XHS_INTERNATIONAL:
+            self._host = "https://webapi.rednote.com"
+            self._domain = "https://www.rednote.com"
+        else:
+            self._host = "https://edith.xiaohongshu.com"
+            self._domain = "https://www.xiaohongshu.com"
+        self.cookie_urls = [self._domain]
+        self.IP_ERROR_STR = "Network connection error, please check network settings or restart"
         self.IP_ERROR_CODE = 300012
-        self.NOTE_ABNORMAL_STR = "笔记状态异常，请稍后查看"
+        self.NOTE_NOT_FOUND_CODE = -510000
+        self.NOTE_ABNORMAL_STR = "Note status abnormal, please check later"
         self.NOTE_ABNORMAL_CODE = -510001
         self.playwright_page = playwright_page
         self.cookie_dict = cookie_dict
         self._extractor = XiaoHongShuExtractor()
-        # 初始化代理池（来自 ProxyRefreshMixin）
+        # Initialize proxy pool (from ProxyRefreshMixin)
         self.init_proxy_pool(proxy_ip_pool)
 
     async def _pre_headers(self, url: str, params: Optional[Dict] = None, payload: Optional[Dict] = None) -> Dict:
-        """请求头参数签名（使用 playwright 注入方式）
+        """请求头参数签名 (使用 xhshow 纯算法)
 
         Args:
-            url: 请求的URL
-            params: GET请求的参数
-            payload: POST请求的参数
+            url: 请求 URI path
+            params: GET 请求参数
+            payload: POST 请求参数
 
         Returns:
-            Dict: 请求头参数签名
+            Dict: 签名后的请求头参数
         """
-        a1_value = self.cookie_dict.get("a1", "")
-
-        # 确定请求数据和 URI
         if params is not None:
             data = params
+            method = "GET"
         elif payload is not None:
             data = payload
+            method = "POST"
         else:
             raise ValueError("params or payload is required")
 
-        # 使用 playwright 注入方式生成签名
-        signs = await sign_with_playwright(
-            page=self.playwright_page,
+        # 使用 xhshow 纯算法生成签名
+        signs = sign_with_xhshow(
             uri=url,
             data=data,
-            a1=a1_value,
+            cookie_str=self.headers.get("Cookie", ""),
+            method=method,
         )
 
         headers = {
@@ -106,31 +112,31 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
         self.headers.update(headers)
         return self.headers
 
-    @retry(stop=stop_after_attempt(3), wait=wait_fixed(1))
+    @retry(stop=stop_after_attempt(3), wait=wait_fixed(1), retry=retry_if_not_exception_type(NoteNotFoundError))
     async def request(self, method, url, **kwargs) -> Union[str, Any]:
         """
-        封装httpx的公共请求方法，对请求响应做一些处理
+        Wrapper for httpx common request method, processes request response
         Args:
-            method: 请求方法
-            url: 请求的URL
-            **kwargs: 其他请求参数，例如请求头、请求体等
+            method: Request method
+            url: Request URL
+            **kwargs: Other request parameters, such as headers, body, etc.
 
         Returns:
 
         """
-        # 每次请求前检测代理是否过期
+        # Check if proxy is expired before each request
         await self._refresh_proxy_if_expired()
 
         # return response.text
         return_response = kwargs.pop("return_response", False)
-        async with httpx.AsyncClient(proxy=self.proxy) as client:
+        async with make_async_client(proxy=self.proxy) as client:
             response = await client.request(method, url, timeout=self.timeout, **kwargs)
 
         if response.status_code == 471 or response.status_code == 461:
             # someday someone maybe will bypass captcha
             verify_type = response.headers["Verifytype"]
             verify_uuid = response.headers["Verifyuuid"]
-            msg = f"出现验证码，请求失败，Verifytype: {verify_type}，Verifyuuid: {verify_uuid}, Response: {response}"
+            msg = f"CAPTCHA appeared, request failed, Verifytype: {verify_type}, Verifyuuid: {verify_uuid}, Response: {response}"
             utils.logger.error(msg)
             raise Exception(msg)
 
@@ -141,25 +147,36 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
             return data.get("data", data.get("success", {}))
         elif data["code"] == self.IP_ERROR_CODE:
             raise IPBlockError(self.IP_ERROR_STR)
+        elif data["code"] in (self.NOTE_NOT_FOUND_CODE, self.NOTE_ABNORMAL_CODE):
+            raise NoteNotFoundError(f"Note not found or abnormal, code: {data['code']}")
         else:
             err_msg = data.get("msg", None) or f"{response.text}"
             raise DataFetchError(err_msg)
 
+    @staticmethod
+    def _build_query_string(params: Dict) -> str:
+        """Build URL query string with encoding matching browser behavior (commas not encoded)"""
+        parts = []
+        for key, value in params.items():
+            value_str = str(value) if value is not None else ""
+            parts.append(f"{key}={quote(value_str, safe=',')}")
+        return "&".join(parts)
+
     async def get(self, uri: str, params: Optional[Dict] = None) -> Dict:
         """
-        GET请求，对请求头签名
+        GET request, signs request headers
         Args:
-            uri: 请求路由
-            params: 请求参数
+            uri: Request route
+            params: Request parameters
 
         Returns:
 
         """
         headers = await self._pre_headers(uri, params)
-        if isinstance(params, dict):
-            # 构建带参数的完整 URL
-            query_string = urlencode(params)
-            full_url = f"{self._host}{uri}?{query_string}"
+        # Build URL manually to ensure query string encoding matches the sign string
+        # (httpx's default params encoding differs from browser/XHS frontend behavior)
+        if params:
+            full_url = f"{self._host}{uri}?{self._build_query_string(params)}"
         else:
             full_url = f"{self._host}{uri}"
 
@@ -169,10 +186,10 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
 
     async def post(self, uri: str, data: dict, **kwargs) -> Dict:
         """
-        POST请求，对请求头签名
+        POST request, signs request headers
         Args:
-            uri: 请求路由
-            data: 请求体参数
+            uri: Request route
+            data: Request body parameters
 
         Returns:
 
@@ -188,10 +205,10 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
         )
 
     async def get_note_media(self, url: str) -> Union[bytes, None]:
-        # 请求前检测代理是否过期
+        # Check if proxy is expired before request
         await self._refresh_proxy_if_expired()
 
-        async with httpx.AsyncClient(proxy=self.proxy) as client:
+        async with make_async_client(proxy=self.proxy) as client:
             try:
                 response = await client.request("GET", url, timeout=self.timeout)
                 response.raise_for_status()
@@ -207,39 +224,56 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
             ) as exc:  # some wrong when call httpx.request method, such as connection error, client error, server error or response status code is not 2xx
                 utils.logger.error(
                     f"[XiaoHongShuClient.get_aweme_media] {exc.__class__.__name__} for {exc.request.url} - {exc}"
-                )  # 保留原始异常类型名称，以便开发者调试
+                )  # Keep original exception type name for developer debugging
                 return None
 
+    async def query_self(self) -> Optional[Dict]:
+        """
+        Query self user info to check login state
+        Returns:
+            Dict: User info if logged in, None otherwise
+        """
+        uri = "/api/sns/web/v1/user/selfinfo"
+        headers = await self._pre_headers(uri, params={})
+        async with make_async_client(proxy=self.proxy) as client:
+            response = await client.get(f"{self._host}{uri}", headers=headers)
+            if response.status_code == 200:
+                return response.json()
+        return None
+
     async def pong(self) -> bool:
         """
-        用于检查登录态是否失效了
+        Check if login state is still valid by querying self user info
         Returns:
-
+            bool: True if logged in, False otherwise
         """
-        """get a note to check if login state is ok"""
-        utils.logger.info("[XiaoHongShuClient.pong] Begin to pong xhs...")
+        utils.logger.info("[XiaoHongShuClient.pong] Begin to check login state...")
         ping_flag = False
         try:
-            note_card: Dict = await self.get_note_by_keyword(keyword="小红书")
-            if note_card.get("items"):
+            self_info: Dict = await self.query_self()
+            if self_info and self_info.get("data", {}).get("result", {}).get("success"):
                 ping_flag = True
         except Exception as e:
             utils.logger.error(
-                f"[XiaoHongShuClient.pong] Ping xhs failed: {e}, and try to login again..."
+                f"[XiaoHongShuClient.pong] Check login state failed: {e}, and try to login again..."
             )
             ping_flag = False
+        utils.logger.info(f"[XiaoHongShuClient.pong] Login state result: {ping_flag}")
         return ping_flag
 
-    async def update_cookies(self, browser_context: BrowserContext):
+    async def update_cookies(self, browser_context: BrowserContext, urls: Optional[list[str]] = None):
         """
-        API客户端提供的更新cookies方法，一般情况下登录成功后会调用此方法
+        Update cookies method provided by API client, usually called after successful login
         Args:
-            browser_context: 浏览器上下文对象
+            browser_context: Browser context object
 
         Returns:
 
         """
-        cookie_str, cookie_dict = utils.convert_cookies(await browser_context.cookies())
+        cookie_str, cookie_dict = await utils.convert_browser_context_cookies(
+            browser_context,
+            urls=urls or self.cookie_urls,
+        )
         self.headers["Cookie"] = cookie_str
         self.cookie_dict = cookie_dict
 
@@ -253,13 +287,13 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
         note_type: SearchNoteType = SearchNoteType.ALL,
     ) -> Dict:
         """
-        根据关键词搜索笔记
+        Search notes by keyword
         Args:
-            keyword: 关键词参数
-            page: 分页第几页
-            page_size: 分页数据长度
-            sort: 搜索结果排序指定
-            note_type: 搜索的笔记类型
+            keyword: Keyword parameter
+            page: Page number
+            page_size: Page data length
+            sort: Search result sorting specification
+            note_type: Type of note to search
 
         Returns:
 
@@ -282,11 +316,11 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
         xsec_token: str,
     ) -> Dict:
         """
-        获取笔记详情API
+        Get note detail API
         Args:
-            note_id:笔记ID
-            xsec_source: 渠道来源
-            xsec_token: 搜索关键字之后返回的比较列表中返回的token
+            note_id: Note ID
+            xsec_source: Channel source
+            xsec_token: Token returned from search keyword result list
 
         Returns:
 
@@ -306,7 +340,7 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
         if res and res.get("items"):
             res_dict: Dict = res["items"][0]["note_card"]
             return res_dict
-        # 爬取频繁了可能会出现有的笔记能有结果有的没有
+        # When crawling frequently, some notes may have results while others don't
         utils.logger.error(
             f"[XiaoHongShuClient.get_note_by_id] get note id:{note_id} empty and res:{res}"
         )
@@ -319,11 +353,11 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
         cursor: str = "",
     ) -> Dict:
         """
-        获取一级评论的API
+        Get first-level comments API
         Args:
-            note_id: 笔记ID
-            xsec_token: 验证token
-            cursor: 分页游标
+            note_id: Note ID
+            xsec_token: Verification token
+            cursor: Pagination cursor
 
         Returns:
 
@@ -347,13 +381,13 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
         cursor: str = "",
     ):
         """
-        获取指定父评论下的子评论的API
+        Get sub-comments under specified parent comment API
         Args:
-            note_id: 子评论的帖子ID
-            root_comment_id: 根评论ID
-            xsec_token: 验证token
-            num: 分页数量
-            cursor: 分页游标
+            note_id: Post ID of sub-comments
+            root_comment_id: Root comment ID
+            xsec_token: Verification token
+            num: Pagination quantity
+            cursor: Pagination cursor
 
         Returns:
 
@@ -362,7 +396,7 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
         params = {
             "note_id": note_id,
             "root_comment_id": root_comment_id,
-            "num": num,
+            "num": str(num),
             "cursor": cursor,
             "image_formats": "jpg,webp,avif",
             "top_comment_id": "",
@@ -379,13 +413,13 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
         max_count: int = 10,
     ) -> List[Dict]:
         """
-        获取指定笔记下的所有一级评论，该方法会一直查找一个帖子下的所有评论信息
+        Get all first-level comments under specified note, this method will continuously find all comment information under a post
         Args:
-            note_id: 笔记ID
-            xsec_token: 验证token
-            crawl_interval: 爬取一次笔记的延迟单位（秒）
-            callback: 一次笔记爬取结束后
-            max_count: 一次笔记爬取的最大评论数量
+            note_id: Note ID
+            xsec_token: Verification token
+            crawl_interval: Crawl delay per note (seconds)
+            callback: Callback after one note crawl ends
+            max_count: Maximum number of comments to crawl per note
         Returns:
 
         """
@@ -427,12 +461,12 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
         callback: Optional[Callable] = None,
     ) -> List[Dict]:
         """
-        获取指定一级评论下的所有二级评论, 该方法会一直查找一级评论下的所有二级评论信息
+        Get all second-level comments under specified first-level comments, this method will continuously find all second-level comment information under first-level comments
         Args:
-            comments: 评论列表
-            xsec_token: 验证token
-            crawl_interval: 爬取一次评论的延迟单位（秒）
-            callback: 一次评论爬取结束后
+            comments: Comment list
+            xsec_token: Verification token
+            crawl_interval: Crawl delay per comment (seconds)
+            callback: Callback after one comment crawl ends
 
         Returns:
 
@@ -445,62 +479,79 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
 
         result = []
         for comment in comments:
-            note_id = comment.get("note_id")
-            sub_comments = comment.get("sub_comments")
-            if sub_comments and callback:
-                await callback(note_id, sub_comments)
+            try:
+                note_id = comment.get("note_id")
+                sub_comments = comment.get("sub_comments")
+                if sub_comments and callback:
+                    await callback(note_id, sub_comments)
 
-            sub_comment_has_more = comment.get("sub_comment_has_more")
-            if not sub_comment_has_more:
-                continue
-
-            root_comment_id = comment.get("id")
-            sub_comment_cursor = comment.get("sub_comment_cursor")
-
-            while sub_comment_has_more:
-                comments_res = await self.get_note_sub_comments(
-                    note_id=note_id,
-                    root_comment_id=root_comment_id,
-                    xsec_token=xsec_token,
-                    num=10,
-                    cursor=sub_comment_cursor,
-                )
-
-                if comments_res is None:
-                    utils.logger.info(
-                        f"[XiaoHongShuClient.get_comments_all_sub_comments] No response found for note_id: {note_id}"
-                    )
+                sub_comment_has_more = comment.get("sub_comment_has_more")
+                if not sub_comment_has_more:
                     continue
-                sub_comment_has_more = comments_res.get("has_more", False)
-                sub_comment_cursor = comments_res.get("cursor", "")
-                if "comments" not in comments_res:
-                    utils.logger.info(
-                        f"[XiaoHongShuClient.get_comments_all_sub_comments] No 'comments' key found in response: {comments_res}"
-                    )
-                    break
-                comments = comments_res["comments"]
-                if callback:
-                    await callback(note_id, comments)
-                await asyncio.sleep(crawl_interval)
-                result.extend(comments)
+
+                root_comment_id = comment.get("id")
+                sub_comment_cursor = comment.get("sub_comment_cursor")
+
+                while sub_comment_has_more:
+                    try:
+                        comments_res = await self.get_note_sub_comments(
+                            note_id=note_id,
+                            root_comment_id=root_comment_id,
+                            xsec_token=xsec_token,
+                            num=10,
+                            cursor=sub_comment_cursor,
+                        )
+
+                        if comments_res is None:
+                            utils.logger.info(
+                                f"[XiaoHongShuClient.get_comments_all_sub_comments] No response found for note_id: {note_id}"
+                            )
+                            break
+                        sub_comment_has_more = comments_res.get("has_more", False)
+                        sub_comment_cursor = comments_res.get("cursor", "")
+                        if "comments" not in comments_res:
+                            utils.logger.info(
+                                f"[XiaoHongShuClient.get_comments_all_sub_comments] No 'comments' key found in response: {comments_res}"
+                            )
+                            break
+                        comments = comments_res["comments"]
+                        if callback:
+                            await callback(note_id, comments)
+                        await asyncio.sleep(crawl_interval)
+                        result.extend(comments)
+                    except DataFetchError as e:
+                        utils.logger.warning(
+                            f"[XiaoHongShuClient.get_comments_all_sub_comments] Failed to get sub-comments for note_id: {note_id}, root_comment_id: {root_comment_id}, error: {e}. Skipping this comment's sub-comments."
+                        )
+                        break  # Break out of the sub-comment acquisition loop of the current comment and continue processing the next comment
+                    except Exception as e:
+                        utils.logger.error(
+                            f"[XiaoHongShuClient.get_comments_all_sub_comments] Unexpected error when getting sub-comments for note_id: {note_id}, root_comment_id: {root_comment_id}, error: {e}"
+                        )
+                        break
+            except Exception as e:
+                utils.logger.error(
+                    f"[XiaoHongShuClient.get_comments_all_sub_comments] Error processing comment: {comment.get('id', 'unknown')}, error: {e}. Continuing with next comment."
+                )
+                continue  # Continue to next comment
         return result
 
     async def get_creator_info(
         self, user_id: str, xsec_token: str = "", xsec_source: str = ""
     ) -> Dict:
         """
-        通过解析网页版的用户主页HTML，获取用户个人简要信息
-        PC端用户主页的网页存在window.__INITIAL_STATE__这个变量上的，解析它即可
+        Get user profile brief information by parsing user homepage HTML
+        The PC user homepage has window.__INITIAL_STATE__ variable, just parse it
 
         Args:
-            user_id: 用户ID
-            xsec_token: 验证token (可选,如果URL中包含此参数则传入)
-            xsec_source: 渠道来源 (可选,如果URL中包含此参数则传入)
+            user_id: User ID
+            xsec_token: Verification token (optional, pass if included in URL)
+            xsec_source: Channel source (optional, pass if included in URL)
 
         Returns:
-            Dict: 创作者信息
+            Dict: Creator information
         """
-        # 构建URI,如果有xsec参数则添加到URL中
+        # Build URI, add xsec parameters to URL if available
         uri = f"/user/profile/{user_id}"
         if xsec_token and xsec_source:
             uri = f"{uri}?xsec_token={xsec_token}&xsec_source={xsec_source}"
@@ -519,13 +570,13 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
         xsec_source: str = "pc_feed",
     ) -> Dict:
         """
-        获取博主的笔记
+        Get creator's notes
         Args:
-            creator: 博主ID
-            cursor: 上一页最后一条笔记的ID
-            page_size: 分页数据长度
-            xsec_token: 验证token
-            xsec_source: 渠道来源
+            creator: Creator ID
+            cursor: Last note ID from previous page
+            page_size: Page data length
+            xsec_token: Verification token
+            xsec_source: Channel source
 
         Returns:
 
@@ -535,6 +586,7 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
             "num": page_size,
             "cursor": cursor,
             "user_id": creator,
+            "image_formats": "jpg,webp,avif",
             "xsec_token": xsec_token,
             "xsec_source": xsec_source,
         }
@@ -549,13 +601,13 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
         xsec_source: str = "pc_feed",
     ) -> List[Dict]:
         """
-        获取指定用户下的所有发过的帖子，该方法会一直查找一个用户下的所有帖子信息
+        Get all posts published by specified user, this method will continuously find all post information under a user
         Args:
-            user_id: 用户ID
-            crawl_interval: 爬取一次的延迟单位（秒）
-            callback: 一次分页爬取结束后的更新回调函数
-            xsec_token: 验证token
-            xsec_source: 渠道来源
+            user_id: User ID
+            crawl_interval: Crawl delay (seconds)
+            callback: Update callback function after one pagination crawl ends
+            xsec_token: Verification token
+            xsec_source: Channel source
 
         Returns:
 
@@ -604,9 +656,9 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
 
     async def get_note_short_url(self, note_id: str) -> Dict:
         """
-        获取笔记的短链接
+        Get note short URL
         Args:
-            note_id: 笔记ID
+            note_id: Note ID
 
         Returns:
 
@@ -624,7 +676,7 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
         enable_cookie: bool = False,
     ) -> Optional[Dict]:
         """
-        通过解析网页版的笔记详情页HTML，获取笔记详情, 该接口可能会出现失败的情况，这里尝试重试3次
+        Get note details by parsing note detail page HTML, this interface may fail, retry 3 times here
         copy from https://github.com/ReaJason/xhs/blob/eb1c5a0213f6fbb592f0a2897ee552847c69ea2d/xhs/core.py#L217-L259
         thanks for ReaJason
         Args:
@@ -637,7 +689,7 @@ class XiaoHongShuClient(AbstractApiClient, ProxyRefreshMixin):
 
         """
         url = (
-            "https://www.xiaohongshu.com/explore/"
+            f"{self._domain}/explore/"
             + note_id
             + f"?xsec_token={xsec_token}&xsec_source={xsec_source}"
         )