fix: make SSL verification opt-in via config, extend fix to all platforms

- Add DISABLE_SSL_VERIFY = False to base_config.py (default: verification on)
- Add tools/httpx_util.py with make_async_client() factory that reads the config
- Replace all httpx.AsyncClient() call sites across all platforms (bilibili,
  weibo, zhihu, xhs, douyin, kuaishou) and crawler_util with make_async_client()
- Extends SSL fix to previously missed platforms: xhs, douyin, kuaishou

Users running behind an intercepting proxy can set DISABLE_SSL_VERIFY = True
in config/base_config.py. All other users retain certificate verification.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Wei Liu
2026-03-18 12:31:49 +13:00
parent eb45a6367f
commit 125e02a4b9
9 changed files with 41 additions and 15 deletions

14
tools/httpx_util.py Normal file
View File

@@ -0,0 +1,14 @@
# -*- coding: utf-8 -*-
import httpx
import config
def make_async_client(**kwargs) -> httpx.AsyncClient:
"""Create an httpx.AsyncClient with project-wide defaults.
Reads DISABLE_SSL_VERIFY from config (default False).
Set DISABLE_SSL_VERIFY = True in config/base_config.py only when running
behind an intercepting proxy (corporate gateway, Burp, mitmproxy, etc.).
"""
kwargs.setdefault("verify", not getattr(config, "DISABLE_SSL_VERIFY", False))
return httpx.AsyncClient(**kwargs)