jiazhizhong/MediaCrawler
1
0
Fork 0
mirror of https://github.com/NanmiCoder/MediaCrawler.git synced 2026-04-20 10:47:37 +08:00
Code Issues Packages Projects Releases Wiki Activity

fix: make SSL verification opt-in via config, extend fix to all platforms

Browse Source 
- Add DISABLE_SSL_VERIFY = False to base_config.py (default: verification on)
- Add tools/httpx_util.py with make_async_client() factory that reads the config
- Replace all httpx.AsyncClient() call sites across all platforms (bilibili,
  weibo, zhihu, xhs, douyin, kuaishou) and crawler_util with make_async_client()
- Extends SSL fix to previously missed platforms: xhs, douyin, kuaishou

Users running behind an intercepting proxy can set DISABLE_SSL_VERIFY = True
in config/base_config.py. All other users retain certificate verification.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Wei Liu
2026-03-18 12:31:49 +13:00
parent eb45a6367f
commit 125e02a4b9
9 changed files with 41 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
config/base_config.py
View File

@@ -120,6 +120,11 @@ FONT_PATH = "./docs/STZHONGS.TTF"
# Crawl interval
CRAWLER_MAX_SLEEP_SEC = 2
# Disable SSL certificate verification. Set to True only when running behind an intercepting
# proxy (corporate gateway, Burp Suite, mitmproxy, etc.) that injects its own certificate.
# WARNING: disabling SSL verification exposes all traffic to MITM attacks.
DISABLE_SSL_VERIFY = False
from .bilibili_config import *
from .xhs_config import *
from .dy_config import *