jiazhizhong/MediaCrawler
1
0
Fork 0
mirror of https://github.com/NanmiCoder/MediaCrawler.git synced 2026-06-07 02:17:25 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(core): 新增控制爬虫 参数起始页面的页数start_page;perf(argparse): 向命令行解析器添加程序参数起始页面页数和关键字

Browse Source
This commit is contained in:
Tianci-King
2024-04-12 00:52:47 +08:00
parent bba9841c26
commit 1115b0d90c
9 changed files with 94 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
config/base_config.py
View File

@@ -27,6 +27,9 @@ SAVE_DATA_OPTION = "json" # csv or db or json
# 用户浏览器缓存的浏览器文件配置
USER_DATA_DIR = "%s_user_data_dir" # %s will be replaced by platform name
# 爬取开始页数 默认从第一页开始
START_PAGE = 1
# 爬取视频/帖子的数量控制
CRAWLER_MAX_NOTES_COUNT = 20