gaoxiaobei
d0d7293926
feat(bilibili): Add flexible search modes and fix limit logic
...
Refactors the Bilibili keyword search functionality to provide more flexible crawling strategies and corrects a flaw in how crawl limits were applied.
Previously, the `ALL_DAY` boolean flag offered a rigid choice for time-based searching and contained a logical issue where `CRAWLER_MAX_NOTES_COUNT` was incorrectly applied on a per-day basis instead of as an overall total.
This commit introduces the `BILI_SEARCH_MODE` configuration option with three distinct modes:
- `normal`: The default search behavior without time constraints.
- `all_in_time_range`: Maximizes data collection within a specified date range, replicating the original intent of `ALL_DAY=True`.
- `daily_limit_in_time_range`: A new mode that strictly enforces both the daily `MAX_NOTES_PER_DAY` and the total `CRAWLER_MAX_NOTES_COUNT` limits across the entire date range.
This change resolves the limit logic bug and gives users more precise control over the crawling process.
Changes include:
- Modified `config/base_config.py` to replace `ALL_DAY` with `BILI_SEARCH_MODE`.
- Refactored `media_platform/bilibili/core.py` to implement the new search mode logic.
2025-07-13 06:07:13 +08:00
gaoxiaobei
cad9fc7af8
feat: Add daily limit for video/post crawling in Bilibili and base config
2025-07-12 14:50:59 +08:00
mirza-samad-ahmed-baig
7edf3bcc15
refactor(bilibili): process creator videos in batches
2025-07-04 21:04:10 +05:00
程序员阿江(Relakkes)
848df2b491
feat: other platfrom support the cdp mode
2025-07-03 17:13:32 +08:00
Bowenwin
66843f216a
finish_all_for_expand_bili
2025-05-22 22:26:30 +08:00
Bowenwin
59619fff0a
finish_all
2025-05-22 22:06:06 +08:00
Bowenwin
44e3d370ff
fix_words
2025-05-22 20:31:48 +08:00
Bowenwin
a356358c21
get_fans_and_get_followings
2025-05-19 19:57:36 +08:00
翟持江
af5a393a7a
Update core.py，删除了其它代码贡献者所添加的try-catch语句，该段try-catch语句将会影响其代码的最终逻辑并令其失效，使其仅能爬取当天一天数据而无法跳转到下一天（原先的逻辑就是try-catch捕获异常从而进入下一天，不要再向该语句中添加捕获异常操作或者finally语句！）
2025-04-19 04:34:24 +08:00
Relakkes
0d715a9f32
fix: bili qrcode login fix
2025-04-08 21:11:40 +08:00
crpa33
a39b571d27
输出到日志-处理视频搜索页任务列表构造的错误
2025-04-02 11:57:28 +08:00
crpa33
eaf14721f8
输出到日志-NoneType导致的推导式错误
2025-04-02 11:48:36 +08:00
crpa33
6b6e2b8ba0
修复NoneType导致的推导式错误
2025-03-27 23:18:01 +08:00
Relakkes
678ce1bfac
fix: bilibili bugfix
2025-02-10 17:13:37 +08:00
翟持江
0364b23b5b
Update core.py，为爬取类型为detail和creator的任务，添加了和search任务一样的，用于转存up主信息的bilibili_store.update_up_info的函数调用
...
正如`search`函数中一样，在调用`get_video_info_task`后，`bilibili_video`和`bilibili_up_info`信息都将获得。
原先的`get_specified_videos`在`detail`任务中仅保存了指定`bilibili_video`的信息，而`bilibili_up_info`信息尚未保存，`creator`任务的`get_creator_videos`中也调用了`get_specified_videos`获取指定创作者下所有的视频信息，同理也未保存`bilibili_up_info`信息。
所以只需为`get_specified_videos`添加一句`await bilibili_store.update_up_info(video_detail)`即可和`search`任务下获得的数据文件个数保持一致，不会缺少对应up主的个人信息。
已测试：
- 原先仅`search`任务下产生`*_creator.csv`、`*_contents.csv`、`*_comments.csv`，而`detail`和`creator`任务下缺少`*_creator.csv`文件。
- 此次提交后将使三种模式下的数据文件个数一致。
2025-01-19 19:55:18 +08:00
翟持江
2d93ec5a82
Update core.py，更改了错误的缩进
2025-01-15 18:33:12 +08:00
翟持江
d2ecd3b11d
Update client.py，将search_video_by_keyword中post_data错误的请求参数进行更新
...
`pubtime_begin`更改为`pubtime_begin_s`，`pubtime_end`更改为`pubtime_end_s`。已测试
2025-01-15 18:21:03 +08:00
翟持江
f2b41b573b
Update core.py，以实现按照 START_DAY 至 END_DAY ，每一天进行筛选，这样能够突破 1000 条视频的限制，最大程度爬取该关键词下的所有视频
...
添加了`get_pubtime_datetime`函数用以获取`pubtime_begin_s`和`pubtime_end_s`参数，并为`search`函数添加了`ALL_DAY`选项，若`ALL_DAY`未开启，则保留原先的搜索策略，但每个关键词最多返回 1000 条数据，若`ALL_DAY`已开启，则使用新策略，按照 START_DAY 至 END_DAY 按照每一天进行筛选，这样能够突破 1000 条视频的限制，最大程度爬取该关键词下的所有视频，新添加的`get_pubtime_datetime`函数仅在`search`中使用，需要用户按安装`datetime`和`pandas`模块。已测试完毕
2025-01-15 18:18:36 +08:00
unknown
7e53c4acfc
All_platform_comments_restrict
2024-10-23 16:32:02 +08:00
Relakkes
9fe3e47b0f
chore: 增加代码学习声明，严格禁止非法、禁止商业、不当用途
2024-10-20 00:43:25 +08:00
Relakkes
aa0f920369
feat: B站搜索接口增加发布日期筛选
2024-10-17 15:11:25 +08:00
Relakkes
d6fb255bdf
fix: xhs note detail error
2024-09-02 21:45:12 +08:00
Relakkes
c70bd9e071
feat: 增加搜索词来源渠道
2024-08-23 08:29:24 +08:00
ZhouXSh
3b2cc44750
新增B站创作者（UP主）信息爬取
2024-07-18 20:11:51 +08:00
helloteemo
d686d17f9b
feat: 支持bilibili视频下载
2024-07-15 19:40:17 +08:00
Relakkes
d3eeccbaac
feat: logger record current search page
2024-06-24 22:24:51 +08:00
Relakkes Yang
a0e5a29af8
fix: weibo bug
2024-06-17 00:25:48 +08:00
nelzomal
111e08602c
feat: support bilibili creator
2024-06-12 16:48:19 +08:00
nelzomal
eace7d1750
improve base config reading command line arg logic
2024-06-09 18:51:36 +08:00
Nan Zhou
0cad36e17b
support bilibili level two comment
2024-05-26 14:10:57 +08:00
Relakkes
e64df93edd
feat: 由于xhs和dy现在检测playwright二维码登录了，大概率会出现滑块或者手机验证，增加登录态检测时间为5min，预留足够的时间手动过验证码。
2024-05-15 23:23:30 +08:00
Relakkes
87eb8aa6a7
fix : #230
2024-04-13 20:18:04 +08:00
Tianci-King
1115b0d90c
feat(core): 新增控制爬虫 参数起始页面的页数start_page;perf(argparse): 向命令行解析器添加程序参数起始页面页数和关键字
2024-04-12 00:52:47 +08:00
leantli
68a60faa7f
chore: 简化判断方式
2024-04-04 00:11:22 +08:00
leantli
133f978477
fix: 修复爬取视频/帖子最大数设置值较低导致不爬取的问题
2024-04-03 12:18:23 +08:00
Relakkes
e950e0d6e3
feat: add abstract api client to all platform
2024-03-30 21:27:25 +08:00
Relakkes
59cd9f67a0
feat: 支持评论模式是否开启爬取选项
2024-03-16 11:52:42 +08:00
Relakkes
894dabcf63
refactor: 数据存储重构，分离不同类型的存储实现
2024-01-14 22:06:31 +08:00
Relakkes
e31aebbdfb
fix: 修复代理Bug
2024-01-13 15:50:02 +08:00
Relakkes
80ecc53cd3
fix : #107
2024-01-03 23:08:30 +08:00
Relakkes
c5b64fdbf5
feat: 微博爬虫帖子搜索完成
2023-12-24 17:57:48 +08:00
Relakkes
aba9f14f50
refactor: 规范日志打印
...
feat: B站指定视频ID爬取（bvid）
2023-12-23 01:04:08 +08:00
Relakkes
273c9a316b
fix: 修复日志打印时参数格式错误
2023-12-22 23:10:44 +08:00
peanutsplash
f17a85305e
添加功能:(哔哩哔哩,快手,小红书)每个视频/帖子抓取评论最大条数限制,评论关键词筛选
2023-12-13 23:53:12 +08:00
Relakkes
97d7a0c38b
feat: Bilibili comment done
2023-12-09 21:10:01 +08:00
Relakkes
1cec23f73d
feat: 代理IP功能 Done
2023-12-08 00:10:04 +08:00
Relakkes
c530bd4219
feat: 代理IP缓存到redis中
2023-12-06 23:49:56 +08:00
Relakkes
f71d086464
fix: B站get_wbi_keys函数类型标注问题
2023-12-05 23:32:35 +08:00
Relakkes
a6e877de42
fix: 修复B站搜索Field命名 bug
...
refactor: ping接口统一更换为pong
2023-12-05 22:54:47 +08:00
Relakkes
8f04943105
feat: B站评论API
2023-12-04 23:16:02 +08:00
