程序员阿江(Relakkes)
b9d30bbabb
fix : #693
2025-08-01 15:55:21 +08:00
程序员阿江(Relakkes)
12450759d8
fix: httpx proxy format error
...
feat: add a ip proxy provider
2025-08-01 01:05:11 +08:00
程序员阿江-Relakkes
0024ce6ab4
feat: upgrade httpx version to 0.28.1
...
fix issue #689
2025-07-31 23:19:08 +08:00
未来可欺
a6fd9ebdbc
简单更改了抖音保存图片与视频的命名方式，一个视频 id 仅对应一个短视频，返回一个 video_download_url，因此不需要使用数字方式进行命名
2025-07-31 23:11:45 +08:00
未来可欺
0b81240aed
升级 httpx 版本至 0.28.1，并修改关键字参数 proxies 至 proxy
2025-07-31 22:48:02 +08:00
未来可欺
9d90e9fc6d
fix issue #689，目前来看，应该是 httpx 库的问题，因为无论是使用同步还是异步版本，构不构造 httpx.***Client 对象来发起请求，返回的响应都是为空，response.content = b''，response.text = ’‘，但换成 requests 库就能正常获取数据了
2025-07-31 22:01:48 +08:00
程序员阿江-Relakkes
a1535289c1
Merge pull request #687 from 2513502304/main
...
添加抖音平台爬取短视频以及帖子图片的逻辑，并规范目前实现爬取媒体的抖音、b站、小红书、微博四个平台存储逻辑
2025-07-30 23:06:35 +08:00
未来可欺
e9f976117a
将配置文件恢复原状
2025-07-30 21:32:00 +08:00
翟持江
082c316345
Merge branch 'NanmiCoder:main' into main
2025-07-30 21:28:29 +08:00
程序员阿江-Relakkes
c61ed57a20
fix: 二维码在部分系统无法显示 #685
...
fix : #685
2025-07-30 21:26:41 +08:00
未来可欺
93a1c27fff
通过测试search模式，修复部分运行时的bug，并对能够爬取媒体的平台设置了较长的超时时间
2025-07-30 21:19:56 +08:00
GokoRuri
87caf07495
fix : #685
2025-07-30 21:14:37 +08:00
未来可欺
a7cc18ec7d
修改部分文档
2025-07-30 18:58:10 +08:00
未来可欺
ecddfbe02c
将store文件夹中后缀名为_video或_image的.py文件统一更名为以_media.py为结尾的命名方式，避免某些平台仅有_video.py文件或_image.py文件的单独实现。之后的所有存储视频或图像的代码均放在此文件中实现
2025-07-30 18:32:08 +08:00
未来可欺
173bc08a9d
添加了抖音存储视频以及图片的逻辑，并将config.py中ENABLE_GET_IMAGES参数更名为ENABLE_GET_MEIDAS，在此基础上略微修改存储逻辑
2025-07-30 18:24:08 +08:00
程序员阿江(Relakkes)
417c39de69
docs: add a sponsor
2025-07-30 16:44:10 +08:00
程序员阿江-Relakkes
b2d52918ae
Merge pull request #684 from 2513502304/main
...
添加抖音笔记图片的存储逻辑
2025-07-30 14:51:46 +08:00
程序员阿江(Relakkes)
8ab1b7ee4c
fix: fixed circular import issue
2025-07-30 14:47:11 +08:00
翟持江
214ccaa294
Update sqlite_tables.sql，同步更新sqlite以支持保存笔记下载地址
2025-07-30 10:48:52 +08:00
翟持江
612a9b53d3
Update tables.sql，同步更新该文件以支持保存笔记下载地址
2025-07-30 10:46:46 +08:00
翟持江
e6db6be1ca
Update __init__.py，添加了提取抖音笔记图片的逻辑
2025-07-30 10:45:38 +08:00
程序员阿江-Relakkes
72a4d194c0
Merge pull request #680 from korruz/main
...
refactor: move format_proxy_info to utils and update crawler classes
2025-07-29 14:21:48 +08:00
korruz
07a6e387ea
refactor: move format_proxy_info to utils and update crawler classes to use it
2025-07-29 14:16:24 +08:00
程序员阿江-Relakkes
2935841872
Merge pull request #676 from andy9468/main
...
add aiosqlite
2025-07-28 11:33:42 +08:00
andy9468
3653fe2d37
add aiosqlite
2025-07-26 16:42:16 +08:00
程序员阿江-Relakkes
dbbc33a0df
Merge pull request #674 from persist-1/chore
...
chore: 增加--help参数中文显示支持；增加"douyin_aweme"表"music_download_url"字段与功能实现
2025-07-25 17:23:03 +08:00
persist-1
19df1734f1
chore: 增加--help参数中文显示支持及douyin_aweme表music_download_url字段\n\n- 为命令行参数增加中文显示支持，提升用户体验\n- 在douyin_aweme表中新增music_download_url字段用于存储视频音乐下载链接\n- 更新相关数据库表结构文件(tables.sql, sqlite_tables.sql)\n- 实现音乐下载URL提取逻辑并集成到数据存储流程
2025-07-24 22:39:53 +08:00
程序员阿江(Relakkes)
fc06c783f5
fix: fixed xhs req headers
2025-07-23 13:28:58 +08:00
程序员阿江(Relakkes)
b41896f4f3
docs: add a sponsor
2025-07-23 13:17:19 +08:00
程序员阿江(Relakkes)
a4d9aaa34a
refactor: xhs update
2025-07-21 21:26:16 +08:00
程序员阿江(Relakkes)
26a43358cb
chore: update config
2025-07-20 14:34:56 +08:00
程序员阿江(Relakkes)
13b00f7a36
refactor: config update
2025-07-18 23:26:52 +08:00
程序员阿江(Relakkes)
122978b35c
Merge pull request #652 from gaoxiaobei/dev
...
feat(bilibili): Add flexible search modes and fix limit logic
2025-07-18 21:41:20 +08:00
gaoxiaobei
8105b053ed
Merge remote-tracking branch 'origin/dev' into devdev
2025-07-18 17:37:29 +08:00
gaoxiaobei
7176956e51
Merge branch 'NanmiCoder:main' into dev
2025-07-18 17:32:04 +08:00
gaoxiaobei
b913db64bb
refactor(config): move platform-specific configs to separate files
...
- Remove platform-specific configurations from base_config.py
- Create separate config files for each platform in their respective directories
- Update import statements in core files to use new platform-specific config modules
- Clean up unused and deprecated configuration options
2025-07-18 17:27:37 +08:00
程序员阿江(Relakkes)
2753e7631e
Merge pull request #664 from cfl-chenfangliang/feat-dyCommentBug
...
feat: 修复抖音二级评论地理位置缺失问题
2025-07-18 15:53:33 +08:00
chenfangliang
aa54dad9a5
feat: 修复抖音二级评论地理位置缺失问题
2025-07-18 10:48:43 +08:00
gaoxiaobei
5daae04c7d
fix: conflict resolution errors
2025-07-17 16:54:17 +08:00
gaoxiaobei
1dc8c1789f
docs(config): update Bilibili search mode options
...
- Clarify the three search mode options for Bilibili
- Add note about setting MAX_NOTES_PER_DAY in bilibili config
2025-07-17 07:51:27 +08:00
gaoxiaobei
29b6cee408
Merge pull request #1 from gaoxiaobei/dev
...
Enhance robustness.
2025-07-17 06:45:55 +08:00
gaoxiaobei
6ced357096
Merge branch 'main' into dev
2025-07-17 06:45:30 +08:00
gaoxiaobei
9fb396c7d1
fix(media_platform): handle edge cases and improve error handling for Bilibili client and crawler
...
- BilibiliClient:
- Improve wbi_img_urls handling for better compatibility
- Add error handling for missing or invalid 'is_end' and 'next' in comment cursor
- BilibiliCrawler:
- Fix daily limit logic for keyword-based searches
- Improve logging and break conditions for max notes count limits
- Ensure proper tracking of total notes crawled for each keyword
2025-07-17 06:40:56 +08:00
gaoxiaobei
fb846e9060
Merge branch 'NanmiCoder:main' into main
2025-07-17 06:39:04 +08:00
程序员阿江(Relakkes)
08c28e6f7b
Merge pull request #658 from persist-1/feature/sqlite-support
...
增加对本地Sqlite数据库的支持（在不便于使用Mysql服务时也能使用数据库进行相关操作）
2025-07-16 20:35:58 +08:00
买定不离手
9457455c18
fix: 修复SQLite数据库初始化问题并重新生成数据库文件
...
- db.py: 在init_table_schema函数中增加损坏数据库文件检查和清理逻辑，确保SQLite初始化时能创建干净的数据库文件
- schema/sqlite_tables.db: 重新生成完整的SQLite数据库文件，包含所有平台的表结构和索引
2025-07-16 19:48:52 +08:00
程序员阿江(Relakkes)
c5509ab91f
fix: update ip proxy valid url for #662
2025-07-16 11:06:58 +08:00
程序员阿江(Relakkes)
c795b1316a
fix: import error for #663
2025-07-16 10:58:11 +08:00
程序员阿江(Relakkes)
3184e0a8d9
fix: sign data too long error #659
2025-07-14 18:43:17 +08:00
gaoxiaobei
4d743f6c17
debug & resume default configuration
2025-07-14 08:00:48 +08:00
