jiazhizhong/MediaCrawler
1
0
Fork 0
mirror of https://github.com/NanmiCoder/MediaCrawler.git synced 2026-06-10 03:47:26 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat: 新增 JSONL 存储格式支持，默认存储格式改为 jsonl

Browse Source 
JSONL（JSON Lines）每行一个 JSON 对象，采用 append 模式写入，
无需读取已有数据，大数据量下性能远优于 JSON 格式。

- 新增 AsyncFileWriter.write_to_jsonl() 核心方法
- 7 个平台新增 JsonlStoreImplement 类并注册到工厂
- 配置默认值从 json 改为 jsonl，CLI/API 枚举同步更新
- db_session.py 守卫条件加入 jsonl，避免误触 ValueError
- 词云生成支持读取 JSONL 文件，优先 jsonl 回退 json
- 原有 json 选项完全保留，向后兼容
- 更新相关文档和测试
This commit is contained in:
程序员阿江(Relakkes)
2026-03-03 23:31:07 +08:00
parent 4331b91fe1
commit 0282e626c9
29 changed files with 245 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README_en.md
View File

@@ -221,7 +221,7 @@ python main.py --help
## 💾 Data Storage
MediaCrawler supports multiple data storage methods, including CSV, JSON, Excel, SQLite, and MySQL databases.
MediaCrawler supports multiple data storage methods, including CSV, JSON, JSONL, Excel, SQLite, and MySQL databases.
📖 **For detailed usage instructions, please see: [Data Storage Guide](docs/data_storage_guide.md)**