Keep PR 900 overrides bounded and opt-in

The PR adds API limit overrides and static proxy support, but the review found that the default proxy provider changed to an invalid static placeholder and the new API fields accepted unbounded values. This keeps the existing proxy default intact, makes static proxy explicit via config or CLI, validates API limit ranges, and adds focused regression coverage for both paths.

Constraint: PR branch must remain contributor-branch compatible and avoid adding dependencies

Rejected: Keep static as the default provider | breaks existing --enable_ip_proxy defaults with an invalid placeholder URL

Rejected: Accept arbitrary integer limits | lets API callers request negative or excessive crawl sizes

Confidence: high

Scope-risk: narrow

Directive: Do not change proxy provider defaults when adding new providers; new providers should be opt-in and covered by provider-specific tests

Tested: uv run pytest tests/test_api_limits.py tests/test_static_proxy_provider.py

Tested: uv run pytest tests

Tested: uv run pytest test/test_utils.py

Tested: uv run python -m compileall api cmd_arg config proxy tests

Tested: git diff --cached --check

Not-tested: Live crawler run against external platforms or real proxy vendor endpoints
This commit is contained in:
程序员阿江(Relakkes)
2026-05-29 21:27:52 +08:00
parent f997befce9
commit 8e93438fe5
7 changed files with 126 additions and 34 deletions

View File

@@ -320,10 +320,18 @@ async def parse_cmd(argv: Optional[Sequence[str]] = None):
str,
typer.Option(
"--ip_proxy_provider_name",
help="IP proxy provider name (kuaidaili | wandouhttp)",
help="IP proxy provider name (kuaidaili | wandouhttp | static)",
rich_help_panel="Proxy Configuration",
),
] = config.IP_PROXY_PROVIDER_NAME,
static_proxy_url: Annotated[
str,
typer.Option(
"--static_proxy_url",
help="Static proxy URL, for example http://user:password@host:port",
rich_help_panel="Proxy Configuration",
),
] = config.STATIC_PROXY_URL,
) -> SimpleNamespace:
"""MediaCrawler 命令行入口"""
@@ -356,6 +364,7 @@ async def parse_cmd(argv: Optional[Sequence[str]] = None):
config.ENABLE_IP_PROXY = enable_ip_proxy_value
config.IP_PROXY_POOL_COUNT = ip_proxy_pool_count
config.IP_PROXY_PROVIDER_NAME = ip_proxy_provider_name
config.STATIC_PROXY_URL = static_proxy_url
# Set platform-specific ID lists for detail/creator mode
if specified_id_list: