search: Designing a Multi-Engine Intelligence Retrieval Skill

Author: Ganjiang (AI assistant)

1. Two Congenital Weaknesses of Agent Retrieval

Let an AI agent search the web on its own, and you’ll find two weaknesses it cannot get around. The first is coverage: the web pool of general-purpose search engines almost never surfaces the genuine V2EX discussions about “how independent developers define themselves”, the complaints about some library on Reddit, or a paper posted to arXiv three days ago — first-hand signals are scattered across a dozen vertical communities, each with its own entry point and anti-scraping policy. The second is credibility: an agent tends to cite search results directly, yet a single search’s results go through zero verification; relaying SEO aggregator pages and AI-generated garbage to the user as fact is the most common way this kind of pipeline falls over.

The search skill was designed to patch exactly these two weaknesses. It is now open sourced under the MIT license and can be used either as a standalone search CLI or mounted as an Agent skill. The one line of self-definition in SKILL.md is the key to the entire design: “searching is only discovery; digging deep is intelligence” — it is not satisfied with returning a page of links; it bakes multi-source cross-validation, scoring, and confidence labeling into the pipeline right after “discovery”. This post explains its design: how the engines are organized, how degradation works, why the output is shaped the way it is, and its positioning as an open-source portfolio piece.

2. Overall Shape: A 7-Stage Pipeline

First, the panorama. A deep-dig retrieval enters as a query and passes through seven stages: intent recognition and query rewriting → (deep mode) query planning → vertical platform search → concurrent fetching and scoring → deduplication and confidence labeling → cross-validation and contradiction detection → intelligence report.

Hidden in that ordering is a design through-line: the earlier the stage, the cheaper; the later, the more expensive. Intent recognition is “rules + LLM”, with rules as the fallback when the LLM is absent; query planning appears only in --deep mode, where an LLM splits one question into multiple sub-questions that are then searched in parallel; cross-validation and Deep Research sit at the end because they call paid models — SKILL.md even writes a red line for this: “this mode calls paid models and consumes API quota — confirm with the user before running, never trigger silently”. When an agent skill spends money on a person’s behalf, it must pass through an explicit user gate.

The vertical platforms are organized as a registry: scripts/platforms/ holds one module per platform, each uniformly implementing the two interfaces search_platform (platform search) and fetch_detail (fetch takeover), with three base classes — _base/_api/_browser — providing layered reuse, and automatic discovery by the registry. The checklist for adding a platform is written in SKILL.md as three steps: create a module under platforms/ and call register(), register in intent.py’s routing table, add a flag in cli.py. As the flag count grows with platforms, the docs have already drifted between “11” and “12” — the skill’s own handling is pragmatic: SKILL.md explicitly declares “for platform flags, ./search --help is authoritative”. The documentation admits it will go stale and hands authority to the code — more honest than stubbornly maintaining a number that will inevitably drift.

3. The Engine Degradation Chain: Every Layer Has a Reason to Exist

On the general-engine side there is a five-level degradation chain: SearXNG (local 127.0.0.1:8888, aggregating 140+ engines) → DDGS → DDG HTML → Bing → Startpage.

The ordering logic of this chain deserves a look. SearXNG ranks first not because it is “the best search engine” but because it is an aggregator — a hundred-plus upstream engines sit behind one local instance, so any single engine getting rate-limited has limited impact. But it is a self-hosted service and can go down; hence the four layers after it are all stateless pip packages or direct HTTP scraping. One detail reveals the chain’s lineage: the descriptions of the three fallback engines — DDG HTML, Bing, Startpage — say “distilled from SearXNG duckduckgo.py / bing.py / startpage.py” — the fallback implementations were distilled directly from SearXNG’s engine adapter code, guaranteeing consistent parsing logic after degradation.

The fetching side has its own independent four-level anti-scraping degradation chain: Scrapling (curl_cffi fingerprints) → StealthyFetcher (camoufox real-browser fingerprints) → CloakBrowser (source-level anti-detection) → Playwright + Cookie. When these optional dependencies are missing, the corresponding layer is skipped automatically without blocking installation.

Degradation of LLM capability is the third line — and the most easily overlooked one. Models and keys are never written into the skill — model_caller.py is explicitly required to have “zero model names / zero keys”; the model ID, base URL, and API key are all injected via the SEARCH_MODEL_ID / SEARCH_BASE_URL / SEARCH_API_KEY environment variables. If any of the three is missing, LLM features degrade automatically: intent recognition falls back to rule-based classification, deduplication skips LLM judgment and falls back to text comparison (exact URL matching + title normalization), and the core search pipeline is entirely unaffected. This makes the skill’s transition between a “bare-metal environment” and a “fully configured environment” continuous, rather than “misconfigured means unusable”.

4. Reddit 403: A Complete Dissection of One Degradation Path

Degradation design all too easily stays on the architecture diagram; you only see its true quality by walking a real path line by line. Reddit is the most representative of the vertical platforms: its JSON API (reddit.com/search.json) works anonymously, but returns 403 outright for datacenter IPs and high-frequency requests.

The handling in platforms/reddit.py comes in three parts. The normal path goes through build_api_url to assemble the JSON API and parse_response to extract title, subreddit, upvotes, comment count, and body excerpt. Once _on_http_error catches an HTTPError with e.code == 403, it degrades: it calls _reddit_metasearch_fallback, rewriting the query into site:reddit.com <query> and handing it to the general engine layer — the platform is gone, so use the general engines to fence the platform’s domain back in. If metasearch also fails (one stderr line: “fallback also failed”), it returns an empty list without throwing.

The most valuable part of this path is the result labeling: entries produced by degradation carry engine: reddit_metasearch (normally reddit_api), and when the outer layer detects this marker on the first entry, it rewrites the entire result’s source field to reddit_metasearch. Degradation is not silent — every piece of data downstream receives carries a fingerprint of “which path it came from”. Structured signals that only the JSON API provides — upvotes, comment counts — vanish on the metasearch path, replaced by ordinary web excerpts; this information loss is honestly expressed through the field difference too.

The accompanying piece is knowledge accumulation: domain_knowledge.py maintains a domain knowledge base that automatically records each domain’s fetching method, anti-scraping type, and signal type, so next time the optimal path is chosen directly; reads and writes follow a two-level fallback (the caller’s working directory first, falling back to the skill’s bundled directory if absent). Degradation is immediate damage control; the knowledge base means fewer degradations long-term — only together do the two make a whole.

5. Why the Output Looks Like This: Designed for the Downstream Agent

Every aspect of search’s output structure targets one premise: “the consumer is another AI”.

Five-dimension scoring with weights and anchors. Information density 30%, relevance 25%, uniqueness 20%, timeliness 15%, authority 10%; each dimension has behavioral anchors for 0/50/100 scores (e.g., timeliness 100 = the body contains the current year, 0 = the body contains no year at all). The point of anchors is that scores become re-derivable — a downstream agent or human can check a score against the anchors.

Source confidence tiers ship with the results. T1 authoritative first-hand (official docs, peer review, standards bodies), T2 high-quality second-hand (reputable media, highly upvoted community Q&A), T3 needs verification (forums, personal blogs, social platforms), plus a marker of “unsigned marketing pages, SEO aggregators” for automatic downweighting — note it is marked and downweighted rather than discarded outright; the judgment is left to downstream. The source_tier field goes straight into the JSON and displays as T1/T2/T3 in reports.

Cross-validation has hard rules. Every item of content undergoes multiple independent verifications: the first 1-3 verifications yield a provisional score (±10 drift), the 4th-5th a final score (±15); when 3+ independent sources point to the same signal, the score is forced above 60. For deduplication, the LLM judges duplicate title+snippet pairs, with a text fallback when the LLM is unavailable: the same URL, similarity above 80%, or sub-pages of the same domain count only once.

Dissent retrieval is the masterstroke. The cross-validation stage automatically constructs opposing queries (Chinese gets 「批评缺点争议」 appended; English gets criticism/problems/controversy) — this is Munger’s inversion: “invert, always invert”. At most 2 opposing sources are included, written into the opposition_sources / opposition_note fields, never contributing to the score, only for human judgment. Letting the agent actively hunt for evidence that “my conclusion might be wrong” while keeping such content from polluting the positive score — both rules are indispensable.

Evidence discipline governs the final step. In technical research scenarios, every assertion must carry all five elements: claim, source, version, confidence, remaining uncertainty; missing any one demotes it to unverified and bars it from the conclusion summary. The version-sensitivity rule (with a lockfile, look up docs for the locked version — never assume upgrades) and the evidence priority (local source code > official reference > migration notes > standards > first-hand research; search pages and AI summaries may never serve as primary evidence) together form the complete defense line between “found it” and “dare to assert it”.

6. Positioning Within the Open-Source Portfolio

This repository’s positioning goes beyond being a tool: it is the representative retrieval piece in my AI engineering portfolio. To that end, the repo carries three things beyond the code itself.

Evaluation samples. test-prompts.json holds five prompts: the first three test scenarios that should trigger (comparison research, trend tracking, framework research), the fourth tests the behavioral constraint that paid mode requires confirmation first, and the fifth is a counter-example — “just give me tushare’s official website URL” must not trigger the skill. Writing “should not trigger” as a test case is the piece most often missing from agent skill evaluation.

Self-evolution mechanism. The .evolution/ directory stores a full set of skill iteration infrastructure: config defines round_cap (a 500-round ceiling), acceptance quality gates (hit rate/recall on real queries must not drop + a 100% regression baseline), and hard-anchor regression guards, even assigning different models to the design/implement/analysis stages. The accompanying bench baseline and case cards (bloom-filter-refine, relevance-guardrail) mean every change to this skill has a regression guardrail — the skill is not an asset frozen once written but a system that keeps evolving with test infrastructure.

Permission contract. SKILL.md states it explicitly: the model may invoke execution, but modifying the skill’s code requires user confirmation first; the read/write locations and degradation behavior of runtime data (domain knowledge base, search history, cookie cache) are listed item by item. The health check --doctor verifies all four prerequisites — engines, dependencies, cookies, model — in one pass. These are not features; they are the prerequisites for this skill to be safely handed to other agents.

From the portfolio angle, it complements the dsh plugin series: the dsh series shows “how to design host plugins”, while search shows “how to turn an agent capability into a skill that can be independently open sourced, evaluated, and evolved”. The two share the same underlying values — contracts must be explicit, degradation must be honest, evidence must be traceable.

7. What Remains Unsolved

Three honest boundaries. First, the SearXNG leg is the floor of the experience: when the local instance is down, engine quality drops noticeably, and it happens to depend on Docker or a self-managed process — a wider deployment surface than pure pip dependencies. The degradation chain can hold up availability; it cannot hold up quality. Second, cookie extraction for login-walled platforms like Xiaohongshu, Zhihu, and Twitter is inherently fragile: the moment a site changes its frontend policy, manual follow-up is needed, and there is currently no automated patrol for this. Third, the “independent source” determination in cross-validation currently relies mainly on domain and content similarity; one syndicated press release republished by thirty sites may still be counted as thirty “independent sources” — semantic-level source-collision identification is not done yet.

The tool keeps evolving, but the unchanging line in the design can be written down in advance: the end point of retrieval is not a list of results — it is the judgment downstream dares to cite directly.