This reproducible Python project stores Crossref metadata in a local SQLite database and exports CSV deliverables. Historical retrieval uses only the Crossref REST API; the optional dynamic keyword workflow uses the local 2batch LLM setup to construct and screen Boolean queries. It does not use Zotero, PDFs, Google Scholar, or OpenAlex.
cd /Users/zhaoxun/scripts/test/PaperSearch
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install -e .
cp .env.example .env
# Edit .env and set a monitored CROSSREF_MAILTO address.config/journals.csv contains the verified print and online ISSNs used by the project. To extend the scope, add a journal row and/or edit the two explicit term lists in src/papersearch/config.py.
Query one DOI exactly in the local data_processed/crossref_articles.sqlite database without an LLM or network request:
papersearch doi-title 10.2307/1907296The command emits stable JSON. A match has found: true and returns doi, title, journal, and year; a miss returns {"found": false, "article": null}. The paperdownload JSTOR fallback uses this interface.
The initial historical retrieval is done once. Its deduplicated master_articles.csv is then loaded into data_processed/crossref_articles.sqlite.
# Only needed for a new empty project; downloads the full historical corpus.
papersearch retrieve-all
papersearch bootstrap
# Compatibility alias for the created-date daily update.
papersearch runpapersearch run is a compatibility alias for papersearch --update and therefore uses the current Crossref-created date plus the previous two days. papersearch process always rebuilds the CSVs from the SQLite database without network access. The database uses DOI first, then normalized title/journal/year only when DOI is missing. Run tests with:
macOS runs the daily job directly through the user LaunchAgent at ~/Library/LaunchAgents/com.zhaoxun.papersearch.daily-update.plist. It invokes the local PaperSearch CLI at 14:00 system time and does not require Codex/GPT task scheduling.
The scheduled entry point is:
# Daily/manual default: today and the previous two Crossref-created dates.
papersearch --update
papersearch -u
# Explicit created-date year, month, or day.
papersearch --update 2026
papersearch -u 2026-08
papersearch -u 202608
papersearch -u 2026/08
papersearch -u 2026-08-09
papersearch -u 20260809
papersearch -u 2026/08/09It checks every journal currently listed in config/journals.csv. With no date value it queries Crossref created dates for the local run date and previous two days. If one or more calendar days elapsed after the last successful automatic update, it expands the recovery range to start seven days before the earliest missed date and end on the current run date. Explicit values accept fixed year-month-day order as YYYY, YYYY-MM, YYYYMM, YYYY/MM, YYYY-MM-DD, YYYYMMDD, or YYYY/MM/DD. The discovery query uses from-created-date and until-created-date, not publication dates: Crossref recommends created for finding new records because publication dates can change and can differ from registration time. Every inclusive range is split into non-overlapping windows of at most seven days, each with its own cached raw response and resumable cursor state.
A run is marked complete in SQLite only after every ISSN cursor is complete, all new records have passed DOI-first deduplication, the schema-validated 2batch translations and summaries and timestamped Markdown report have been written, and the database upsert has succeeded. One 2batch row per new paper returns title_zh, abstract_zh, summary_en, and summary_zh; no GPT scheduler performs translation or summarization. The retriever identifies itself with mailto plus User-Agent, runs serially below the limits returned in Crossref response headers, honors Retry-After, uses bounded exponential backoff when needed, and treats 403/other permanent client errors as failed runs. Run metadata is stored in automatic_update_runs; the append-only summary is logs/automatic_update_runs.jsonl; request CSVs and full logs are under logs/automatic_update/; resumable Crossref cursors and 2batch artifacts remain under state/. The default report directory is /Users/zhaoxun/Library/CloudStorage/坚果云-425841907@qq.com/临时/PaperSearch. Each report groups all newly identified papers by journal and includes the original and Chinese title, authors, DOI, Crossref's created first-registration timestamp, bilingual abstract when Crossref provides one, and bilingual 2batch summaries. The created timestamp is stored separately from publication dates. Crossref notes that abstracts retain publisher or author copyright.
pytest -qpapersearch search performs no network requests. It builds/refreshes a SQLite FTS5 index over title, abstract, and subject, then produces the independent four-route outputs 01_direct_insurance_complaints.csv, 02_insurance_consumer_friction.csv, 03_insurance_complaint_data.csv, 04_seed_citation_candidates.csv, 06_abstract_still_missing.csv, and 07_manual_screening.csv. Direct complaints, consumer friction, and regulator/data papers are never combined into one score. subject cannot by itself admit an article.
papersearch regtech makes no Crossref requests. It searches the existing SQLite FTS5 database and writes regtech_direct.csv, regtech_mechanism.csv, suptech.csv, regtech_all_candidates.csv, regtech_llm_candidates.csv, and regtech_summary.csv. The project uses RegTech（监管科技） for firm-side compliance/regulatory technology, and SupTech（监管技术，监管机构侧） for supervisory authorities' technology applications. The editable, auditable terms live in config/regtech_terms.json.
Direct terminology is admitted only from title or abstract. A mechanism candidate requires a specific regulatory/compliance workflow (for example, regulatory reporting, AML/KYC, supervisory data) and an explicit technology marker to co-occur in the same title/abstract sentence; subject is evidence only and never admits a record. For high-recall LLM screening, regtech_llm_candidates.csv is deliberately broader: it retains any title/abstract combination of broad regulatory/supervisory and technology terms, labels it llm_review_required, and makes no local relevance assertion. These remain metadata candidates, not a complete conceptual universe, because Crossref abstracts and subjects are incomplete.
papersearch enrich-abstracts is optional. S2_API_KEY in .env is used when available, but the command can use the public endpoint without it. It only submits DOI batches (maximum 500) for insurance-title records whose Crossref abstract is absent, caches every Semantic Scholar response under data_raw/semantic_scholar/, and stores a separate Semantic Scholar abstract without overwriting Crossref metadata. After enrichment, run papersearch search again.
After the local SQLite database has been bootstrapped, pass one JSON object whose single key is the core term and whose value is a list of annotations:
papersearch-keywords --search-type modifier_head '{"RegTech":["广义的监管所使用的技术手段，包括外部的SEC、法院等，也包括内部的内控、内审、合规等","必须是技术，算法、程序、模型等"]}'The core-term key is the sole conceptual anchor and is assumed to be a modifier-head construction AB. Definition generation happens entirely in step 1. Five independent candidate definitions first explain the integrated concept and its boundaries. A single Ann-guided correction stage then reconciles all five candidates and emits the definitive structured contract: English definition prose, atomic required conditions, optional indicators, explicit inclusions, explicit exclusions, and boundary rules. Every downstream retrieval and classification call reuses that same final object; no later stage may silently add a condition or narrow the definition. Ann entries are not direct search terms. The generated Boolean lists are still used for --keywords, while the final structured definition is used for --rag.
For ordinary --keywords runs, the original annotations act as user_requirements only while the final structured definition is created; this is the same definition workflow used by RAG, and the requirements are not passed downstream as a second competing policy. Relevance scoring and classification then use the validated structured_core_definition. Step 4 sends only articles whose mean relevance score is greater than zero to the reusable 2batch LLM committee. Three members review each selected article independently; material disagreement or a fatal objection triggers bounded reconsideration and automatic escalation to five members. The committee's schema-validated class, review subtypes, and English final commentary are definitive, after which the commentary is translated to Chinese for export. Final workbook names are explicitly mode-labelled: _keywords.xlsx, _rag.xlsx, or _rag_iter.xlsx, with an additional _broad tag when applicable.
--boolean is keywords-only. It stops after local Boolean screening, skips relevance scoring and every classification/commentary stage, translates the title of every locally screened article in one schema-validated batch, and writes one timestamped _keywords_boolean.xlsx. Its columns are 标题翻译, DOI, 期刊, 年度, 篇名, and 摘要.
--direct is a zero-LLM keywords mode. Supply one or more terms directly after --keywords; they are combined with OR and searched locally without rewriting. Literal matching is case-insensitive and requires a non-word boundary on both sides, so a term can occur at the start or end or next to spaces, hyphens, and punctuation, but not inside another word. * wildcards and explicit /pattern/i regular expressions are also supported. Add --title to require the match in the title, or --abstract to require it in the abstract; these two field options are mutually exclusive. The mode writes one timestamped _keywords_direct.xlsx with DOI, 期刊, 年度, 篇名, and 摘要, and cannot be combined with --boolean, --broad, --rag, --iter, or --exact.
RAG uses a 3-sigma similarity threshold by default. Override it per run with any non-negative finite number, for example --sigma 2, --sigma 2.5, or --sigma 3:
papersearch --rag '{"RegTech":["广义监管技术","必须是技术"]}' --sigma 2.5Use --exact to skip all LLM Boolean-query design. It searches only the core-term key; annotations are not searched directly:
papersearch-keywords --exact '{"RegTech":["仅按核心词进行精确检索"]}'
# Reuse a saved final_keyword_list.json and skip query design
papersearch-keywords /Users/zhaoxun/Downloads/20260719_162549_regtech-regulatory-technology/final_keyword_list.json
# When strict scoring yields too few papers, retain more potentially related papers
papersearch-keywords --broad '{"RegTech":["广义监管技术","必须是技术"]}'
# Export all locally screened papers with Chinese title translations only
papersearch-keywords --boolean '{"RegTech":["广义监管技术","必须是技术"]}'
# Search unchanged terms locally with OR logic and no LLM calls
papersearch --keywords "SEC" "CSRC" "Securities and Exchange Commission" "China Securities Regulatory Commission" --direct
# Require one of those terms to occur in the title (use --abstract for abstract-only matching)
papersearch --keywords "SEC" "CSRC" "Securities and Exchange Commission" "China Securities Regulatory Commission" --direct --title
# Iteratively redesign the RAG definition, sigma-screen only new papers, and merge classifications
papersearch --rag '{"RegTech":["广义监管技术","必须是技术"]}' --sigma 3 --iter 5--iter ROUNDS repeats definition generation and sigma screening. ROUNDS controls how many consecutive completed rounds without a new relation-class-2 article are required for convergence; for example, --iter 5 stops after five such rounds. Writing only --iter remains valid and defaults to three rounds. In this mode only, step 1 uses temperature 1 (the ordinary mode remains at 0.5). After sigma screening, only articles not retained by an earlier round go directly to the combined commentary-and-classification request; the three-round relevance scoring stage is not run. During iteration, classes 1–2 are retained and deduplicated by DOI, with normalized title + journal + year as the fallback identity; class 0 is not retained and remains eligible for sigma screening and reclassification in the next round. Once convergence is reached, every retained article is reviewed by the reusable 2batch LLM committee. Three members first conduct blind reviews using English intermediate commentary; a high-risk decision requires unanimity. Material disagreement or a fatal objection triggers one bounded reconsideration, and unresolved cases automatically add the boundary reviewer and evidence auditor to form a five-member committee whose high-risk threshold is four votes. The final adjudicator rereads the structured definition, title, abstract, assessments, objections, and vote context, then emits the definitive integer class, schema-validated review subtypes, and Chinese final commentary. Committee inputs are processed in resumable 100-article parts, with actual rendered prompts, responses, retry histories, and costs kept in the checkpoint directory until completion. The final workbook omits the numeric relation-class column: 综述类别 displays class 0 as 可能相关, class 1 as 背景, and class 2 as 核心： followed by its validated subtypes. Completion leaves one timestamped _rag_iter.xlsx, without a containing output folder, similarity columns, or LLM-score columns; the checkpoint directory is then deleted.
The default modifier_head search type creates a named JSON Boolean tree for a general modifier-head concept. It first generates the default AB definition, refines it with Ann, designs direct_topic, then passes that validated part together with the original list to two parallel prompts: modifier (the qualifying or attributive concept) and head_concept (the qualified central object). The final tree is always composed as:
The default-definition stage shows only its progress bar in the terminal; its verbose intermediate definition is hidden. The final definition after Ann refinement remains visible because it is the definition used by all downstream stages.
direct_topic OR (modifier AND head_concept)--exact skips LLM design and searches only the original terms as one OR group. Query grammars are dispatched through SEARCH_TYPES; a future X+Y search can add its own grammar and prompt set without changing the modifier-head logic.
Terms support * wildcards and /pattern/i regular expressions. Each of the three generated lists is produced independently five times, then merged into a case-insensitively de-duplicated union before final composition. Every one of those LLM requests has its own JSON schema validation and receives at most 10 attempts.
All LLM prompt and validation definitions are directly editable JSON files in config/keyword_search_llm/:
core_definition_prompt.json— the annotation-free default AB definition template.core_definition_refinement_prompt.json— the Ann update/qualification stage that preserves the default AB structure.direct_topic_prompt.json,modifier_prompt.json, andhead_concept_prompt.json— the three Boolean-query design prompts and their matching schemas.review_prompt.jsonandreview_schema.json— three-round relevance scoring. The reviewer returns only one integer field,relevance_score(0–10).commentary_prompt.jsonandcommentary_schema.json— Chinese commentary for non-zero scores.
Prompt files contain system_prompt and user_prompt_template. Stage-one prompts receive {core_term} and {annotations} separately, followed by generated context such as {core_definition}, {modifier_role}, {head_role}, and {direct_condition}. Review prompts continue to use {title}, {abstract}, and {mean_score} as applicable.
The terminal prints a compact horizontal summary of the Boolean rule before local filtering. Each candidate is labelled with its retrieval basis: direct_topic when it contains a direct target-concept term, otherwise modifier_head_match. The reviewer uses only the title and abstract, and judges relevance exclusively against the complete direct_topic term list; modifier and head-concept terms are retrieval aids only.
Candidates are then sent through three independent, schema-validated LLM relevance-score rounds. The output is an integer on this scale: 0 unrelated; 1–3 incidental or disjoint mention; 4–6 plausible but requiring review; 7–8 clearly studies the target concept; 9–10 directly studies it. The arithmetic mean across the three rounds is retained. Papers with a mean score above zero receive one additional Chinese-commentary call at [4/5]; RAG-mode commentary has thinking disabled. 2batch displays one aggregate LLM requests progress bar for each round. The right side of every LLM stage header is derived from that stage's generated call config, so model, temperature, thinking, rounds, and aggregation switch with the actual mode instead of a separate display constant. At the very end, PaperSearch prints [5/5] Run summary with separate DeepSeek consumption and Elapsed time subsections. The first groups money-only costs by task and total; the second reports aligned per-step wall times and a total. Intermediate files are kept in a temporary hidden folder under /Users/zhaoxun/Downloads; after a successful run that folder is deleted and only YYYYMMDD_HHMMSS_<terms>_<mode>.xlsx remains.
Definition RAG retrieval uses the top 4,096 vector neighbors as its reference sample. It computes that sample's mean and sample standard deviation and retains every article whose raw vector similarity is strictly greater than mean + sigma × std. Sigma defaults to 3 and can be changed with --sigma. There is no reranking stage or rerank score. The Similarity table reports both the 4,096-row reference sample and the dynamically selected sigma sample; the n column contains each sample's actual size.
In the final RAG workbook, 相似度得分 is the article's z-score relative to the top-4,096 reference sample, expressed directly in standard deviations rather than Min-Max normalized. LLM打分 is the original 0–10 mean across the three scoring rounds and is not divided by 10.
Every LLM request receives at most 10 attempts in total: one initial request and up to nine retries when its JSON fails schema validation.
config/update_policy.json is the explicit, manual completeness policy. Its current complete_through_year is 2025: records published in 2025 or earlier have literature_completeness=complete; 2026 records have literature_completeness=provisional. Every article CSV includes both complete_through_year and literature_completeness. This cutoff never advances automatically: change it only after explicitly requesting a refresh and confirmation of that year, then run papersearch process.
For the one-time historical build, the retriever calls GET /v1/journals/{ISSN}/works with filter=type:journal-article, rows=1000, and cursor pagination. Routine updates use inclusive from-created-date and until-created-date filters split into non-overlapping weekly windows. ISSN requests are intentionally serial. Each completed page is appended to scope-specific JSONL and its next cursor/page/completion flag is written atomically to state/, allowing interruption and resumption. Requests send a polite User-Agent plus mailto, learn the current Crossref rate/concurrency headers, honor Retry-After, use bounded fallback backoff when the header is absent, and retry transient connection/timeout/5xx/invalid-response failures with exponential backoff and jitter.
All output files are UTF-8-SIG CSVs in data_processed/:
master_articles.csv— deduplicated articles from all journals.insurance_articles.csv— master records whosetitle,abstract, orsubjectmatches the fixed insurance list.insurance_complaint_articles.csv— strict subset of insurance records also matching the fixed complaint list.duplicate_report.csv— all duplicate groups and their DOI/fallback merge basis.journal_summary.csv— counts, year range, and metadata coverage by journal.retrieval_log.csv— request/page outcome, retry count, and error text.
DOIs are lowercased and stripped of DOI URL/prefix variants before deduplication. Records lacking a DOI are deduplicated on normalized title, journal, and year. The raw Crossref ISSN, queried ISSN, and raw JSONL source filename remain in the outputs.
Crossref abstract and subject coverage is incomplete. Therefore the two samples represent exact fixed-term matches in available Crossref metadata only; they are not a complete conceptual universe of insurance or complaints research. Missing abstracts and subjects are never inferred or filled.