跳至正文
OpenAgentSkill
浏览已开放 · 目录记录日期 2026-11-21 · 自动更新和账户服务暂时关闭。
搜索与研究

literature-live-session-pipeline

Orchestrate lawful literature collection through live Microsoft Edge sessions across ScienceDirect, CNKI, and Web of Science, then create an EndNote X9 library, import RIS metadata, and group records by section_paths from a manifest CSV. Use when browser login or bot verification blocks direct fetching, when the user wants separate browser sessions per source, or when a paper batch must be downloaded into a folder and mirrored into a grouped EndNote library.

Given-Dream · 已有 AI 审核记录

安装与使用

运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。

npx skills add Given-Dream/sciencedirect-live-session-fetcher --skill literature-live-session-pipeline

技能说明


name: literature-live-session-pipeline description: Orchestrate lawful literature collection through live Microsoft Edge sessions across ScienceDirect, CNKI, and Web of Science, then create an EndNote X9 library, import RIS metadata, and group records by section_paths from a manifest CSV. Use when browser login or bot verification blocks direct fetching, when the user wants separate browser sessions per source, or when a paper batch must be downloaded into a folder and mirrored into a grouped EndNote library.

Literature Live Session Pipeline

Use this skill when the browser session is the source of truth and the batch needs both files and reference-management structure.

Quick Start

  1. Prepare the manifest CSV. Read references/input-schema.md. The core columns are number, title, doi, source_hint, entry_url, candidate_urls, and section_paths. If the source document is a Word file with EndNote EN.CITE fields, use scripts/extract_manifest_from_endnote_docx.py first.

  2. Launch a source-specific Edge session. Use scripts/launch_edge_live_session.ps1. Keep separate ports for sciencedirect, cnki, and wos.

  3. Let the user finish the manual browser step. They must sign in lawfully, pass verification pages, open one representative record, and keep the window open.

  4. Probe the current page before the full batch. Use scripts/probe_live_session.py. If the probe still reports a challenge page, stop and let the user finish the browser step manually.

  5. Create the local EndNote library input. Use scripts/build_ris_from_manifest.py to generate RIS. Use scripts/create_endnote_library.py if a fresh library is needed.

  6. Import the RIS and group records in EndNote. Use scripts/endnote_pipeline.py. It can create the library, import RIS, and assign records to flat groups named from section_paths.

  7. Fetch the PDFs. Route CNKI rows to scripts/run_cnki_pdf_live_fetch.ps1. This route opens the exact-title detail page and clicks only the visible PDF下载 control; it never selects CAJ. Route other publishers to scripts/run_devtools_multi_source_fetch.ps1, which wraps scripts/devtools_multi_source_serial_fetch.py.

Commands

Launch a ScienceDirect browser session:

powershell -ExecutionPolicy Bypass -File "$HOME\.codex\skills\literature-live-session-pipeline\scripts\launch_edge_live_session.ps1" -Source sciencedirect

Launch a CNKI browser session:

powershell -ExecutionPolicy Bypass -File "$HOME\.codex\skills\literature-live-session-pipeline\scripts\launch_edge_live_session.ps1" -Source cnki

Probe a live session:

python "$HOME\.codex\skills\literature-live-session-pipeline\scripts\probe_live_session.py" --debug-port 9222 --url "https://example.com/article"

Build RIS from the manifest:

python "$HOME\.codex\skills\literature-live-session-pipeline\scripts\build_ris_from_manifest.py" `
  --input-csv C:\path\to\manifest.csv `
  --ris-out C:\path\to\batch\manifest.ris `
  --json-out C:\path\to\batch\manifest.records.json

Extract a manifest from an EndNote-embedded DOCX:

python "$HOME\.codex\skills\literature-live-session-pipeline\scripts\extract_manifest_from_endnote_docx.py" `
  --docx C:\path\to\manuscript.docx `
  --manifest-out C:\path\to\batch\manifest.csv `
  --report-json C:\path\to\batch\manifest.extract.json

Create or update the EndNote library:

python "$HOME\.codex\skills\literature-live-session-pipeline\scripts\endnote_pipeline.py" `
  --manifest-csv C:\path\to\manifest.csv `
  --ris-path C:\path\to\batch\manifest.ris `
  --library-name "UHS_THMD_batch" `
  --library-root C:\path\to\batch `
  --report-json C:\path\to\batch\endnote_report.json

Run the download batch:

powershell -ExecutionPolicy Bypass -File "$HOME\.codex\skills\literature-live-session-pipeline\scripts\run_devtools_multi_source_fetch.ps1" `
  -InputCsv C:\path\to\manifest.csv `
  -OutDir C:\path\to\batch\downloads `
  -DebugPort 9222 `
  -InterItemSleepSeconds 6

Run a CNKI PDF-only batch after the user has completed login or any visible verification:

powershell -ExecutionPolicy Bypass -File "$HOME\.codex\skills\literature-live-session-pipeline\scripts\run_cnki_pdf_live_fetch.ps1" `
  -InputCsv C:\path\to\cnki-manifest.csv `
  -OutDir C:\path\to\batch\cnki-downloads `
  -DebugPort 9223 `
  -ReuseExistingPage `
  -InterItemSleepSeconds 7

Operational Notes

  • Treat Web of Science as a record router, not a guaranteed PDF host. Expect it to resolve DOI or publisher links first.
  • For CNKI, use the dedicated PDF-only route: search result -> exact-title detail page -> exact visible text PDF下载. Do not use element IDs because CNKI can assign the same id="cajDown" to both CAJ and PDF anchors.
  • CNKI keeps dormant CAPTCHA markup on normal pages. A prompt found only in raw HTML or body text is not proof of a real slider; require a verification URL/title or a rendered prompt that intersects the viewport.
  • CNKI may write _blank downloads to the Windows Downloads folder even after a batch download directory is configured. The CNKI route watches both locations and copies, rather than removes, a valid system-download source.
  • Treat a PDF permissions-encryption flag as metadata, not automatic failure. Accept it when the file opens without a password, has at least two readable pages, and matches the expected title.
  • Keep separate Edge user-data directories by source. This is the default behavior of the launch script and
阅读完整来源 ↗