mechanical-preprocess
Bash-first mechanical pattern cleanup pipeline — handles em-dash removal, forbidden words, and repetitive structures at scale BEFORE AI agents touch the text. Processes 30+ chapters efficiently.
felipelobomotta-blip · 已有静态检查记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add felipelobomotta-blip/book-genesis-studio --skill mechanical-preprocess技能说明
name: mechanical-preprocess description: Bash-first mechanical pattern cleanup pipeline — handles em-dash removal, forbidden words, and repetitive structures at scale BEFORE AI agents touch the text. Processes 30+ chapters efficiently.
Mechanical Preprocess
PURPOSE
AI agents are bad at processing entire manuscripts for mechanical fixes. Context windows overflow, attention drifts, and the agent "forgets" the rules by chapter 15. This skill solves that by splitting the work:
- Bash handles the 80% — pattern matching, counting, safe replacements
- AI handles the 20% — judgment calls on ambiguous cases, reviewing diffs
This is the ONLY skill in the pipeline that uses bash scripting as its primary tool. It exists because some problems are engineering problems, not language problems.
WHEN TO RUN
- After: All chapters have been through prose-craft (complete draft exists)
- Before: dialogue-polish, chaos-engine, or any AI-based editing pass
- Trigger: Orchestrator calls this once when the full manuscript draft is ready
- Re-run: After any major rewrite pass that may reintroduce mechanical patterns
REQUIRED INPUTS
- Chapter files — all chapter drafts in
chapters/directory (e.g.,chapters/chapter-01.mdthroughchapters/chapter-[N].md) - voice-dna.md — for:
- Forbidden word list
- Em-dash policy (max per chapter, allowed contexts)
- Any other mechanical rules (e.g., max semicolons per chapter, banned constructions)
- foundation.md — for genre context (affects which patterns are acceptable)
PROCESS
PHASE 1: SCAN (Pure bash -- no AI)
Run these scans against every chapter file. All commands use standard Unix tools (grep, wc, awk, sed).
1.1 Em-Dash Census
# Count em-dashes per chapter (both spaced and unspaced variants)
for f in chapters/chapter-*.md; do
echo "$(basename $f): $(grep -oP '(\x{2014}| — )' "$f" | wc -l) em-dashes"
done
Categorize each em-dash by context:
- Independent clause separator —
[complete sentence] — [complete sentence](TARGET FOR REMOVAL) - Parenthetical aside —
word — aside — continuation(TARGET: convert to commas or restructure) - Dialogue interruption —
"I was going to—"(KEEP — this is correct usage) - List/appositive —
three things — money, power, fame — were gone(EVALUATE case by case)
1.2 Pattern #11 Census (Formulaic Constructions)
# "not because X but because Y"
grep -cnP 'not because.*but because' chapters/chapter-*.md
# "the kind of X that Y"
grep -cnP 'the kind of \w+ that' chapters/chapter-*.md
# "not as X but as Y"
grep -cnP 'not as \w+ but as' chapters/chapter-*.md
# "there was something X about Y"
grep -cnP 'there was something \w+ about' chapters/chapter-*.md
# "it was the sort of X that Y"
grep -cnP 'it was the (sort|kind|type) of' chapters/chapter-*.md
# "as if X were Y" (excessive simile construction)
grep -cnP 'as if .{5,40} were' chapters/chapter-*.md
# Doubled constructions "X and X" where both are abstract
grep -cnP '(grief|loss|pain|joy|love|fear|hope|shame|guilt|rage) and (grief|loss|pain|joy|love|fear|hope|shame|guilt|rage)' chapters/chapter-*.md
1.3 Forbidden Word Census
Extract the forbidden word list from voice-dna.md and scan:
# For each forbidden word in the list
for word in "palpable" "tangible" "visceral" "whilst" "gaze" "orbs" "ministrations" "utilize" "plethora" "myriad" "ubiquitous" "dichotomy" "juxtaposition" "paradigm" "trajectory"; do
echo "--- $word ---"
grep -cnP "\b${word}\b" chapters/chapter-*.md
done
Note: The actual forbidden word list comes from voice-dna.md. The list above is a default fallback. Always prioritize the project-specific list.
1.4 Repetition Scan
# Sentence-start repetition: same first word in consecutive sentences
# (catches "She did X. She did Y. She did Z." patterns)
# Extract first word of each sentence per chapter
for f in chapters/chapter-*.md; do
grep -oP '(?<=\. |^)[A-Z][a-z]+' "$f" | uniq -cd | sort -rn | head -20
done
# Paragraph-start repetition
for f in chapters/chapter-*.md; do
grep -oP '(?<=\n\n)[A-Z][a-z]+' "$f" | uniq -cd | sort -rn | head -10
done
# Word frequency outliers (words appearing 3x+ per 1000 words beyond normal frequency)
for f in chapters/chapter-*.md; do
total=$(wc -w < "$f")
echo "=== $(basename $f) ($total words) ==="
tr '[:upper:]' '[:lower:]' < "$f" | tr -cs '[:alpha:]' '\n' | sort | uniq -c | sort -rn | \
awk -v total="$total" '$1 > (total/1000)*3 && length($2) > 4 {print}'
done
1.5 Adverb Density
# Count -ly adverbs per chapter
for f in chapters/chapter-*.md; do
total=$(wc -w < "$f")
adverbs=$(grep -oP '\b\w+ly\b' "$f" | wc -l)
echo "$(basename $f): $adverbs adverbs in $total words ($(echo "scale=1; $adverbs*1000/$total" | bc)/1000 words)"
done
Target: fewer than 15 adverbs per 1000 words for literary fiction. Adjust per genre.
1.6 Scan Report
Compile all counts into evaluations/mechanical-scan-report.md:
# Mechanical Scan Report
## Em-Dash Count
| Chapter | Total | Clause Sep | Parenthetical | Dialogue | Other |
|---------|-------|------------|---------------|----------|-------|
| ch-01 | 14 | 8 | 4 | 2 | 0 |
## Pattern #11 Count
| Chapter | not-because | kind-of-that | not-as-but-as | something-about | sort-of | as-if-were | doubled-abstract |
|---------|-------------|-------------|---------------|-----------------|---------|------------|-----------------|
| ch-01 | 2 | 1 | 0 | 3 | 1 | 4 | 0 |
## Forbidden Words
| Word | ch-01 | ch-02 | ... | Total |
|------------|-------|-------|-----|-------|
| palpable | 1 | 0 | | 1 |
## Repetition Flags
[per chapter list of repeated sentence starts and high-frequency words]
## Adverb Density
| Chapter | Adverbs | Words | Rate/1000 | Status
阅读完整来源 ↗