Internal Engineering Memo Pipeline Automation Confidence: High Last Revised 25-04-2026

The Website
Factory. A finite, failure-tolerant pipeline for replicating niche sites at scale.

Stop letting the CEO agent invent the process. The recipe lives in YAML, the state lives in Postgres, n8n drives the engine, and Paperclip executes the work. Each website moves through ten deterministic stages — independently, with retries, validation gates, and human escalation when machines run out of judgement.

§ 01

Research

Researcher

━━▶

§ 02

Strategy

Strategist

━━▶

§ 03

Content

Writer

━━▶

§ 04

Design

━━▶

§ 05

Build

Developer

━━▶

§ 06

Legal

━━▶

§ 07

Analytics

Google

━━▶

§ 08

Tester

━━▶

§ 09

Deploy

DevOps

━━▶

§ 10

Audit

Tester

━━▶

END

Complete

∞ forever

§ 02 / Main Principle

Do not let the CEO agent invent the process. The process should live in a deterministic workflow definition — what must happen, where each website is, and who advances it.

§ 03

A factory of four parts

The recipe is in GitHub. The state is in Postgres. The engine is n8n. The hands are Paperclip agents. Humans appear only where judgement is required.

Component	Responsibility
Niche DB	Source list of websites to process — the inbox of the factory.
GitHub Playbook	Stores workflow YAML, issue templates, checklists, and validators. The recipe.
Postgres	Stores workflow run state and stage history. The memory of every job.
n8n	Runs intake, monitor, validation, retry, and escalation. The engine.
Paperclip	Creates companies, issues, agents, and audit trail. The control plane.
Agents	Execute scoped Paperclip issues — research, write, design, build, deploy.
Humans	Approve secrets, billing, legal, and risky deploys. Nothing routine.

The CEO/PM agent should handle exceptions, ambiguity, prioritization, and delegation — not remember the entire factory process from a prompt.

§ 04

Recipe versus run-state

Two different concepts, two different homes. Conflating them is how factories collapse into chaos.

Workflow Definition

The recipe

Stage IDs, agent roles, issue templates, required inputs and outputs, validation checklists, retry policy, timeouts, and what happens next on success or failure.

Lives in: GitHub YAML + Markdown templates.

Workflow Run State

The current step

One specific website moving through the recipe. Source URL, current stage, status, Paperclip issue ID, attempts, last error, timestamps.

Lives in: Postgres tables.

# ─── recipe excerpt ───
id: niche_site_replication_v1
version: 1

stages:
  - id: research
    agent: Researcher
    issue_template: templates/research.md
    required_outputs:
      - market_synthesis.md
      - competitors.json
      - keyword_map.md
    validation:
      - has_competitor_list
      - has_keyword_map
      - has_market_synthesis
    on_success: strategy      # ▶ next stage
    on_failure: retry
    max_attempts: 3
    timeout_hours: 24

-- one row per website / job
create table niche_jobs (
  id                    uuid primary key default gen_random_uuid(),
  source_url            text not null,
  target_domain         text,
  workflow_id           text not null default 'niche_site_replication_v1',
  status                text not null default 'queued',
  current_stage         text,
  priority              integer not null default 100,
  paperclip_company_id  uuid,
  paperclip_project_id  uuid,
  parent_issue_id       uuid,
  current_issue_id      uuid,
  attempts              integer not null default 0,
  last_error            text,
  created_at            timestamptz not null default now(),
  started_at            timestamptz,
  completed_at          timestamptz
);

§ 05

Five small workflows, not one giant one

A monolithic graph collapses under its own weight. Split the engine into specialised, restartable jobs.

Intake: Reads candidate websites from your DB. Filters out duplicates. Creates niche_jobs rows in queued.
Bootstrap: Analyses source URL, picks a target domain, creates Paperclip company / project / agents, opens the parent issue and the first research issue.
Pipeline Driver: Runs every five minutes. For each active job: load the recipe, check the current Paperclip issue, validate outputs, advance the stage, schedule retries, or escalate.
Human Escalation: Spawns a Paperclip "human review" issue with reason, logs, and the exact decision required. Optionally pings Slack / Discord.
Completion: When final-audit passes: mark parent issue done, set status=completed, archive, ignore forever.

# every 5 minutes
for job in active_jobs():
    workflow = load_yaml(job.workflow_id)
    stage    = workflow.stages[job.current_stage]
    issue    = paperclip.get(job.current_issue_id)

    if issue.open:
        continue                          # still working

    if issue.blocked:
        mark_human_review(job)
        continue

    if not validate(stage, issue):
        if job.attempts < stage.max_attempts:
            schedule_retry(job, backoff=exponential)
        else:
            mark_human_review(job)
        continue

    if stage.on_success == "complete":
        finalise(job)                       # ✓ stop forever
    else:
        create_next_issue(job, stage.on_success)

§ 06

Gates the agent cannot bluff past

A stage doesn't pass because an agent ticked the box. It passes because the artifacts exist and the checks succeed.

G-01Research

competitor list
market synthesis
keyword map
recommended pages
top opportunities

G-02Strategy

sitemap
page list
content briefs
build spec
acceptance criteria

G-03Design

header specified
footer specified
mobile nav
CTA sections
legal links
responsive layout

G-04Build

build passes
required pages exist
header & footer
no placeholder text
preview URL works
screenshots

G-05Legal

privacy page
imprint / contact
cookie logic
footer legal links

G-06Analytics & SEO

title & meta
canonical tags
sitemap.xml
robots.txt
internal links
GSC + GA4

G-07QA

mobile passes
desktop passes
forms work
links work
Lighthouse
no critical visual issues

G-08Deploy

production URL works
smoke test passes
DNS checked
rollback notes
final approval

§ 07

Retry, then escalate

Most failures are transient. Some need a human. The system must know which is which.

↻ retry automatically

agent hit a rate limit
temporary API error
command crashed
Paperclip heartbeat failed
Cloudflare / Spaceship transient error
generic timeout

⚠ escalate to human

account credit exhausted
API key / secret missing
billing required
legal approval needed
DNS / domain decision is risky
same stage failed too many times

retry:
  max_attempts: 3
  backoff:
    type: exponential
    initial_minutes: 10
    max_minutes: 120
  retry_on:
    - rate_limit
    - transient_api_error
    - agent_crash
    - timeout
  do_not_retry_on:
    - missing_secret
    - billing_required
    - legal_approval_required
    - unclear_requirements

§ 08

Throttle, or burn

Without limits, the factory eats its own credits, agents collide, and domains get bought in panic.

Active Jobs (max)

5parallel

Research (max)

10jobs

Build (max)

2jobs

Deploy (max)

1job

Per-Agent (max)

1–2tasks

§ 09

The most important change

From semi-automation to a reliable factory is the difference between asking the CEO and reading the recipe.

Before / fragile

"@CEO, next phase is ready. Please create issues."

After / deterministic

"Research passed validation. Workflow says next stage is Strategy. Creating Strategy issue assigned to Strategist."

§ 10

Implementation in six phases

Build the floor before the walls. Each phase produces something useful even if the next phase never happens.

P-01

Make state explicit

Create niche_jobs and niche_job_stages. Stop relying on Paperclip issue hierarchy as your only memory.

P-02

Build the playbook repo

Workflow YAML, stage templates, checklists, output schemas. Versioned, reviewable, diff-able.

P-03

Replace CEO guessing

n8n reads the workflow definition and creates the exact next issue. The CEO is no longer a router.

P-04

Add validation

Start with required-artifact checks. Grow into builds, browser tests, Lighthouse, HTML inspections, analytics tag detection.

P-05

Retry & escalation

Exponential backoff for transient failures. Human-review issues for everything that needs judgement, money, or signatures.

P-06

Completion stop

Final audit passes → status completed, parent issue done, archive forever. The site is no longer the factory's problem.

§ 11

One website, start to finish

A real walk through the line — no manual prompts, no CEO guessing, no forgetting.

# INTAKE
n8n.intake → niche_jobs ← { source_url: "https://klassenplan.de", status: "queued" }

# BOOTSTRAP
analyse_site() → choose_target_domain() → paperclip.create_company()
                → create_agents() → create_parent_issue()
                → create_research_issue()
niche_jobs.current_stage = "research"
niche_jobs.status        = "waiting_agent"

# RESEARCH DONE
researcher.complete(issue) → monitor.detect()
                           → validate(research_outputs)
                              ├─ pass → create_strategy_issue()
                              └─ fail → retry  /  human_review

# PIPELINE ADVANCES
strategy → content → design → build → legal → analytics → qa → deploy → final_audit

# COMPLETE
final_audit.pass → niche_jobs.status      = "completed"
                 → niche_jobs.completed_at = now()
                 → parent_issue           = done
                 → ▶ stop forever