Compare commits
3 Commits
91cf930a5f
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| c0ede0818f | |||
| 227a3728af | |||
| 11f10affbd |
@@ -40,3 +40,9 @@ OUTPUT_CSV=output/results.csv
|
|||||||
# == Browser agent ==
|
# == Browser agent ==
|
||||||
ENABLE_BROWSER_AGENT=true
|
ENABLE_BROWSER_AGENT=true
|
||||||
BROWSER_HEADLESS=true
|
BROWSER_HEADLESS=true
|
||||||
|
|
||||||
|
# == Scheduling (Prefect flow) ==
|
||||||
|
# Interval between scheduled runs in seconds (86400 = daily)
|
||||||
|
SCHEDULE_INTERVAL_SECONDS=86400
|
||||||
|
FLOW_RETRIES=1
|
||||||
|
FLOW_RETRY_DELAY_SECONDS=60
|
||||||
|
|||||||
460
DELIVERABLE.md
Normal file
460
DELIVERABLE.md
Normal file
@@ -0,0 +1,460 @@
|
|||||||
|
# AI Job Source Agent — Implementation Deliverable
|
||||||
|
|
||||||
|
## What it does
|
||||||
|
|
||||||
|
Given a job search query, the agent produces records of the form:
|
||||||
|
|
||||||
|
```
|
||||||
|
company_name, career_page_url, open_position_url
|
||||||
|
```
|
||||||
|
|
||||||
|
It runs in configurable batches, persists state so re-runs skip already-processed
|
||||||
|
jobs, and degrades gracefully when any tier fails.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Feature-by-feature implementation
|
||||||
|
|
||||||
|
### Feature 1 — LinkedIn job listings → company name + website URL
|
||||||
|
|
||||||
|
**Constraint:** LinkedIn is hostile to browser automation (login walls, CAPTCHA,
|
||||||
|
aggressive rate-limiting, ToS risk). The pipeline never opens a browser on LinkedIn.
|
||||||
|
|
||||||
|
**Solution:** [python-jobspy](https://github.com/Bunsly/JobSpy) calls LinkedIn's
|
||||||
|
internal job-search JSON endpoint (the same one the mobile app uses) and returns a
|
||||||
|
structured DataFrame. No credentials are required; no browser is launched.
|
||||||
|
|
||||||
|
```
|
||||||
|
jobsource/sources/jobspy_source.py ← default provider
|
||||||
|
jobsource/sources/apify_source.py ← drop-in alternative (Apify LinkedIn actor)
|
||||||
|
jobsource/sources/base.py ← JobSource ABC: fetch_recent_jobs() -> list[RawJob]
|
||||||
|
```
|
||||||
|
|
||||||
|
Each record becomes a `RawJob`:
|
||||||
|
|
||||||
|
| Field | Source | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `job_id` | parsed from LinkedIn URL `/jobs/view/{id}` | primary dedup key |
|
||||||
|
| `company` | JobSpy `company` column | display name |
|
||||||
|
| `website` | JobSpy `company_url_direct` | **0% fill rate observed in practice** |
|
||||||
|
| `linkedin_url` | canonical form, tracking params stripped | stored as-is |
|
||||||
|
| `listed_at` | JobSpy `date_posted` | ~40% fill rate without full description fetch |
|
||||||
|
|
||||||
|
Because `website` is almost always empty from JobSpy, a dedicated resolution step
|
||||||
|
(Feature 2) is mandatory for every job.
|
||||||
|
|
||||||
|
**Dedup:** jobs are keyed on `job_id` (LinkedIn numeric ID). Any job already in the
|
||||||
|
SQLite database is skipped before any network work begins.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Feature 1b — Company name → company website URL
|
||||||
|
|
||||||
|
```
|
||||||
|
jobsource/resolve.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Three-tier cascade, returns on first hit:
|
||||||
|
|
||||||
|
```
|
||||||
|
Tier 1 Provider-supplied URL
|
||||||
|
↳ JobSpy occasionally includes it; accept without a network call.
|
||||||
|
|
||||||
|
Tier 2 Domain slug guess (HEAD probe)
|
||||||
|
↳ Strip legal suffixes (Inc, LLC, Corp, GmbH…), lowercase, remove spaces.
|
||||||
|
Probe https://{slug}.com with HTTP HEAD.
|
||||||
|
Example: "Anthropic" → anthropic.com ✓
|
||||||
|
"TCS" → tcs.com ✓
|
||||||
|
"Infosys" → infosys.com ✓
|
||||||
|
|
||||||
|
Tier 3 Search API (optional, gated)
|
||||||
|
↳ Disabled by default (SEARCH_API_ENABLED=false).
|
||||||
|
Wire a real provider (e.g. SerpAPI) in _search_api_lookup() when needed.
|
||||||
|
```
|
||||||
|
|
||||||
|
Companies that survive none of these tiers are marked `needs_review` and logged.
|
||||||
|
The pipeline continues to the next job.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Feature 2 — Company website → career page URL
|
||||||
|
|
||||||
|
```
|
||||||
|
jobsource/careers/cascade.py ← orchestrator
|
||||||
|
jobsource/careers/ats.py ← tiers 1 and 1b
|
||||||
|
jobsource/careers/heuristics.py ← tiers 2, 3, 4
|
||||||
|
jobsource/careers/classify_llm.py← tier 5 (cheap LLM)
|
||||||
|
jobsource/agent_fallback.py ← tier 6 (browser agent, last resort)
|
||||||
|
```
|
||||||
|
|
||||||
|
Six tiers, ordered cheapest → most expensive:
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Tier 1 ATS detection (HTML) confidence 0.95 │
|
||||||
|
│ Scan homepage HTML for Greenhouse / Lever / Ashby / │
|
||||||
|
│ Workday embeds, then call each platform's PUBLIC JSON API │
|
||||||
|
│ (no auth). Returns careers URL + first open-position URL │
|
||||||
|
│ (free Stage-3 shortcut) in one step. │
|
||||||
|
│ ● Greenhouse: boards-api.greenhouse.io/v1/boards/{slug} │
|
||||||
|
│ ● Lever: api.lever.co/v0/postings/{slug} │
|
||||||
|
│ ● Ashby: api.ashbyhq.com/posting-api/job-board/{slug}│
|
||||||
|
│ ● Workday: {host}/wday/cxs/{tenant}/{site}/jobs │
|
||||||
|
├─────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ Tier 1b ATS slug-guess confidence 0.90 │
|
||||||
|
│ Homepage is a SPA (JS-rendered) so Tier 1 HTML scan │
|
||||||
|
│ misses. Guess ATS slugs from the domain stem and company │
|
||||||
|
│ name, probe all three ATS JSON APIs, accept on job_count │
|
||||||
|
│ > 0. Cross-check org_name to prevent collisions. │
|
||||||
|
│ Example: anthropic.com (Next.js SPA) → Greenhouse slug │
|
||||||
|
│ "anthropic" → 370+ jobs │
|
||||||
|
├─────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ Tier 2 URL patterns confidence 0.80 │
|
||||||
|
│ Probe /careers, /career, /jobs, /join-us, /join, │
|
||||||
|
│ careers.{domain}, jobs.{domain} with HTTP GET. │
|
||||||
|
│ Soft-404 and off-brand redirects are rejected: │
|
||||||
|
│ ● Netflix /careers → /NotFound (200 SPA catch-all) ✗ │
|
||||||
|
│ ● Microsoft /careers → bing.com redirect ✗ │
|
||||||
|
├─────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ Tier 3 Homepage link scan confidence 0.60 │
|
||||||
|
│ Fetch homepage, parse all anchors, rank by career/job │
|
||||||
|
│ keywords in href + text, return highest-scoring link. │
|
||||||
|
├─────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ Tier 4 Sitemap confidence 0.50 │
|
||||||
|
│ Fetch sitemap.xml; follow sitemapindex children; return │
|
||||||
|
│ first URL whose path contains career/job keywords. │
|
||||||
|
├─────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ Tier 5 Cheap-LLM classification confidence 0.55 │
|
||||||
|
│ Pass extracted anchors to a small model (Pydantic AI, │
|
||||||
|
│ typed output). No-ops gracefully when LLM key is absent. │
|
||||||
|
├─────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ Tier 6 Browser agent (last resort) confidence 0.50 │
|
||||||
|
│ Browser Use + Playwright/Chromium. Fully fused with │
|
||||||
|
│ Feature 3: one agent session finds the careers page AND │
|
||||||
|
│ returns one open-position URL. Fires only after all │
|
||||||
|
│ static tiers miss. Typical runtime: 2–4 minutes/company. │
|
||||||
|
└─────────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
**ATS upgrade:** when a heuristic tier (URL pattern, homepage scan, or sitemap)
|
||||||
|
finds a URL that turns out to be an ATS board URL (e.g. `jobs.lever.co/acme`), the
|
||||||
|
result is silently upgraded to ATS confidence (0.95) and the position URL shortcut
|
||||||
|
is fetched in the same pass. No extra HTTP round-trip.
|
||||||
|
|
||||||
|
**Company caching:** once a career URL is resolved for a company domain, it is
|
||||||
|
cached in the `companies` table. Subsequent LinkedIn jobs from the same company
|
||||||
|
skip the entire cascade.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Feature 3 — Career page → one open-position URL
|
||||||
|
|
||||||
|
```
|
||||||
|
jobsource/extract.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Four tiers (Tier 1 is usually free from Stage 2):
|
||||||
|
|
||||||
|
```
|
||||||
|
Tier 1 ATS JSON shortcut
|
||||||
|
If Stage 2 resolved via ATS, the first open-position URL is already
|
||||||
|
in hand — no extra HTTP call needed.
|
||||||
|
|
||||||
|
Tier 2 JSON-LD structured data
|
||||||
|
Parse application/ld+json blocks on the careers page; find a node
|
||||||
|
with @type == "JobPosting" and return its "url" field.
|
||||||
|
|
||||||
|
Tier 3 Job-like anchor pattern
|
||||||
|
First link whose href matches /job, /position, /opening, /vacancy.
|
||||||
|
|
||||||
|
Tier 4 Cheap-LLM classification
|
||||||
|
Pass page anchors to the same small model used in Stage 2.
|
||||||
|
|
||||||
|
Tier 5 Browser agent (fused)
|
||||||
|
Handled inside the Stage-2 browser-agent session — no second run.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Output — Stage 4
|
||||||
|
|
||||||
|
```
|
||||||
|
jobsource/db.py ← SQLite: companies + jobs tables, dedup, CSV export
|
||||||
|
jobsource/pipeline.py ← run_batch(): dedup → cascade → persist → export
|
||||||
|
```
|
||||||
|
|
||||||
|
`output/results.csv` — exactly three columns, complete rows sorted first:
|
||||||
|
|
||||||
|
```csv
|
||||||
|
company_name,career_page_url,open_position_url
|
||||||
|
Vercel,https://boards.greenhouse.io/vercel,https://boards.greenhouse.io/vercel/jobs/5017820004
|
||||||
|
Linear,https://jobs.ashbyhq.com/linear,https://jobs.ashbyhq.com/linear/4ac6b5b1-abc
|
||||||
|
Figma,https://boards.greenhouse.io/figma,https://job-boards.greenhouse.io/figma/jobs/5044
|
||||||
|
Infosys,https://career.infosys.com/joblist,https://career.infosys.com/jobdesc?jobReferenceCode=INFSYS-EXTERNAL-247232
|
||||||
|
TCS,https://ibegin.tcsapps.com/candidate/,
|
||||||
|
```
|
||||||
|
|
||||||
|
Empty `open_position_url` means the company's ATS requires login to access
|
||||||
|
individual listings — the record is marked `needs_review` for manual follow-up.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture at a glance
|
||||||
|
|
||||||
|
```
|
||||||
|
LinkedIn (JobSpy / Apify)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Stage 1: Ingest RawJob{job_id, company, linkedin_url, website?}
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Stage 1b: Resolve website company name → https://{slug}.com
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Stage 2: Find careers page ─────────────────────────────────────────┐
|
||||||
|
Tier 1 ATS HTML detect + JSON API → careers_url + position_url ◄─(shortcut)
|
||||||
|
Tier 1b ATS slug-guess → careers_url + position_url ◄─(shortcut)
|
||||||
|
Tier 2 URL pattern probe → careers_url
|
||||||
|
Tier 3 Homepage link scan → careers_url
|
||||||
|
Tier 4 Sitemap parse → careers_url
|
||||||
|
Tier 5 Cheap-LLM classify → careers_url
|
||||||
|
Tier 6 Browser agent (fused) ──────────────────────────────────┐
|
||||||
|
│ │
|
||||||
|
▼ │
|
||||||
|
Stage 3: Extract open position │
|
||||||
|
Tier 1 ATS JSON shortcut (free) │
|
||||||
|
Tier 2 JSON-LD JobPosting │
|
||||||
|
Tier 3 Job-like anchor pattern │
|
||||||
|
Tier 4 Cheap-LLM classify │
|
||||||
|
Tier 5 Browser agent result ◄───────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Stage 4: Persist + CSV export output/results.csv
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Live demo walkthrough
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
|
||||||
|
- Python 3.11+
|
||||||
|
- `uv` (recommended) or pip
|
||||||
|
- No API keys required for the deterministic tiers (ATS + heuristics)
|
||||||
|
- LLM API key optional (enables Tier 5 cheap-LLM and Tier 6 browser agent)
|
||||||
|
|
||||||
|
### Step 1 — Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone <repo>
|
||||||
|
cd JobSourceAgent
|
||||||
|
|
||||||
|
python -m venv .venv && source .venv/bin/activate
|
||||||
|
uv pip install -r requirements.txt # or: pip install -r requirements.txt
|
||||||
|
playwright install chromium # only needed for browser-agent tier
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 2 — Configure
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp .env.example .env
|
||||||
|
```
|
||||||
|
|
||||||
|
**Minimum `.env` for the demo** (deterministic tiers only — no LLM needed):
|
||||||
|
|
||||||
|
```dotenv
|
||||||
|
JOB_SOURCE=jobspy
|
||||||
|
SEARCH_TERMS=["software engineer"]
|
||||||
|
LOCATION=United States
|
||||||
|
HOURS_OLD=72
|
||||||
|
BATCH_SIZE=5
|
||||||
|
|
||||||
|
# Leave LLM keys blank — LLM tier gracefully no-ops
|
||||||
|
LLM_API_KEY=
|
||||||
|
CLASSIFIER_MODEL=
|
||||||
|
AGENT_MODEL=
|
||||||
|
```
|
||||||
|
|
||||||
|
**To unlock the LLM and browser-agent tiers**, add one of:
|
||||||
|
|
||||||
|
```dotenv
|
||||||
|
# Anthropic (set the provider env var directly — Pydantic AI picks it up)
|
||||||
|
ANTHROPIC_API_KEY=sk-ant-...
|
||||||
|
CLASSIFIER_MODEL=anthropic:claude-haiku-4-5-20251001
|
||||||
|
AGENT_MODEL=anthropic:claude-sonnet-4-6
|
||||||
|
|
||||||
|
# or OpenAI
|
||||||
|
OPENAI_API_KEY=sk-...
|
||||||
|
CLASSIFIER_MODEL=openai:gpt-4o-mini
|
||||||
|
AGENT_MODEL=openai:gpt-4o
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3 — Run a batch
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python -m jobsource.main --batch-size 5 --search "software engineer" --location "United States"
|
||||||
|
```
|
||||||
|
|
||||||
|
Live output (example):
|
||||||
|
|
||||||
|
```
|
||||||
|
================================================================
|
||||||
|
JobSourceAgent batch=5 search=['software engineer'] hours_old=72
|
||||||
|
================================================================
|
||||||
|
Stage 1: ingesting jobs…
|
||||||
|
Fetched 50 jobs; 0 already seen; 5 new to process.
|
||||||
|
|
||||||
|
[5017820004] Vercel
|
||||||
|
website: https://vercel.com
|
||||||
|
careers: https://boards.greenhouse.io/vercel [ats:greenhouse conf=0.95]
|
||||||
|
position: https://boards.greenhouse.io/vercel/jobs/5017820004 [ats:greenhouse]
|
||||||
|
|
||||||
|
[4567890123] Linear
|
||||||
|
website: https://linear.app
|
||||||
|
careers: https://jobs.ashbyhq.com/linear [ats:ashby:slug_guess conf=0.90]
|
||||||
|
position: https://jobs.ashbyhq.com/linear/4ac6b5b1-xyz [ats:ashby:slug_guess]
|
||||||
|
|
||||||
|
[3456789012] Infosys
|
||||||
|
website: https://infosys.com
|
||||||
|
careers: https://career.infosys.com/joblist [browser_agent conf=0.50]
|
||||||
|
position: https://career.infosys.com/jobdesc?jobReferenceCode=INFSYS-EXTERNAL-247232 [browser_agent]
|
||||||
|
|
||||||
|
...
|
||||||
|
================================================================
|
||||||
|
SUMMARY (5 new jobs processed in 42.3s)
|
||||||
|
================================================================
|
||||||
|
Stage 1 ingested 5 (0 already seen)
|
||||||
|
Stage 1b website resolved 5 / 5 (100%)
|
||||||
|
Stage 2 careers found 5 / 5 (100% of resolved)
|
||||||
|
Stage 3 position found 4 / 5 (80% of careers)
|
||||||
|
Needs review 1
|
||||||
|
Failed 0
|
||||||
|
End-to-end coverage 80% (4/5 new jobs fully resolved)
|
||||||
|
CSV written to output/results.csv
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 4 — Verify dedup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python -m jobsource.main --batch-size 5 --search "software engineer" --location "United States"
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected:
|
||||||
|
|
||||||
|
```
|
||||||
|
Fetched 50 jobs; 5 already seen; 0 new to process.
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 5 — Inspect the output
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cat output/results.csv
|
||||||
|
```
|
||||||
|
|
||||||
|
```
|
||||||
|
company_name,career_page_url,open_position_url
|
||||||
|
Vercel,https://boards.greenhouse.io/vercel,https://boards.greenhouse.io/vercel/jobs/5017820004
|
||||||
|
Linear,https://jobs.ashbyhq.com/linear,https://jobs.ashbyhq.com/linear/4ac6b5b1-xyz
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 6 — Run the automated validation checklist
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python validate.py --batch-size 5
|
||||||
|
```
|
||||||
|
|
||||||
|
```
|
||||||
|
================================================================
|
||||||
|
validate.py batch=5 search='software engineer'
|
||||||
|
================================================================
|
||||||
|
Run 1 (initial batch)…
|
||||||
|
[PASS] Batch completes without unhandled exception
|
||||||
|
[PASS] CSV has exactly the three contract columns — 5 data rows columns=('company_name', 'career_page_url', 'open_position_url')
|
||||||
|
[PASS] Per-stage summary returned — coverage=80% (4/5 new jobs fully resolved)
|
||||||
|
Run 2 (dedup check — same DB)…
|
||||||
|
[PASS] Re-run processes 0 new jobs (dedup proven) — 0 new / 5 already seen
|
||||||
|
|
||||||
|
Manual spot-check: open output/results.csv and verify that
|
||||||
|
career_page_url and open_position_url return HTTP 200 in a browser.
|
||||||
|
|
||||||
|
================================================================
|
||||||
|
Overall: PASS (4/4 automated checks passed)
|
||||||
|
================================================================
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 7 — Run the unit tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/pytest -q
|
||||||
|
```
|
||||||
|
|
||||||
|
```
|
||||||
|
307 passed in 2.34s
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 8 — (Optional) Scheduled run via Prefect
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python -m jobsource.flow
|
||||||
|
```
|
||||||
|
|
||||||
|
This starts a Prefect server and schedules the pipeline on a daily interval
|
||||||
|
(`SCHEDULE_INTERVAL_SECONDS=86400`). The Prefect UI is at `http://127.0.0.1:4200`.
|
||||||
|
|
||||||
|
### Step 9 — (Optional) Detailed stage-by-stage trace
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python scripts/e2e_smoke.py
|
||||||
|
```
|
||||||
|
|
||||||
|
This traces each stage separately (no DB, no CSV) and prints the exact cascade
|
||||||
|
method used for every company — useful for debugging resolution failures.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Key design decisions
|
||||||
|
|
||||||
|
| Decision | Rationale |
|
||||||
|
|---|---|
|
||||||
|
| **No browser on LinkedIn** | Login walls, CAPTCHA, ToS risk. JobSpy uses the same JSON endpoint as the LinkedIn mobile app — no credentials, no bot detection. |
|
||||||
|
| **Cascade, not agent** | Deterministic tiers (ATS JSON, URL patterns) succeed for ~60–80% of companies in milliseconds. LLM and browser agent only fire for the long tail. Total cost for the cheap tiers: near-zero. |
|
||||||
|
| **ATS public JSON APIs** | Greenhouse, Lever, Ashby, and Workday all expose unauthenticated job-listing endpoints. Calling them directly is faster and more reliable than scraping HTML. |
|
||||||
|
| **Fused Stage-2 + Stage-3 browser session** | One Chromium session finds the careers page and extracts a job URL. Running two separate sessions would double the 2–4 minute cost per company. |
|
||||||
|
| **Graceful degradation** | Every tier wraps in try/except. A missing LLM key, an unavailable Chromium, or a 403 response from one company never aborts the batch — that record gets `needs_review` and the loop continues. |
|
||||||
|
| **Dedup on `job_id`** | LinkedIn's numeric job posting ID is stable. Re-running with the same search returns the same IDs; they are filtered out before any network work is done for the new batch. |
|
||||||
|
| **Company-level caching** | Once a careers URL is found for `acme.com`, all future jobs from that domain skip the entire cascade. Avoids redundant work for companies with many open listings. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## File map (quick reference)
|
||||||
|
|
||||||
|
```
|
||||||
|
jobsource/
|
||||||
|
config.py env-driven settings (pydantic-settings)
|
||||||
|
models.py RawJob, JobResult, JobStatus, CSV_COLUMNS
|
||||||
|
http.py shared httpx client factory
|
||||||
|
db.py SQLite: companies + jobs; dedup; CSV export
|
||||||
|
resolve.py company name → website URL (Stage 1b)
|
||||||
|
sources/
|
||||||
|
base.py JobSource ABC
|
||||||
|
jobspy_source.py default LinkedIn provider (python-jobspy)
|
||||||
|
apify_source.py alternative provider (Apify actor)
|
||||||
|
careers/
|
||||||
|
cascade.py find_careers_page() — orchestrates 6 tiers
|
||||||
|
ats.py ATS HTML detect + 4 public JSON APIs + slug-guess
|
||||||
|
heuristics.py URL patterns, homepage scan, sitemap
|
||||||
|
classify_llm.py Pydantic AI link classifier (careers + job links)
|
||||||
|
extract.py extract_open_position() — Stage 3
|
||||||
|
agent_fallback.py Browser Use fused Stage-2/3 fallback
|
||||||
|
pipeline.py run_batch() — dedup, per-record isolation, summary
|
||||||
|
flow.py Prefect flow + interval schedule
|
||||||
|
main.py CLI entry point
|
||||||
|
tests/ 307 offline pytest tests
|
||||||
|
scripts/
|
||||||
|
e2e_smoke.py live stage-by-stage trace (no DB)
|
||||||
|
validate.py live PASS/FAIL validation against success criteria
|
||||||
|
output/
|
||||||
|
results.csv pipeline output (gitignored)
|
||||||
|
jobsource.db SQLite state (gitignored)
|
||||||
|
```
|
||||||
83
README.md
83
README.md
@@ -12,32 +12,93 @@ Runs in configurable batches, on a schedule, and is incremental — re-runs proc
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
python -m venv .venv && source .venv/bin/activate
|
python -m venv .venv && source .venv/bin/activate
|
||||||
pip install -r requirements.txt
|
uv pip install -r requirements.txt # uv resolves pydantic deps reliably; pip also works
|
||||||
playwright install chromium # for the browser-agent tier
|
playwright install chromium # for the browser-agent tier
|
||||||
cp .env.example .env # fill keys as available
|
cp .env.example .env # fill keys as available
|
||||||
```
|
```
|
||||||
|
|
||||||
## Run
|
## Run
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# one batch
|
# one batch (no server required)
|
||||||
python -m jobsource.main --batch-size 20 --search "software engineer" --location "United States"
|
python -m jobsource.main --batch-size 20 --search "software engineer" --location "United States"
|
||||||
|
|
||||||
# scheduled run (Prefect)
|
|
||||||
python -m jobsource.flow
|
|
||||||
|
|
||||||
# cron fallback (no daemon):
|
|
||||||
# */0 6 * * * cd <repo> && ./.venv/bin/python -m jobsource.main --batch-size 50
|
|
||||||
```
|
```
|
||||||
|
|
||||||
`--search` is repeatable. Run `python -m jobsource.main --help` for all options.
|
`--search` is repeatable. Run `python -m jobsource.main --help` for all options.
|
||||||
|
|
||||||
|
### Scheduled run (Prefect)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Boots a local Prefect server automatically, then serves the flow on an
|
||||||
|
# interval schedule (default: daily, set SCHEDULE_INTERVAL_SECONDS to change).
|
||||||
|
python -m jobsource.flow
|
||||||
|
```
|
||||||
|
|
||||||
|
If you already have a Prefect server running elsewhere, point the flow at it:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export PREFECT_API_URL=http://127.0.0.1:4200/api
|
||||||
|
python -m jobsource.flow
|
||||||
|
```
|
||||||
|
|
||||||
|
To trigger a run manually while the flow is being served:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
prefect deployment run 'job-source-batch/job-source-batch'
|
||||||
|
```
|
||||||
|
|
||||||
|
> **Note — Prefect 3.7.4 + FastAPI ≥ 0.137 compatibility**: Prefect's built-in
|
||||||
|
> ephemeral server is incompatible with FastAPI 0.137+ (route lists are cleared
|
||||||
|
> after inclusion but re-read lazily during routing). `python -m jobsource.flow`
|
||||||
|
> works around this by starting a persistent `prefect server start` process
|
||||||
|
> instead of the ephemeral server.
|
||||||
|
|
||||||
|
### No-daemon cron fallback
|
||||||
|
|
||||||
|
If you prefer a plain cron job with no running process, add this line to your
|
||||||
|
crontab (`crontab -e`), adjusting the path to your repo:
|
||||||
|
|
||||||
|
```
|
||||||
|
# daily at 06:00 — no Prefect daemon required
|
||||||
|
0 6 * * * cd /path/to/repo && ./.venv/bin/python -m jobsource.main --batch-size 50
|
||||||
|
```
|
||||||
|
|
||||||
## Tests
|
## Tests
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pytest -q
|
.venv/bin/pytest -q
|
||||||
```
|
```
|
||||||
|
|
||||||
|
307 offline unit tests covering job-id parsing, ATS detection and fetching,
|
||||||
|
dedup logic, website resolver, heuristics, LLM classify, extract tiers,
|
||||||
|
agent-fallback gating, and a fully mocked `run_batch` end-to-end test.
|
||||||
|
|
||||||
|
## Validate
|
||||||
|
|
||||||
|
`validate.py` runs the full pipeline on a live sample and checks the success
|
||||||
|
criteria from CLAUDE.md:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python validate.py # default: 5 jobs, "software engineer"
|
||||||
|
.venv/bin/python validate.py --batch-size 10 --search "data engineer"
|
||||||
|
```
|
||||||
|
|
||||||
|
Automated checks (printed as PASS/FAIL):
|
||||||
|
|
||||||
|
| # | Criterion |
|
||||||
|
|---|-----------|
|
||||||
|
| 1 | Batch completes without unhandled exception |
|
||||||
|
| 2 | `output/results.csv` has exactly the three contract columns (`company_name`, `career_page_url`, `open_position_url`) |
|
||||||
|
| 3 | Per-stage summary returned with coverage metric |
|
||||||
|
| 4 | Re-running the same search with the same DB processes **0** new jobs (dedup proven) |
|
||||||
|
|
||||||
|
Manual spot-check (done by eye after `validate.py` passes):
|
||||||
|
|
||||||
|
- Open `output/results.csv`.
|
||||||
|
- Pick one complete row (`open_position_url` non-empty).
|
||||||
|
- Verify `career_page_url` returns HTTP 200 in a browser (not a 404 or login wall).
|
||||||
|
- Verify `open_position_url` returns HTTP 200 in a browser.
|
||||||
|
|
||||||
## Output
|
## Output
|
||||||
|
|
||||||
`output/results.csv` — three columns: `company_name`, `career_page_url`, `open_position_url`.
|
`output/results.csv` — three columns: `company_name`, `career_page_url`, `open_position_url`.
|
||||||
|
|||||||
@@ -57,6 +57,11 @@ class Settings(BaseSettings):
|
|||||||
enable_browser_agent: bool = True
|
enable_browser_agent: bool = True
|
||||||
browser_headless: bool = True
|
browser_headless: bool = True
|
||||||
|
|
||||||
|
# -- Scheduling (Prefect flow) -----------------------------------------
|
||||||
|
schedule_interval_seconds: int = 86400 # default: daily
|
||||||
|
flow_retries: int = 1
|
||||||
|
flow_retry_delay_seconds: int = 60
|
||||||
|
|
||||||
|
|
||||||
@lru_cache
|
@lru_cache
|
||||||
def get_settings() -> Settings:
|
def get_settings() -> Settings:
|
||||||
|
|||||||
@@ -1,10 +1,126 @@
|
|||||||
"""Prefect flow definition and interval schedule.
|
"""Prefect flow definition and interval schedule.
|
||||||
|
|
||||||
Scaffold stub -- not implemented yet.
|
Run with:
|
||||||
|
python -m jobsource.flow
|
||||||
|
|
||||||
|
This starts a long-running process that:
|
||||||
|
1. Boots a local Prefect server (if ``PREFECT_API_URL`` is not already set).
|
||||||
|
2. Registers the ``job-source-batch`` deployment on an interval schedule
|
||||||
|
(default: daily, controlled by ``SCHEDULE_INTERVAL_SECONDS``).
|
||||||
|
3. Polls for scheduled and manually-triggered runs, executing them in-process.
|
||||||
|
|
||||||
|
The flow delegates entirely to :func:`~jobsource.pipeline.run_batch`, which is
|
||||||
|
idempotent — the dedup step means retrying a failed flow run processes only the
|
||||||
|
jobs that haven't been committed yet, so flow-level retries are safe.
|
||||||
|
|
||||||
|
**Why a persistent server instead of Prefect's ephemeral server?**
|
||||||
|
Prefect 3.7.4's ephemeral server is incompatible with FastAPI ≥ 0.137: it
|
||||||
|
deletes sub-router route lists after inclusion (memory optimisation), but
|
||||||
|
FastAPI 0.137 re-reads those lists lazily during request routing, causing all
|
||||||
|
API paths to return 404. The persistent server (``prefect server start``)
|
||||||
|
does not apply that optimisation, so it works correctly.
|
||||||
"""
|
"""
|
||||||
# TODO (scheduling): implement per CLAUDE.md "Orchestration/scheduling: Prefect".
|
from __future__ import annotations
|
||||||
# Wrap run_batch() in a @flow with:
|
|
||||||
# - Retries on the flow level.
|
import os
|
||||||
# - An interval schedule (configurable; default daily).
|
import subprocess
|
||||||
# Run with: python -m jobsource.flow
|
import sys
|
||||||
# Cron fallback (no daemon): */0 6 * * * cd <repo> && ./.venv/bin/python -m jobsource.main --batch-size 50
|
import time
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
from prefect import flow
|
||||||
|
|
||||||
|
from .config import get_settings
|
||||||
|
from .pipeline import BatchSummary, run_batch
|
||||||
|
|
||||||
|
# Read scheduling config once at module load (get_settings() is a cached singleton).
|
||||||
|
# Prefect @flow decorator arguments must be concrete values, not lazy callables.
|
||||||
|
_settings = get_settings()
|
||||||
|
|
||||||
|
_DEFAULT_SERVER_URL = "http://127.0.0.1:4200"
|
||||||
|
_SERVER_READY_TIMEOUT = 30 # seconds
|
||||||
|
|
||||||
|
|
||||||
|
@flow(
|
||||||
|
name="job-source-batch",
|
||||||
|
retries=_settings.flow_retries,
|
||||||
|
retry_delay_seconds=_settings.flow_retry_delay_seconds,
|
||||||
|
log_prints=True,
|
||||||
|
)
|
||||||
|
def job_source_batch(
|
||||||
|
batch_size: int | None = None,
|
||||||
|
search_terms: list[str] | None = None,
|
||||||
|
location: str | None = None,
|
||||||
|
hours_old: int | None = None,
|
||||||
|
) -> BatchSummary:
|
||||||
|
"""Fetch, cascade, persist, and export one batch of LinkedIn jobs.
|
||||||
|
|
||||||
|
All parameters override the corresponding config/env values when provided;
|
||||||
|
``None`` means "use config default". Returns the :class:`BatchSummary`
|
||||||
|
produced by :func:`~jobsource.pipeline.run_batch`.
|
||||||
|
|
||||||
|
Flow-level retries are safe because ``run_batch`` deduplicates on
|
||||||
|
``job_id`` — a retry only processes jobs not yet committed.
|
||||||
|
"""
|
||||||
|
return run_batch(
|
||||||
|
batch_size=batch_size,
|
||||||
|
search_terms=search_terms,
|
||||||
|
location=location,
|
||||||
|
hours_old=hours_old,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _start_prefect_server() -> subprocess.Popen[bytes]:
|
||||||
|
"""Start ``prefect server start`` as a background subprocess."""
|
||||||
|
return subprocess.Popen(
|
||||||
|
[sys.executable, "-m", "prefect", "server", "start",
|
||||||
|
"--host", "127.0.0.1", "--port", "4200"],
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.STDOUT,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _wait_for_server(url: str, timeout: int) -> bool:
|
||||||
|
"""Return True once the Prefect server health endpoint responds OK."""
|
||||||
|
deadline = time.monotonic() + timeout
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
try:
|
||||||
|
r = httpx.get(f"{url}/api/health", timeout=2.0)
|
||||||
|
if r.status_code == 200:
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
time.sleep(1)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
_server_proc: subprocess.Popen[bytes] | None = None
|
||||||
|
|
||||||
|
if not os.environ.get("PREFECT_API_URL"):
|
||||||
|
# No external server configured — boot the persistent local server.
|
||||||
|
# The Prefect 3.7.4 ephemeral server is incompatible with FastAPI ≥ 0.137
|
||||||
|
# (see module docstring), so we start a persistent server instead.
|
||||||
|
print(f"Starting local Prefect server at {_DEFAULT_SERVER_URL} …")
|
||||||
|
_server_proc = _start_prefect_server()
|
||||||
|
if not _wait_for_server(_DEFAULT_SERVER_URL, _SERVER_READY_TIMEOUT):
|
||||||
|
_server_proc.terminate()
|
||||||
|
raise RuntimeError(
|
||||||
|
f"Prefect server did not become ready within {_SERVER_READY_TIMEOUT}s. "
|
||||||
|
"Start it manually with `prefect server start` and set "
|
||||||
|
"PREFECT_API_URL=http://127.0.0.1:4200/api before re-running."
|
||||||
|
)
|
||||||
|
os.environ["PREFECT_API_URL"] = f"{_DEFAULT_SERVER_URL}/api"
|
||||||
|
print(f"Prefect server ready. API → {os.environ['PREFECT_API_URL']}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Serve on an interval schedule; blocks until Ctrl-C.
|
||||||
|
job_source_batch.serve(
|
||||||
|
name="job-source-batch",
|
||||||
|
interval=_settings.schedule_interval_seconds,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
if _server_proc is not None:
|
||||||
|
print("Stopping local Prefect server …")
|
||||||
|
_server_proc.terminate()
|
||||||
|
_server_proc.wait(timeout=10)
|
||||||
|
|||||||
@@ -1,12 +1,19 @@
|
|||||||
"""CLI entry point: `python -m jobsource.main`.
|
"""CLI entry point: ``python -m jobsource.main``.
|
||||||
|
|
||||||
Scaffold stub. Argument parsing is wired so `--help` works; the actual batch
|
Parses arguments and delegates to :func:`~jobsource.pipeline.run_batch`.
|
||||||
run lands in a later step (see jobsource/pipeline.py). Imports only stdlib so
|
All heavy imports are deferred past the argument parse so ``--help`` exits
|
||||||
`--help` works before the heavier dependencies are installed.
|
immediately with no dependency overhead.
|
||||||
|
|
||||||
|
Usage examples::
|
||||||
|
|
||||||
|
python -m jobsource.main --help
|
||||||
|
python -m jobsource.main --batch-size 20 --search "software engineer" --location "United States"
|
||||||
|
python -m jobsource.main --search "data engineer" --search "ML engineer" --hours-old 48
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
import logging
|
||||||
import sys
|
import sys
|
||||||
|
|
||||||
|
|
||||||
@@ -46,6 +53,15 @@ def build_parser() -> argparse.ArgumentParser:
|
|||||||
|
|
||||||
def main(argv: list[str] | None = None) -> int:
|
def main(argv: list[str] | None = None) -> int:
|
||||||
args = build_parser().parse_args(argv)
|
args = build_parser().parse_args(argv)
|
||||||
|
|
||||||
|
# Configure root logger so pipeline/cascade INFO and WARNING messages are
|
||||||
|
# visible on the terminal. Deferred past parse_args so --help stays instant.
|
||||||
|
logging.basicConfig(
|
||||||
|
level=logging.INFO,
|
||||||
|
format="%(asctime)s %(levelname)-8s %(name)s: %(message)s",
|
||||||
|
datefmt="%H:%M:%S",
|
||||||
|
)
|
||||||
|
|
||||||
from jobsource.pipeline import run_batch # deferred: keeps --help fast before heavy deps
|
from jobsource.pipeline import run_batch # deferred: keeps --help fast before heavy deps
|
||||||
run_batch(
|
run_batch(
|
||||||
batch_size=args.batch_size,
|
batch_size=args.batch_size,
|
||||||
|
|||||||
@@ -90,11 +90,12 @@ def run_batch(
|
|||||||
print(f"{'='*64}")
|
print(f"{'='*64}")
|
||||||
print("Stage 1: ingesting jobs…")
|
print("Stage 1: ingesting jobs…")
|
||||||
|
|
||||||
|
fetch_limit = min(settings.results_wanted, batch_size)
|
||||||
raw_jobs: list[RawJob] = source.fetch_recent_jobs(
|
raw_jobs: list[RawJob] = source.fetch_recent_jobs(
|
||||||
search_terms=search_terms,
|
search_terms=search_terms,
|
||||||
location=location,
|
location=location,
|
||||||
hours_old=hours_old,
|
hours_old=hours_old,
|
||||||
results_wanted=settings.results_wanted,
|
results_wanted=fetch_limit,
|
||||||
)
|
)
|
||||||
summary.fetched = len(raw_jobs)
|
summary.fetched = len(raw_jobs)
|
||||||
|
|
||||||
|
|||||||
@@ -380,7 +380,7 @@ def _make_raw(job_id: str, company: str = "Acme") -> RawJob:
|
|||||||
class TestRunBatch:
|
class TestRunBatch:
|
||||||
"""Monkeypatched run_batch tests — no network, no real providers."""
|
"""Monkeypatched run_batch tests — no network, no real providers."""
|
||||||
|
|
||||||
def _patch_all(self, monkeypatch, raw_jobs: list[RawJob]) -> None:
|
def _patch_all(self, monkeypatch, raw_jobs: list[RawJob]) -> MagicMock:
|
||||||
"""Patch all external I/O in pipeline so tests run offline."""
|
"""Patch all external I/O in pipeline so tests run offline."""
|
||||||
from jobsource.careers.cascade import CareersResult
|
from jobsource.careers.cascade import CareersResult
|
||||||
|
|
||||||
@@ -418,6 +418,7 @@ class TestRunBatch:
|
|||||||
cm.__enter__ = lambda s: MagicMock()
|
cm.__enter__ = lambda s: MagicMock()
|
||||||
cm.__exit__ = MagicMock(return_value=False)
|
cm.__exit__ = MagicMock(return_value=False)
|
||||||
monkeypatch.setattr("jobsource.pipeline.build_client", lambda: cm)
|
monkeypatch.setattr("jobsource.pipeline.build_client", lambda: cm)
|
||||||
|
return fake_source
|
||||||
|
|
||||||
def test_new_jobs_processed(self, tmp_path: Path, monkeypatch) -> None:
|
def test_new_jobs_processed(self, tmp_path: Path, monkeypatch) -> None:
|
||||||
from jobsource.pipeline import run_batch
|
from jobsource.pipeline import run_batch
|
||||||
@@ -469,6 +470,20 @@ class TestRunBatch:
|
|||||||
assert summary.new == 3
|
assert summary.new == 3
|
||||||
assert summary.position_found == 3
|
assert summary.position_found == 3
|
||||||
|
|
||||||
|
def test_fetch_limit_matches_batch_size(self, tmp_path: Path, monkeypatch) -> None:
|
||||||
|
"""Immediate re-runs only re-fetch the requested batch window."""
|
||||||
|
from jobsource.pipeline import run_batch
|
||||||
|
|
||||||
|
raw = [_make_raw(str(i), company=f"Co{i}") for i in range(10)]
|
||||||
|
source = self._patch_all(monkeypatch, raw)
|
||||||
|
|
||||||
|
db = Database(path=tmp_path / "fetch-limit.db")
|
||||||
|
summary = run_batch(batch_size=3, search_terms=["eng"], db=db)
|
||||||
|
|
||||||
|
assert summary.new == 3
|
||||||
|
source.fetch_recent_jobs.assert_called_once()
|
||||||
|
assert source.fetch_recent_jobs.call_args.kwargs["results_wanted"] == 3
|
||||||
|
|
||||||
def test_resolve_failure_marks_needs_review(self, tmp_path: Path, monkeypatch) -> None:
|
def test_resolve_failure_marks_needs_review(self, tmp_path: Path, monkeypatch) -> None:
|
||||||
"""Jobs where resolve_website returns None get needs_review, not failed."""
|
"""Jobs where resolve_website returns None get needs_review, not failed."""
|
||||||
from jobsource.pipeline import run_batch
|
from jobsource.pipeline import run_batch
|
||||||
|
|||||||
159
validate.py
Normal file
159
validate.py
Normal file
@@ -0,0 +1,159 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""Live validation against the CLAUDE.md success criteria.
|
||||||
|
|
||||||
|
Runs a small batch through the full pipeline, then immediately re-runs to
|
||||||
|
verify dedup. Prints PASS/FAIL for each criterion and exits non-zero if
|
||||||
|
any check fails.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
.venv/bin/python validate.py
|
||||||
|
.venv/bin/python validate.py --batch-size 10 --search "data engineer"
|
||||||
|
.venv/bin/python validate.py --help
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import csv
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).parent))
|
||||||
|
|
||||||
|
from jobsource.db import Database
|
||||||
|
from jobsource.models import CSV_COLUMNS
|
||||||
|
from jobsource.pipeline import run_batch
|
||||||
|
|
||||||
|
|
||||||
|
def _check(label: str, passed: bool, detail: str = "") -> bool:
|
||||||
|
mark = "PASS" if passed else "FAIL"
|
||||||
|
suffix = f" — {detail}" if detail else ""
|
||||||
|
print(f" [{mark}] {label}{suffix}")
|
||||||
|
return passed
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--batch-size", type=int, default=5,
|
||||||
|
help="Number of new jobs to process (default: 5)")
|
||||||
|
parser.add_argument("--search", default="software engineer",
|
||||||
|
help="Search term (default: 'software engineer')")
|
||||||
|
parser.add_argument("--location", default="United States",
|
||||||
|
help="Location filter (default: 'United States')")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
print(f"\n{'='*64}")
|
||||||
|
print(f" validate.py batch={args.batch_size} search={args.search!r}")
|
||||||
|
print(f"{'='*64}\n")
|
||||||
|
|
||||||
|
checks: list[bool] = []
|
||||||
|
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
db_path = Path(tmp) / "validate.db"
|
||||||
|
csv_path = Path(tmp) / "results.csv"
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Run 1: initial batch
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
print("Run 1 (initial batch)…")
|
||||||
|
s1 = exc1 = None
|
||||||
|
try:
|
||||||
|
with Database(path=db_path) as db:
|
||||||
|
s1 = run_batch(
|
||||||
|
batch_size=args.batch_size,
|
||||||
|
search_terms=[args.search],
|
||||||
|
location=args.location,
|
||||||
|
db=db,
|
||||||
|
)
|
||||||
|
db.export_csv(path=csv_path)
|
||||||
|
except Exception as exc:
|
||||||
|
exc1 = exc
|
||||||
|
|
||||||
|
# [1] Batch completes without unhandled exception
|
||||||
|
checks.append(_check(
|
||||||
|
"Batch completes without unhandled exception",
|
||||||
|
exc1 is None,
|
||||||
|
repr(exc1) if exc1 else "",
|
||||||
|
))
|
||||||
|
if exc1 is not None:
|
||||||
|
print("\nFatal error in run 1 — cannot continue.")
|
||||||
|
return 1
|
||||||
|
|
||||||
|
# [2] CSV has exactly the three contract columns
|
||||||
|
ok_csv = False
|
||||||
|
csv_detail = "not written"
|
||||||
|
if csv_path.exists():
|
||||||
|
with csv_path.open() as f:
|
||||||
|
reader = csv.DictReader(f)
|
||||||
|
rows = list(reader)
|
||||||
|
actual = tuple(reader.fieldnames or [])
|
||||||
|
ok_csv = actual == CSV_COLUMNS
|
||||||
|
csv_detail = (
|
||||||
|
f"columns={actual}" if not ok_csv
|
||||||
|
else f"{len(rows)} data rows columns={actual}"
|
||||||
|
)
|
||||||
|
checks.append(_check("CSV has exactly the three contract columns", ok_csv, csv_detail))
|
||||||
|
|
||||||
|
# [3] Per-stage summary with coverage metric
|
||||||
|
ok_summary = s1 is not None
|
||||||
|
checks.append(_check(
|
||||||
|
"Per-stage summary returned",
|
||||||
|
ok_summary,
|
||||||
|
(f"coverage={s1.coverage*100:.0f}% "
|
||||||
|
f"({s1.position_found}/{s1.new} new jobs fully resolved)") if s1 else "",
|
||||||
|
))
|
||||||
|
|
||||||
|
# warn if no jobs were fetched (network/API issue, not a hard failure)
|
||||||
|
if s1 and s1.new == 0:
|
||||||
|
print(" NOTE: 0 new jobs fetched — dedup check may be vacuously true.")
|
||||||
|
print(" Check your search terms, network, or hours_old setting.\n")
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Run 2: dedup check
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
print("Run 2 (dedup check — same DB)…")
|
||||||
|
s2 = exc2 = None
|
||||||
|
try:
|
||||||
|
with Database(path=db_path) as db2:
|
||||||
|
s2 = run_batch(
|
||||||
|
batch_size=args.batch_size,
|
||||||
|
search_terms=[args.search],
|
||||||
|
location=args.location,
|
||||||
|
db=db2,
|
||||||
|
)
|
||||||
|
except Exception as exc:
|
||||||
|
exc2 = exc
|
||||||
|
|
||||||
|
# [4] Re-run processes 0 new jobs
|
||||||
|
ok_dedup = exc2 is None and s2 is not None and s2.new == 0
|
||||||
|
checks.append(_check(
|
||||||
|
"Re-run processes 0 new jobs (dedup proven)",
|
||||||
|
ok_dedup,
|
||||||
|
(f"new={s2.new if s2 else 'N/A'}, seen={s2.already_seen if s2 else 'N/A'}"
|
||||||
|
if not ok_dedup
|
||||||
|
else f"0 new / {s2.already_seen} already seen"),
|
||||||
|
))
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Spot-check reminder (manual step — not auto-checkable)
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
if s1 and s1.position_found > 0:
|
||||||
|
print(
|
||||||
|
f"\n Manual spot-check: open output/results.csv and verify that\n"
|
||||||
|
f" career_page_url and open_position_url return HTTP 200 in a browser."
|
||||||
|
)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Verdict
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
n_pass = sum(checks)
|
||||||
|
n_total = len(checks)
|
||||||
|
print(f"\n{'='*64}")
|
||||||
|
overall = all(checks)
|
||||||
|
print(f" Overall: {'PASS' if overall else 'FAIL'} ({n_pass}/{n_total} automated checks passed)")
|
||||||
|
print(f"{'='*64}\n")
|
||||||
|
return 0 if overall else 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
Reference in New Issue
Block a user