Compare commits
3 Commits
91cf930a5f
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| c0ede0818f | |||
| 227a3728af | |||
| 11f10affbd |
@@ -40,3 +40,9 @@ OUTPUT_CSV=output/results.csv
|
||||
# == Browser agent ==
|
||||
ENABLE_BROWSER_AGENT=true
|
||||
BROWSER_HEADLESS=true
|
||||
|
||||
# == Scheduling (Prefect flow) ==
|
||||
# Interval between scheduled runs in seconds (86400 = daily)
|
||||
SCHEDULE_INTERVAL_SECONDS=86400
|
||||
FLOW_RETRIES=1
|
||||
FLOW_RETRY_DELAY_SECONDS=60
|
||||
|
||||
460
DELIVERABLE.md
Normal file
460
DELIVERABLE.md
Normal file
@@ -0,0 +1,460 @@
|
||||
# AI Job Source Agent — Implementation Deliverable
|
||||
|
||||
## What it does
|
||||
|
||||
Given a job search query, the agent produces records of the form:
|
||||
|
||||
```
|
||||
company_name, career_page_url, open_position_url
|
||||
```
|
||||
|
||||
It runs in configurable batches, persists state so re-runs skip already-processed
|
||||
jobs, and degrades gracefully when any tier fails.
|
||||
|
||||
---
|
||||
|
||||
## Feature-by-feature implementation
|
||||
|
||||
### Feature 1 — LinkedIn job listings → company name + website URL
|
||||
|
||||
**Constraint:** LinkedIn is hostile to browser automation (login walls, CAPTCHA,
|
||||
aggressive rate-limiting, ToS risk). The pipeline never opens a browser on LinkedIn.
|
||||
|
||||
**Solution:** [python-jobspy](https://github.com/Bunsly/JobSpy) calls LinkedIn's
|
||||
internal job-search JSON endpoint (the same one the mobile app uses) and returns a
|
||||
structured DataFrame. No credentials are required; no browser is launched.
|
||||
|
||||
```
|
||||
jobsource/sources/jobspy_source.py ← default provider
|
||||
jobsource/sources/apify_source.py ← drop-in alternative (Apify LinkedIn actor)
|
||||
jobsource/sources/base.py ← JobSource ABC: fetch_recent_jobs() -> list[RawJob]
|
||||
```
|
||||
|
||||
Each record becomes a `RawJob`:
|
||||
|
||||
| Field | Source | Notes |
|
||||
|---|---|---|
|
||||
| `job_id` | parsed from LinkedIn URL `/jobs/view/{id}` | primary dedup key |
|
||||
| `company` | JobSpy `company` column | display name |
|
||||
| `website` | JobSpy `company_url_direct` | **0% fill rate observed in practice** |
|
||||
| `linkedin_url` | canonical form, tracking params stripped | stored as-is |
|
||||
| `listed_at` | JobSpy `date_posted` | ~40% fill rate without full description fetch |
|
||||
|
||||
Because `website` is almost always empty from JobSpy, a dedicated resolution step
|
||||
(Feature 2) is mandatory for every job.
|
||||
|
||||
**Dedup:** jobs are keyed on `job_id` (LinkedIn numeric ID). Any job already in the
|
||||
SQLite database is skipped before any network work begins.
|
||||
|
||||
---
|
||||
|
||||
### Feature 1b — Company name → company website URL
|
||||
|
||||
```
|
||||
jobsource/resolve.py
|
||||
```
|
||||
|
||||
Three-tier cascade, returns on first hit:
|
||||
|
||||
```
|
||||
Tier 1 Provider-supplied URL
|
||||
↳ JobSpy occasionally includes it; accept without a network call.
|
||||
|
||||
Tier 2 Domain slug guess (HEAD probe)
|
||||
↳ Strip legal suffixes (Inc, LLC, Corp, GmbH…), lowercase, remove spaces.
|
||||
Probe https://{slug}.com with HTTP HEAD.
|
||||
Example: "Anthropic" → anthropic.com ✓
|
||||
"TCS" → tcs.com ✓
|
||||
"Infosys" → infosys.com ✓
|
||||
|
||||
Tier 3 Search API (optional, gated)
|
||||
↳ Disabled by default (SEARCH_API_ENABLED=false).
|
||||
Wire a real provider (e.g. SerpAPI) in _search_api_lookup() when needed.
|
||||
```
|
||||
|
||||
Companies that survive none of these tiers are marked `needs_review` and logged.
|
||||
The pipeline continues to the next job.
|
||||
|
||||
---
|
||||
|
||||
### Feature 2 — Company website → career page URL
|
||||
|
||||
```
|
||||
jobsource/careers/cascade.py ← orchestrator
|
||||
jobsource/careers/ats.py ← tiers 1 and 1b
|
||||
jobsource/careers/heuristics.py ← tiers 2, 3, 4
|
||||
jobsource/careers/classify_llm.py← tier 5 (cheap LLM)
|
||||
jobsource/agent_fallback.py ← tier 6 (browser agent, last resort)
|
||||
```
|
||||
|
||||
Six tiers, ordered cheapest → most expensive:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ Tier 1 ATS detection (HTML) confidence 0.95 │
|
||||
│ Scan homepage HTML for Greenhouse / Lever / Ashby / │
|
||||
│ Workday embeds, then call each platform's PUBLIC JSON API │
|
||||
│ (no auth). Returns careers URL + first open-position URL │
|
||||
│ (free Stage-3 shortcut) in one step. │
|
||||
│ ● Greenhouse: boards-api.greenhouse.io/v1/boards/{slug} │
|
||||
│ ● Lever: api.lever.co/v0/postings/{slug} │
|
||||
│ ● Ashby: api.ashbyhq.com/posting-api/job-board/{slug}│
|
||||
│ ● Workday: {host}/wday/cxs/{tenant}/{site}/jobs │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ Tier 1b ATS slug-guess confidence 0.90 │
|
||||
│ Homepage is a SPA (JS-rendered) so Tier 1 HTML scan │
|
||||
│ misses. Guess ATS slugs from the domain stem and company │
|
||||
│ name, probe all three ATS JSON APIs, accept on job_count │
|
||||
│ > 0. Cross-check org_name to prevent collisions. │
|
||||
│ Example: anthropic.com (Next.js SPA) → Greenhouse slug │
|
||||
│ "anthropic" → 370+ jobs │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ Tier 2 URL patterns confidence 0.80 │
|
||||
│ Probe /careers, /career, /jobs, /join-us, /join, │
|
||||
│ careers.{domain}, jobs.{domain} with HTTP GET. │
|
||||
│ Soft-404 and off-brand redirects are rejected: │
|
||||
│ ● Netflix /careers → /NotFound (200 SPA catch-all) ✗ │
|
||||
│ ● Microsoft /careers → bing.com redirect ✗ │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ Tier 3 Homepage link scan confidence 0.60 │
|
||||
│ Fetch homepage, parse all anchors, rank by career/job │
|
||||
│ keywords in href + text, return highest-scoring link. │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ Tier 4 Sitemap confidence 0.50 │
|
||||
│ Fetch sitemap.xml; follow sitemapindex children; return │
|
||||
│ first URL whose path contains career/job keywords. │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ Tier 5 Cheap-LLM classification confidence 0.55 │
|
||||
│ Pass extracted anchors to a small model (Pydantic AI, │
|
||||
│ typed output). No-ops gracefully when LLM key is absent. │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ Tier 6 Browser agent (last resort) confidence 0.50 │
|
||||
│ Browser Use + Playwright/Chromium. Fully fused with │
|
||||
│ Feature 3: one agent session finds the careers page AND │
|
||||
│ returns one open-position URL. Fires only after all │
|
||||
│ static tiers miss. Typical runtime: 2–4 minutes/company. │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**ATS upgrade:** when a heuristic tier (URL pattern, homepage scan, or sitemap)
|
||||
finds a URL that turns out to be an ATS board URL (e.g. `jobs.lever.co/acme`), the
|
||||
result is silently upgraded to ATS confidence (0.95) and the position URL shortcut
|
||||
is fetched in the same pass. No extra HTTP round-trip.
|
||||
|
||||
**Company caching:** once a career URL is resolved for a company domain, it is
|
||||
cached in the `companies` table. Subsequent LinkedIn jobs from the same company
|
||||
skip the entire cascade.
|
||||
|
||||
---
|
||||
|
||||
### Feature 3 — Career page → one open-position URL
|
||||
|
||||
```
|
||||
jobsource/extract.py
|
||||
```
|
||||
|
||||
Four tiers (Tier 1 is usually free from Stage 2):
|
||||
|
||||
```
|
||||
Tier 1 ATS JSON shortcut
|
||||
If Stage 2 resolved via ATS, the first open-position URL is already
|
||||
in hand — no extra HTTP call needed.
|
||||
|
||||
Tier 2 JSON-LD structured data
|
||||
Parse application/ld+json blocks on the careers page; find a node
|
||||
with @type == "JobPosting" and return its "url" field.
|
||||
|
||||
Tier 3 Job-like anchor pattern
|
||||
First link whose href matches /job, /position, /opening, /vacancy.
|
||||
|
||||
Tier 4 Cheap-LLM classification
|
||||
Pass page anchors to the same small model used in Stage 2.
|
||||
|
||||
Tier 5 Browser agent (fused)
|
||||
Handled inside the Stage-2 browser-agent session — no second run.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Output — Stage 4
|
||||
|
||||
```
|
||||
jobsource/db.py ← SQLite: companies + jobs tables, dedup, CSV export
|
||||
jobsource/pipeline.py ← run_batch(): dedup → cascade → persist → export
|
||||
```
|
||||
|
||||
`output/results.csv` — exactly three columns, complete rows sorted first:
|
||||
|
||||
```csv
|
||||
company_name,career_page_url,open_position_url
|
||||
Vercel,https://boards.greenhouse.io/vercel,https://boards.greenhouse.io/vercel/jobs/5017820004
|
||||
Linear,https://jobs.ashbyhq.com/linear,https://jobs.ashbyhq.com/linear/4ac6b5b1-abc
|
||||
Figma,https://boards.greenhouse.io/figma,https://job-boards.greenhouse.io/figma/jobs/5044
|
||||
Infosys,https://career.infosys.com/joblist,https://career.infosys.com/jobdesc?jobReferenceCode=INFSYS-EXTERNAL-247232
|
||||
TCS,https://ibegin.tcsapps.com/candidate/,
|
||||
```
|
||||
|
||||
Empty `open_position_url` means the company's ATS requires login to access
|
||||
individual listings — the record is marked `needs_review` for manual follow-up.
|
||||
|
||||
---
|
||||
|
||||
## Architecture at a glance
|
||||
|
||||
```
|
||||
LinkedIn (JobSpy / Apify)
|
||||
│
|
||||
▼
|
||||
Stage 1: Ingest RawJob{job_id, company, linkedin_url, website?}
|
||||
│
|
||||
▼
|
||||
Stage 1b: Resolve website company name → https://{slug}.com
|
||||
│
|
||||
▼
|
||||
Stage 2: Find careers page ─────────────────────────────────────────┐
|
||||
Tier 1 ATS HTML detect + JSON API → careers_url + position_url ◄─(shortcut)
|
||||
Tier 1b ATS slug-guess → careers_url + position_url ◄─(shortcut)
|
||||
Tier 2 URL pattern probe → careers_url
|
||||
Tier 3 Homepage link scan → careers_url
|
||||
Tier 4 Sitemap parse → careers_url
|
||||
Tier 5 Cheap-LLM classify → careers_url
|
||||
Tier 6 Browser agent (fused) ──────────────────────────────────┐
|
||||
│ │
|
||||
▼ │
|
||||
Stage 3: Extract open position │
|
||||
Tier 1 ATS JSON shortcut (free) │
|
||||
Tier 2 JSON-LD JobPosting │
|
||||
Tier 3 Job-like anchor pattern │
|
||||
Tier 4 Cheap-LLM classify │
|
||||
Tier 5 Browser agent result ◄───────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
Stage 4: Persist + CSV export output/results.csv
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Live demo walkthrough
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.11+
|
||||
- `uv` (recommended) or pip
|
||||
- No API keys required for the deterministic tiers (ATS + heuristics)
|
||||
- LLM API key optional (enables Tier 5 cheap-LLM and Tier 6 browser agent)
|
||||
|
||||
### Step 1 — Install
|
||||
|
||||
```bash
|
||||
git clone <repo>
|
||||
cd JobSourceAgent
|
||||
|
||||
python -m venv .venv && source .venv/bin/activate
|
||||
uv pip install -r requirements.txt # or: pip install -r requirements.txt
|
||||
playwright install chromium # only needed for browser-agent tier
|
||||
```
|
||||
|
||||
### Step 2 — Configure
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
**Minimum `.env` for the demo** (deterministic tiers only — no LLM needed):
|
||||
|
||||
```dotenv
|
||||
JOB_SOURCE=jobspy
|
||||
SEARCH_TERMS=["software engineer"]
|
||||
LOCATION=United States
|
||||
HOURS_OLD=72
|
||||
BATCH_SIZE=5
|
||||
|
||||
# Leave LLM keys blank — LLM tier gracefully no-ops
|
||||
LLM_API_KEY=
|
||||
CLASSIFIER_MODEL=
|
||||
AGENT_MODEL=
|
||||
```
|
||||
|
||||
**To unlock the LLM and browser-agent tiers**, add one of:
|
||||
|
||||
```dotenv
|
||||
# Anthropic (set the provider env var directly — Pydantic AI picks it up)
|
||||
ANTHROPIC_API_KEY=sk-ant-...
|
||||
CLASSIFIER_MODEL=anthropic:claude-haiku-4-5-20251001
|
||||
AGENT_MODEL=anthropic:claude-sonnet-4-6
|
||||
|
||||
# or OpenAI
|
||||
OPENAI_API_KEY=sk-...
|
||||
CLASSIFIER_MODEL=openai:gpt-4o-mini
|
||||
AGENT_MODEL=openai:gpt-4o
|
||||
```
|
||||
|
||||
### Step 3 — Run a batch
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m jobsource.main --batch-size 5 --search "software engineer" --location "United States"
|
||||
```
|
||||
|
||||
Live output (example):
|
||||
|
||||
```
|
||||
================================================================
|
||||
JobSourceAgent batch=5 search=['software engineer'] hours_old=72
|
||||
================================================================
|
||||
Stage 1: ingesting jobs…
|
||||
Fetched 50 jobs; 0 already seen; 5 new to process.
|
||||
|
||||
[5017820004] Vercel
|
||||
website: https://vercel.com
|
||||
careers: https://boards.greenhouse.io/vercel [ats:greenhouse conf=0.95]
|
||||
position: https://boards.greenhouse.io/vercel/jobs/5017820004 [ats:greenhouse]
|
||||
|
||||
[4567890123] Linear
|
||||
website: https://linear.app
|
||||
careers: https://jobs.ashbyhq.com/linear [ats:ashby:slug_guess conf=0.90]
|
||||
position: https://jobs.ashbyhq.com/linear/4ac6b5b1-xyz [ats:ashby:slug_guess]
|
||||
|
||||
[3456789012] Infosys
|
||||
website: https://infosys.com
|
||||
careers: https://career.infosys.com/joblist [browser_agent conf=0.50]
|
||||
position: https://career.infosys.com/jobdesc?jobReferenceCode=INFSYS-EXTERNAL-247232 [browser_agent]
|
||||
|
||||
...
|
||||
================================================================
|
||||
SUMMARY (5 new jobs processed in 42.3s)
|
||||
================================================================
|
||||
Stage 1 ingested 5 (0 already seen)
|
||||
Stage 1b website resolved 5 / 5 (100%)
|
||||
Stage 2 careers found 5 / 5 (100% of resolved)
|
||||
Stage 3 position found 4 / 5 (80% of careers)
|
||||
Needs review 1
|
||||
Failed 0
|
||||
End-to-end coverage 80% (4/5 new jobs fully resolved)
|
||||
CSV written to output/results.csv
|
||||
```
|
||||
|
||||
### Step 4 — Verify dedup
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m jobsource.main --batch-size 5 --search "software engineer" --location "United States"
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
```
|
||||
Fetched 50 jobs; 5 already seen; 0 new to process.
|
||||
```
|
||||
|
||||
### Step 5 — Inspect the output
|
||||
|
||||
```bash
|
||||
cat output/results.csv
|
||||
```
|
||||
|
||||
```
|
||||
company_name,career_page_url,open_position_url
|
||||
Vercel,https://boards.greenhouse.io/vercel,https://boards.greenhouse.io/vercel/jobs/5017820004
|
||||
Linear,https://jobs.ashbyhq.com/linear,https://jobs.ashbyhq.com/linear/4ac6b5b1-xyz
|
||||
...
|
||||
```
|
||||
|
||||
### Step 6 — Run the automated validation checklist
|
||||
|
||||
```bash
|
||||
.venv/bin/python validate.py --batch-size 5
|
||||
```
|
||||
|
||||
```
|
||||
================================================================
|
||||
validate.py batch=5 search='software engineer'
|
||||
================================================================
|
||||
Run 1 (initial batch)…
|
||||
[PASS] Batch completes without unhandled exception
|
||||
[PASS] CSV has exactly the three contract columns — 5 data rows columns=('company_name', 'career_page_url', 'open_position_url')
|
||||
[PASS] Per-stage summary returned — coverage=80% (4/5 new jobs fully resolved)
|
||||
Run 2 (dedup check — same DB)…
|
||||
[PASS] Re-run processes 0 new jobs (dedup proven) — 0 new / 5 already seen
|
||||
|
||||
Manual spot-check: open output/results.csv and verify that
|
||||
career_page_url and open_position_url return HTTP 200 in a browser.
|
||||
|
||||
================================================================
|
||||
Overall: PASS (4/4 automated checks passed)
|
||||
================================================================
|
||||
```
|
||||
|
||||
### Step 7 — Run the unit tests
|
||||
|
||||
```bash
|
||||
.venv/bin/pytest -q
|
||||
```
|
||||
|
||||
```
|
||||
307 passed in 2.34s
|
||||
```
|
||||
|
||||
### Step 8 — (Optional) Scheduled run via Prefect
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m jobsource.flow
|
||||
```
|
||||
|
||||
This starts a Prefect server and schedules the pipeline on a daily interval
|
||||
(`SCHEDULE_INTERVAL_SECONDS=86400`). The Prefect UI is at `http://127.0.0.1:4200`.
|
||||
|
||||
### Step 9 — (Optional) Detailed stage-by-stage trace
|
||||
|
||||
```bash
|
||||
.venv/bin/python scripts/e2e_smoke.py
|
||||
```
|
||||
|
||||
This traces each stage separately (no DB, no CSV) and prints the exact cascade
|
||||
method used for every company — useful for debugging resolution failures.
|
||||
|
||||
---
|
||||
|
||||
## Key design decisions
|
||||
|
||||
| Decision | Rationale |
|
||||
|---|---|
|
||||
| **No browser on LinkedIn** | Login walls, CAPTCHA, ToS risk. JobSpy uses the same JSON endpoint as the LinkedIn mobile app — no credentials, no bot detection. |
|
||||
| **Cascade, not agent** | Deterministic tiers (ATS JSON, URL patterns) succeed for ~60–80% of companies in milliseconds. LLM and browser agent only fire for the long tail. Total cost for the cheap tiers: near-zero. |
|
||||
| **ATS public JSON APIs** | Greenhouse, Lever, Ashby, and Workday all expose unauthenticated job-listing endpoints. Calling them directly is faster and more reliable than scraping HTML. |
|
||||
| **Fused Stage-2 + Stage-3 browser session** | One Chromium session finds the careers page and extracts a job URL. Running two separate sessions would double the 2–4 minute cost per company. |
|
||||
| **Graceful degradation** | Every tier wraps in try/except. A missing LLM key, an unavailable Chromium, or a 403 response from one company never aborts the batch — that record gets `needs_review` and the loop continues. |
|
||||
| **Dedup on `job_id`** | LinkedIn's numeric job posting ID is stable. Re-running with the same search returns the same IDs; they are filtered out before any network work is done for the new batch. |
|
||||
| **Company-level caching** | Once a careers URL is found for `acme.com`, all future jobs from that domain skip the entire cascade. Avoids redundant work for companies with many open listings. |
|
||||
|
||||
---
|
||||
|
||||
## File map (quick reference)
|
||||
|
||||
```
|
||||
jobsource/
|
||||
config.py env-driven settings (pydantic-settings)
|
||||
models.py RawJob, JobResult, JobStatus, CSV_COLUMNS
|
||||
http.py shared httpx client factory
|
||||
db.py SQLite: companies + jobs; dedup; CSV export
|
||||
resolve.py company name → website URL (Stage 1b)
|
||||
sources/
|
||||
base.py JobSource ABC
|
||||
jobspy_source.py default LinkedIn provider (python-jobspy)
|
||||
apify_source.py alternative provider (Apify actor)
|
||||
careers/
|
||||
cascade.py find_careers_page() — orchestrates 6 tiers
|
||||
ats.py ATS HTML detect + 4 public JSON APIs + slug-guess
|
||||
heuristics.py URL patterns, homepage scan, sitemap
|
||||
classify_llm.py Pydantic AI link classifier (careers + job links)
|
||||
extract.py extract_open_position() — Stage 3
|
||||
agent_fallback.py Browser Use fused Stage-2/3 fallback
|
||||
pipeline.py run_batch() — dedup, per-record isolation, summary
|
||||
flow.py Prefect flow + interval schedule
|
||||
main.py CLI entry point
|
||||
tests/ 307 offline pytest tests
|
||||
scripts/
|
||||
e2e_smoke.py live stage-by-stage trace (no DB)
|
||||
validate.py live PASS/FAIL validation against success criteria
|
||||
output/
|
||||
results.csv pipeline output (gitignored)
|
||||
jobsource.db SQLite state (gitignored)
|
||||
```
|
||||
83
README.md
83
README.md
@@ -12,32 +12,93 @@ Runs in configurable batches, on a schedule, and is incremental — re-runs proc
|
||||
|
||||
```bash
|
||||
python -m venv .venv && source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
playwright install chromium # for the browser-agent tier
|
||||
cp .env.example .env # fill keys as available
|
||||
uv pip install -r requirements.txt # uv resolves pydantic deps reliably; pip also works
|
||||
playwright install chromium # for the browser-agent tier
|
||||
cp .env.example .env # fill keys as available
|
||||
```
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
# one batch
|
||||
# one batch (no server required)
|
||||
python -m jobsource.main --batch-size 20 --search "software engineer" --location "United States"
|
||||
|
||||
# scheduled run (Prefect)
|
||||
python -m jobsource.flow
|
||||
|
||||
# cron fallback (no daemon):
|
||||
# */0 6 * * * cd <repo> && ./.venv/bin/python -m jobsource.main --batch-size 50
|
||||
```
|
||||
|
||||
`--search` is repeatable. Run `python -m jobsource.main --help` for all options.
|
||||
|
||||
### Scheduled run (Prefect)
|
||||
|
||||
```bash
|
||||
# Boots a local Prefect server automatically, then serves the flow on an
|
||||
# interval schedule (default: daily, set SCHEDULE_INTERVAL_SECONDS to change).
|
||||
python -m jobsource.flow
|
||||
```
|
||||
|
||||
If you already have a Prefect server running elsewhere, point the flow at it:
|
||||
|
||||
```bash
|
||||
export PREFECT_API_URL=http://127.0.0.1:4200/api
|
||||
python -m jobsource.flow
|
||||
```
|
||||
|
||||
To trigger a run manually while the flow is being served:
|
||||
|
||||
```bash
|
||||
prefect deployment run 'job-source-batch/job-source-batch'
|
||||
```
|
||||
|
||||
> **Note — Prefect 3.7.4 + FastAPI ≥ 0.137 compatibility**: Prefect's built-in
|
||||
> ephemeral server is incompatible with FastAPI 0.137+ (route lists are cleared
|
||||
> after inclusion but re-read lazily during routing). `python -m jobsource.flow`
|
||||
> works around this by starting a persistent `prefect server start` process
|
||||
> instead of the ephemeral server.
|
||||
|
||||
### No-daemon cron fallback
|
||||
|
||||
If you prefer a plain cron job with no running process, add this line to your
|
||||
crontab (`crontab -e`), adjusting the path to your repo:
|
||||
|
||||
```
|
||||
# daily at 06:00 — no Prefect daemon required
|
||||
0 6 * * * cd /path/to/repo && ./.venv/bin/python -m jobsource.main --batch-size 50
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
pytest -q
|
||||
.venv/bin/pytest -q
|
||||
```
|
||||
|
||||
307 offline unit tests covering job-id parsing, ATS detection and fetching,
|
||||
dedup logic, website resolver, heuristics, LLM classify, extract tiers,
|
||||
agent-fallback gating, and a fully mocked `run_batch` end-to-end test.
|
||||
|
||||
## Validate
|
||||
|
||||
`validate.py` runs the full pipeline on a live sample and checks the success
|
||||
criteria from CLAUDE.md:
|
||||
|
||||
```bash
|
||||
.venv/bin/python validate.py # default: 5 jobs, "software engineer"
|
||||
.venv/bin/python validate.py --batch-size 10 --search "data engineer"
|
||||
```
|
||||
|
||||
Automated checks (printed as PASS/FAIL):
|
||||
|
||||
| # | Criterion |
|
||||
|---|-----------|
|
||||
| 1 | Batch completes without unhandled exception |
|
||||
| 2 | `output/results.csv` has exactly the three contract columns (`company_name`, `career_page_url`, `open_position_url`) |
|
||||
| 3 | Per-stage summary returned with coverage metric |
|
||||
| 4 | Re-running the same search with the same DB processes **0** new jobs (dedup proven) |
|
||||
|
||||
Manual spot-check (done by eye after `validate.py` passes):
|
||||
|
||||
- Open `output/results.csv`.
|
||||
- Pick one complete row (`open_position_url` non-empty).
|
||||
- Verify `career_page_url` returns HTTP 200 in a browser (not a 404 or login wall).
|
||||
- Verify `open_position_url` returns HTTP 200 in a browser.
|
||||
|
||||
## Output
|
||||
|
||||
`output/results.csv` — three columns: `company_name`, `career_page_url`, `open_position_url`.
|
||||
|
||||
@@ -57,6 +57,11 @@ class Settings(BaseSettings):
|
||||
enable_browser_agent: bool = True
|
||||
browser_headless: bool = True
|
||||
|
||||
# -- Scheduling (Prefect flow) -----------------------------------------
|
||||
schedule_interval_seconds: int = 86400 # default: daily
|
||||
flow_retries: int = 1
|
||||
flow_retry_delay_seconds: int = 60
|
||||
|
||||
|
||||
@lru_cache
|
||||
def get_settings() -> Settings:
|
||||
|
||||
@@ -1,10 +1,126 @@
|
||||
"""Prefect flow definition and interval schedule.
|
||||
|
||||
Scaffold stub -- not implemented yet.
|
||||
Run with:
|
||||
python -m jobsource.flow
|
||||
|
||||
This starts a long-running process that:
|
||||
1. Boots a local Prefect server (if ``PREFECT_API_URL`` is not already set).
|
||||
2. Registers the ``job-source-batch`` deployment on an interval schedule
|
||||
(default: daily, controlled by ``SCHEDULE_INTERVAL_SECONDS``).
|
||||
3. Polls for scheduled and manually-triggered runs, executing them in-process.
|
||||
|
||||
The flow delegates entirely to :func:`~jobsource.pipeline.run_batch`, which is
|
||||
idempotent — the dedup step means retrying a failed flow run processes only the
|
||||
jobs that haven't been committed yet, so flow-level retries are safe.
|
||||
|
||||
**Why a persistent server instead of Prefect's ephemeral server?**
|
||||
Prefect 3.7.4's ephemeral server is incompatible with FastAPI ≥ 0.137: it
|
||||
deletes sub-router route lists after inclusion (memory optimisation), but
|
||||
FastAPI 0.137 re-reads those lists lazily during request routing, causing all
|
||||
API paths to return 404. The persistent server (``prefect server start``)
|
||||
does not apply that optimisation, so it works correctly.
|
||||
"""
|
||||
# TODO (scheduling): implement per CLAUDE.md "Orchestration/scheduling: Prefect".
|
||||
# Wrap run_batch() in a @flow with:
|
||||
# - Retries on the flow level.
|
||||
# - An interval schedule (configurable; default daily).
|
||||
# Run with: python -m jobsource.flow
|
||||
# Cron fallback (no daemon): */0 6 * * * cd <repo> && ./.venv/bin/python -m jobsource.main --batch-size 50
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
import httpx
|
||||
from prefect import flow
|
||||
|
||||
from .config import get_settings
|
||||
from .pipeline import BatchSummary, run_batch
|
||||
|
||||
# Read scheduling config once at module load (get_settings() is a cached singleton).
|
||||
# Prefect @flow decorator arguments must be concrete values, not lazy callables.
|
||||
_settings = get_settings()
|
||||
|
||||
_DEFAULT_SERVER_URL = "http://127.0.0.1:4200"
|
||||
_SERVER_READY_TIMEOUT = 30 # seconds
|
||||
|
||||
|
||||
@flow(
|
||||
name="job-source-batch",
|
||||
retries=_settings.flow_retries,
|
||||
retry_delay_seconds=_settings.flow_retry_delay_seconds,
|
||||
log_prints=True,
|
||||
)
|
||||
def job_source_batch(
|
||||
batch_size: int | None = None,
|
||||
search_terms: list[str] | None = None,
|
||||
location: str | None = None,
|
||||
hours_old: int | None = None,
|
||||
) -> BatchSummary:
|
||||
"""Fetch, cascade, persist, and export one batch of LinkedIn jobs.
|
||||
|
||||
All parameters override the corresponding config/env values when provided;
|
||||
``None`` means "use config default". Returns the :class:`BatchSummary`
|
||||
produced by :func:`~jobsource.pipeline.run_batch`.
|
||||
|
||||
Flow-level retries are safe because ``run_batch`` deduplicates on
|
||||
``job_id`` — a retry only processes jobs not yet committed.
|
||||
"""
|
||||
return run_batch(
|
||||
batch_size=batch_size,
|
||||
search_terms=search_terms,
|
||||
location=location,
|
||||
hours_old=hours_old,
|
||||
)
|
||||
|
||||
|
||||
def _start_prefect_server() -> subprocess.Popen[bytes]:
|
||||
"""Start ``prefect server start`` as a background subprocess."""
|
||||
return subprocess.Popen(
|
||||
[sys.executable, "-m", "prefect", "server", "start",
|
||||
"--host", "127.0.0.1", "--port", "4200"],
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
)
|
||||
|
||||
|
||||
def _wait_for_server(url: str, timeout: int) -> bool:
|
||||
"""Return True once the Prefect server health endpoint responds OK."""
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
r = httpx.get(f"{url}/api/health", timeout=2.0)
|
||||
if r.status_code == 200:
|
||||
return True
|
||||
except Exception:
|
||||
pass
|
||||
time.sleep(1)
|
||||
return False
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
_server_proc: subprocess.Popen[bytes] | None = None
|
||||
|
||||
if not os.environ.get("PREFECT_API_URL"):
|
||||
# No external server configured — boot the persistent local server.
|
||||
# The Prefect 3.7.4 ephemeral server is incompatible with FastAPI ≥ 0.137
|
||||
# (see module docstring), so we start a persistent server instead.
|
||||
print(f"Starting local Prefect server at {_DEFAULT_SERVER_URL} …")
|
||||
_server_proc = _start_prefect_server()
|
||||
if not _wait_for_server(_DEFAULT_SERVER_URL, _SERVER_READY_TIMEOUT):
|
||||
_server_proc.terminate()
|
||||
raise RuntimeError(
|
||||
f"Prefect server did not become ready within {_SERVER_READY_TIMEOUT}s. "
|
||||
"Start it manually with `prefect server start` and set "
|
||||
"PREFECT_API_URL=http://127.0.0.1:4200/api before re-running."
|
||||
)
|
||||
os.environ["PREFECT_API_URL"] = f"{_DEFAULT_SERVER_URL}/api"
|
||||
print(f"Prefect server ready. API → {os.environ['PREFECT_API_URL']}")
|
||||
|
||||
try:
|
||||
# Serve on an interval schedule; blocks until Ctrl-C.
|
||||
job_source_batch.serve(
|
||||
name="job-source-batch",
|
||||
interval=_settings.schedule_interval_seconds,
|
||||
)
|
||||
finally:
|
||||
if _server_proc is not None:
|
||||
print("Stopping local Prefect server …")
|
||||
_server_proc.terminate()
|
||||
_server_proc.wait(timeout=10)
|
||||
|
||||
@@ -1,12 +1,19 @@
|
||||
"""CLI entry point: `python -m jobsource.main`.
|
||||
"""CLI entry point: ``python -m jobsource.main``.
|
||||
|
||||
Scaffold stub. Argument parsing is wired so `--help` works; the actual batch
|
||||
run lands in a later step (see jobsource/pipeline.py). Imports only stdlib so
|
||||
`--help` works before the heavier dependencies are installed.
|
||||
Parses arguments and delegates to :func:`~jobsource.pipeline.run_batch`.
|
||||
All heavy imports are deferred past the argument parse so ``--help`` exits
|
||||
immediately with no dependency overhead.
|
||||
|
||||
Usage examples::
|
||||
|
||||
python -m jobsource.main --help
|
||||
python -m jobsource.main --batch-size 20 --search "software engineer" --location "United States"
|
||||
python -m jobsource.main --search "data engineer" --search "ML engineer" --hours-old 48
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import sys
|
||||
|
||||
|
||||
@@ -46,6 +53,15 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
|
||||
# Configure root logger so pipeline/cascade INFO and WARNING messages are
|
||||
# visible on the terminal. Deferred past parse_args so --help stays instant.
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)-8s %(name)s: %(message)s",
|
||||
datefmt="%H:%M:%S",
|
||||
)
|
||||
|
||||
from jobsource.pipeline import run_batch # deferred: keeps --help fast before heavy deps
|
||||
run_batch(
|
||||
batch_size=args.batch_size,
|
||||
|
||||
@@ -90,11 +90,12 @@ def run_batch(
|
||||
print(f"{'='*64}")
|
||||
print("Stage 1: ingesting jobs…")
|
||||
|
||||
fetch_limit = min(settings.results_wanted, batch_size)
|
||||
raw_jobs: list[RawJob] = source.fetch_recent_jobs(
|
||||
search_terms=search_terms,
|
||||
location=location,
|
||||
hours_old=hours_old,
|
||||
results_wanted=settings.results_wanted,
|
||||
results_wanted=fetch_limit,
|
||||
)
|
||||
summary.fetched = len(raw_jobs)
|
||||
|
||||
|
||||
@@ -380,7 +380,7 @@ def _make_raw(job_id: str, company: str = "Acme") -> RawJob:
|
||||
class TestRunBatch:
|
||||
"""Monkeypatched run_batch tests — no network, no real providers."""
|
||||
|
||||
def _patch_all(self, monkeypatch, raw_jobs: list[RawJob]) -> None:
|
||||
def _patch_all(self, monkeypatch, raw_jobs: list[RawJob]) -> MagicMock:
|
||||
"""Patch all external I/O in pipeline so tests run offline."""
|
||||
from jobsource.careers.cascade import CareersResult
|
||||
|
||||
@@ -418,6 +418,7 @@ class TestRunBatch:
|
||||
cm.__enter__ = lambda s: MagicMock()
|
||||
cm.__exit__ = MagicMock(return_value=False)
|
||||
monkeypatch.setattr("jobsource.pipeline.build_client", lambda: cm)
|
||||
return fake_source
|
||||
|
||||
def test_new_jobs_processed(self, tmp_path: Path, monkeypatch) -> None:
|
||||
from jobsource.pipeline import run_batch
|
||||
@@ -469,6 +470,20 @@ class TestRunBatch:
|
||||
assert summary.new == 3
|
||||
assert summary.position_found == 3
|
||||
|
||||
def test_fetch_limit_matches_batch_size(self, tmp_path: Path, monkeypatch) -> None:
|
||||
"""Immediate re-runs only re-fetch the requested batch window."""
|
||||
from jobsource.pipeline import run_batch
|
||||
|
||||
raw = [_make_raw(str(i), company=f"Co{i}") for i in range(10)]
|
||||
source = self._patch_all(monkeypatch, raw)
|
||||
|
||||
db = Database(path=tmp_path / "fetch-limit.db")
|
||||
summary = run_batch(batch_size=3, search_terms=["eng"], db=db)
|
||||
|
||||
assert summary.new == 3
|
||||
source.fetch_recent_jobs.assert_called_once()
|
||||
assert source.fetch_recent_jobs.call_args.kwargs["results_wanted"] == 3
|
||||
|
||||
def test_resolve_failure_marks_needs_review(self, tmp_path: Path, monkeypatch) -> None:
|
||||
"""Jobs where resolve_website returns None get needs_review, not failed."""
|
||||
from jobsource.pipeline import run_batch
|
||||
|
||||
159
validate.py
Normal file
159
validate.py
Normal file
@@ -0,0 +1,159 @@
|
||||
#!/usr/bin/env python
|
||||
"""Live validation against the CLAUDE.md success criteria.
|
||||
|
||||
Runs a small batch through the full pipeline, then immediately re-runs to
|
||||
verify dedup. Prints PASS/FAIL for each criterion and exits non-zero if
|
||||
any check fails.
|
||||
|
||||
Usage:
|
||||
.venv/bin/python validate.py
|
||||
.venv/bin/python validate.py --batch-size 10 --search "data engineer"
|
||||
.venv/bin/python validate.py --help
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
|
||||
from jobsource.db import Database
|
||||
from jobsource.models import CSV_COLUMNS
|
||||
from jobsource.pipeline import run_batch
|
||||
|
||||
|
||||
def _check(label: str, passed: bool, detail: str = "") -> bool:
|
||||
mark = "PASS" if passed else "FAIL"
|
||||
suffix = f" — {detail}" if detail else ""
|
||||
print(f" [{mark}] {label}{suffix}")
|
||||
return passed
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--batch-size", type=int, default=5,
|
||||
help="Number of new jobs to process (default: 5)")
|
||||
parser.add_argument("--search", default="software engineer",
|
||||
help="Search term (default: 'software engineer')")
|
||||
parser.add_argument("--location", default="United States",
|
||||
help="Location filter (default: 'United States')")
|
||||
args = parser.parse_args()
|
||||
|
||||
print(f"\n{'='*64}")
|
||||
print(f" validate.py batch={args.batch_size} search={args.search!r}")
|
||||
print(f"{'='*64}\n")
|
||||
|
||||
checks: list[bool] = []
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
db_path = Path(tmp) / "validate.db"
|
||||
csv_path = Path(tmp) / "results.csv"
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Run 1: initial batch
|
||||
# ------------------------------------------------------------------
|
||||
print("Run 1 (initial batch)…")
|
||||
s1 = exc1 = None
|
||||
try:
|
||||
with Database(path=db_path) as db:
|
||||
s1 = run_batch(
|
||||
batch_size=args.batch_size,
|
||||
search_terms=[args.search],
|
||||
location=args.location,
|
||||
db=db,
|
||||
)
|
||||
db.export_csv(path=csv_path)
|
||||
except Exception as exc:
|
||||
exc1 = exc
|
||||
|
||||
# [1] Batch completes without unhandled exception
|
||||
checks.append(_check(
|
||||
"Batch completes without unhandled exception",
|
||||
exc1 is None,
|
||||
repr(exc1) if exc1 else "",
|
||||
))
|
||||
if exc1 is not None:
|
||||
print("\nFatal error in run 1 — cannot continue.")
|
||||
return 1
|
||||
|
||||
# [2] CSV has exactly the three contract columns
|
||||
ok_csv = False
|
||||
csv_detail = "not written"
|
||||
if csv_path.exists():
|
||||
with csv_path.open() as f:
|
||||
reader = csv.DictReader(f)
|
||||
rows = list(reader)
|
||||
actual = tuple(reader.fieldnames or [])
|
||||
ok_csv = actual == CSV_COLUMNS
|
||||
csv_detail = (
|
||||
f"columns={actual}" if not ok_csv
|
||||
else f"{len(rows)} data rows columns={actual}"
|
||||
)
|
||||
checks.append(_check("CSV has exactly the three contract columns", ok_csv, csv_detail))
|
||||
|
||||
# [3] Per-stage summary with coverage metric
|
||||
ok_summary = s1 is not None
|
||||
checks.append(_check(
|
||||
"Per-stage summary returned",
|
||||
ok_summary,
|
||||
(f"coverage={s1.coverage*100:.0f}% "
|
||||
f"({s1.position_found}/{s1.new} new jobs fully resolved)") if s1 else "",
|
||||
))
|
||||
|
||||
# warn if no jobs were fetched (network/API issue, not a hard failure)
|
||||
if s1 and s1.new == 0:
|
||||
print(" NOTE: 0 new jobs fetched — dedup check may be vacuously true.")
|
||||
print(" Check your search terms, network, or hours_old setting.\n")
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Run 2: dedup check
|
||||
# ------------------------------------------------------------------
|
||||
print("Run 2 (dedup check — same DB)…")
|
||||
s2 = exc2 = None
|
||||
try:
|
||||
with Database(path=db_path) as db2:
|
||||
s2 = run_batch(
|
||||
batch_size=args.batch_size,
|
||||
search_terms=[args.search],
|
||||
location=args.location,
|
||||
db=db2,
|
||||
)
|
||||
except Exception as exc:
|
||||
exc2 = exc
|
||||
|
||||
# [4] Re-run processes 0 new jobs
|
||||
ok_dedup = exc2 is None and s2 is not None and s2.new == 0
|
||||
checks.append(_check(
|
||||
"Re-run processes 0 new jobs (dedup proven)",
|
||||
ok_dedup,
|
||||
(f"new={s2.new if s2 else 'N/A'}, seen={s2.already_seen if s2 else 'N/A'}"
|
||||
if not ok_dedup
|
||||
else f"0 new / {s2.already_seen} already seen"),
|
||||
))
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Spot-check reminder (manual step — not auto-checkable)
|
||||
# ------------------------------------------------------------------
|
||||
if s1 and s1.position_found > 0:
|
||||
print(
|
||||
f"\n Manual spot-check: open output/results.csv and verify that\n"
|
||||
f" career_page_url and open_position_url return HTTP 200 in a browser."
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Verdict
|
||||
# ------------------------------------------------------------------
|
||||
n_pass = sum(checks)
|
||||
n_total = len(checks)
|
||||
print(f"\n{'='*64}")
|
||||
overall = all(checks)
|
||||
print(f" Overall: {'PASS' if overall else 'FAIL'} ({n_pass}/{n_total} automated checks passed)")
|
||||
print(f"{'='*64}\n")
|
||||
return 0 if overall else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user