# AI Job Source Agent — Implementation Deliverable ## What it does Given a job search query, the agent produces records of the form: ``` company_name, career_page_url, open_position_url ``` It runs in configurable batches, persists state so re-runs skip already-processed jobs, and degrades gracefully when any tier fails. --- ## Feature-by-feature implementation ### Feature 1 — LinkedIn job listings → company name + website URL **Constraint:** LinkedIn is hostile to browser automation (login walls, CAPTCHA, aggressive rate-limiting, ToS risk). The pipeline never opens a browser on LinkedIn. **Solution:** [python-jobspy](https://github.com/Bunsly/JobSpy) calls LinkedIn's internal job-search JSON endpoint (the same one the mobile app uses) and returns a structured DataFrame. No credentials are required; no browser is launched. ``` jobsource/sources/jobspy_source.py ← default provider jobsource/sources/apify_source.py ← drop-in alternative (Apify LinkedIn actor) jobsource/sources/base.py ← JobSource ABC: fetch_recent_jobs() -> list[RawJob] ``` Each record becomes a `RawJob`: | Field | Source | Notes | |---|---|---| | `job_id` | parsed from LinkedIn URL `/jobs/view/{id}` | primary dedup key | | `company` | JobSpy `company` column | display name | | `website` | JobSpy `company_url_direct` | **0% fill rate observed in practice** | | `linkedin_url` | canonical form, tracking params stripped | stored as-is | | `listed_at` | JobSpy `date_posted` | ~40% fill rate without full description fetch | Because `website` is almost always empty from JobSpy, a dedicated resolution step (Feature 2) is mandatory for every job. **Dedup:** jobs are keyed on `job_id` (LinkedIn numeric ID). Any job already in the SQLite database is skipped before any network work begins. --- ### Feature 1b — Company name → company website URL ``` jobsource/resolve.py ``` Three-tier cascade, returns on first hit: ``` Tier 1 Provider-supplied URL ↳ JobSpy occasionally includes it; accept without a network call. Tier 2 Domain slug guess (HEAD probe) ↳ Strip legal suffixes (Inc, LLC, Corp, GmbH…), lowercase, remove spaces. Probe https://{slug}.com with HTTP HEAD. Example: "Anthropic" → anthropic.com ✓ "TCS" → tcs.com ✓ "Infosys" → infosys.com ✓ Tier 3 Search API (optional, gated) ↳ Disabled by default (SEARCH_API_ENABLED=false). Wire a real provider (e.g. SerpAPI) in _search_api_lookup() when needed. ``` Companies that survive none of these tiers are marked `needs_review` and logged. The pipeline continues to the next job. --- ### Feature 2 — Company website → career page URL ``` jobsource/careers/cascade.py ← orchestrator jobsource/careers/ats.py ← tiers 1 and 1b jobsource/careers/heuristics.py ← tiers 2, 3, 4 jobsource/careers/classify_llm.py← tier 5 (cheap LLM) jobsource/agent_fallback.py ← tier 6 (browser agent, last resort) ``` Six tiers, ordered cheapest → most expensive: ``` ┌─────────────────────────────────────────────────────────────────────┐ │ Tier 1 ATS detection (HTML) confidence 0.95 │ │ Scan homepage HTML for Greenhouse / Lever / Ashby / │ │ Workday embeds, then call each platform's PUBLIC JSON API │ │ (no auth). Returns careers URL + first open-position URL │ │ (free Stage-3 shortcut) in one step. │ │ ● Greenhouse: boards-api.greenhouse.io/v1/boards/{slug} │ │ ● Lever: api.lever.co/v0/postings/{slug} │ │ ● Ashby: api.ashbyhq.com/posting-api/job-board/{slug}│ │ ● Workday: {host}/wday/cxs/{tenant}/{site}/jobs │ ├─────────────────────────────────────────────────────────────────────┤ │ Tier 1b ATS slug-guess confidence 0.90 │ │ Homepage is a SPA (JS-rendered) so Tier 1 HTML scan │ │ misses. Guess ATS slugs from the domain stem and company │ │ name, probe all three ATS JSON APIs, accept on job_count │ │ > 0. Cross-check org_name to prevent collisions. │ │ Example: anthropic.com (Next.js SPA) → Greenhouse slug │ │ "anthropic" → 370+ jobs │ ├─────────────────────────────────────────────────────────────────────┤ │ Tier 2 URL patterns confidence 0.80 │ │ Probe /careers, /career, /jobs, /join-us, /join, │ │ careers.{domain}, jobs.{domain} with HTTP GET. │ │ Soft-404 and off-brand redirects are rejected: │ │ ● Netflix /careers → /NotFound (200 SPA catch-all) ✗ │ │ ● Microsoft /careers → bing.com redirect ✗ │ ├─────────────────────────────────────────────────────────────────────┤ │ Tier 3 Homepage link scan confidence 0.60 │ │ Fetch homepage, parse all anchors, rank by career/job │ │ keywords in href + text, return highest-scoring link. │ ├─────────────────────────────────────────────────────────────────────┤ │ Tier 4 Sitemap confidence 0.50 │ │ Fetch sitemap.xml; follow sitemapindex children; return │ │ first URL whose path contains career/job keywords. │ ├─────────────────────────────────────────────────────────────────────┤ │ Tier 5 Cheap-LLM classification confidence 0.55 │ │ Pass extracted anchors to a small model (Pydantic AI, │ │ typed output). No-ops gracefully when LLM key is absent. │ ├─────────────────────────────────────────────────────────────────────┤ │ Tier 6 Browser agent (last resort) confidence 0.50 │ │ Browser Use + Playwright/Chromium. Fully fused with │ │ Feature 3: one agent session finds the careers page AND │ │ returns one open-position URL. Fires only after all │ │ static tiers miss. Typical runtime: 2–4 minutes/company. │ └─────────────────────────────────────────────────────────────────────┘ ``` **ATS upgrade:** when a heuristic tier (URL pattern, homepage scan, or sitemap) finds a URL that turns out to be an ATS board URL (e.g. `jobs.lever.co/acme`), the result is silently upgraded to ATS confidence (0.95) and the position URL shortcut is fetched in the same pass. No extra HTTP round-trip. **Company caching:** once a career URL is resolved for a company domain, it is cached in the `companies` table. Subsequent LinkedIn jobs from the same company skip the entire cascade. --- ### Feature 3 — Career page → one open-position URL ``` jobsource/extract.py ``` Four tiers (Tier 1 is usually free from Stage 2): ``` Tier 1 ATS JSON shortcut If Stage 2 resolved via ATS, the first open-position URL is already in hand — no extra HTTP call needed. Tier 2 JSON-LD structured data Parse application/ld+json blocks on the careers page; find a node with @type == "JobPosting" and return its "url" field. Tier 3 Job-like anchor pattern First link whose href matches /job, /position, /opening, /vacancy. Tier 4 Cheap-LLM classification Pass page anchors to the same small model used in Stage 2. Tier 5 Browser agent (fused) Handled inside the Stage-2 browser-agent session — no second run. ``` --- ### Output — Stage 4 ``` jobsource/db.py ← SQLite: companies + jobs tables, dedup, CSV export jobsource/pipeline.py ← run_batch(): dedup → cascade → persist → export ``` `output/results.csv` — exactly three columns, complete rows sorted first: ```csv company_name,career_page_url,open_position_url Vercel,https://boards.greenhouse.io/vercel,https://boards.greenhouse.io/vercel/jobs/5017820004 Linear,https://jobs.ashbyhq.com/linear,https://jobs.ashbyhq.com/linear/4ac6b5b1-abc Figma,https://boards.greenhouse.io/figma,https://job-boards.greenhouse.io/figma/jobs/5044 Infosys,https://career.infosys.com/joblist,https://career.infosys.com/jobdesc?jobReferenceCode=INFSYS-EXTERNAL-247232 TCS,https://ibegin.tcsapps.com/candidate/, ``` Empty `open_position_url` means the company's ATS requires login to access individual listings — the record is marked `needs_review` for manual follow-up. --- ## Architecture at a glance ``` LinkedIn (JobSpy / Apify) │ ▼ Stage 1: Ingest RawJob{job_id, company, linkedin_url, website?} │ ▼ Stage 1b: Resolve website company name → https://{slug}.com │ ▼ Stage 2: Find careers page ─────────────────────────────────────────┐ Tier 1 ATS HTML detect + JSON API → careers_url + position_url ◄─(shortcut) Tier 1b ATS slug-guess → careers_url + position_url ◄─(shortcut) Tier 2 URL pattern probe → careers_url Tier 3 Homepage link scan → careers_url Tier 4 Sitemap parse → careers_url Tier 5 Cheap-LLM classify → careers_url Tier 6 Browser agent (fused) ──────────────────────────────────┐ │ │ ▼ │ Stage 3: Extract open position │ Tier 1 ATS JSON shortcut (free) │ Tier 2 JSON-LD JobPosting │ Tier 3 Job-like anchor pattern │ Tier 4 Cheap-LLM classify │ Tier 5 Browser agent result ◄───────────────────────────────────┘ │ ▼ Stage 4: Persist + CSV export output/results.csv ``` --- ## Live demo walkthrough ### Prerequisites - Python 3.11+ - `uv` (recommended) or pip - No API keys required for the deterministic tiers (ATS + heuristics) - LLM API key optional (enables Tier 5 cheap-LLM and Tier 6 browser agent) ### Step 1 — Install ```bash git clone cd JobSourceAgent python -m venv .venv && source .venv/bin/activate uv pip install -r requirements.txt # or: pip install -r requirements.txt playwright install chromium # only needed for browser-agent tier ``` ### Step 2 — Configure ```bash cp .env.example .env ``` **Minimum `.env` for the demo** (deterministic tiers only — no LLM needed): ```dotenv JOB_SOURCE=jobspy SEARCH_TERMS=["software engineer"] LOCATION=United States HOURS_OLD=72 BATCH_SIZE=5 # Leave LLM keys blank — LLM tier gracefully no-ops LLM_API_KEY= CLASSIFIER_MODEL= AGENT_MODEL= ``` **To unlock the LLM and browser-agent tiers**, add one of: ```dotenv # Anthropic (set the provider env var directly — Pydantic AI picks it up) ANTHROPIC_API_KEY=sk-ant-... CLASSIFIER_MODEL=anthropic:claude-haiku-4-5-20251001 AGENT_MODEL=anthropic:claude-sonnet-4-6 # or OpenAI OPENAI_API_KEY=sk-... CLASSIFIER_MODEL=openai:gpt-4o-mini AGENT_MODEL=openai:gpt-4o ``` ### Step 3 — Run a batch ```bash .venv/bin/python -m jobsource.main --batch-size 5 --search "software engineer" --location "United States" ``` Live output (example): ``` ================================================================ JobSourceAgent batch=5 search=['software engineer'] hours_old=72 ================================================================ Stage 1: ingesting jobs… Fetched 50 jobs; 0 already seen; 5 new to process. [5017820004] Vercel website: https://vercel.com careers: https://boards.greenhouse.io/vercel [ats:greenhouse conf=0.95] position: https://boards.greenhouse.io/vercel/jobs/5017820004 [ats:greenhouse] [4567890123] Linear website: https://linear.app careers: https://jobs.ashbyhq.com/linear [ats:ashby:slug_guess conf=0.90] position: https://jobs.ashbyhq.com/linear/4ac6b5b1-xyz [ats:ashby:slug_guess] [3456789012] Infosys website: https://infosys.com careers: https://career.infosys.com/joblist [browser_agent conf=0.50] position: https://career.infosys.com/jobdesc?jobReferenceCode=INFSYS-EXTERNAL-247232 [browser_agent] ... ================================================================ SUMMARY (5 new jobs processed in 42.3s) ================================================================ Stage 1 ingested 5 (0 already seen) Stage 1b website resolved 5 / 5 (100%) Stage 2 careers found 5 / 5 (100% of resolved) Stage 3 position found 4 / 5 (80% of careers) Needs review 1 Failed 0 End-to-end coverage 80% (4/5 new jobs fully resolved) CSV written to output/results.csv ``` ### Step 4 — Verify dedup ```bash .venv/bin/python -m jobsource.main --batch-size 5 --search "software engineer" --location "United States" ``` Expected: ``` Fetched 50 jobs; 5 already seen; 0 new to process. ``` ### Step 5 — Inspect the output ```bash cat output/results.csv ``` ``` company_name,career_page_url,open_position_url Vercel,https://boards.greenhouse.io/vercel,https://boards.greenhouse.io/vercel/jobs/5017820004 Linear,https://jobs.ashbyhq.com/linear,https://jobs.ashbyhq.com/linear/4ac6b5b1-xyz ... ``` ### Step 6 — Run the automated validation checklist ```bash .venv/bin/python validate.py --batch-size 5 ``` ``` ================================================================ validate.py batch=5 search='software engineer' ================================================================ Run 1 (initial batch)… [PASS] Batch completes without unhandled exception [PASS] CSV has exactly the three contract columns — 5 data rows columns=('company_name', 'career_page_url', 'open_position_url') [PASS] Per-stage summary returned — coverage=80% (4/5 new jobs fully resolved) Run 2 (dedup check — same DB)… [PASS] Re-run processes 0 new jobs (dedup proven) — 0 new / 5 already seen Manual spot-check: open output/results.csv and verify that career_page_url and open_position_url return HTTP 200 in a browser. ================================================================ Overall: PASS (4/4 automated checks passed) ================================================================ ``` ### Step 7 — Run the unit tests ```bash .venv/bin/pytest -q ``` ``` 307 passed in 2.34s ``` ### Step 8 — (Optional) Scheduled run via Prefect ```bash .venv/bin/python -m jobsource.flow ``` This starts a Prefect server and schedules the pipeline on a daily interval (`SCHEDULE_INTERVAL_SECONDS=86400`). The Prefect UI is at `http://127.0.0.1:4200`. ### Step 9 — (Optional) Detailed stage-by-stage trace ```bash .venv/bin/python scripts/e2e_smoke.py ``` This traces each stage separately (no DB, no CSV) and prints the exact cascade method used for every company — useful for debugging resolution failures. --- ## Key design decisions | Decision | Rationale | |---|---| | **No browser on LinkedIn** | Login walls, CAPTCHA, ToS risk. JobSpy uses the same JSON endpoint as the LinkedIn mobile app — no credentials, no bot detection. | | **Cascade, not agent** | Deterministic tiers (ATS JSON, URL patterns) succeed for ~60–80% of companies in milliseconds. LLM and browser agent only fire for the long tail. Total cost for the cheap tiers: near-zero. | | **ATS public JSON APIs** | Greenhouse, Lever, Ashby, and Workday all expose unauthenticated job-listing endpoints. Calling them directly is faster and more reliable than scraping HTML. | | **Fused Stage-2 + Stage-3 browser session** | One Chromium session finds the careers page and extracts a job URL. Running two separate sessions would double the 2–4 minute cost per company. | | **Graceful degradation** | Every tier wraps in try/except. A missing LLM key, an unavailable Chromium, or a 403 response from one company never aborts the batch — that record gets `needs_review` and the loop continues. | | **Dedup on `job_id`** | LinkedIn's numeric job posting ID is stable. Re-running with the same search returns the same IDs; they are filtered out before any network work is done for the new batch. | | **Company-level caching** | Once a careers URL is found for `acme.com`, all future jobs from that domain skip the entire cascade. Avoids redundant work for companies with many open listings. | --- ## File map (quick reference) ``` jobsource/ config.py env-driven settings (pydantic-settings) models.py RawJob, JobResult, JobStatus, CSV_COLUMNS http.py shared httpx client factory db.py SQLite: companies + jobs; dedup; CSV export resolve.py company name → website URL (Stage 1b) sources/ base.py JobSource ABC jobspy_source.py default LinkedIn provider (python-jobspy) apify_source.py alternative provider (Apify actor) careers/ cascade.py find_careers_page() — orchestrates 6 tiers ats.py ATS HTML detect + 4 public JSON APIs + slug-guess heuristics.py URL patterns, homepage scan, sitemap classify_llm.py Pydantic AI link classifier (careers + job links) extract.py extract_open_position() — Stage 3 agent_fallback.py Browser Use fused Stage-2/3 fallback pipeline.py run_batch() — dedup, per-record isolation, summary flow.py Prefect flow + interval schedule main.py CLI entry point tests/ 307 offline pytest tests scripts/ e2e_smoke.py live stage-by-stage trace (no DB) validate.py live PASS/FAIL validation against success criteria output/ results.csv pipeline output (gitignored) jobsource.db SQLite state (gitignored) ```