AI Job Source Agent

For recently posted LinkedIn jobs, produces records of the form:

company_name, career_page_url, open_position_url

Runs in configurable batches, on a schedule, and is incremental — re-runs process only new jobs.

Setup

python -m venv .venv && source .venv/bin/activate
uv pip install -r requirements.txt   # uv resolves pydantic deps reliably; pip also works
playwright install chromium           # for the browser-agent tier
cp .env.example .env                  # fill keys as available

Run

# one batch (no server required)
python -m jobsource.main --batch-size 20 --search "software engineer" --location "United States"

--search is repeatable. Run python -m jobsource.main --help for all options.

Scheduled run (Prefect)

# Boots a local Prefect server automatically, then serves the flow on an
# interval schedule (default: daily, set SCHEDULE_INTERVAL_SECONDS to change).
python -m jobsource.flow

If you already have a Prefect server running elsewhere, point the flow at it:

export PREFECT_API_URL=http://127.0.0.1:4200/api
python -m jobsource.flow

To trigger a run manually while the flow is being served:

prefect deployment run 'job-source-batch/job-source-batch'

Note — Prefect 3.7.4 + FastAPI ≥ 0.137 compatibility: Prefect's built-in ephemeral server is incompatible with FastAPI 0.137+ (route lists are cleared after inclusion but re-read lazily during routing). python -m jobsource.flow works around this by starting a persistent prefect server start process instead of the ephemeral server.

No-daemon cron fallback

If you prefer a plain cron job with no running process, add this line to your crontab (crontab -e), adjusting the path to your repo:

# daily at 06:00 — no Prefect daemon required
0 6 * * * cd /path/to/repo && ./.venv/bin/python -m jobsource.main --batch-size 50

Tests

.venv/bin/pytest -q

307 offline unit tests covering job-id parsing, ATS detection and fetching, dedup logic, website resolver, heuristics, LLM classify, extract tiers, agent-fallback gating, and a fully mocked run_batch end-to-end test.

Validate

validate.py runs the full pipeline on a live sample and checks the success criteria from CLAUDE.md:

.venv/bin/python validate.py                          # default: 5 jobs, "software engineer"
.venv/bin/python validate.py --batch-size 10 --search "data engineer"

Automated checks (printed as PASS/FAIL):

#	Criterion
1	Batch completes without unhandled exception
2	`output/results.csv` has exactly the three contract columns (`company_name`, `career_page_url`, `open_position_url`)
3	Per-stage summary returned with coverage metric
4	Re-running the same search with the same DB processes 0 new jobs (dedup proven)

Manual spot-check (done by eye after validate.py passes):

Open output/results.csv.
Pick one complete row (open_position_url non-empty).
Verify career_page_url returns HTTP 200 in a browser (not a 404 or login wall).
Verify open_position_url returns HTTP 200 in a browser.

Output

output/results.csv — three columns: company_name, career_page_url, open_position_url. Complete rows (status position_found) are sorted first.

3.3 KiB Raw Blame History