106 lines
3.3 KiB
Markdown
106 lines
3.3 KiB
Markdown
# AI Job Source Agent
|
|
|
|
For recently posted LinkedIn jobs, produces records of the form:
|
|
|
|
```
|
|
company_name, career_page_url, open_position_url
|
|
```
|
|
|
|
Runs in configurable batches, on a schedule, and is incremental — re-runs process only new jobs.
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
python -m venv .venv && source .venv/bin/activate
|
|
uv pip install -r requirements.txt # uv resolves pydantic deps reliably; pip also works
|
|
playwright install chromium # for the browser-agent tier
|
|
cp .env.example .env # fill keys as available
|
|
```
|
|
|
|
## Run
|
|
|
|
```bash
|
|
# one batch (no server required)
|
|
python -m jobsource.main --batch-size 20 --search "software engineer" --location "United States"
|
|
```
|
|
|
|
`--search` is repeatable. Run `python -m jobsource.main --help` for all options.
|
|
|
|
### Scheduled run (Prefect)
|
|
|
|
```bash
|
|
# Boots a local Prefect server automatically, then serves the flow on an
|
|
# interval schedule (default: daily, set SCHEDULE_INTERVAL_SECONDS to change).
|
|
python -m jobsource.flow
|
|
```
|
|
|
|
If you already have a Prefect server running elsewhere, point the flow at it:
|
|
|
|
```bash
|
|
export PREFECT_API_URL=http://127.0.0.1:4200/api
|
|
python -m jobsource.flow
|
|
```
|
|
|
|
To trigger a run manually while the flow is being served:
|
|
|
|
```bash
|
|
prefect deployment run 'job-source-batch/job-source-batch'
|
|
```
|
|
|
|
> **Note — Prefect 3.7.4 + FastAPI ≥ 0.137 compatibility**: Prefect's built-in
|
|
> ephemeral server is incompatible with FastAPI 0.137+ (route lists are cleared
|
|
> after inclusion but re-read lazily during routing). `python -m jobsource.flow`
|
|
> works around this by starting a persistent `prefect server start` process
|
|
> instead of the ephemeral server.
|
|
|
|
### No-daemon cron fallback
|
|
|
|
If you prefer a plain cron job with no running process, add this line to your
|
|
crontab (`crontab -e`), adjusting the path to your repo:
|
|
|
|
```
|
|
# daily at 06:00 — no Prefect daemon required
|
|
0 6 * * * cd /path/to/repo && ./.venv/bin/python -m jobsource.main --batch-size 50
|
|
```
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
.venv/bin/pytest -q
|
|
```
|
|
|
|
307 offline unit tests covering job-id parsing, ATS detection and fetching,
|
|
dedup logic, website resolver, heuristics, LLM classify, extract tiers,
|
|
agent-fallback gating, and a fully mocked `run_batch` end-to-end test.
|
|
|
|
## Validate
|
|
|
|
`validate.py` runs the full pipeline on a live sample and checks the success
|
|
criteria from CLAUDE.md:
|
|
|
|
```bash
|
|
.venv/bin/python validate.py # default: 5 jobs, "software engineer"
|
|
.venv/bin/python validate.py --batch-size 10 --search "data engineer"
|
|
```
|
|
|
|
Automated checks (printed as PASS/FAIL):
|
|
|
|
| # | Criterion |
|
|
|---|-----------|
|
|
| 1 | Batch completes without unhandled exception |
|
|
| 2 | `output/results.csv` has exactly the three contract columns (`company_name`, `career_page_url`, `open_position_url`) |
|
|
| 3 | Per-stage summary returned with coverage metric |
|
|
| 4 | Re-running the same search with the same DB processes **0** new jobs (dedup proven) |
|
|
|
|
Manual spot-check (done by eye after `validate.py` passes):
|
|
|
|
- Open `output/results.csv`.
|
|
- Pick one complete row (`open_position_url` non-empty).
|
|
- Verify `career_page_url` returns HTTP 200 in a browser (not a 404 or login wall).
|
|
- Verify `open_position_url` returns HTTP 200 in a browser.
|
|
|
|
## Output
|
|
|
|
`output/results.csv` — three columns: `company_name`, `career_page_url`, `open_position_url`.
|
|
Complete rows (status `position_found`) are sorted first.
|