phase6-Prefect

This commit is contained in:
ldy
2026-06-18 00:32:26 -04:00
parent 91cf930a5f
commit 11f10affbd
5 changed files with 192 additions and 18 deletions

View File

@@ -20,18 +20,49 @@ cp .env.example .env # fill keys as available
## Run
```bash
# one batch
# one batch (no server required)
python -m jobsource.main --batch-size 20 --search "software engineer" --location "United States"
# scheduled run (Prefect)
python -m jobsource.flow
# cron fallback (no daemon):
# */0 6 * * * cd <repo> && ./.venv/bin/python -m jobsource.main --batch-size 50
```
`--search` is repeatable. Run `python -m jobsource.main --help` for all options.
### Scheduled run (Prefect)
```bash
# Boots a local Prefect server automatically, then serves the flow on an
# interval schedule (default: daily, set SCHEDULE_INTERVAL_SECONDS to change).
python -m jobsource.flow
```
If you already have a Prefect server running elsewhere, point the flow at it:
```bash
export PREFECT_API_URL=http://127.0.0.1:4200/api
python -m jobsource.flow
```
To trigger a run manually while the flow is being served:
```bash
prefect deployment run 'job-source-batch/job-source-batch'
```
> **Note — Prefect 3.7.4 + FastAPI ≥ 0.137 compatibility**: Prefect's built-in
> ephemeral server is incompatible with FastAPI 0.137+ (route lists are cleared
> after inclusion but re-read lazily during routing). `python -m jobsource.flow`
> works around this by starting a persistent `prefect server start` process
> instead of the ephemeral server.
### No-daemon cron fallback
If you prefer a plain cron job with no running process, add this line to your
crontab (`crontab -e`), adjusting the path to your repo:
```
# daily at 06:00 — no Prefect daemon required
0 6 * * * cd /path/to/repo && ./.venv/bin/python -m jobsource.main --batch-size 50
```
## Tests
```bash