Job Search Pipeline – Local-First Discovery and Scoring

Challenge

Job hunting for a hybrid design-and-engineering role is a search problem with a brutal decay rate. The listings worth applying to are scattered across dozens of applicant tracking systems that never appear in one place, and the best of them close fast — one senior design engineering role I was tracking closed within hours of my confirming it was open. Aggregator sites lag the source boards by days, which is precisely the window that matters.

The obvious solutions all had a catch. Off-the-shelf automation platforms wanted a monthly subscription for what amounts to a cron job. Scraping services charged per call for data that employers already publish for free. And the tools that promised to auto-apply on my behalf were solving the wrong problem loudly — volume is not the bottleneck when you have no referral network; relevance is.

Solution

I built the pipeline as a set of small, boring, local programs. The fetcher is standard-library Python with no dependencies at all, which means it has no install step, no version drift, and nothing that can break on a machine I have not touched in a month.

• Collect from the source, not the aggregator. Ashby, Greenhouse, and Lever all expose public JSON job-board endpoints. Roughly 35 company boards are polled directly, alongside RSS feeds and four keyless JSON aggregators, so the copy is the employer's own and it is fresher than any repost.
• Dedupe into a store that doubles as the tracker. Every posting ever seen lands in one SQLite table keyed by a stable source identifier. The same table carries application status, so the deduplication layer and the "where did I apply" spreadsheet are the same object and can never disagree.
• Separate matching from judgment. Cheap regex lane-matching runs first and throws out the obvious misses across five parallel search lanes. Only survivors reach the scoring pass, which reads a plain-Markdown rubric — reward and downrank rules, salary floors, hard constraints — and produces a 0–100 score with a status.
• Make the rubric a file, not code. Scoring criteria live in a Markdown document I edit in a text editor. Change the file, and the next run scores differently. No redeploy, no code change, no arguing with a config schema.
• Fail loudly per source, never globally. Each board fetch is individually wrapped; a renamed company slug or a flaky feed records an error and the run continues. Scraper-based sources sit in a separate program entirely so their inevitable flakiness can never block the clean API fetch.

A scheduled task runs the whole chain each morning and reports back with anything that scored high enough to act on today.

Decisions Worth Defending

Most of what makes this system work is what it refuses to do.

• It never submits an application. This was the first requirement and it is not a limitation. The system finds, scores, and drafts; a human reads and sends. Auto-apply optimizes the one metric that does not matter and risks the reputation attached to my name.
• It does not scrape LinkedIn while logged in. The largest commercial LinkedIn data API was sued and permanently shut down under a settlement requiring deletion of its scraped data. A logged-in scraper also risks restriction of the single account a job search most depends on. The sanctioned path — parsing the job-alert emails LinkedIn already sends you — gets the same matching for none of the risk.
• The email ingest is strictly read-only. It fetches with BODY.PEEK and tracks progress with a local UID cursor rather than marking mail read, so running it twice is a no-op and the inbox looks untouched afterward. Alert links also carry per-send tracking parameters, so URLs are canonicalized down to their job ID before hashing — without that step, the same job inserts a fresh row every single day and deduplication silently fails.
• An always-on workflow server was removed, not added. The original design used a self-hosted automation platform as the chassis. It turned out to have never actually run, and its one unique capability — IMAP polling — was about forty lines of standard-library Python. Deleting the dependency removed an entire class of failure where the server is simply not up.

Outcome

The pipeline runs daily and has been in continuous use since August 2026.

• Roughly 4,900 postings scanned per run across ~35 boards and nine source types
• 384 roles tracked in the store, triaged into apply-now, surfaced, and excluded
• Five parallel lanes, so a single run serves several different job-title strategies at once
• Zero recurring cost — no subscriptions, no per-call scraping fees, no hosted services

The part I did not expect to matter most was the rubric being a text file. Treating the scoring criteria as an editable document rather than application logic turned the system into something I could argue with. When the results skewed wrong, I edited a sentence instead of a function, and the next morning's digest reflected it.

The same resume-as-data idea runs alongside it: my CV lives as a YAML document that renders to a typeset one-page PDF, so tailoring for a specific role is a diff rather than a fight with a word processor.