From Script to System: Building a Career Data Pipeline I Can Trust

Each successful worker execution produces a verifiable receipt. That lets the system distinguish between a scheduled service, an actual execution, a successful run, a stale worker, a failed run, and a healthy run that found nothing new. The monitoring layer derives health from those receipts rather than assuming a running timer means everything downstream succeeded.

During validation, that distinction mattered. A worker was classified as STALE when its latest verified receipt aged past the allowed window. After a legitimate new execution produced a fresh receipt, the next rollup returned it to HEALTHY. The status changed because the evidence changed.

A discovery system sees the same opportunities repeatedly, so blindly appending every result would quickly destroy the usefulness of the dataset. The workers therefore normalize and deduplicate before writing. In one production replay, all 25 previously processed records were recognized as duplicates and zero additional rows were written.

That is not a run that accomplished nothing. The system checked its inputs, compared them with authoritative state, determined that nothing was new, and left the dataset unchanged. Idempotency became one of the most useful tests in the project.

Production cutover exposed a checkpoint update that targeted the wrong spreadsheet row. Rather than accepting the inconsistent state, the workflow was returned to the previous verified checkpoint, the defect was corrected, and the cycle was rerun. The search rotation advanced only after a successful receipt confirmed the corrected execution.

That failure changed how I think about automation: state should advance because work was verified, not simply because code attempted the work.

Once the workers began operating against live sources, the system successfully added new opportunities to the data layer. Field observation also exposed classification cases where ordinary employment postings could enter a freelance-oriented path. The transport and persistence worked, but the semantics still needed refinement.

That is now treated as patchwork rather than a reason to redesign the architecture: run → observe → measure → identify weakness → patch → validate → continue running.

The system can discover opportunities, score fit, connect requirements to portfolio evidence, identify gaps, and prepare application materials. It is intentionally not allowed to submit an application on its own.

That boundary has already proven useful. When an opportunity was approved for package preparation, the workflow still refused to proceed because its underlying readiness state was invalid. Investigation exposed a spreadsheet formula spill problem. After the affected range was repaired, the records rebuilt correctly, but the opportunity still had to satisfy its remaining prerequisites. Human approval did not give the software permission to misrepresent its own state.