BRAEDEN KEENA

A client came to Upwork needing help pulling together records for a legal proceeding. What the case actually needed was full-scale data engineering: 7,626 messages and records were scattered across five incompatible export formats — Gmail, iPhone SMS, WhatsApp, Facebook Messenger, and a webmail account — each with its own timestamp convention, its own missing fields, and no shared identifier to link them.

The pipeline parses each source’s native export format, then normalizes every timestamp to a single timezone with correct daylight-saving handling — a naive UTC conversion silently shifts events by an hour on either side of a DST boundary, which is disqualifying when timing is part of the record. Duplicate and fragmentary records are reconciled across sources into one chronological timeline, and every surviving record carries a provenance tag back to its original source file, so any line in the final output can be traced to exactly where it came from.

The deliverable was a formatted PDF report generated automatically from the reconciled dataset, plus a dedicated QA audit pass — a second, independent check of the pipeline’s output against the source exports before delivery — because a consolidation this size is only useful if it’s also correct.

A static public-records intelligence dashboard and daily extraction pipeline built on DeKalb County delinquent tax listings.

County tax listings are published as a monolithic 9,915-page fixed-width mainframe PDF file (DQ205GADEK). The python extraction engine ingests the document offline line-by-line, detecting and auto-correcting 32 column-overflow shifts via a 5-column checksum verification algorithm. The pipeline enforces 100% line accounting — zero dropped rows, zero malformed records — backed by 90 automated offline test fixtures.

The web interface presents aggregate financial insights without publishing personal PII:

• Tax Year Trend Explorer: Multi-metric trend analysis showing how 56.9% ($35.7M) of outstanding debt sits in the 2025 cycle, while 26,000+ 2018 records remain open as long-tail “zombie properties”.
• Pareto Economic Breakdown: Highlighting how 77.8% of rows are micro-bills under $100, whereas just 1,023 bills (> $5,000) hold over $11.3M in debt.
• Interactive Policy Collection Simulator: Live threshold tool allowing policy researchers to test collection targets ($100, $500, $1,000, $5,000+) to calculate recovered revenue ($M), percentage of county debt cleared, and owner contact efficiency.
• Georgia Tax Status Classification: Clear breakdown detailing Fi.Fa. Lien Execution (F), School Tax Delinquency (S), Assessment Appeals (A), and Claim Transfers (T).

A federal procurement monitoring engine modeled on defense logistics (DLA/DIBBS) RFQ tracking. GovScout ingests live solicitations from the SAM.gov Opportunities API v2, filters them by FSC/PSC codes, detects pricing-availability signals, and extracts structured part data into SQLite.

• Pricing Signal Classifier: Evaluates solicitation copy against weighted keyword families (online_pricing, historical_pricing, quote_workflow, competitive) to generate a 0–100 pricing feasibility score.
• Structured Part Extractor: Automatically parses National Stock Numbers (NSN \b\d{4}-\d{2}-\d{3}-\d{4}\b), manufacturer part numbers (P/N), and required item quantities from raw text.
• Dual Mode Architecture: Operates in Live Mode (REST API client with 0.5s rate-limiting, 429 backoff, and state persistence) and Demo Mode (bundled offline dataset for zero-network testing).
• SQLite Engine & Report Generator: Upserts normalized solicitations with duplicate detection and exports CSV feeds and console digests grouped by signal priority.

Businesses cited by a health inspector for a broken or dirty ice machine have a dated, documented problem with a specific piece of equipment. Those citations are public record. Turning them into a ranked call list is a day of work. Making sure the list is right took the rest of the time.

Two publishers, two entirely different problems

Santa Clara County publishes on Socrata — a real JSON API, three tables joined client-side, about thirty requests to cover 9,711 businesses.

Washington DC publishes through Tyler’s Digital Health Department platform, which has no API at all. One GET returns one server-rendered inspection report, and all four query parameters are required, including an undocumented tenant id. Inspection ids are sequential integers. None of it is written down anywhere; it was reverse-engineered.

Both feed one deduplicated, scored list, and every lead carries the market it came from.

The join key is misspelled, and correcting it breaks everything silently

Santa Clara’s inspections table names its join key inpsection_id — p and s transposed. The violations table spells the same key correctly. Both are right; they disagree with each other, because Socrata derives field names from the uploaded column header and the typo is baked into the dataset.

Correcting the spelling doesn’t raise an error. It produces an empty lead list, which looks exactly like a quiet week. The constant carries a large comment and is pinned by a test named test_join_key_typo_is_deliberate, so a future tidy-up has to argue with a failing test rather than with nothing.

DC has the same shape of trap from the other direction: requesting an inspection id that was never issued returns HTTP 200 with a normal-looking page. The only usable “this record doesn’t exist” test is a string check on the body.

Florida: researched in full, then not built

Florida’s mechanism was worked out completely — the free bulk extract, the per-district partitioning, the licensure file that joins phone numbers, all thirty-five of its columns enumerated to confirm what it does and does not carry. It carries no email address.

No adapter was written. The finding was that Florida is better served by a public records request than by scraping, and the document explaining why is in the repository next to the adapters that were built. Deciding not to build something is a result, and it is only useful if it is written down where the next person will find it.

Rate limiting, and correcting my own first version

Version one used random.uniform(1.0, 2.0) between requests and described this in its own comments as “human-like delays” with “zero bot-detection footprint.”

That posture is wrong for a documented public API that publishes a crawl delay. The rewrite honours the published Crawl-delay: 1, sends a descriptive User-Agent with a contact address rather than a browser string, honours Retry-After on 429, backs off exponentially on 5xx, and caps the entire run at 400 requests so a bug cannot become a hammering. Both jurisdictions share one budget, so adding a second market did not silently double the traffic.

The correct behaviour on a public API is to identify yourself and stay inside the published limit, not to look like a person.

What the source doesn’t have, reported as its own output

Every run writes a coverage report and a request trace: what was retrieved, what failed, what was deliberately skipped, and which fields the source simply does not populate. An empty result has to distinguish no data exists from the request failed from I never asked — three things that look identical in a lead list and mean completely different things.

Two more that are stated because leaving them out would mislead. Santa Clara’s data carries no explicit open licence — Delaware’s equivalent is marked public domain and San Francisco’s carries PDDL, and Santa Clara’s metadata carries neither. Access is permitted and the records are public, but there is no licence grant to point at. And the dataset goes back exactly two years and no further, because it was created in August 2024 — there are no older records being filtered out, the source does not have any.

Two fields where one would have been easier

jurisdiction is the publisher’s boundary and the deduplication key. market is the sales territory and the label a human reads. Collapsing them into one field would either put a county name on someone’s call list or use a marketing label as a data key.

The first version exported everything as “San Jose.” It is a county dataset covering twenty-three cities.

The daily run refuses to send stale mail

The emailer reads a CSV. It has no way of knowing whether that file was written thirty seconds ago or last Tuesday. On one run the collector found 246 leads and wrote none of them, because the file was open in a spreadsheet and locked — the pipeline reported that correctly, but an emailer running blind straight afterwards would have sent the previous list with the current date on it.

So the wrapper sends only if the file was written by that run, and the digest covers net-new leads only. On a day with nothing new it stays silent, which is the honest output and the one that keeps the mail worth opening.

This began as a store-ID finder. A five-digit site number that the team I was on needed daily wasn’t published anywhere, so I built the thing that found them — and the market data came along for free. It grew into a national directory of advertised self-storage prices and the research that falls out of tracking them over time.

A seven-pass discovery scraper collected publicly published rates for 55,332 advertised price points across 4,664 stores in 43 states, running itself daily through GitHub Actions with commits landing unattended. The pipeline had safety rails: a run aborted rather than publishing if store count dropped below a floor or fell more than 10% against the previous run. It took 48 consecutive daily snapshots with no missed days, and the rate-change log holds 396,402 advertised price changes and 148,870 promotion changes. When a large operator merger added roughly 1,100 stores overnight — a ~30% jump — the pipeline absorbed it cleanly and surfaced the shift in the trend data instead of choking on it.

Collection ended 25 August 2026. Before adding a second operator to the dataset I sat down to read their terms of use, which raised the obvious question of when I had last read the first operator’s. Theirs prohibits automated collection. I stopped the same night, disabled the scheduled job, and the site now stands as a dated archive rather than a live directory. The engineering below is what the project was; none of it depends on the collection continuing.

What 48 days of daily snapshots turned up

Advertised rates do not drift. They sit completely still for days and then move in coordinated waves. On 18 of 46 days not a single tracked rate changed; on thirteen days, tens of thousands did. Nothing observed falls between those two states — the busiest quiet day logged 2,443 changes and the smallest wave logged 12,792.

The largest wave repriced 88.7% of tracked inventory in one day. Individual stores move decisively and in opposite directions within the same wave, at three to five times the rate a shuffled null predicts, so the national median can sit flat while most of the book moves under it.

One 5×5 unit shows the mechanism in seven rows. Its advertised list rate changed eight times in six weeks while the promotional badge on the page never changed. Between 20 and 22 August the advertised saving grew from $123 to $176 — and the first month went from $46 to $66. The discount is a fixed percentage; the number it is a percentage of is not.

The analysis I care most about is a matched-pairs design. Averaging prices confounds the attribute with the market — climate-controlled units are more common in expensive metros, so a naive average credits the feature for the city. Instead it compares units of the same size, at the same store, differing in exactly one attribute (climate control, floor level, drive-up), so location and local demand cancel out and what’s left is closer to the price of the attribute itself. Climate control carries roughly an 18–22% premium on that basis. Records are linked across sources by 150m haversine distance plus ≥60% name-token overlap.

Everything is cut by size (locker through 10x30 and parking). 414 ZIP3 markets are computed nationwide rather than hand-picked, each with a client-side drill-down: store count, units per store, promo share, median price by size with price-per-sqft against the national figure, trend sparklines, and a renter leverage score. Movers tables surface the largest availability drops and restocks and the largest 10x10 price moves over a rolling window, and a per-store price-history view charts a single store’s advertised prices over time by unit size. Moving the detail data into a sidecar fetched on first interaction kept the whole drill-down to +3.6KB gzip on initial load.

How the numbers were checked

Every failure below was found in my own data, by me, and every one of them had been quietly wrong for some length of time before it was caught. They are on the page because the corrections are the part worth reading.

A number with no command behind it decays. A hand-typed coverage figure turned out to be wrong by 5.6×, and had been wrong for an unknown period — nothing in the pipeline could say how long, because nothing had ever recomputed it. Every published figure is now emitted by a committed script on each run, and the script refuses to write its output file if any value comes back null. A stale number never drifts randomly; it drifts in the direction that flatters the page.

Counts and statistics have to describe the same rows. 56 rows in the rate log recorded a price moving from zero — a unit arriving in the dataset, not a repricing. They were excluded from the medians but still counted in the daily totals, so two numbers on the same page were computed over two different populations. The filter now runs before the tally rather than after it.

Derive the window from the thing that defines it. The observation window was originally read off the change log, which meant days with no changes weren’t days at all. It reported 41 days and 13 zero-change days. Taken from the snapshot dates instead — the actual record of when the scraper ran — the real figures were 44 and 16. Three days had vanished because nothing happened on them.

A step in the data is not a movement in the market. Twice the tracked population changed size sharply: 441 stores entered on 9 July and 1,096 on 23 July, the second being a merger. Averages across those dates compare two different populations. The breaks are detected from the store counts rather than asserted from the news, and every chart that crosses one draws it as a labelled rule — because the misreading being prevented is someone glancing at the shape of a line.

Zero is not the same as nothing. 9,164 of 34,416 store-size listings are quarantined from the movers tables, each with a named reason recorded to a CSV. The one that matters most: a store whose advertised availability drops to exactly zero and stays there has stopped publishing availability, not rented out its entire inventory. Ranked naively it would be the largest mover on the page and the least likely to mean what it says. The same logic runs in reverse for stores that begin publishing.

Test the pattern against a null. The claim that individual stores move coherently — decisively, and in opposite directions inside the same wave — is measured against a shuffled null rather than eyeballed off a chart. Stores move together at three to five times the rate the shuffle predicts.

Fail loudly, in the right direction. A run aborts rather than publishes if the store count drops below a floor or falls more than 10% against the previous run, so a source changing its markup surfaces as a stopped pipeline the same day instead of as quietly missing rows. The failure mode that costs you is the one that still produces plausible output.

Worth stating plainly: these are advertised online rates, not what any given renter ends up paying. Coverage is whatever the discovery pass found published, so it is a sample rather than a census, and matched pairs control for store and size — not for everything. The dataset is closed as of 25 August 2026 and will not be extended.

Built because I wanted one place for everything World Cup 2026 — every morsel of news, plus which bars and fan events were happening near me. It became my own daily driver during the tournament, and drew around 1,500 unique visitors a month while the World Cup was on.

News is aggregated from 16+ feeds and ranked by a time-decay scoring formula that weighs source credibility, team and player entity matches, and whether a team is playing that day — with near-duplicate coverage clustered instead of repeated. The venue map merges Yelp, Foursquare, OpenStreetMap and Google Places into single records using geographic proximity and name-token similarity, each carrying a confidence score based on how many independent sources agree.

A daily GitHub Actions pipeline regenerates the site — 48 team pages, 99 match pages, 38 city pages and a 187-URL sitemap — for programmatic SEO across every team, match, and host city.

The publisher’s existing charts were the specification. One of their published files came along as a reference, and the job was to produce output that matched it — the same pinned Plotly build already loaded by their CMS, the same two stacked panels, the same colours and marker shapes, the same $ prefix on the sales axis.

Sales arrive in the spreadsheet stated in millions and appear in the published charts as whole dollars. Getting that exactly right meant reproducing 97885.166233 as 97885000000 — rounding to the nearest million, then multiplying — rather than something a rounding difference away from it.

The question that came up mid-job

381 of the 428 companies have all five years. 47 do not — 27 are missing 2021, 20 are missing 2021 and 2022. The client raised it partway through: chart those companies over the years they actually have, with the title adjusted.

The easy thing is to plot a missing year as zero. That draws a chart where revenue collapses to the floor and climbs back out, which is a picture of a company in crisis rather than a company that wasn’t in the ranking yet. A year a company did not report is not a year it sold nothing.

So blank years are never plotted as zero. Leading and trailing blanks are trimmed away and the chart is titled over the range it actually covers — Sales & Units 2022–2025 for a four-year company, with no empty stretch at the left.

A blank in the middle of a run is different: it cannot be trimmed away without closing a gap that exists, and joining 2022 straight to 2024 would imply a continuity that isn’t there. Those stay as a break in the line. No company in this year’s sheet has one — the code handles it anyway, because next year’s might.

Written to be run by someone else

The handover assumes no Python and no command line. Install it, install two libraries once, drop next year’s spreadsheet in the folder, run one command. When the years roll forward, one line changes and the chart titles update themselves.

The script reads columns by header text rather than by position, so a reordered sheet doesn’t silently produce wrong charts. If an expected header is missing it stops and names the column — the alternative being 428 files that look fine and aren’t.

Six company names contain characters Windows will not accept in a filename — RE/MAX, Checkers/Rally's, The UPS Store/Mail Boxes Etc. among them. Those are substituted in the filename only. The chart title and the underlying data are left exactly as the publisher wrote them.

How the work was won

The posting included the spreadsheet and one sample of the output they wanted. Instead of describing an approach, I spent thirty minutes building three real charts from their own data and attached the files to the proposal.

He replied inside a minute, and said it was the only proposal he had opened out of the fifteen or twenty he’d received. Three files out of four hundred and twenty-eight is small enough to give away and specific enough to prove the rest exists.