Technical Investigation · Civic Data Integrity
The Verification Filter.
Anyone can scrape 50,000 rows of garbage. Filtering it down to four prospects that each carry their own documented gaps is where the real engineering happens.
Two Numbers, Not One Funnel
The engine and the audit are different measurements and collapsing them misrepresents both.The engine ingests and resolves everything the feeds publish. The audit takes one batch of that output and refuses most of it. The second number is small on purpose.
1. The Naive Myth vs. The Engineering Reality
A common premise in sales intelligence sounds seductive: "Just write a python script to scrape building permits for Certificates of Occupancy and call them before competitors know they exist."
"1,200 New Commercial Leads Every Month!"
Scrapes every row with the word "Commercial", grabs the phone number off the PDF, and floods the sales team with 500 dial targets a week.
- Calls electrical subcontractors who installed lighting
- Pitches dry cleaners when the permit was for the dental office next door
- Wastes hundreds of calls on speculative landlord "white-boxes" with no tenant
"4 Prospects, Each With Its Documented Gaps"
Separates property owners from tenant operators, isolates multi-suite strip mall addresses, and refuses to declare a commercial buyer without license corroboration.
- Generates Secretary of State lookup URLs so entity identity is resolved by hand rather than guessed
- Disambiguates individual retail suites sharing a single street parcel
- Hands sales reps actionable leads with explicit mandatory limitations
2. Tyler EnerGov: Reverse-Engineering Multi-Tenant Municipalities
Across the Kansas City metropolitan area, municipal data infrastructure ranges from open civic transparency to hostile vendor firewalls. Even when neighboring cities buy the exact same off-the-shelf software suite—Tyler Technologies EnerGov Self-Service—the operational realities vary drastically. Some deployments served automated queries without friction; a vendor-cloud instance served one full collection and then blocked every refresh after it.
While Tyler EnerGov portals share an underlying JSON search endpoint, the API refuses all connections without four mandatory, undocumented request headers:
tenantid, tenantname, tyler-tenanturl, tyler-tenant-cultureThese values cannot be derived mathematically; they must be verified via actual network inspection per deployment. Across the verified live deployments feeding our pipeline, the API enforced completely distinct naming conventions and divergent module architectures:
| Municipality | Tenant ID | Tenant Name Header | Target Module | Architectural Discovery |
|---|---|---|---|---|
| Olathe, KS | 1 | Olathe, KS | Permits | Display name with punctuation. Commercial activity resided in Permits (Plans held lot splits). |
| Overland Park, KS | 1 | OverlandParkKSProd | Plans | CamelCase slug. Permits was 92% Right-of-Way with zero COs across 5,000 records; commercial COs lived in Plans. |
| Kansas City, MO (CompassKC) | 7 | KansasCityMOProd | Plans | Tenant ID 7 (not 1). Breaks assumption of uniform IDs across identical vendor installs. |
Olathe, OP, KCMO
Deployed on municipal infrastructure. Once the undocumented headers were mapped, these servers accepted an identifying User-Agent (KCPermitFeed/1.0) and returned thousands of records without rate limiting or blocking. Not being blocked is a server behaviour, not a grant of permission — terms were read separately.
Leawood, KS
Vendor-hosted on Tyler's central cloud (tylerhost.net), so the path is /apps/selfservice rather than the city-hosted /energov_prod/selfservice. One complete collection succeeded on 26 August — 3,086 permits and 1,618 licences. Every refresh since has been blocked, so that evidence is now stale and the city sits outside the live set.
Independence, MO
No vendor portal to reverse-engineer. Independence publishes a direct, unauthenticated Open Data REST API (apps.independencemo.gov) that ingested cleanly — no undocumented headers, no blocking.
3. Where the Work Actually Went: The Forensic Failure Ledger
The most dangerous failures in civic data pipelines do not throw HTTP 500 crashes; they succeed politely while silently corrupting downstream business decisions. Here are four critical failure modes caught and engineered around in this pipeline:
The Silent 10,000-Record Cap
The Elasticsearch behind these portals refuses past 10,000 results — from + size must stay under index.max_result_window. A query matching more than that returns HTTP 200 with 10,000 rows and zero warning flag. The overflow is not reported, not flagged, and not distinguishable from a complete answer.
10000 / page_size — and refuses to request past it, reporting the cap instead of returning a truncated page set that looks complete.The Date Filter That Does Nothing
A portal accepted date-range criteria in its JSON search payload and ignored them server-side. Measured, not assumed: a 90-day window returned the same 124,214 rows as no window at all — the filter is decorative, and a caller who trusts it believes they pulled a quarter when they pulled the lifetime database.
Strip Mall Multi-Suite Collisions
In suburban retail corridors, a single parcel address can carry many separately tenanted suites. Naive scrapers merge all permits into one master entity, claiming an existing tenant just signed a new lease.
Speculative Landlord "Ghost Shells"
Property owners routinely pull commercial alteration permits for "white-box" shell renovations with no signed tenant. Treating a commercial remodel as an active business leads sales teams to pitch empty buildings.
4. The Reconciliation Audit: 37 → 10 → 4
In a production deliverable, trust requires accounting for every input record. Below is the exact reconciliation panel and status classification from our public pilot evaluation:
Accounting Assertion: 4 (prospect) + 4 (review) + 2 (excluded) = 10 distinct projects (10 = 10).
Multi-structure parcels (e.g. apartment subdivisions) and multi-phase commercial finishes consolidate into unified facility identities.
The Four Evaluated Prospects & Mandatory Limitations
Notice what our pipeline asserts: these are designated as prospects, never hyped as guaranteed buyers. Each opportunity documents the dated civic event alongside its explicit verification limitations:
Caravel Autism Health
11140 Locust St, Kansas City, MO 64131
Mandatory Limitation: Facility grand opening confirmed; cleaning demand and vendor procurement authority unverified.
Midwest Surgery Center of Kansas City LLC
6750 W 93rd St, Ste 120, Overland Park, KS 66212
Mandatory Limitation: Role is recorded in a public federal source; personal purchasing authority is unverified, as is the date of patient reception.
Scovell Office Addition & Renovation
8035 Ward Parkway Plaza, Kansas City, MO 64114
Mandatory Limitation: Certificate supplies a dated reason to investigate; does not establish an unfilled cleaning contract.
Pho & Boil
7702 Shawnee Mission Pkwy, Overland Park, KS
Mandatory Limitation: Planned opening and owner reported in local news; verified doors-open date, reachability, and commercial vendor authority remain unconfirmed.
5. The Discipline: Keeping the Claim Smaller than the Evidence
In public record extraction, the temptation is always to inflate confidence: promote "probably" to "verified", call unconfirmed phone numbers "direct contacts", and turn preliminary building permits into "guaranteed sales leads".
This pipeline refuses that shortcut. Records that lack business names stay joined to licenses. Ambiguous strip mall addresses stay flagged for manual review. And the prospects that survive the verification filter are handed over with their gaps explicitly documented—so when a client makes a call, they are acting on defensible facts rather than pipeline hallucinations.
Inspect the Full 10-Project Evaluation
The production report includes full SHA-256 source reconciliation hashes, source file names and row ordinals, Tyler EnerGov and KCMO record IDs, and explicit mandatory limitations for every candidate.
Open Complete Deliverable Report (HTML) ↗