Using LLMs to keep track · Abigail Haddad
How often do workforce cuts or DOGE show up in sole-source justifications? 390 of 9,710. 41 name DOGE specifically.
I look at a lot of data
1
Get the data (GitHub actions)
2
Clean it (sometimes LLMs)
3
Lives on Hugging Face
4
Website
167,864 comments. 96% opposed.
1% of 38,036 postings.
27.5% gone inside a year. CBP lost 10–11%.
12 bills this Congress would amend it. Two passed the House.
Backup · J&A pipeline
Reading one notice
about $0.004
A day of new notices (about 31)
about $0.13
Writing the daily email
about $0.10 a day
Model calls only. Estimates from the repo’s docs, October 2026.
Backup · J&A pipeline
1
The model gives a quote from the document for each value.
2
A script checks that the quote is really in the document.
3
If the quote isn’t there, the value is thrown out.
Regex cleans up spaces, line breaks and curly quotes first, so a real quote still matches. It also pulls the dollar figure out of the quote to check it against the value.
Backup · Structured output
I define the fields I want with Pydantic. The model has to return exactly those.
class EducationClassification(BaseModel):
category: EducationCategory
reasoning: str
key_quote: str
key_quote: “Must be copied verbatim — do not paraphrase.”
Backup · J&A pipeline
A random sample of notices, each field compared against the document, using a better model
Field
Right, of 32
Wrong
Signature date
32
0
Estimated value
31
1
Vendor
31
0
Authority cited
30
0