Here’s how I found it:

1. There’s an API for past USAJobs postings. It doesn’t have direct hire information, but you can use it to construct the links for each job posting, like this.

2. Once you have those links, you can scrape them and pull all of the text on the pages.

3. Once you have that, you can look for mentions of “direct hire”—but you can’t just count those up. You also have to figure out which broader text patterns actually mean they’re using direct hire, versus which ones are using the phrase some other way.

To do this, you need to know a little about the data set, and then you have to write some code. Most policy-relevant data questions are like this: the data is public, but that doesn’t mean it’s easy to answer the question.

And if you’re a journalist or Congressional staffer, you’re probably not doing this. You don’t have time, and you’re a generalist in a lot of things, not an expert in the data set I happen to have spent large parts of the last decade working on. (I was a data scientist for the government, first at Army and then in the DHS AI Corps.)

So here's what I'm doing: building open-source tooling for answering policy questions with public data. If you’re a journalist or Congressional staffer interested in something answerable with specific public data sets—the budget data Joe Carlile is making available, the spending data from MakeGov, EHRI federal personnel data from OPM, or USAJobs data—I will do my best to answer it for you.

And if you’re a researcher or data person who wants to volunteer—particularly if you know budget or spending data better than I do, or there are other data sets you want to offer to represent—please reach out. I’ll happily hand that part off.

The Data Sets

These are the data sets I can work with:

EHRI: This is civil servant personnel data: monthly employment, accessions, and separations. It goes back to 2005, and it’s where to go if you want to answer questions like “how many lawyers work at DOJ, and how has that changed in the last year?”

This is also the easiest to work with yourself: OPM has done a fantastic job with their business intelligence tools, so you may be able to get what you need just from the table builder without coding or talking to anyone. There’s also a link to the data dictionary on that page, and that can get you started with what the variables mean. But if you have additional questions—some examples of recent ones I’ve gotten include “how do you identify middle management?” and “if we’re trying to tell a story about human capital and difficulty replacing people, what would we look at?”—I can help.

USAJobs: This is the listing of federal jobs data. It isn’t all federal jobs—some agencies don’t list everything on USAJobs. And there’s also no 1:1 relationship between job listings and hires: listings can result in zero, one, or many hires.

Despite these issues, USAJobs is a gold mine for answering certain questions about federal hiring that you can’t get from EHRI. USAJobs lets you see titles, the full text of duties and requirements, and other text data like those direct hire designations. It’s also a leading indicator of hiring, because hiring takes a long time, so trends may show up in USAJobs before they show up in accessions data.

If you want a sense of what this data looks like—or want to poke at it yourself—I’ve built a site for exploring the postings: usajobs-historical.vercel.app. And this will let you do your own self-service analytics: you can filter by variables like title, occupation, agency, and opening date, and download both summary statistics and the raw data with links to each job posting. But it doesn’t have the full job listing text data and, like with EHRI, you may have other questions.

Budget data: Joe Carlile, formerly of OMB, has made various data sets available, and this is my source for budget data: OMB’s SF-132 apportionment data, SF-133 execution files, and the President’s Budget appendix are the main ones I foresee demand for. Others he’s hosting, like the OMB spend plans, are difficult-to-parse PDFs; I don’t think I can be useful for those. You can see more on his site. And if this is an interest of yours, you should sign up for his newsletter and potentially hire him—he’s the actual guy on this.

Contracts and spending data: A huge amount of federal contracting data lives on USAspending and SAM.gov. I get some of this from a pipeline I built that pulls in USAspending data and some from MakeGov, which wraps the official sources. The questions you can ask include: what vendors are doing business in my Congressional district? What states and counties are they registered in? What’s being awarded non-competitively? How has that changed recently? If there’s a specific FAR rule you want to know what’s happening under, tell me that.

What I Can Offer

If you’re a journalist or Congressional staffer and you ask me a question that I think I can answer with one of those data sets, you’ll get a public GitHub repo with code that processes the data and gets you whatever number, chart, or map you asked for. It will explain in detail what was done and, as best I can, the assumptions I made. It will link to public data so you can verify it yourself—for instance, if it relies on USAspending data about a particular vendor, it’ll contain that vendor’s USAspending link.

If it’s about USAJobs or EHRI, it stands a really, really good chance of being accurate. If it’s about budget or spending data—I’m doing my best, but the more details you can give me about what you want, the more likely you are to get what you’re looking for. If I later find out I did something wrong, I’ll update the code and let you know as soon as I know.

Also tell me your deadline. If it’s USAJobs or EHRI, turnaround might be hours or faster. If it’s budget or contracts—especially if it’s complicated to me—it could take longer.

On citing me: in general, you shouldn’t need to, because I’m giving you the code and the public data. If you want to, that’s generally fine—unless it’s something I’m not sure of, in which case I’ll tell you that.

I am not offering to make you websites or other permanent infrastructure. I own way too much of that already.

What I Can’t Offer

Don’t tell me anything private. I don’t want constituent data. I am not on FedRAMP. I’m a person doing this with Claude Code because I believe in data and democracy and those things being related.

My preference is that everything I do goes on GitHub and becomes public. I think it’s possible to abstract this out enough to keep any inquiries private—for instance, if you care about your specific Congressional district, we can pull data for all districts, not just yours. I’m also fine with waiting some period of time before making code public—more like a week than a year.

This matters to me partly because I’m committed to working in public, and partly because if you ever want to hand this over to another data scientist or developer, a public repo will make that easier for you.

How to Reach Out

Try me on Signal at abigail.haddad_43—that’s the best way to get a response. If you’re reaching out on Signal and I can’t easily verify who you are, I’ll ask you to send me an email from an official email address or find me on LinkedIn. If this gets overwhelming for me, I’ll set up some form of ticketing system that makes these easier to track.

Also, this is an experiment! I have no idea how it’s going to play out, so let’s see.

Partly, I was trying to figure out what information was being encoded: within the overall color groups, did the shade or shape of individual tiles mean anything? I decided they couldn’t. I also figured out that part of what made it so compelling was that the green and gray tiles came from distinct shape families — the greens had more large outliers, and were far less rectangular than the grays.

That was all I got — five or ten minutes of staring at it, and I never saw it again.

But I couldn’t get it out of my head. For days after, I was pulling out my laptop at the conference and trying to understand the math of tile generation, wanting to make something similar for DC.

It was hard. The standard ways of characterizing tile style for mosaics didn’t work. With Claude Code, we measured the actual tiles from a photo online, mapped them to coordinates, built a generator tuned to match certain shape metrics — and the output still looked wrong. What finally worked was keying directly off the shapes themselves and defining our own metrics. The gray tiles, for instance, have a lot of 80–100 degree angles — not exactly 90, but close enough that they read as rectangular. I guided Claude Code to that, and I was happy with how it ended up.

Noticing becomes a project

My mom once let a stray cat onto her porch during a storm. Later that night a second one showed up and insisted on coming in, and that's how we ended up with two cats. Side projects work the same way: you get interested in X, you do X, you notice the gaps and the new questions, and at some point you have more than you can manage. From the original map, I digressed into whether 3D printing any of it was feasible (it's not, at a price I'd pay), then into the math of tiling more broadly, then into population density, and now I'm working on another map combining the mosaics with using color to denote population.

So when people ask how to find side projects, I sort of joke: at this point I have to try harder not to find them. You take one in and suddenly the whole neighborhood knows you’re a sucker. But really, I think it’s a pattern of thought you develop that starts with noticing certain kinds of things in the world. Noticing “I find this incredibly compelling” or “I want to know how this works” or “something is missing here” — and that last one can mean a lot of things, from “this community needs an event” to “someone should answer this question.” It’s noticing, and then going from that to “I could do something about this.”

The first projects are the hardest because you haven’t built that reflex yet. At this point, matching what I observe to “I could build something here” is annoyingly everywhere — and Claude Code has dramatically expanded my sense of what I’m capable of building, which is part of why mosaic tiles are apparently in scope for me now. (Though if I hadn’t already done a lot of map work right before that trip, I’m not sure I would have felt equipped to try.)

Another thing that’s shifted: I used to start projects as open-ended explorations, which meant they went on indefinitely and never shipped. Now I tend to have a clear picture of what I’m making before I start, and I think that makes it feel lower-stakes: maybe I’m fostering the stray cat rather than adopting it.

Hit at least one of these criteria

A project needs to hit at least one of these for me.

Is it fascinating or beautiful? The mosaic was, immediately and obviously. Sometimes I pull up the finished maps just to look at them. There are things I’m wired to find compelling, and that set has grown over the last few years — partly because Claude Code lets me get into subjects that used to require a lot of prerequisite learning before they were accessible to me.

Is it useful to me to make this? Sometimes this is about learning something I need to have: I did work with sam.gov and USASpending because I needed to understand federal procurement data well enough to answer certain types of questions about it. Or last year, when I was actively job-hunting, I took a day off specifically to build something I could talk about in interviews.

Is it useful for this to exist? Sometimes I just want a thing to exist in the world, either for me or for someone else. In those cases, if I can get someone else to do it, that’s fantastic.

The people you meet along the way

One of my side projects is that I run a monthly happy hour for civic tech people. It started partly because I was a fed for the first part of 2025, it was bad, and I wanted in-person spaces to connect that didn’t yet exist.

And being in spaces where people talk about things they care about is one of the best ways to find the next project — someone names a problem, other people riff on it, and you see the gap. But it’s also a flywheel: you make something, you share it, you meet people through it, and those people pull you into new spaces with new problems and new ideas.

What LLMs do and don’t do (yet?)

With the mosaic, Claude Code struggled early on. It couldn’t tell that the tiles it was generating didn’t look like what I wanted, and it kept insisting they did. I was doing a lot of hand-holding. But once I’d worked out the approach with it, it generated the code much more quickly and better than I would have.

The way I’d describe the current dynamic: the models are good at execution, and much of the interesting work — figuring out what to make, diagnosing what’s wrong, deciding what matters — still largely comes from me. This is a description of the technology as it is right now, not an eternal claim about what humans bring that machines never will. Model quality is improving fast. But at the moment, there’s still a lot of fun in the gap for me — especially the ability to dabble in math I never formally learned, to have an intuition about what should happen when I change a parameter and then find out if I’m right.

It doesn’t need to be good

Seven years ago, I modeled sock loss. My kids were small, I wasn’t sleeping much, and I had become convinced that I was losing the first sock of each pair at a disproportionate rate — that my personal sock-loss distribution was non-random in a specific and maddening way. So I modeled it: if you lose socks uniformly at random, what share end up unpaired as a function of the percentage of socks you’ve lost? This is a trivial problem. But it was a lot of fun, and I’m glad I did it.

I’ve also made things that were technically bad — a genuinely awful RAG chatbot for answering questions about making Quarto presentations, for instance. I learned things about RAG chatbots in the process. Someone later built something dramatically better that incorporated that functionality, and seeing it I thought: oh, obviously I was not the person who was going to do a good job with that. But that was fine.

If you’re concerned about quality — you want feedback on how you’re approaching something technically — ask for it. I almost never offer this unsolicited, but I find it very useful when I get this from other people.

Don’t go to the pet store

How do I find projects? I notice them — and then I let them in. Because I source my own from the world around me instead of pulling pre-packaged ones off Kaggle, they've never once bored me. The projects no one else would have thought to do — the ones that are specifically yours — are the ones you'll most enjoy working on and the ones people most enjoy hearing about. Earnest fascination is contagious. Follow the thing you can't stop looking at, and you'll ask the best questions and build the best things out of it.

For years, I’ve now been using this problem as my own benchmark for ChatGPT1, and then Claude, and then Claude Code. I give the model the blog post and the latest data — but none of the code I wrote — and see if it can correctly write new code to solve the problem.

Prior to last week, this had gone badly. It’s not that this is especially complicated. But it is a weird problem: the data is messy in multiple unexpected ways, and I use a Python package to solve it that’s not exactly obscure, but it’s also not pandas.

The last time I tried this with Claude Code, in late 2025, we went back and forth with it confidently telling me it had solved the problem — and then I’d look at the numbers it had found, and they would be immediately and obviously incorrect. (Think: negative numbers of people.) I spent hours going back and forth giving it hints, but it still failed.

So when Fable came out, and I heard it was really good at one-shotting complicated problems, I tried once again giving it just the data and my blog post. And with almost no additional guidance from me — I told it one fact about how the newest data was differently-structured than the file I’d written about — it correctly wrote the code. And when prompted lightly, it improved the code from how I’d done it, finding a different package I didn’t know about that could do part of the work at a much faster pace2.

That a description of a solution can increasingly generate working code — no coding or subject matter expertise required — changes the math on disclosure of exploits. If you’re counting on limited expertise or will being the thing that protects your data or your system, that continues to be an ever-worse bet.

The Original Problem

DC releases a huge amount of standardized test score data every year. Much of it is suppressed to protect student privacy — they don’t want people to be able to identify individual students’ test scores by knowing facts about them, like their school, grade, and race.

But the specific way the suppression is done never made sense to me.

The published numbers are linearly related — that is, they can be reduced to systems of linear equations. Data appears at multiple levels of aggregation: performance levels 1–5 (but also rolled up as 3+ and 4+), school, school × grade, and school × grade × race. State- and LEA-level data (Local Education Agency — the public school system and each charter school or network) add further constraints. By setting up and solving these systems, you can recover a lot of the missing values.

What the Exploit Gets You

There are no names in these files — you can't look anyone up. But individual test results can be recovered in very specific circumstances: when a demographic slice at a school is small or homogeneous enough that every child in it received the same score. If you already know a specific child belongs to that group, you now know whether they tested proficient. At the finest cross-tabs — school × grade × demographic — roughly 7,500 results become readable this way. Most of what leaks is "this child did not reach proficiency." In far more cases, though, you can rule out that a child scored at a particular level — most often, the top score.

But this is not really the point of the post: you can think it’s really bad for this information to be inferrable, or not bad at all. The point is that what protected this data wasn’t the suppression design — it was that almost nobody who might want this data could write the code to exploit it. That barrier is now gone.

Who Does This Matter For?

If the only thing standing between your data, your system, or your process and misuse is “most people couldn’t figure out how to do this” — you should go test that assumption with the latest model. And if it fails now, keep testing it as new models come out.

For instance: if you’re a company that gives take-home exercises to job candidates, put yours into Claude Code and see what comes back. If you’ve ever written up a vulnerability in prose and not published proof-of-concept code because you figured that was enough protection, test it. Anywhere you’ve said “yes, but you’d need to know X to do that to our system” — where X is something a new model might be able to do — that sentence is worth revisiting and continuing to revisit.

I’ve been using this problem as my own benchmark because I wanted to know when that line got crossed. Well, that happened.

When I wrote this up in 2024, I didn’t share the code or the recovered data. It felt wrong. I knew realistically that probably no one else was going to go through the trouble of figuring out how to code this thing, because that would have required a pretty niche skill set and the willingness to spend a real chunk of time doing it.

And I’m still not publishing the code or the recovered data. Partly I’m afraid of getting into legal trouble, partly it still just feels wrong to publish children’s data, even if I’m not the person responsible for safeguarding it and it’s been over three years on this.

But here’s what I can tell you. I found this problem in 2022 and I had to solve it. I did a proof of concept in Excel first to make sure what I thought I saw was indeed there. Then — and this was before ChatGPT — I sat down and wrote some egregiously bad code that weekend as fast as I could to get to the basic answer. In the years after that I rebuilt it properly, added tests, and studied the problem. I know this topic well.

And if you’re curious, you can take my old blog post, put it into Fable, and in an afternoon, you will have access to all of the same code and results that I do3.

I’ve done these tasks, and some of them I’ve even done well. If you insist on this role being one person because that’s what you have money or the billet to hire for, you should hire an engineer. By an engineer, I mean someone who has written a lot of code, is familiar with software development practices, and is going to be comfortable hands-on building out tooling, even if they’re not coming from a traditional software development background.

This is because there is no such thing as an ‘AI workflow’ for automation that doesn’t include traditional engineering. There are only data pipelines with AI components. Building AI tools is engineering+, not engineering-.

Let’s Look at This Job Posting

Here’s an excerpt from the skills and experience section of a recent posting I came across, lightly anonymized.

None of the following words or phrases appear anywhere in this posting: software, code, engineering, data pipeline. And yet: connecting to external tools via APIs is coding. Collecting, cleaning, and reporting on large amounts of data is data engineering. Building a CRM with robust data modeling and normalization logic is an engineering project. These tasks have nothing to do with AI beyond potentially enabling it — they’re completely standard engineering tasks.

The bullet I want to spend the most time on is “designing AI workflows to automate processes.” There is no such thing as an AI workflow for automation — there are only data pipelines with AI components, and you may not even know if you need an AI component until you sit down and scope it.

What Is an ‘AI Workflow’?

The simplest AI workflow I’ve ever done professionally looked like this:

1. Pull data from an existing database via a SQL query

2. Deduplicate and clean the data, which involved making specific calls about what constituted a duplicate

3. Send each row to an LLM via API calls, along with a prompt, and get the results back

4. Build an evaluation harness to try out different prompts and measure accuracy against human labels

5. Build tables and visualizations to display the results

6. Handle data storage so when new data comes in, you don’t waste API calls on data you’ve already run

7. Push to GitHub

This is basically just data science. Knowing only how to make an API call to an LLM does not get you anything without the rest of it.

And most of the workflows I build that incorporate AI are considerably more complicated from an engineering perspective. I’m usually not handed the data — I’m building a pipeline to pull it from wherever it lives, which frequently involves web scraping. My data cleaning and analysis workflow nearly always has more moving parts. I’m often building a website to display the results. More real automation — things that run in real-time in response to a user action, or that integrate with your existing databases, or that make a decision for you — require even more of a software-building skill set1.

How much can you vibe-code this? To some extent. But we’re not at the point where you don’t need some background. If your toolbox is only AI, you will reach for AI whether or not it’s the right tool, and you probably don’t even have a good sense of which AI component to use.

What about no-code tools? For some bounded, well-defined tasks, those will work. But for the kinds of custom pipelines with AI components that I see in real life, the tooling just isn’t there yet.

Web scraping a site that doesn’t want you to scrape it, where the formatting varies considerably page-to-page, may require a workflow that combines LLM API calls to discover structure along with data processing, if what you want is a repeatable, transparent process2. Using agents and also building an evaluation harness that can tell you what tool calls they’re making and how variable this is across runs requires code.

The problem with hiring someone who’s only comfortable with no-code tools isn’t that they can’t build anything — it’s that you’re going to hit a wall, and you can’t know before you invest the time scoping out projects how much of a limitation this will present.

This matters for what skills you look for and how you describe the role, and it matters for salary: if you set the salary at “generalist business analyst,” you will not get this skill set.

Don’t Be Like Data Science

Data science had a significant mismatch between requirements, duties, and desired outcomes, and it did not go well. Organizations hired data scientists expecting someone to come in and “make their data useful.” What the data scientists got instead was data engineering, data cleaning, and building dashboards — and everyone was disappointed.

And “make our data useful” was also never something one data scientist could come in and do. It takes organizational buy-in, a data infrastructure that’s already in place, and a lot of scoping before you can determine whether any particular project is feasible and what it needs. The same is true now with AI. Hiring an “AI person” and expecting them to transform how your organization works will produce the same result: a frustrated person doing work they didn’t sign up for, and an org that’s not getting what it thought it was buying.

What does being ready look like? Some of it is data access: your data needs to live somewhere that allows integrations with their tooling, and you need to be willing to grant them.

Some of it is willingness to change. For organizations that run heavily on Excel for analytics, a lot of the available gains from AI only materialize if you’re willing to move to code-based workflows. An engineer can help you get there, but they can’t make your staff do that.

And some of it is just your time. AI coding tools allow for building quickly, which means the person you hire needs fast feedback loops. If stakeholders aren’t available to weigh in, that’s a bottleneck.

“AI” is not a job the way “software developer” or “data engineer” is a job. It’s a problem space. Figure out what problems you’re trying to solve and whether you’re set up to solve them — and if you are, hire an engineer.

I was joking, sort of. Leaflet is a JavaScript library that integrates with OpenStreetMap, which provides the map tiles I was using. I can tell you exactly what I built with it — a choropleth map, a proportional-symbol dot map, and a flag-symbol map (one country flag per owner, sized by assessed value).

But if you ask me about the code itself, I can’t tell you much. I can’t tell you the syntax. I can’t tell you which Leaflet functions I called, or even very much about JavaScript as a language. That’s because, with Claude Code, the level of abstraction I’m working at is very different from when I was writing code by hand. Sometimes I glance at a few lines of code while Claude Code is writing it, or when we’re refactoring together. But I almost never open an actual file and read it from start to finish.

I think this is reasonable. And I think if you’re an engineering leader — or anyone who talks to people about coding practices — part of your job is asking which practices still make sense given where the tooling is and where it’s going. What were we trying to get by reading code, and are there better ways of getting it now? How does this depend on what we’re building and on the stage of development of the product? The answers to these are different than they were two years ago, and they’re going to keep changing.

Modeling vs. understanding

I’m told that coding sorting algorithms is a standard activity in introductory computer science classes — there are sorting algorithms of different degrees of efficiency, and students come out of it understanding how they differ and how to build each of them.

Even before LLMs, that wouldn’t have been especially useful to my work as a data scientist. The things I needed to know about sorting algorithms were essentially: what’s the syntax, and how do they behave on my data? I can still recall a couple of the parameters in the sort function that I used the most, and at one point I probably needed to know what happens if you sort a series with both numbers and letters. But that’s as far as I would have gone.

Are data scientists bad coders because they don’t know which algorithms their tools are using to sort, and couldn’t derive them?

I don’t think they are. What coders always need isn’t full understanding — it’s to be able to model their code at a level appropriate to their goals.

What I mean by modeling is: I hold a picture of what my code does at the level of inputs, outputs, and behavior. If I run this pipeline on this data, here’s what I expect to come out. Here’s what would surprise me. Here’s where it’s most likely to break, and here’s what I’d see if it did.

I used to get that in part by reading my code. How I get it is different now. And how much of it I need depends on what I’m building.

For a prototype — something exploratory I’m building to figure out what I want to make — I might barely need it at all. What I’m making might just be a glorified wire frame, meant to show the layout and what happens when you click. For something in production, I need a lot more: I know from experience that, while Claude Code can help enormously with this, I still need to model my code well enough to catch issues and build on it. How much that changes as the tooling improves, I don’t know.

Your attention is now expensive

I write so many more tests than I used to.

It’s not that I never wrote tests before LLMs. It’s that they weren’t a regular part of my practice as a data scientist. Now they’re a default. Anything breaks, write a test for it. The cost of writing the test is cheap, and the benefit — never manually testing that thing again, never worrying it broke without my noticing — is high.

This is because my attention is now what’s expensive.

Old practices were optimizing against the cost of writing code. That cost is gone. Most things in the development workflow that used to be expensive — tests, refactoring, documentation, exploration — are now cheap. The only expensive thing left is your attention.

That changes which practices are worth investing in. The answer to “what should we do more of?” is: almost everything that protects code quality without costing a lot of your own attention.

Write more tests, no matter what kind of code you’re writing. Write them as a default, not a chore at the end of a project. Refactor more often, in smaller chunks, as a normal part of working. Open more pull requests, in smaller pieces, and if you’re running an engineering team, raise the bar for what gets merged — because the cost of improving code has dropped.

Your attention is the budget. Spend it on what only your attention can do: what to build, whether the artifact is meeting the goal, where the design is wrong. Don’t spend it on the things the tools now do for you.

And for each practice we use, we should be able to answer: what are we trying to get from this, and is this still the best way to get it?

I got asked to give that lightning talk about four days before the event, and I put it together in about an hour. The point I wanted to make was the utility of the data and the ease of building with it. That’s all. It was uncomfortable having only a surface-level understanding of my code, but that really was the only level I needed to make that point — and if I’d needed to have more, I wouldn’t have had time to make it.

To people who haven’t gotten on board

There’s a lot of evidence about what these tools can do. The benchmarks have been moving for two years. There’s testimony from developers across the field. For me, there’s my own 2,000+ hours. All of it points the same direction, which is that these tools are just really good.

But I still see a lot of dismissal from software developers, including sneering at people who don’t read their code.

I have some sympathy for where the dismissal comes from. When you have a skill that feels central to your professional — and maybe personal — identity, and now a piece of software can do it, and so can people who don’t have your skills but use that software — it’s hard. This is something a lot of us are dealing with right now.

But nonetheless, that is what happened and is continuing to happen.

And if you’re going to stay in this field — and stay in a position of leadership — you’re doing the people who depend on you a disservice if you don’t acknowledge what’s happening. They need your help figuring out what comes next: what good practice looks like in this new shape, what we should be teaching, what we should be hiring for, how to make the cheap practices stick. That work needs people who care about doing things right and are willing to update their picture of what that looks like.

I wasn’t thinking about how this name I picked as a bit was going to be on my business cards, my LinkedIn, my tax forms, and my bios when I give talks. That’s because I didn’t think I would still be doing this full-time a year from now. I assumed I would do some part-time consulting and then go back to a full-time salaried role. But I haven’t, and I’ve had a much better year for it.

This is about why I chose that and what it’s cost me.

What I wasn’t going to give up

AI Corps — before it collapsed — had a lot of things that I loved. I got to work on multiple projects at once, sometimes by raising my hand to volunteer, or by putting some work in to build and then saying “hey, I think we should do more of this.” I had technical autonomy in the areas where it matters to me. I was getting to grow my career in the ways I wanted, and work on things that mattered to me. I had a level of compensation I was happy with, and I was able to work from home.

When I left, I didn’t want to give up any of those things. But the salaried jobs that had all of those attributes weren’t going to hire me. I’d gotten engineering interviews through recruiters and my network, but the interviewers could tell immediately I wasn’t what they were looking for. I don’t have a STEM or CS background, and I haven’t worked as a software engineer.

Meanwhile, unpaid side projects I was doing were leading to meeting people who wanted to work with me, and former coworkers were reaching out as well. When someone’s hiring for a role, there are a lot of boxes they want to check, and I don’t typically fulfill those. But when they see what I do — when they can go, “Oh, there’s this person who does these interesting things” — they’re not checking off boxes anymore, and that’s a much better lane for me.

It’s also lower-risk for them: they don’t have to commit to 40 hours a week. And the comp piece can work out better as well. There are organizations that wouldn’t pay any engineer the full-time equivalent of what I make, but will make room for a contractor on a fixed number of hours for a specific project.

What I do for a living

My niche is full-stack data engineering with Claude Code. I build the data pipeline. I do the analysis, frequently with LLMs for the text analysis part, and now increasingly with agents. I also build the front end in JavaScript. I’m not the best person you could hire for any one of these parts, but I’m good enough at all of them for various kinds of data projects. You can also put me in front of a client to take requirements, and I can brief or write up what I made.

What am I actually very good at? Architecting pipelines with LLMs. I’ve been doing it for about as long as anyone has. But I don’t want to only do that, and I don’t want to do it full-time on one specific pipeline.

My specialty is public, government data. If it’s in an API, on a website, or scattered across Excel files or PDFs, I’ll get it. And like with nearly all of the technical work, I’m an intermediate generalist: I’m not the best person to analyze any of the data sets I work with. The best people tend to be at the relevant federal agency, or building their own startups and consulting practices around the data, or they’re data SMEs who aren’t engineers.

Having a brand like this is useful. It indicates where to focus my unpaid career-development time, like for side projects or talks. It tells me what to say yes to and how to describe myself. And it tells potential clients what not to hire me for: I’m not the right person for building software, because you can get someone cheaper and better at it, and I tell people that.

Marketing: none, or a lot

I got lucky early on with a client. We connected when I was on DRP. I offered to do something for free; they insisted on paying me for it, which was a good sign. They guarantee me up to 40 hours a week and they’re great to work with in a bunch of ways. I have one other substantial client, who I’ve been half-time with at various points, and then a handful of others I’ve done smaller projects for. I typically bill about 40 hours a week total, but the mix varies.

In a sense, I haven’t done any marketing to find clients. I haven’t posted on LinkedIn that I’m available. I mentioned a couple of times in talks that you can hire me, but no one ever did hire me out of that, so I stopped. The people I’ve worked for have been former coworkers or people in my network. A couple of times I’ve pitched a specific project to a specific group I thought would be interested, but I wouldn’t really describe that as marketing.

But the other way of looking at this is that I’m constantly marketing, in the sense that my life and work and hobbies have basically collapsed into the same thing: coding, writing, being in spaces where I meet people interested in the kinds of things I do, both online and in person. I’ve given four talks in the last three months. I co-organize a couple of meetups. I post far too much on LinkedIn. I have an upsetting number of coding side projects.

What kinds of spaces am I in? Someone asked me recently what spaces were useful for networking, and I didn’t have a great answer. That’s because the best spaces are specific to your professional interests. They’re the ones that bring you into contact with people who might want to collaborate or hire you, or who you might want to hire. They’re probably not public, but public spaces are how you funnel into them. If you can’t find the space, it might not exist yet, and maybe you need to start it.

More on logistics

I file quarterly taxes. I registered my LLC and I got business insurance. I have a solo 401(k) that I fund from both the employer and employee side. I use Toggl to track hours by project, then enter them into whatever invoicing system the client uses.

Without staff, the administrative load isn’t that different from being a salaried employee. The biggest issue is budgeting for both sides of payroll taxes and retirement contributions.

The other big piece is that billing hourly makes the cost of every choice visible to me. Taking a break or going to a conference feels expensive because it displaces work I could be billing for. In some ways the difference from salaried work is illusory — I’ve had employers that paid out PTO, so there was still an hourly cost to taking time off, I just didn’t feel it as acutely. In other ways, there really is a difference: no one is going to subsidize my conference trips, and there’s work that in a salaried position would be part of the job, but now I have to choose between skipping it or eating the hours myself. For instance, a focus group on federal hiring is very much somewhere I want to be, but I’m also calculating the cost to attend and deciding if I can justify it to myself.

But the same mechanism that makes breaks feel expensive is what makes it possible to do work that wouldn’t fit in any salaried job. When the new OPM data dropped early this year, that was going to be a real chunk of my time for the next few days because I wanted to get to know it and to share that knowledge. I put together a call where I could go over it with some folks, in the middle of a work day, because I wanted to. No one was paying me for it, but that also meant I wasn’t using time that should have been an employer’s. I have trouble imagining anyone who would pay me for a salaried position and still let me do all of the side work I want to do at the times when I want to do it.

Always being on

There was a stretch at AI Corps, a few months, when I was spending most of my evenings and weekends not doing anything that looked like work. I was writing the occasional blog post, working on the occasional side project, but mostly I was hanging out with my kids, watching Netflix, or going to the gym. Then January 2025 came, I realized I was likely to need a new job pretty soon, and that was the end of my weekends. It’s been like that ever since.

The supposed benefit of not having a salaried position — that you can set your own hours — is true in a limited way. I’ll duck out some morning or afternoon most weeks. But I’m spending more hours on things that are like work than I think is healthy.

Some of that is fear. I’m afraid to let up, because everything with AI is moving fast and I’m afraid of being left behind. The fear isn’t about whether I can pay the mortgage this month. It’s about whether I’ll still be able to do this work in two years if I stop putting in the time to learn and build.

But a lot of it isn’t fear. I’m not responding on a Saturday afternoon to someone’s question about how to locate mid-level management in federal personnel data out of fear that if I don’t, I won’t be employable. I’m doing it because I want to. Claude Code is addictive. The work is interesting, and I also think it’s important that people be able to access and understand the data I work with. The talks and the side projects and the meetups and the Signal chats and the writing — I love doing all of it.

A year in

I feel extremely lucky that this is how my year has played out. A lot of people who left the federal government have struggled to replace their income or find anything comparable. If you worked in international aid, there isn’t a big industry for that outside of government, and the jobs that do exist now have enormous competition.

My area of interest sort of went the other way. I started using Claude Code just before I left AI Corps, and that let me work at a speed and across a range of the tech stack that made what I do now possible. Interest in government data and LLM pipelines was also growing, and I sit at that intersection. I don’t think that would have helped me get a salaried job, because what people were hiring for just didn’t adjust that quickly, but it definitely helped me get paid work.

One of the things keeping me here is that I don’t want to be dependent on one institution the way I was at AI Corps. In a short amount of time, it went from being remote to in-person; the hours were now entirely set; the kinds of work we did and some of the tools we could use changed; and my professional development opportunities also became more limited. And that wasn’t unique to AI Corps. The same pattern keeps happening elsewhere: earlier this year, when DoD made the designation that Claude was a supply chain risk, I was thinking about how if I had been in government and lucky enough to get Claude Code access, I would be freaking out at the prospect of the way I work being taken away. Having multiple clients means spreading risk: I don’t feel dependent or at anyone’s mercy the way I felt at DHS. It’s a real relief.

I also don’t know how I’d go back to a normal salaried role where you apply for a job and they compare you to a set of criteria. The longer I do this, the less legible I become. I don’t have a title, I don’t have a clear level of seniority, and the set of things I do every week doesn’t correspond well to any particular role that I’m aware of. And for now, that’s okay. I’m not trying to go back.

Some people make this kind of life look graceful. I make it look like I haven’t slept enough and my GitHub Actions just failed this morning and I don’t know why yet. But it’s mine, I chose it, and I’m so glad I get to do it.

On the other side, there are the folks promoting these new hiring programs and talking up government tech modernization. They’re political appointees or others closely associated with the administration — not career civil servants.

Meanwhile, you're not hearing from the career folks who are still there, because they aren’t allowed to talk publicly about the challenges of the last year, why they've stayed, whether they're optimistic or not, or why they think you should join them.

What I have to say isn't as valuable as what you'd hear from them, but unlike them, I’m allowed to write what I want to.

This is a companion to my previous post about the new federal tech hiring programs — Tech Force, Data Science Fellows, and the rest. That piece was about what to look for if you apply to those specifically. This one is more general: how to think about whether to take a government job at all right now, particularly one in tech.

I don't know what the answer is for you, but here are the questions I think you should be asking.

Am I going to be asked to do something that goes against my values?

Maybe. It depends on the agency, the role, and your values.

For a lot of roles, the day-to-day is going to feel like any other large organization with a lot of bureaucracy. You’re modernizing a legacy system, building a pipeline, creating a dashboard.

But as part of that, you might be building tools for an agency whose broader mission or leadership you have problems with. You might be part of a narrative — “we’re finally modernizing government” — that you don’t think is accurate and is serving purposes you’re not comfortable with. The way you’re building might also go against your values — building in ways that aren’t accessible, don’t involve work with users, aren’t open-source, or just generally aren’t very good.

There are also other kinds of issues. How likely is it that your role will involve interacting with political leadership, and how comfortable is that going to be for you? Or, do you want to say “Department of War” all day long?

If you’re going into a senior role, there’s another version of this: you might think you’re going in to be a moderating influence, or to protect your team. That can be true. But your ability to steer things or protect people is likely going to be more limited than you think, and “they need me, it would be worse without me” can become a story you tell yourself while lending credibility to something you shouldn’t be lending credibility to.

Most of these aren’t entirely brand-new issues, but there are certainly teams and agencies where they’ve gotten worse during the last year.

But it's also true that if you care about the government working well, the most direct way to make that happen is to be inside, building. That's always been the case for government tech, and it hasn't stopped being true.

If you’re going into a role where something against your values is likely to come up, have some lines. Write them down, because once you’re in a role, it’s easy to justify whatever it takes to stay in it.

Am I going to be able to be effective?

The gap between “we’re modernizing government!” and shipping something useful is enormous.

Government tech has always had this problem. Someone announces a big initiative, there’s a launch event, maybe a blog post, and then it quietly dies two years later when the funding runs out or the champion leaves.

I don’t think this has improved. Overall capacity is significantly down. A lot of technologists either got pushed out or decided the circus wasn’t worth it and left on their own.

But things are still happening. There are teams that are doing important work and doing it well. I wish they would talk publicly more.

When you interview, ask what they’ve shipped recently. Ask what the dev environment looks like. Are people writing code, or managing contractors and writing reports? What’s the tech stack? If there’s code on GitHub for that agency, look at it.

Ask those questions, look at what they’ve built, and talk to people who are there or recently left, not just the ones trying to hire you.

Is this going to be an awful place to work?

Look, it might be.

Government tech was frequently depressing even in the best of times.

I’ve already written the litany of things that happened last year, and you’ve probably more or less heard it before anyway. It’s not over. A couple of people I know who made it through last year just had a terrible, demoralizing week because of actions reminiscent of the chaos of 2025. I can say that without worrying about getting them in trouble, because I’m sure that’s not uniquely identifying.

Some offices are still pretty insulated right now. Others went through a terrible stretch last year and have since settled down. Is that going to be stable? I don't know.

Is anyone going to want to hire me after I work for this government?

It’s true that “I worked for the federal government in 2026” may prompt a different set of questions in an interview than “I worked for the federal government in 2019” would have — but it depends what you’re doing. Going to work for ICE is going to look pretty different on a resume than going to work on FAFSA. The younger you are and the more briefly you’re there, the easier it will be to take it off your resume later on if it’s an issue. Seniority also matters: if you’re in a named leadership role, that’s googleable.

I don’t know the answer to this one — but if you’re doing something good and impactful and in line with your values, take the job and make your case later.

Am I going to get the career growth I want?

Government tech has always had a version of this problem — you can end up spending years on a tech stack or a way of building that just does not transfer to the private sector, or even to what more technical government jobs want.

For instance, right now in government very few engineers have access to the best AI coding tools—a lot of folks are using GitHub Copilot, which isn’t as good, and many have access to nothing.

This might or might not matter to you. My position—that I’d be reluctant to take a job where I can’t use Claude Code—is an extreme one: a lot of people would not put that anywhere near the top of their list of things to care about when evaluating a job.

Other considerations: whether there are people who can mentor you or who you can learn from technically, and whether you’re going to be hands-on building vs. managing contractors.

Think about your next best alternatives. Find out as much as you can about tooling and what your role involves.

Am I going to be asked to break the law?

In most roles, this is not going to be a major concern. But it’s something to think about. Here are a couple of ways it might come up:

Talking about work off the record. Discussing substantive decisions — not coordinating carpool to a meeting — in Signal chats or other channels that aren’t FOIAable, and purposely not making them official records, violates the Federal Records Act. You could be disciplined for this.

Data access. Accessing data you’re not supposed to have, or giving someone else access, can violate several laws. There’s a lot of litigation around data access and privacy right now — recent federal court action over improper sharing of confidential tax data, for instance. These cases typically target agencies and systems, not rank-and-file technologists. But even if you’re not a senior official, you can end up in court filings, IG reports, or public scrutiny if you mishandle data. This one is hard, because there’s not always a clear data owner or other point of contact to run this by. If you’re concerned, getting permission in writing on official systems is a good thing to do.

Other issues might come up too, but the advice is the same for all of it: get it in writing, on official systems. “My boss told me to do it verbally” is not a position you want to be in. “The agency general counsel told me in their official capacity that this was legal” is a much better one. But if you’re at the point where you need a formal legal opinion to feel okay about what you’re being asked to do, that’s a pretty bad sign.

This is going to be difficult to get a sense of before you join. But for most jobs, I would not worry about this.

Taking the job, leaving the job

For me, the answer is no — I’m not trying to go back. But some people I know and respect a great deal are still working in government, or want to return. And I’m glad for that. The government is doing things that matter regardless of your politics, and there are people still there trying to do them well.

You don’t need anyone’s permission to take a government job right now, and you don’t need anyone’s permission to leave if it stops being worth it.

This is fine if you only want a few files. But I wanted hundreds of them, for myself and to make available for other people with questions about this data. Downloading them manually means leaving no record of what you did, and doing it all over again if the data updates from Version 1 to Version 2. And if you make a mistake, are you going to realize it?

I didn’t pull it manually, but I still got the data. I’ll explain how in a moment. But first, some context.

Stata, SAS, SPSS, and MATLAB are proprietary tools for statistical and mathematical analysis. You write code in them, though they have point-and-click interfaces as well. They’ve been around for decades, they’re taught in graduate programs, and a lot of people use them for analytical work.

When I say you should switch to Python from those tools, I’m using Python as a stand-in for something broader: open-source, general-purpose programming languages. R. JavaScript, whatever. The point here isn’t that Python is the One True Language2. You might be able to do everything you need in R, or you might, as is increasingly common, combine several languages.

But Python is the biggest and most general-purpose of these languages and the one I use most, so that’s what I’ll focus on. The argument is really: join the world where tools work together, you can build anything you want, and you can share what you built. AI coding assistance is making it much easier: the switching costs fell, so there’s even less reason to stay where you are than there was a year ago.

The OPM data

The data got released on a Thursday. By the time I could look at it, I’d already done my full-time job, run a meetup, and come home. I wanted to get this data up before the next morning, when I was hosting a call to share initial thoughts and go over what was available in the drag-and-drop tools vs. the raw data.

I also wanted to sleep.

Because of how the OPM site loads, to download the data automatically, I needed something that could control a web browser. There’s a Python library called Playwright that does this beautifully. Anything you can do manually in Chrome, it can do with a script. A couple hours later, I had the files I needed, uploaded to Hugging Face where they can be easily downloaded or worked with in-place. I included a Jupyter notebook on GitHub demonstrating how to pull from Hugging Face and analyze the data. Claude Code helped enormously with every step of this.

Besides just the data, what does this get me? A workflow that’s transparent: every step is shared on GitHub. It’s repeatable. Other people can run it. I can build on it: when new versions come out, I can tweak the code. If I wanted, I could set up a GitHub Action to check daily for updates automatically and upload them to Hugging Face.

Last week, a reporter flagged a possible issue in the data. I replicated it, built a notebook reproducing the issue, and sent a link to OPM. They can run it themselves, because Google Colab makes that free for anyone. And they can follow what I did, because they have employees who know Python.

This is the infrastructure that exists. If you’re using Python.

You can’t do that in Stata, SAS, SPSS, or MATLAB

Browser automation? Stata can’t. SAS can’t. SPSS can’t. MATLAB’s answer is “call Python from MATLAB.”

Upload the data to Hugging Face? You’re going to be stuck using the GUI to drag-and-drop it.

Create a notebook anyone can run for free online? None of these allow that.

Put it on GitHub Actions to run automatically? Python comes pre-installed on GitHub Actions and can be run for free. None of these proprietary tools have that.

The pattern: even where these tools have some capability, it requires calling Python, or expensive licenses, or both. You only get the tools someone built for you—and those are narrower, because you’re a smaller and more specialized group.

What else opens up

Once you’re in Python, what counts as “data” expands dramatically.

Data no longer has to be a CSV or an API call wrapped up in a nice bow. It can be PDFs you found by automating Google searches. It can be Excel files someone built intending them to be gone through by hand, but you want years of them, you want to process their different formats and tabs, and you want a pipeline that grabs new ones and alerts you if they don’t match what you’re expecting.

Anything you can access online can be data, if you want it hard enough.

And then there’s what you can build to show your findings. I work in JavaScript now, and there are still plenty of websites I wouldn’t trust myself to build yet: anything with user authentication, a back end, payments. But if you’re a data person who wants sites with interactive visualizations? The JavaScript D3 library is what data journalists and data storytellers use. It’s the gold standard for communicating your results and letting other people play with your model or your findings.

And the sharing infrastructure isn’t just about code or results. It’s about data too. Hugging Face, git large file storage, all the tools software developers have been building let you easily share data. If your tools include Python, there’s a whole data-sharing ecosystem.

“I have no obligation to believe anything they write”

Researcher and statistician Andrew Gelman has this line I think about a lot: “The authors of research papers have no obligation to share their data and code, and I have no obligation to believe anything they write.”

This is how I feel about anyone trying to persuade the public of something while coding in languages that aren’t free to run. Are there people using Python who also don’t share their code? Sure, and that’s bad too. But if you’re using a proprietary language, you can’t share in a useful way even if you want to.

Part of this is audience. If you share a Stata .do file, only people with Stata licenses can run it. If you’re okay limiting yourself to the subset of your field who have licenses and know how to use them, fine. But if you want your work accessible to journalists, policymakers, researchers in adjacent fields, the public—you need tools they can use.

And part of it is structural. Users of proprietary statistical software are far less likely to share code at all because the tools don’t encourage it. Your code probably isn’t in a Git repo. It’s less likely to run end-to-end without manual steps. The workflow of “write reproducible code, put it on GitHub, let anyone run it” isn’t what these ecosystems were built for. Python and R communities developed around reproducibility and sharing. The proprietary tools didn't, so reproducibility in those ecosystems is the exception, not the norm.

Even if your code should never be public, the same problems apply internally: you can't run each other's work without licenses, can't easily hand it off, can't do anything outside this ecosystem.

The exceptions

Are there things you genuinely can’t do outside your proprietary software? Yes, but the list is short. SAS is still dominant for pharmaceutical submissions to the FDA, although there’s a growing R community. MATLAB's Simulink has capabilities for control systems hardware deployment that Python can't match. But Python now has PuLP for linear and integer programming, and PyTorch for machine learning—and Python has become the standard in ML.

And if you’re an economist cleaning data and running regressions, or even doing more complex statistical work3? You’re probably using it because you learned it in grad school and switching felt hard. Well, it’s not hard anymore. You can install Claude Code and ask it to translate your .do file into Python, run it, and compare the results. I’ve done this. If you can do the statistical analysis, you can figure out the migration as well.

And even where regulators demand a specific tool, or your analysis needs something not yet built in R or Python, you still benefit from using those languages. I used to work on a team where we had to use ACL, another proprietary analysis software, for compliance purposes: the auditors wanted the logs. So I wrote Python that generated the ACL code and ran both workflows in parallel, comparing results. Back then I didn’t know how to automate that comparison or have Python kick off the ACL, but now I do. The specialized tool can be the last mile, not the whole journey.

There’s a whole world out there

I know coders who prefer R but work in Python because that’s what the market asks them to do. I’ve rarely met anyone who learned Python or R after switching from Stata or SAS and said, “I wish I were still working in those.” Not everyone gets to choose their tools, but if you want to move to a job outside this ecosystem someday, or if you want to help your organization do better work, you need to know what else is out there.

There’s a learning curve, but it’s way less than it used to be. I haven’t written a line of code myself in months—Claude Code does it. The AI coding assistants are good enough now that “learning Python” increasingly means learning to think about architecture and workflows: how do you want data to flow, what should update automatically, how should pieces connect, what should you test? You no longer need to master syntax to work in these ecosystems.

And when you get there, I think you’ll like it. There’s data you haven’t been able to touch because it doesn’t come in a nice CSV. There are visualizations you haven’t been able to build because your tools don’t do interactive. There’s an entire infrastructure for sharing your work and your data that you’re not part of. Half of all developers use Python. Tens of millions of people, building tools for each other, for decades.

But maybe you’re reading this thinking: I don’t need browser automation. I don’t care about Hugging Face. I’m not trying to share anything with the public and I never want to make a website. Fine. But do you ever struggle to remember what you did six months ago? Do you have a 1,500-line .do file you’re afraid to touch? Do you copy-paste the same code between projects? Do you dread handing off work to a coauthor? Do you redo the same manual steps every time new data comes in?

General-purpose programming languages solve these problems too. Functions you can abstract out and reuse. Version control that tracks every change. Package managers that handle dependencies. The big flashy stuff—the browser automation, the sharing infrastructure—is what I’ve been talking about because it’s cool. But the boring stuff matters too, and it’s also better in Python.

The moat is disappearing

Unless you control access to unique data, the barriers to entry for analysis in your field have fallen. A grad student or policy analyst or random blogger with Claude Code can use your work and spin up something real with the same data you’re using, without being limited by their tools. And they can also access other data that you’re not able to.

The world where you can build a full pipeline in a weekend, pull from multiple datasets, test it, and share everything openly is here. The people who are excited about what’s becoming possible aren’t waiting.

My background is Python and data science, not front end. If you ask me how to explore a dataset or pick a good visualization, I'll give you some curse-of-knowledge answer that's useless to a beginner. But I've only been building websites with Claude Code for eight months, so there’s definitely no curse of knowledge here. Through trial and error (and suggestions), this is where I've landed and why in terms of technical practices.

What I’m Building and How

These tips are somewhat specific to what I build, which is data pipelines with static JavaScript front ends.

A static site is a site with no back end—no server running code, no database being pulled from. It’s just HTML, CSS, and JavaScript files sitting somewhere. When a user visits your site, their browser downloads those files and runs the JavaScript locally. Everything happens in that browser.

For almost all of these, I also have a Python pipeline that processes data and outputs data files. The pipeline runs on my computer (or, as the project matures, on a schedule via GitHub Actions), regenerates the file with new data, and pushes the updated data to GitHub. From there, Netlify—a service that hosts static sites for free—automatically deploys it.

If instead you’re building a website with user accounts and a database, this still applies, but you’ve got a lot more to worry about on top of that.

Put It on the Internet–Continuously

Don't be me eight months ago, dragging and dropping folders. Set up Netlify to do continuous deployment from GitHub: you push code and data to GitHub, and Netlify automatically rebuilds and deploys the site.

This matters because deployment should be easy, boring, and automatic.

Netlify has a generous free tier. Your site might fail to build at first, but each time after that will be easier.

Use Someone Else’s Components

Don't build things from scratch when you can use someone else's code. Look for libraries—pre-built tools you can drop into your project. Periodically ask Claude: "Is there a library for this?" Otherwise, the AI will happily write you 200 lines of custom code when a library does it in five.

Here are a few that I like:

1. DataTables: For tables with sorting, filtering, search, and CSV export. And make sure you’re actually using it for everything: I was spending half my time on the front end just on filters—getting Claude to write custom filter code, realizing the filter code didn’t actually work, debugging filter code. Now, as much as possible, I use the filtering native to DataTables.

2. D3: If you want interactive data visualization that looks like what data journalists and other data viz pros use, you want D3. I tried to build a bubble chart in Python—where the size of each bubble represents a value—and it just wasn’t happening for me. The Python visualization libraries are great for static charts, but for anything interactive, I’m fighting them the whole way on appearance. With D3, I had a working interactive bubble chart in an evening. Yes, there are wrappers for D3 in other languages, but if you’re using tools like Claude Code anyway, you can just use D3 directly.

3. Bootstrap: For layout and basic visual stuff—buttons, cards, forms, making things look reasonable on mobile. Full disclosure: I’ve only used this twice, but I was able to delete hundreds of lines of CSS without making my sites look or operate worse.

Put Your Colors and Sizes in One Place

CSS—Cascading Style Sheets—is how you control what your website looks like. It’s separate from HTML, which is the structure and content. CSS handles colors, fonts, spacing, layout.

CSS lets you define variables—values you name once and use everywhere. When the color scheme changes, you change a few lines in one file instead of hunting through your whole repository.

I keep styles that show up on every page (tables, buttons, popup boxes) in a separate CSS file. Styles that are unique to one page can live directly in that page’s HTML.

Customizing your CSS is also how you stop your site from looking like every other AI-generated page, if that’s something you want.

Test Your Website

I test my sites now because I shipped broken stuff and didn’t notice until later.

There are two kinds of tests I use: data tests and front end tests. Data tests make sure the data is in the format I need and I didn’t drop any of it since the last time the pipeline ran.

I initially didn’t write front end tests, but they turned out to be easier than I thought, so now I do.

Four categories of front end tests catch most of my “it broke and I didn’t notice” problems:

1. Page loads without error. Catches broken imports, missing files.

2. Data loads. Makes sure the table isn’t empty.

3. Search/filters change results. Your search box looks nice, but does typing in it do anything? This catches when the user interface exists but isn’t connected to the data.

4. Search finds something you know exists. If your state filter has “Illinois” and your data has “IL”, you’ll get zero results. A test catches that mismatch.

You can run these as part of your GitHub Actions deployment setup: that is, GitHub can run both your data and your front end tests and only deploy to Netlify if they all pass.

For front-end tests, you'll need a headless browser. I use Playwright. It loads your page, clicks buttons, types in search boxes, and checks for expected behavior. And Claude Code is good at writing these tests if you describe what behavior you want to verify; I use it with the Playwright MCP Server.

Make Things Shareable via URL

This is more specific, but it’s a nice pattern that I’m using now and I really like.

You can put user selections into the URL, like yoursite.com/data?year=2024. Now that view is shareable. Someone can bookmark it. They can send the link to a colleague. Whereas if you store state only in the browser’s memory, it can’t be shared.

If You’re Coming from Python

I was brand-new to JavaScript, but I wasn’t new to coding, and so there were also some patterns I was able to bring over that were useful.

Don’t repeat yourself applies here too. If you’re writing the same code in multiple places, pull it into a function or a separate file.

Config files are nice. You can have a config.json that defines things like which fields to show or what the column headers should be. That way when those change, you change the config, not the code.

Keep your data separate from your display. Your data lives in a JSON file, not embedded in the HTML. When your pipeline regenerates the JSON, you don’t have to touch the website code.

Why This Matters (Or Doesn’t)

If you’re building a one-time site for fun that you’ll never touch again, most of this doesn’t matter. Ship it, move on.

But if you're planning to keep building on something—adding features, updating data, fixing bugs—there's a concept worth knowing: technical debt. Technical debt is what happens when you say "I'll fix this later" forty times and then, suddenly, it's later. It's the code you copied and pasted instead of abstracting into a function, so now you have to change it in multiple places. It's the CSS scattered across various files, so you can't find where that shade of blue is coming from. It's the deployment process that requires you to remember various manual steps including commenting out your code.

Technical debt isn’t always bad. Sometimes you take on debt intentionally because you need to ship fast. But the messier your code base gets, the harder every additional change becomes.

Everything in this post is about minimizing technical debt:

• Using libraries means less code you wrote and have to maintain.

• Abstracting shared code means changing code in one place when you need to fix something.

• CSS variables mean updating colors once.

• Tests mean catching bugs before you deploy.

• Continuous deployment means not forgetting to push updates—and having a process that runs those tests before deployment.

Every line you write is a line you might have to debug later. The goal isn’t writing a lot of code—it’s writing as little as possible to do what you want.

I’m having fun building these sites, and I want to keep having fun. Technical debt is the opposite of fun.

And if you ever hand your project over to a developer, you want it to be a good transition—not one where they’re cursing you and your vibe coding.

For instance: a coworker at the same level as me emailed me an org chart showing that I now reported to her. She told me I couldn’t share it with anyone. Naturally, I did. Turns out I did not, in fact, report to her.

But the drama wasn’t even the biggest issue. The real problem—and this has been the case for every technical job I’ve done in government—was figuring out, between me and management, how to identify projects that were appropriate for my skill set, possible to do given our team’s technical and personnel constraints, and useful to my agency.

On paper, this should be simple: there are technical problems, you bring in people with the skills to solve them, voila. In practice, it’s not. Which is how I’ve found myself, at various points, wrangling spreadsheets instead of coding, managing contractors, and building models that were never going to get used.

At my most recent civil service job at the short-lived AI Corps at the Department of Homeland Security, this was done well—but I didn’t realize at the time how much work was happening behind the scenes to enable that. We had leadership with previous experience with digital services teams in government, which meant we had a significant advantage in doing it right. And it was still a challenge.

And that was before everything went sideways.

What Happened Last Year

The federal government ended 2025 down roughly 250,000 people. Technical folks were not excluded from that number. 18F got dismantled—that was the team that built tools like cloud.gov. The U.S. Digital Service (USDS) got taken over by DOGE, fired some people, and saw significant resignations. Some agencies conducted terminations of probationary employees—anyone hired or promoted in the past year or two.

At Homeland Security, some engineers were fired for being probationary. More left semi-voluntarily, through a combination of early retirement and buyouts as the carrot, and return to office (including for employees hired to be fully remote, which included everyone on AI Corps), forced reassignments requiring relocation, and a fear of additional layoffs as the stick.

I loved the work at AI Corps. Then the chaos started. Projects stopped. We were told by leadership that there weren’t plans to eliminate our team, but also that individuals likely would be fired—and the indications were all negative that things would be functional again.

And then came the Sunday afternoon email ordering us to report back to work five days a week or be fired starting the next morning.

I left shortly after that.

And Now They’re Hiring Again

Last month, the Office of Personnel Management (OPM) announced several new programs: U.S. Tech Force (1,000 fellows annually), Data Science Fellows (250), and Project Management Fellows (250). These are all short-term roles where OPM assists with the hiring, but actual assignments are to various agencies across the executive branch.

The goal? “Modernize the federal government” and “tackle the government’s most complex and large-scale challenges.”

These are good goals. If you want to work on them, I hope you find a role where you can do that. And I also hope these programs are successful.

But they might not be. So if you’re going to apply, go in with your eyes open about what to look for.

Why This Work Is Hard (Even in Good Times)

Government tech modernization is difficult. OPM spent approximately $290 million on a project to automate federal employee retirement claims before canceling it in 2011. The Coast Guard spent $67 million over five years trying to replace its aging health records system before scrapping the effort; as of 2018 it was still paper-based. The VA spent $127 million over nine years on a scheduling system that never launched. The FAA’s air traffic control systems include multiple systems over 30 years old; its ongoing modernization effort, NextGen, has faced decades of delays, and some critical systems won’t be modernized until 2035.

I think of the core problem as responsibility without authority.

For instance, it’s your job to modernize the agency’s data systems. But you’re reliant on contractors to do much of the work because you’re not staffed to build everything the agency wants fixed by yourselves. The contracts have already been competed and awarded, most likely to the vendor putting forth the Lowest Priced Technically Acceptable proposal, not the best. The payment structure also likely incentivizes billing hours rather than building good products.

Additionally, the roles and skill sets the contractors were hired for may not be the ones you need and the work may not be scoped in a way that’s useful to you. And if it’s not working out, you can’t just change the contract or bring in different people. That takes months or years, if you can do it at all. (And you probably can’t, because you don’t know how and your office doesn’t even own the contracts.) And good luck if you try to get involved in pushing the contractor to hire or screen differently.

Or maybe the contract is fine, but you need existing code from the previous contractor or else you’re way behind. Where does the code live? Not on any server you’re going to get access to, because whoever was managing that previous contract never asked for it.

You also can’t change hiring regulations for civil servants. You may not even be able to get HR to do things that are totally legal.

And then there’s risk aversion about the wrong things. A lot of people are terrified of a process violation, of something that could look bad in an audit. But they’re not as worried about the system being terrible and not working for users, or wasting years and millions of dollars on something that doesn’t solve the problem.

Meanwhile, your leadership is under pressure for short-term wins. They just brought in a bunch of expensive, highly visible staff. They need to show results. But they may not know where the opportunities to be effective are—what’s realistic to accomplish, what would require years of groundwork, what’s blocked by forces outside their control, like Congress. They may want innovation but settle for innovation theater.

Finally, a lot of people at the agency you work in and whose assistance you need are predisposed to dislike you. You’re likely younger, you’re paid more, and your mission (to fix the bad stuff) is an implicit criticism of them. I’m not saying everyone sees it that way, but “they thought they were the smartest people in the room and they didn’t listen to us” is something that gets said so regularly about digital services teams in government that it’s hard to tell when it’s true and when it’s not1.

The limitation on making progress in government is rarely engineering skills. Building the thing is usually the easiest part. It’s everything else that’s hard.

And all of that is just the starting point, in a good year2.

Why This Particular Moment Is Harder

Think about the year civil servants—the people you will be working with and need buy-in from3, should you do any of these programs—just completed.

Russell Vought, who ran the Office of Management and Budget in Trump’s first term, said in a private 2023 speech that in a second Trump administration, “We want the bureaucrats to be traumatically affected. When they wake up in the morning, we want them to not want to go to work because they are increasingly viewed as the villains.”

“We want to put them in trauma,” he said.

And that’s pretty much what happened in 2025. People were told with no notice to return to offices with no space for them. Folks you’d helped recruit and persuade to join the government were fired, or left because they had good reason to think they would be. Staff were called into meetings with DOGE representatives who wouldn’t turn their cameras on and asked to justify their jobs and offer judgment on their coworkers. Hit pieces were written calling out specific employees in ways that appeared coordinated with the people making personnel cuts.

And all of that was just a small subset of the things that happened to people I know personally.

So when you show up as a new Tech Force fellow, there’s going to be additional skepticism besides just the normal kind. People just watched their colleagues with a lot more experience in your fields get pushed out or fired. And here you come, early career, only here for a short time, and hired to do...what exactly?

Also, it's not as if everything has settled down: you'll either be subject to ongoing actions or you'll be insulated from them. Either is a problem.

And there’s another layer of instability: contract actions. Last year saw widespread cancellations of federal contracts and delays in awards and extensions. At Homeland Security, contract actions of $100,000 or above still require sign-off from the Secretary. You will not be directly affected, but the contractors you work side-by-side with and depend on? They may be.

Finally, if the starting assumption from the agency leadership that brought you in—and potentially even from your managers brought in from industry to manage you—is that the government doesn’t work because civil servants are stupid or lazy, you’re starting from a terrible position. You need to build trust with people to get things done. That’s hard when the message from above is contempt and when you’re under pressure to put out a narrative, both in the agency and externally, that minimizes the contributions of career civil servants.

Here’s what to look for when you apply to reduce the likelihood that you’re walking into a situation like that.

What to Look for If You Apply

When you’re interviewing, look to see that someone running this program at your agency has an actual theory of change about building things in government and a history of successfully doing so. Get names and find out what their backgrounds are.

Ask questions like:

• What conversations have happened with agency leadership? How have they prepared for us?

• Have projects been scoped yet? What does that process look like?

• Regarding those projects, what’s been tried before at this agency and what are we doing differently? Who is currently working on them and how are we going to work with them?

• What infrastructure exists for us? Is there a git server? Are we going to be allowed to use AI coding tools, and if so, what?

• What’s the process for user research? Are we hiring or working with UX researchers or designers? How will we talk to the people who use these systems?

Do not just assume this has been figured out. I’m looking at OPM’s long list of participating agencies and I would be surprised if when that was released, they’d had extensive conversations with each one to assess that they’re going to make productive use of you.

I’m sure agency leadership said “by all means, send us technical people”, but did it go beyond that, or will it before they start hiring? And if the agencies don’t have good plans, will OPM back down from their very ambitious, very public hiring goals?

Normally, this could get deferred slightly, and they could work things out after you show up. But these are one and two-year terms. If there aren’t projects scoped by the time you get there, you could easily burn through a big chunk of your term waiting for something useful to do—or discover that the people you’re working for don’t know how to find that.

My prediction: Experiences will vary wildly agency to agency. Some agencies will do this well and others will not.

What You’re Signing Up For

A lot of tech within the federal government is bad, and the challenges to improvement aren’t inherently insurmountable. Even last year, good things were built.

And I don’t think the premise or structure of these programs is wrong. Centralized hiring is absolutely the way to go, because getting hiring announcements out and screening resumes is a huge lift that most agencies don’t have the capacity to do. (Does OPM? They can staff up to do it, and it’s easier to solve this for one agency than for all of them.)

I also think it’s good that they’re making these positions explicitly short-term. No one should be joining the government now planning on having job security. And it would not surprise me if there are problems that get solved faster than they would have previously—for instance, I think if you join, you’ll quickly get the computers you need with the software you need, which isn’t nothing.

Finally, making these programs high-profile, working with industry, targeting folks early enough in their career that they can be paid very competitively on the GS pay scale—this is all fine.

But if your goal with joining these programs is solving actual problems for actual people—and I hope it is, if you’re thinking of applying—there are also significant risks here, and reasons to be skeptical. The usual structural challenges of government tech still exist, compounded by everything else that’s happened.

If you apply, go in with your eyes open. Ask hard questions. Look into the backgrounds of folks running these programs at the agency you’d be assigned to. Don’t assume it’s going to work out. And understand that you’re not just signing up for a technical challenge; you’re signing up for a political and organizational one too.

I held my last real data science job in 2023. Equipped with what exists now—I use Claude Code, but there are other similar tools like Codex and Cursor—I could have done that 40-hour-a-week job in under ten hours. And that’s with the meetings and the emails.

This isn’t a sustainable equilibrium, and at some point, employers are going to figure it out.

What I Did as a Data Scientist

Here are some of the kinds of tasks I did in that role as a data scientist:

1. Talking to people: briefing clients, managing small projects, getting information from subject matter experts who knew the data.

2. Creating documents: PowerPoints, short papers, code documentation.

3. Statistics: deciding what machine learning model to use, or how to solve problems involving distance minimization, or how to reduce the rate of flagging potential errors without causing more real errors to slip through.

4. Coding: writing Python and SQL to clean data and build reports – filtering, merging, aggregating. Figuring out how to get my code to post results in Teams or send emails. Making my code run faster on very large data sets. Figuring out how some API worked – for geocoding, or getting user stats for our dashboards – and using it. This was by far the biggest part of what I did.

By GPT-4, LLMs were already useful for coding and somewhat useful for the kind of statistics I was doing. At this point, they completely smoke me at this kind of work.

Here’s what I mean by this: Claude Code can find documentation better and faster than I can. It can synthesize information from multiple sources better and faster. It can write code, run it, read the error message, and fix the problem in the time it would take me to find a relevant Stack Overflow page and read a few responses. It can look at a problem I’ve never seen before and guess, usually correctly, how it’s supposed to work based on patterns it’s seen in a million other codebases.*

This is no longer just about taking a well-described query and generating the code for it.

For instance, in 2023, I spent a few weeks trying to figure out the best way to do a specific geographical analysis, and I only got there because an engineer made a great suggestion at a brown bag I was giving. Now, that could have been one conversation with an LLM, and I would have gotten there much faster.

Or, there was the time when I was trying to figure out the best model to use for a machine learning problem–I had some ideas going in, but ultimately I got the answer from figuring out what outcomes I cared about and then systematically testing out different types of models to see how they performed on those outcomes. Claude Code could have done that with minimal input.

A lot of what separates pretty good data scientists from bad ones is tolerance for tedium. Willingness to read the documentation. Patience to debug. The discipline to write clean code instead of the spaghetti you might be able to get away with. Maybe you can’t become a great data scientist on the basis of that—I don’t think I was one— but being willing to just go in there and get the thing done no matter how annoying it was, and even better, get it done in a way that’s repeatable and clean and that someone else could understand—a lot of people can’t or don’t do that.

If you can automate that—and you can—then the gap between a good data scientist and a bad one gets a lot smaller. This job, as it was done, as I was doing it, becomes not just able to be done faster, but able to be done well by someone who previously wouldn’t have done a good job. It becomes far less skilled. The curiosity piece still matters—asking the right questions, really digging into the data, not believing your too-good-to-be-true results until you’ve sufficiently tried to disprove them—but even there, the barrier is lower than it was.

The Conversation I Keep Having

When I talk about this—for instance, in the context of how important it is to get government employees coding assistant tools, or how useful this has been for me—I get a couple kinds of pushback.

The first is “but it hallucinates”—as if that ends the conversation. Yes, LLMs make things up. So I deal with that. When I’m doing research, I ask for citations, including links and quotes, and I check them. For code, I write a lot more tests than I used to so I can verify the code is doing what I think it’s doing—which, like all the coding I do, is now much easier. There’s a learning curve, but this is not an unsolvable problem. And when people tell me LLMs aren’t reliable enough, I want to ask: compared to what? Have you seen the code humans write?

The second isn’t really an objection—it’s more that any conversation about LLMs being useful gets pulled into everything else. Doomerism. Environmental impact. Is AGI coming? Is there a bubble? (Maybe, but there was an internet bubble and we didn’t all stop using the Internet.) How awful are these tech bros?

I’m not dismissing all of those as unimportant, but they’re different questions from the one I’m trying to answer here, which is: what can these tools do right now, and what does that mean for work like mine and workers like me?

Most of the informed skepticism I hear comes from software engineers talking about enterprise software—large codebases, complex dependencies, production systems. That’s fair, or at least, I’m not the right person to argue this point because that’s not my domain.

But data science code isn’t enterprise software. We’re talking about smaller projects, often just one person. It’s not software that ships to users—it’s pipelines and reports that typically run on your machine or in a cloud environment your employer controls.

Because of that, the question of huge disruption in my field isn’t “are these tools perfect for all coding?” It’s “could someone much less skilled, armed with Claude Code, do better than the person currently in this job?” And having been a data scientist and worked with data scientists, I think the answer is often yes.

If you’re skeptical of this, I’d say: talk to your developer friends who are seriously engaging with these tools. The ones who are using Claude Code or Cursor all day, every day. Ask them what their work looks like now versus two years ago. Ask them if they’d go back. And try the latest models and tools yourself for research and data analysis. Treat it like any new technology, where you assume you’ll need to put some time into learning how to use it.

Where I Am Now

I was recently assisting a colleague who was newer to LLMs. They were trying to get a program to work and asking ChatGPT to help. They told it the software had errored, but they didn’t think to paste in the error message. It reminded me: I’ve been talking to these things constantly for years. Most people haven’t. The steps that have become automatic for me—what to ask, what context to give, how to check the output—none of that is obvious yet. And neither is it obvious what they can do.

That’s the gap. Not capability—we’re past that, at least in data science. It’s that most people haven’t learned how to use these tools, and most organizations haven’t figured out how to integrate them. The lack of labor market disruption so far is a matter of implementation, not capability. That’s what’s buying time.

Years later, with a lot more experience, I’m still currently not finding work through traditional application processes. When I left my job with the federal government in May, I decided it was going to be a better use of my time to build things than to send out resumes. I sent a few applications—Anthropic, still waiting to hear from you!—but by and large, it wasn’t something I even tried.

The short story: it worked out. I wound up with contract work I wanted, and it came entirely from a mix of side projects, networking via side projects, and previous coworkers. The mechanism is different now than it was in grad school—I have a network, and I can’t overstate how much that helped—but the pattern is the same as for that earlier job: make things, tell people.

And for that first job, I got lucky—I happened to be doing research someone was interested in. But after that, side projects were consistently what made the difference. Side projects let you make more of your own luck. You get to choose what you work on and what story you’re able to tell, instead of hoping your coursework or current job happens to match what someone wants and what you want to be doing. Side projects help in two ways—they can lead to opportunities through networking, but they also make your traditional applications much stronger when you do just apply.

I write this because college students and people making career changes get in touch with me asking for advice, and I find myself being weirdly reluctant to tell anyone they should spend their free time on side projects, even though it’s what I do. And in a strong hiring market, traditional applications with coursework and internships work fine. But this is not that market. Managers are looking at hundreds or thousands of interchangeable resumes with the same coursework and the same ‘proficient in Python and SQL.’ You can’t retroactively change the school you went to or what your grades were, but you can build interesting, new things.

Why This Works

Side projects have a few built-in advantages that can be very hard to get otherwise, depending on your background:

1. Problem definition and solving. They let you figure out what problem is worth solving and how to solve it—as opposed to coursework, where you’re generally assigned problems that someone has figured out are solvable already. In this sense, this mimics work a lot more closely.

2. Clear individual contribution. They let you build something by yourself, so people can tell it was you, and in a way that you can make fully public—as opposed to work projects you’re not allowed to share and where it’s hard to tell what your role was.

3. Demonstrating genuine interest. Side projects are strong signal that you care about that topic.

4. Showing agency. You trusted yourself that something was worth doing and that you could do it. And then you did it.

These reasons also point to why some side projects work better than others. If you’re analyzing a data set that’s been done to death already, it’s harder to tell what you specifically contributed vs. took from someone else in terms of both the approach and the execution. It also doesn’t give you the chance to pick something that’s unique to you and your interests. This doesn’t mean you shouldn’t do that kind of side project—maybe you’re just doing this to learn a particular skill, in which case it doesn’t matter what anyone else thinks of it. But if you’re avoiding doing your own thing because you think every project has been done already and so you can’t possibly do anything new, this is false, and you absolutely can.

And all of these advantages matter whether you’re networking your way into opportunities or strengthening a traditional application.

How to Do It

Finding ideas

I spend a lot of time in places where people are talking about problems they have or topics they’re curious about. Data Science DC meetups, Signal chats where people talk about government tech, LinkedIn. When something really catches my attention, I start trying to figure out if there’s something small I can build to answer a question or solve a problem.

If you’re not in these spaces yet, start going to local tech meetups, free conferences, hackathons—whatever exists near you2. Listen to the talks. Pay attention to what problems people mention and what tools they use. For instance, you’ll see that GitHub isn’t optional even if your classes didn’t teach you how to use it properly. R is still used, but more frequently on teams where Python is as well. Polars and DuckDB are increasingly common. A lot of work involves LLMs. The tools people are using in their work and giving talks about are the tools you should be using in your side projects3.

A recent project: I’d been hanging out in spaces where people talk about government open source, and I’d also recently worked with the GitHub API to pull commit data. I added those two things up and spent a couple evenings making a site that pulls data from government GitHub accounts to see what agencies are making code public. It wasn’t a big idea at all, it was just noticing two things I already thought about and connecting them.

That’s the pattern. Someone posts on LinkedIn. Or mentions something at a meetup. You read an article and think “wait, is that actually true?” And instead of just wondering about it, you go build something.

Telling people about it

Making the thing is only half of it. The other half is telling people.

I go back to the same spaces where I found ideas and share what I built. But also, I’ve messaged strangers to offer them data, and sometimes they say yes. I write blog posts so that later, when I’m talking to someone about that topic, I have something to send them.

When you show up in these spaces—especially if you’re early in your career—lead with what you’re working on or what you’re curious about, not with “I’m looking for a job”. “I was messing around with MCP servers and here’s what I built” is memorable and gives a sense of your interests and capabilities. “I’m looking for a data analyst role” does not.

And public events are just the first step. A lot of opportunities happen in private groups and conversations, but you’ll never meet the people who might invite you unless you engage with the public spaces first.

Telling people about it also includes putting it on your resume, along with links.

What You Get Out of This

I’m not saying this is how job hunting should work. And if you can find the job you want without doing any of this—the side projects or the networking—that’s great. I’m writing this because I don’t reliably know how to do that, so I do this instead, and I’m happy with how it’s worked out for me over the years.

Part of why I’m happy with it is that building things I find interesting is a lot less grim than sending out applications. It doesn’t feel like wasted time. And each project builds your skills—both the technical ones, but also the pattern of seeing a problem and making something that responds to it.

This was new for me. I’m coming from data science, and I’d built reports which updated regularly and sent emails, but most of them ran inside Databricks—a system someone else maintained, pulling from data tables someone else updated—and they never had their own websites. This was the first time I was building something like this all the way through by myself. At the same time, these were also side projects, and I didn’t want to spend a lot of money on them.

Initially, I just ran the update scripts manually when I remembered. Then I got tired of that and wanted it all online. Then I realized I needed tests because things were breaking and I wouldn’t figure it out until I checked the website—or, worse, someone else told me. My setup came together piece by piece as I needed it.

Here’s where I landed.

GitHub Large File Storage (LFS)

The problem: I had too much data to commit to GitHub the normal way: GitHub won’t let you commit files over 100MB, and my Parquet files with years of job listings were bigger than that. But I wasn’t ready to set up a database—that felt like overkill.

How it works: GitHub LFS stores large files separately and keeps pointers in Git. You tell Git which file types to track with LFS, and it handles the rest.

My implementation: All my big data files go into LFS. When my GitHub Actions workflow runs—more on that later—it pulls those files to process them. This way, the data stays versioned and can live on GitHub.

When to use it: When you’re accumulating data over time and need version control but aren’t ready for a database. It costs $5/month for 50GB of storage and bandwidth; some months I pay more, when people are cloning my repos.

GitHub Actions

The problem: I needed my pipeline to run automatically. Running scripts manually on my laptop when I remembered wasn’t sustainable.

How it works: GitHub Actions runs code on GitHub’s servers based on schedules you set. For instance, you write a workflow file that says “run this Python script every day at 4 AM.” Actions spins up a virtual machine, checks out your code, installs dependencies, runs your scripts, and can push results back to your repo or even create and resolve pull requests to other branches.

My implementation: I have several actions that run daily and a couple that are more infrequent. They check out the repo with LFS files, install dependencies, run the data collection scripts, then run tests.

What typically breaks when I’m first setting these up are dependencies and permissions.

For dependencies, you spell out everything. Not just Python packages, but if you’re web scraping with Playwright, you need Playwright AND the browser AND system dependencies. If you’re calling APIs, you need API keys in GitHub Secrets. Everything your local environment has, Actions needs explicitly declared.

For permissions, the workflow also needs explicit grants to write to your repo or create pull requests. Without these, your workflow fails.

Writing tests: I test what I’m worried might break AND what would be bad if it broke. For instance, my tests check that data files exist, aren’t empty, and have required columns. I don’t have tests for things like my JavaScript filters, because I’ve decided that what I’m most concerned about is data integrity, and I wasn’t quite ready to also add website testing as well.

When something breaks in an unexpected way, add a test for it. When I accidentally deleted some of my data, I added a test that checks for every job listing that was in the files last time this action ran. If a test fails, the workflow stops.

When to use it: When you need scheduled, automated work without running servers. Public repos get unlimited Action minutes. Private repos get 2,000 free minutes per month. Each job has a 6-hour timeout.

Multiple Branches

The problem: I needed stage/prod separation without servers. But how do you test changes when everything happens in the same repository and you have no budget?

How it works: Use branches as environments. Your pipeline pushes updates to a staging branch. Tests run automatically. Only if tests pass does anything merge to main, then push to a production branch that your website deploys from. (Your exact branch structure may differ.)

My implementation: Each day, my workflow creates a fresh staging branch from main, runs the pipeline, and commits results. Tests run. If they pass, the workflow auto-creates a pull request to main and auto-merges it. Finally, it pushes to my production branch, which my website deploys.

The flow: staging → tests → auto-merge to main → push to production → website deploys.

When to use it: When you need to test before you deploy, but you’re not running servers. The branch-based approach gives you stage/prod separation without infrastructure costs.

Netlify

The problem: I needed to host websites that updated automatically, stayed fast, and didn’t cost money.

How it works: Netlify hosts static sites—just HTML, CSS, and JavaScript files, no backend fetching from a database or running Python. You can set it up so that Netlify watches a specific GitHub branch, and whenever you push to that branch, it automatically deploys your site.

My implementation: Netlify monitors only my production branch. When GitHub Actions pushes there (after tests pass), Netlify builds the site and then deploys. I stay within the free tier.

Static sites are simpler, which means less to break. But there’s a constraint: you can’t load huge amounts of data all at once. Load a 50MB JSON file on page load and it will crawl—or fail outright.

The data split trick: When I have a lot of raw data, I split it into multiple files—one for each filter value. For USAJobs, that means I create separate files with the raw jobs data for each federal department. To see raw data, users must filter by department first, which triggers loading just that department’s file. Each file is small, so it loads quickly.

When to use it: When you want simple deployment and don’t need a backend. Netlify’s free tier: 100GB bandwidth per month and 300 build minutes.

D3 and DataTables

The problem: With my sites, I needed interactivity—users filtering data, searching tables, clicking on visualizations. Python can’t run in the browser. JavaScript is what runs client-side, which means it’s what you need for static sites to do anything interactive.

The secondary benefit: I also wanted very specific visualizations. For static graphs, Python has great options. But for interactive web visualizations—bubble maps, filterable charts—I wasn’t happy with what I was able to make.

How it works: JavaScript runs in the browser. D3 is a JavaScript library for data-driven visualizations. DataTables is a library that adds sorting, filtering, and search to HTML tables. The ecosystem is huge: if you need a specific chart or interaction, someone has probably built a component you can use.

My implementation: I use DataTables for tables—it gives me instant search and filter functionality without building it myself. For graphics, I use D3. I still don’t know JavaScript—I use Claude Code to write it1. But I knew enough from Python to recognize patterns like “I should abstract this out because I keep using it” or “use DataTables for search instead of building your own.”

When to use it: When you’re building sites and need interactivity and specific data visualizations. Use existing components instead of building everything yourself.

What This Costs

Almost everything here is free. GitHub Actions is free for public repos. Netlify’s free tier covers all the hosting I need. The only real cost is GitHub LFS.

The more significant cost was my time. Getting permissions right, understanding when tests needed to run, realizing I needed branch-based environments—that took a lot of trial and error. But as I learned, the incremental time it took for each new site or action got lower.

Now it runs without me. The pipeline updates daily, tests catch the problems that I care about the most, bad data never reaches production, and the website stays current.

Last week, one explained to me their data pipeline—which was a very standard version of this—AND another person told me about a tool their organization deployed that lets users do this without any coding.

Point made! I think I can stop writing about this pattern. So this is, instead, about a handful of other text-processing tricks I use LLMs for as part of data pipelines.

Most of these you could fold into your unstructured text pipeline; a couple work alongside tools like regular expressions or semantic clustering. They’re all variations on the same principle: if it’s a repetitive text processing task that’s easy-to-explain and doesn’t require a ton of judgement, you can write that into your data pipeline and have an LLM do it for you.

1. Column Discovery

The problem: Your pipeline processes data that’s fairly standardized, but column names and formats may vary between datasets.

The trick: Describe what you’re looking for to an LLM, give it information about the dataset (column names, the first few rows, information on how many values are missing and what the most common values are—basically, whatever you’d give a human analyst), and have it populate a JSON file mapping your standard field names to what this specific dataset calls them. Use that JSON in the rest of your pipeline.

Real example: In my comment analyzer for regulations.gov bulk data, column names are usually the same, but not always. Because of this, I have an automated step that figures out the mapping: in this case, I just give it the first five rows, because that seems to work. The alternative would be maintaining an exhaustive list of every variation and either hard-coding them all or using pattern matching to anticipate future changes, or else doing this manually each time.

When to use it: When you have semi-standardized data with naming variations. You can run this with user validation (showing them the mapping and letting them confirm or correct it) or without—but if it’s without, I think you should build in some automated validation that you found the right columns (like checking that the data types make sense), which will still be easier than trying to do column identification without the LLM.

2. Theme Discovery and Prompt Writing

The problem: You need to classify text, but the relevant classification buckets you’re looking for, like what themes are present in the data, are different for each dataset.

The trick: Give the LLM a prompt describing what you’re looking for, plus a random sample of your text. It synthesizes the themes faster than you could, determining what buckets to classify the text into and writing the exact prompts. Your pipeline then takes this output and uses it to populate both the prompt text and the Pydantic class structure, so the LLM can only choose from the themes it identified, not make up new ones.

Real example: For my public comment analysis, the arguments people make of course vary by regulation. I let the LLM read a sample, identify the main arguments according to a very specific set of rules, and write the prompts and values that will be used to classify all the comments.

When to use it: When classification categories aren’t predetermined—this could be themes in comments, types of errors in logs, categories of customer feedback, or any other situation where the buckets themselves need to be discovered from the data. Depending on the use case, you might let users validate the LLM’s choices about buckets or prompts if they want something different.

3. Hybrid LLM/Regular Expressions

The problem: You have too much data to send each document to an LLM, maybe because of cost or processing time, but you still want to extract things that would be tedious to hard-code manually.

The trick: Sample from your documents and use an LLM to extract what you’re looking for. Deduplicate the results, optionally set a threshold (like “appears in at least 2% of documents”), then use string matching or regular expressions to find those items in your full dataset.

Real example: Processing resumes at scale—extracting programming languages, frameworks, tools. The LLM tells you what’s in the data without you spending hours building and maintaining lists. You can also model the likelihood that additional sampling would find new items, and monitor over time to catch new terms appearing in resumes.

When to use it: This works for terms that aren’t one-offs in the documents you’re analyzing. For instance, programming languages like “Python” appear the same way across many resumes, so once you’ve identified them with the LLM, you can use pattern matching to find them everywhere else. But it won’t work as well for things with too many possible values—like college names or job titles. So for certain types of entities (names, places, organizations), you’d need a different approach—like a smaller model specifically designed for entity recognition, like GLiNER, possibly fine-tuned with labeled examples you got from the LLM.

4. Matching with LLMs

The problem: You need to match items between two datasets, but exact matching won’t work and lexical matching, or comparing how similar strings are based on the actual characters or words, isn’t smart enough.

The trick: Do a first pass with lexical matching and heuristics like date proximity or other filtering rules to get close matches. For anything below your confidence threshold, pass the top candidates to an LLM to make the final call.

Real example: In my congressional hearing matcher, I’m matching YouTube videos to their Congress.gov records. For instance, “Health Subcommittee Markup of 23 Bills” versus “23 Pieces of Legislation” use completely different words but refer to the same event. Lexical matching and date filtering narrow it down, then the LLM adjudicates unclear cases via a more nuanced matching process.

When to use it: When you’re matching things where the relationship isn’t captured by business rules. This is a variation of the standard text pipeline, but I’m including it because the possible outputs do vary for each ‘document’: your lexical/business rule process limits the options that the LLM can return.

5. Naming Clusters with LLMs

The problem: You’ve done semantic clustering on your text data, but the clusters are just numbers or unlabeled groups.

The trick: After clustering, pass representative examples and the list of the most common or most-distinguishing words or phrases from each cluster to an LLM and have it generate human-readable labels that capture what the cluster is about.

Real example: In my Schedule F comment analysis, I clustered semantically similar comments together. Then I pass the LLM samples and common words from each cluster and it describes them. This is faster than reading through clusters myself, and especially useful when your clusters may change over time with new data.

When to use it: After any clustering or grouping where you need human-interpretable labels. The LLM is doing synthesis work you’d otherwise do manually.

The Pattern

These tricks all follow the same logic: they’re tasks you can give clear instructions for, and they’re not inherently complicated—but they’d still take you time to do manually. An LLM can do them quickly and frequently well enough, and you can add in a human validation step if you want.

I’m especially interested in LLM/non-LLM hybrid patterns like #3. The hybrid approach splits the work: the LLM samples your documents and extracts terms, then you use those terms with regular expressions to search your full dataset. You get the LLM’s ability to identify what’s actually in your data without manually building lists, combined with speed and transparency for the bulk processing.

This structure also builds in error tolerance. If your LLM sample occasionally misses “Ruby,” you’ll probably see it in the next sample batch. You can also check completeness by comparing your found terms with your samples—and because you can model the likelihood of finding new terms by continuing to sample more, you can also quantify your sampling coverage. It’s also an example of using the LLM to label data for use with smaller models—similar to using LLMs to extract entities and use those labels to test or fine-tune your entity model, except in this case your “model” is just the presence or absence of each word.

Working with different kinds of text data pipelines makes these patterns easier to see. The more you do it, the faster you recognize when you need a new one.

In the early days of ChatGPT, I had projects where half my prompt was begging the model to ONLY SAY YES OR NO or GIVE THIS TO ME AS A LIST. Sometimes it worked. Often it didn’t, and I was stuck cleaning up the output. The model would give me a paragraph about why something might be yes or no, when all I needed was the answer.

Then the Marvin library came out and fixed this for OpenAI models: instead of begging or lecturing, you could just describe the behavior of the LLM and your output would come back formatted correctly. I heard Adam Azzam talk about it at a meetup, and that weekend I rebuilt my project. Suddenly, I could define exactly what structure I wanted back, and the model would comply.

Now this general pattern—where you describe the behavior you want—is built into many LLMs natively. Here’s what it looks like, using LiteLLM, a library that provides a consistent interface across different LLM providers. In the following example, we use an LLM to characterize text as support, oppose, or neutral, and we get a list of the main arguments it's making.

from pydantic import BaseModel, Field
from enum import Enum
import litellm
import json

litellm.api_key=your_api_key # Replace this with your actual key!

class Stance(str, Enum):
    SUPPORT = “support”
    OPPOSE = “oppose”
    NEUTRAL = “neutral”

class CommentAnalysis(BaseModel):
    stance: Stance = Field(
        description=”’support’ if favoring proposal, ‘oppose’ if against, ‘neutral’ if no clear position”
    )
    key_points: list[str] = Field(
        description=”Main arguments from the comment”
    )

comment_text = “”“I can’t believe anyone is considering this.
It will hurt small businesses and increase costs for consumers.
Read an economics textbook, losers.”“”

response = litellm.completion(
    model=”gpt-4o-mini”,
    messages=[
        {”role”: “system”, “content”: “Analyze public comments on regulations. Determine stance (support/oppose/neutral) and extract key points.”},
        {”role”: “user”, “content”: f”Analyze this comment: {comment_text}”}
    ],
    response_format=CommentAnalysis,
    temperature=0.0
)

That’s the core of it. The model returns the structure you defined, with stance limited to your three options, and with a list of arguments.

Important caveat: This is just about getting your output structured correctly—the content might still be wrong!

Why This Works So Well

Three components make this pattern work:

• Pydantic declares fields, like Stance, and types, like string or list

• Enums constrain choices (yes/no, support/oppose/neutral, high/medium/low)

• LiteLLM provides a portable interface so you aren’t rewriting provider-or model-specific glue to call the LLM.

A Real Example: Matching Congressional Hearings to YouTube

For the Congressional hackathon recently, I needed to match YouTube videos of hearings (the first data set) with their official Congress.gov records (the second data set). The problem is that, while we have the titles and dates in both, they frequently don’t match exactly across data sets.

For instance, “Health Subcommittee Markup of 23 Bills” and “23 Pieces of Legislation” use almost entirely different words, but refer to the same event.

My approach:

1. First pass: Heuristics. Date matching and lexical title similarity (comparing the actual words, not the meaning)

2. Second pass: LLM adjudication. For uncertain cases, hand the top matches to an LLM and ask it to decide.

For the structured output, I asked for a match ID number (or None, if there’s no good match), a confidence level, and "reasoning”, or an explanation of the answer.

For this project, I didn’t actually use the confidence or reasoning fields for anything. The confidence score isn’t calibrated—it hasn’t been trained against labeled examples, so it’s just a guess. And the reasoning field isn’t the model’s actual decision process, but a post-hoc justification. Both can still be handy for debugging or flagging cases to review.

Where I’ve Used This

I’ve used variations of this pattern for:

• Analyzing public comments: First, discovering what themes people discuss, then classifying each comment using those themes

• Processing research papers: Extracting summaries, key contributions, methods used

• Evaluating synthetic data: Checking if LLM-generated content meets specific criteria

Same building block, different structure.

When It Doesn’t Work

I had one dataset where this pattern just wouldn’t work with certain providers: I wasn't able to consistently get back the structure I wanted. It worked fine with one model but failed with others. I tried multiple different tools for getting structured output across providers until I gave up and said “well, I guess I’m just using this with Ollama,” which is a tool for running LLMs locally on your own machine.

But since then I’ve used this on probably 200,000 documents across a dozen projects, and that’s the only failure I’ve gotten. Your mileage will vary with provider and data weirdness. If you see failures—where models keep breaking your schema despite using structured output—you can try heavier-duty tools.

For instance, there’s Instructor, which does both output validation (as opposed to just passing the class to the model) and automatic retries if there are failure. I don’t use it because I haven’t needed it, but it’s out there if you do.

Also, not every model supports structured output. LiteLLM provides an API endpoint for checking whether a particular model supports it, which is helpful when you’re trying different models.

The Bigger Pattern

This structured output technique is part of a bigger pattern: most LLM applications I see aren’t chatbots or RAG systems. They’re document processing pipelines that ask the same questions of many documents.

When you need to process a pile of text and extract the same information from each one, this pattern is a good one. Same questions, many documents, structured answers.

The code for my YouTube matcher is on GitHub if you want to see the full implementation. But honestly, the pattern above is most of what you need to know. Define your structure, constrain your choices, and let the model do the work.

Around that time, I decided to try Claude Code, the extension for Claude that runs in your development environment. I’d been using LLMs for coding since GPT-3.5, but apart from a short-lived experiment with GitHub Copilot, my workflow hadn’t changed: grab code from ChatGPT or Claude’s GUI, test it, iterate, paste back in. But I liked Claude, and suddenly I had the time to experiment.

There were some bumps, and it wasn’t good at everything, but going from GUI to Claude Code was as big a leap as when I first started using LLMs for coding. It let me build out what I wanted to build.

What I Built

What took shape this summer was a focus on making federal data more accessible. There’s a ton of really useful federal data out there that tells a lot of stories, but it’s not always the easiest to work with.

My biggest projects were with FedScope data, the official source on federal civil servants, and USAJobs data, which contains most federal job listings. I also worked with budget data—how much each agency is appropriated by Congress, and how much they’re obligating to spend—plus some public comment analysis and a demo for the Congressional hackathon that matches committee hearings with their YouTube videos.

Most of this work was data engineering—shifting data between locations and formats, like APIs, zipped text files, or Excel workbooks—and cleaning it up. Some was light web development: sites that might be interactive, but where everything runs in the browser, not on a back end or through a database. Claude Code helped in various ways:

• When scraping questionnaires for federal job listings, switching from Selenium to Playwright took under half an hour and significantly sped up my scraping.

• For matching the congressional hearings to YouTube videos, Claude suggested a package for getting YouTube metadata so I could stop using the very rate-limited YouTube API, and then implemented it.

• Across all of this, Claude hardened pipelines with logging, error handling, and small things like caffeinate to keep my laptop awake overnight.

Not everything landed. I tried it for slides and data analysis, with much less success. It can draft slides, but I rewrite all of them. And its instincts for data visualization when asked to analyze a data set or even answer specific questions are terrible: overcomplicated charts, unreadable outputs, zero intuition for what the data is about.

Still, the sheer volume of what I was able to build—quickly, and in new domains—was different from anything I’d done before.

Bubble Charts and Bubble Math

Here’s one example of attempting something new. I needed to visualize budget data: agency, amount of funds appropriated, and the percentage of those appropriated funds which had been obligated.

Pre-LLMs, I’d have reached for the Python plotly package. If I was feeling ambitious, I might have tried Dash, the tool for making plotly figures into interactive dashboards. With LLMs, I might have worked in JavaScript, but I would still have felt constrained. With Claude Code, I started from a different premise and therefore a different question: I can build whatever visualization I want. What’s the best way to show the data?

That led to an interactive D3 bubble chart with multiple aggregation levels and filters responsive to user selections—built in one evening. And I don’t know JavaScript even a little. Initially, I also tried a two-color scheme for the inside and outside of the bubble using different colors for agency vs. subagency, but it was too busy, so I scrapped it. The luxury was being able to try and discard experiments so quickly.

A week later, I noticed some bubbles weren’t placed correctly on the x-axis. Bubble math is its own rabbit hole: balancing accurate placement on the x-axis against minimizing overlap. I ended up writing a grid search optimization for bubble parameters. It’s not good yet, and I may never fix it—but I wouldn’t even have thought to try before.

These are the problems I want to work on: showing complicated data clearly, not wrestling with syntax.

What Makes This Different

Claude Code lives in your environment. It sees everything—file structure, error messages, git state—and can act across it. That volume of context dwarfs what I’d copy-paste into a GUI.

Take minor refactoring: Claude can rename files, create folders, move things around, and update paths. I’d never bother doing that with an LLM if I had to copy and paste everything in. Or git errors: I used to debug them via Google and self-loathing. Now Claude fixes them, and sets up GitHub Actions I wouldn’t have attempted before. Suddenly, improvements are cheap.

Another difference is that programming almost feels social. It’s more like coding with someone—like a rubber duck that talks back. So part of why I built more was that I spent more time doing it. I’ve always had the “I’ll be done debugging this soon” problem, where you look up and it’s an hour later. Claude Code made it worse because it made coding more engaging.

This heat map with my git commits—not all of them, but most—is an example of something I never would have spun up without Claude Code. But experimenting with visualizations of git commits for a blog post becomes totally reasonable when it takes 15 minutes.

Where It Fails

Like LLMs in general, Claude Code is strong on well-documented patterns—D3 charts, git parsing, GitHub Actions—but weak on data intuition and messy, real-world problems. I spent a day guiding it through a slightly novel dataset I already knew the answer to, just to see if it could reason its way there. It never got close, and it kept insisting it had solved it.

Budget data was another case: I went in circles until I realized the problem was that my obligations data had real dates, while outlays—the actual spending—was cumulative, a snapshot from whenever I pulled it. Claude Code was never going to figure that out.

And even when it’s on the right track, it creates its own headaches: spawning new scripts instead of editing existing ones, sprinkling in try-catch blocks and functionality I never asked for and didn’t want.

The hardest failures came from deploying. I’d never launched websites before—let alone ones that updated automatically with new data—and every small break felt urgent, something I had to fix immediately no matter the time or what else I was doing. The problem was that building faster than I was learning how to test or maintain. Some of that was the speed Claude Code enabled; some of it was just side projects turning into tools. Eventually, I set up a more professional workflow for my sites with automated data updates: separate branches, and no more pushing to production without passing tests. Claude Code helped me get there, too.

What This Means

I’ve never liked low-code or no-code tools. They put you in a box—you can only build what someone else already implemented. Claude Code is the opposite. It opens boxes. It let me try frameworks, languages, and approaches I never would have touched before.

People sometimes cite the METR study to me, showing Claude Code didn’t help software developers much. I don’t know why I had such a different experience. My best guess is that it’s because in that study, developers were working on codebases they already knew well. For me, the value was mostly not that: Claude let me try things I otherwise wouldn’t have. Also, I’m not a software developer.

I am trying to retrain myself to slow down, look at the data more, and reinvest saved time in testing. It’s hard.

But even so: I’m grateful. I had this summer to choose what to build and put in the hours to do it. Now people email me asking for data. I start a new job soon, and I don’t know how I’ll keep it going. But that’s a good problem to have—and Claude Code is part of why I have it.

This post covers how to programmatically access that data. Sometimes it’s easy, sometimes it’s not. Want to know how many jobs an agency has listed by occupational code right now? Sign up for an API key, make one request, and you're done. Want to know the qualifications for every job listed in the last year? You'll spend a few days scraping data, and then you’ll have to figure out how to parse it correctly.

USAJobs is job listing data, not hiring data. A single posting might result in zero hires, one hire, or multiple hires. There's a Total Positions field that shows how many people the agency wants to hire, but it might have a value of “few” or “many”. There's also a Position Offering Status field that tracks whether a job is open, closed, or has made selections. But "selections made" just means the agency picked someone—not how many, and not that those people said yes or started working. If you need data on who was hired (not just what was posted), you'll want FedScope, which I'll cover later.

Three Ways to Get the Data

There are three general ways to get the data: the Historical Jobs API, the Current Jobs API, and web scraping. You can also scrape most of the assessment questionnaires.

1. Historical Jobs API: Goes Back Far, Limited Fields

The Historical Jobs API has fields like job title, department and agency, opening and closing date, occupational series, location, pay grade, and salary range. What it doesn’t have is any of the rich unstructured text data in the listing, like duties or qualifications, which are in the Current Jobs API.

The fields here are a subset of what’s in the Current Jobs API, except for the Position Offering Status field.1

You can pull comprehensive data by making an API call for each day. No API key is needed.

Data coverage note: While there are some earlier postings, the data is extremely sparse until March 2017.

2. Current Jobs API: Easy and Detailed

The Current Jobs API gives you active job postings with the structured data fields, plus all the rich text content: duties, education requirements, qualification requirements, how to apply instructions, etc.

The first catch: 10,000 result cap per query, and you can only query the previous x days (not specific dates). To get everything currently posted, slice queries by agency or occupation to stay under the 10,000 limit.

The other catch: once the job is no longer current, it won’t be in that API anymore and if you want the good text data, you’re going to have to scrape it.

To use the API, you’ll need an API key from developer.usajobs.gov.

3. Scraping: Everything, But It’s Complicated

Need full content for jobs that are no longer current, or something that’s on the listing but not in either API? You're in scraping territory. This gets you everything that's publicly available on that listing page.

First, get job IDs from one of the APIs. Then construct URLs like:

https://www.usajobs.gov/job/[job-id]

The issues with scraping include:

• Speed: Took me ~24 hours for four months of data. It’s tempting to get aggressive with parallelization, but they will block you

• Inconsistent HTML: Fields like "Qualifications" vary across listings

• Storage: Full HTML files are huge—convert to text when possible

• Historical limits: Need job IDs from APIs to construct URLs—can't scrape without those

Even if you are scraping, you should still get everything you can from the API instead of trying to parse it from the HTML.

Bonus: Assessment Questionnaires

Many federal jobs require applicants to fill out assessment questionnaires. If you need to analyze the actual content of these questionnaires, you're looking at a bit more complexity.

Most questionnaires are hosted on USAStaffing or Monster Government.

Monster Government questionnaires can be scraped with regular HTTP requests after transforming the URL format. USAStaffing requires a browser emulator like Playwright due to dynamic loading.

I haven't tried to find the other sites that questionnaires are hosted on in order to scrape them, but if you know of any and want me to try, let me know.

Tips for Working with the APIs

• Pagination: Make sure you're getting multiple pages for longer queries. The Historical Jobs API returns a continuation URL (check for paging.next in the response) that you follow until it's null. The Current Jobs API uses page numbers—increment the Page parameter until you get no more results.

• Search limitation: You cannot query by job ID in either API (the USAJOBSControlNumbers/MatchedObjectId). You have to use other fields like date ranges or agencies to find jobs, then extract the job IDs from the results.

• Data gaps: If a listing is just a front page for applying elsewhere, those fields won't be complete in the API either.

• Deduplication and field harmonization: There is significant overlap between both APIs—there are current jobs in the Historical Jobs API—so if you want both, you will have to deduplicate. And since the two APIs use completely different field names for the same data, and in some cases format data differently, you’ll also need to harmonize the fields.

Examples of Data Approaches

There are several different approaches you might take, depending on the problem you’re trying to solve. Here are a few examples:

Data Storage Decisions

Your storage and structure decisions flow from what you're trying to accomplish.

My approach: Since the Current and Historical Jobs APIs have different fields—with the current one containing rich text content that doesn't exist in the historical one—I couldn't harmonize them into one schema without losing data. But I still wanted to query across both datasets. So I keep all raw data from both APIs and create overlay fields for the parts that overlap. This preserves all the rich content from the Current Jobs API while allowing queries across both datasets. It’s messy—this is the kind of structure I’d never choose unless I really needed to keep everything.

File format: I use Parquet files because I'm hosting everything on GitHub using Large File Storage (LFS), and file size really matters for staying within free tier limits. Parquet compresses incredibly well—better than CSV or JSON—making it feasible to share years of federal job data. Plus, Parquet works great with pandas and DuckDB, and handles nested API structures without flattening.

Here are a few other examples:

Other Data Sources

While USAJobs captures the vast majority of federal civilian listings, there are a few other places to look depending on what you need:

• FedScope: Shows what agencies actually hired (as opposed to what they posted), but it's released with significant delays due to federal hiring timelines and processing time. It also lacks the detailed job-level information you get from USAJobs: you can get occupational code, location, grade, agency, and supervisory status, but not anything like duties.

• PLUM book: The Policy and Supporting Positions book lists political appointments. It has traditionally been published every four years after presidential elections, though recent legislation now requires annual updates.

• Federal Hiring Dashboard: The Federal Hiring and Selection Outcome Dashboard provides some information on hiring outcomes, but there's no raw data available for download and the most recent data is from June.

• Federal Hiring Assessments and Selection Outcome Dataset: Additional hiring outcomes data. The data is extensive, but it hasn't been updated since FY 2024.

• Contractor positions: There isn't a centralized source for contractor positions. This is a really difficult problem to solve.

Most of my USAJobs code is available here, but this repo doesn't do scraping: get in touch if you want access to my scraping repo. And if you build something interesting with this data, I'd love to hear about it.

What AI Made Easier

When agencies propose certain types of new regulations, they're required to collect and respond to public comments. The idea is that agencies should address all substantive arguments being made. But when there are tens of thousands of comments or more, manual analysis becomes difficult and expensive.

Before large language models, automating this analysis was severely limited. You could search for keywords or use smaller models to cluster similar comments and identify topics. But these methods struggle with varied wording or complicated arguments.

Large language models do much more. You can more reliably categorize comments as agree/disagree and identify very granular themes, even when the wording varies widely. Multimodal models like Gemini also improve processing attachments. They can extract text from scanned PDFs, images, and other documents that were previously difficult to handle automatically.

For Schedule F, I processed 35,000 comments through OpenAI's cheapest model for under $20. Processing attachments through Gemini cost under a dollar for files that failed to process locally using other tools. It didn't work on all of them, but it was a big improvement from previous models. This was fast, cheap, and good—and it could be better with more time and evaluation, while still being much faster and cheaper than human analysis.

This example represents processing unstructured text data, or a workflow where we want to ask the same questions of every document. In these comments, I was looking for a specific set of themes, a representative quote, and agree/disagree categorization. For other types of documents, we'd be asking different questions.

LLMs represent a genuine advance in data processing capabilities. Organizations have tons of documents that have become much easier to analyze. I saw a tiny fraction of the possible use cases when I was a machine learning engineer at DHS, but they're endless.

The Bigger Win: Fixing Data Collection Upfront

But what's better than processing unstructured data? Not having to process unstructured text data because you were able to collect structured data upfront.

For comment data, the government could improve data collection in two ways.

First, instead of accepting any type of file attachment, uploading files could be limited to text documents, Word documents, or non-scanned PDFs. Some of the attachments people send are truly not readable in any way, like dark, scanned PDFs. It's not good user interface design to accept attachments we can’t use.

Second, in addition to allowing raw data (in attachments or not), agencies could add a field where they ask users to summarize their comment or say the most important thing. Where it makes sense, they could add an approve/reject selection button.

However, there are still many cases where data is inherently unstructured, and so you’re stuck parsing it if you want to understand it. For instance, if you're analyzing emails to determine how to respond to Freedom of Information Act (FOIA) requests, the unstructured nature of that email data is unavoidable.

Better Interfaces for Existing Data

Better interfaces are another non-AI way to make data more accessible. For instance, the easiest way to get public comment data is to fill out a form and get a massive spreadsheet (CSV) file that does not contain the text of the attachments. It's great that this exists. But even without any AI, it would be straightforward to build frontends to make the data easier to search, including the attachments.

This applies to much government data. Either it lacks programming interfaces (APIs) and the front ends are difficult or limited, like FedScope data (which is currently getting a redesign). Or like USAJobs data, where there's no graphical interface for historical data. Additionally, if you want any of the rich text data, like job duties or requirements, you can't get it from the API at all — you're going to be scraping inconsistent web data.

People struggle to access information that's technically public and available, and that’s a problem. There's a lot of room for the government and non-governmental groups to build out tooling to make data more accessible.

This is also an issue within government, where a lot of data also gets passed via spreadsheet. It could be more accessible (and version-controlled!) if it lived in a central repository with some basic table functionality.

Harder to Build From the Inside

When I was a DHS employee, I wanted to build tooling to analyze public comments. But if I'd gotten the go-ahead for it, I would have found it more challenging than on the outside. This was true even though I did sometimes have access to LLM APIs, so I could query LLMs like the one I used for my text processing—which many technical employees within government don’t have, including ones with clear use cases for it.

But outside of DHS, I can also use whatever coding assistants I want to help with coding—like Claude Code, which I used for this project. Coding assistants are enormously useful to me because they integrate into my system directly. But even though the comments data was public, there are no coding assistants I could have used at DHS: I would have been copying and pasting code between my computer and the available LLM, which is a much worse way of using LLMs in coding.

I get that there's a balance here between security and access. But the balance now is one of too much risk aversion: in government, you never get in trouble for saying no to new technology. Because of this, there is not enough urgency around making sure that civil servants have access to the best tooling, even if it's inexpensive enough that we're using it for our side projects.

Evaluation and Trust

In contrast to a lot of narratives, I think AI actually provides an opportunity for more transparency—but that doesn't happen automatically.

For instance, I didn't get every comment categorized correctly.

But you can see what prompts I used to categorize it, because they're on GitHub. You can see the evaluation frameworks I built and discarded, because they're in the git history. If I'd categorized this manually in a spreadsheet, I still wouldn't have gotten it perfect. And you wouldn't get any insight into the process.

As the government uses these tools, I think it's important that at a minimum they live on internal code repositories. This is as opposed to proprietary/drag-and-drop interfaces where it's difficult for even the government clients—the people the models are being built for—to see exactly what happened. And when possible, code should be open-sourced for the public to see it.

And regardless of whether it’s appropriate to open-source the code, you should at least be able to share some info about your evaluation framework. The 2024 Report on Select Use Cases of Face Recognition and Face Capture Technology from DHS was a good example of this.

AI isn't magic—it's a useful tool in some circumstances and the wrong answer in others. Our job as technologists is to work with users to determine what they need and to provide transparency so stakeholders can make informed decisions. But the government should give their employees and contractors the technical infrastructure to do that well.

A version of this appeared at Reform for Results.

The Department of Justice recently issued guidance encouraging federal agencies to use "artificial intelligence and machine translation to communicate with individuals who are limited English proficient." The memo specifically calls for "responsible use" of these technologies to "produce cost-effective methods for bridging language barriers."

But the memo provides no details on what "responsible use" means. There's also little guidance in memo M-25-21 on Accelerating Federal Use of AI — or the revoked Biden AI executive order — regarding responsible use for translation models specifically. This leaves agencies to figure out implementation details on their own.

So, what does responsible AI translation look like?

Responsible use of machine translation requires use-case specific evaluation both prior to and subsequent to deployment, as well as using those results to inform what happens. Here's what agencies should do":

Test on your content for your specific use cases. Don't rely on general model performance benchmarks. Create representative samples of the documents for each of your use cases. A tool that works well for routine correspondence might fail on technical documents.

Compare all your options for each use case. Test multiple machine translation systems against human translators. Have qualified evaluators rate translations. In addition to needing different samples for each use case, you might also need different evaluation criteria or cut-off scores. Evaluation doesn’t just show how often models fail, but also in which circumstances — information that can shape your deployment plans.

Act based on the results. You might find that machine translation works well for some languages but not others, or that using uncertainty scores to flag low-confidence translations for human review works better than an all-or-nothing approach. Though confidence scoring is trickier here than with some other models, it's something you can pursue.

Continue evaluating after deployment. Your documents and needs will evolve. Regular re-evaluation with a subset of new documents ensures the model continues to meet your requirements.

The real problem: deployment without evaluation

My primary concern is that agencies will deploy machine translation tools with minimal testing, relying on vendor claims or general benchmarks that may not apply to their specific use cases. The government didn't have enough technical expertise prior to President Donald Trump’s inauguration, and it's only gotten worse since. And even when contractors are doing the work, federal government civilians are ultimately the ones responsible for vendor selection, contract wording, and what deliverables to request.

During my time as an AI/ML engineer at the Department of Homeland Security’s AI Corps — where I drafted a guide on test and evaluation for AI/ML models — I saw tremendous variance across government, from agencies with well-informed evaluation plans that asked all the right questions about model performance to teams that couldn’t do proper evaluation because the government wasn’t giving them access to post-deployment data. I expect implementation of the DOJ guidance will be similarly inconsistent — some agencies will conduct thorough evaluations while others will deploy tools with minimal testing.

More rules aren’t the answer, either

But to the skeptics of machine translation, I also have a message: Even now, agencies aren't always choosing between machine translation and good human translation. In some cases, they're choosing between unevaluated machine translation and none at all, where documents just don't get translated, or translated in a timely way.

For instance, at a federal agency I worked with, employees were using Google Translate for somewhat sensitive material. A secure machine translation tool with even basic evaluation would have solved both the security problem and supplied more information about what types of documents were fine to machine translate versus which needed human review.

Additionally, evaluation standards should be driven by use case requirements, not the technology being used. While a focus on AI evaluation is important, this same attention to performance standards should apply regardless of whether work is done by humans, machines, or via hybrid approaches.

Finally, the lack of specific implementation guidance from both this White House and the previous one is appropriate. More prescriptive requirements wouldn't solve the underlying capacity problems — but they would be interpreted in some agencies in the most restrictive ways possible. For instance, a White House memo requiring evaluation of machine translation models could easily be interpreted at lower levels as requiring that this evaluation happen prior to allowing technologists to so much as install an open-source language model and try it out. And meanwhile, documents aren’t getting translated at all — or they’re getting sent to Google Translate.

So even as an evaluation advocate, I don't see the guidance's vagueness as the problem. When agencies stumble on machine translation deployment, it'll be because of lack of institutional capacity, and that wouldn't be fixed by being told to do evaluation.

But here's the thing: LinkedIn might actually be useful to you professionally. And you don't have to do any of that stuff to get value from it.

LinkedIn Solves Real Problems

A lot of career-related problems are actually information problems. What roles should I apply for? What's happening in my field that I should keep up with? Who are the people with the skills and interests I'm looking for to fill roles at my organization?

I know "networking" feels gross and transactional to many people. But there's no master directory for any of this information. You use what's available. LinkedIn isn't perfect, but it's often the most efficient option. Here are a few things it can do for you:

• Discovery: See what people in your field are talking about, find tools you didn't know existed, learn about events you might want to attend.

• Visibility: Let people know what you're working on, interested in, or thinking about—and if you're job searching.

• Messaging: People can find and contact you about opportunities, and you can reach out when you need information or introductions.

I've learned about tools I later used from LinkedIn first. I've reconnected with people I'd lost touch with. My last job? I applied because I saw people talking about it on LinkedIn. And the one before, I found because a recruiter messaged me on LinkedIn—and I wasn’t at all active then, beyond having an up-to-date profile.

Legibility Without Cringe

Yes, there are LinkedIn posts that make you cringe. There's an algorithm, and some things that perform well are kind of gross. But lots of posts aren't that. Some are just… fine.

Things I've seen in my feed recently include:

• "Everyone says 'We want the best people.' But your hiring process still looks like this."

• "Gov agencies hate risk. Prove your solution works small before going big."

• "Want to learn Rust with me? I'm teaching a workshop at Cascadia R Conference in June."

• "What's your unpopular opinion about AI careers? I'll go first: Most AI jobs don't need deep learning."

None of these are super clickbaity. But they're legible. That is, they answer the question "Why should I care?" right up front. Instead of making people work to figure out what you're talking about, they’re making it easy for you to decide whether to engage.

Here are some examples of my own posts that follow this principle:

Sharing useful resources:

• "Want raw FedScope data? Can't get it? Check out my git repo."

• "📊 FedScope Employment data is the official dataset on federal government personnel. I'm trying to make it easier to use."

Asking for technical help:

• "I've got 150 million rows of data, zero budget, and might want to build a GUI for people to filter/aggregate/download slices. Help!"

Promoting events:

• "JULY 8th: Whether you're deep into a new project, debugging a rogue script, or just curious what everyone's building these days, join us at Prefect in Foggy Bottom."

LinkedIn only shows the first couple of lines of your post, so you need those initial words plus your image1 to communicate what your post is about immediately. That's what people who are new to LinkedIn often miss—they bury the point instead of leading with it.

Don't Worry About the Numbers

If you're not a "social media person," the idea of posting and getting feedback from strangers in the form of likes or comments (or no likes or comments!) might feel uncomfortable. But you can decide how much to care about the numbers—and you don't have to take them personally.

I once posted about using LLMs to help with Python syntax. It wasn't especially interesting. But it got 355,217 impressions and reached 196,366 people. I will probably never get that kind of engagement again—and that's fine.

The way I think about this: be outcome-independent. Focus on clarity, not performance. I try to make the first lines clear, and I usually add an image, but I'm not optimizing for engagement. You can do more or less optimization, depending on your goals.

How to Get Started

If you're interested, here's how to start:

Step 1: Fix the basics. Update your profile so it's accurate. Add a photo. Connect with former coworkers and other people you know.

Step 2: Lurk. Follow people in your field who post things you care about. Notice what they’re doing.

Step 3: Share2. Start posting when you have something to say. That might be asking for help with a tool, sharing an article you liked, or reposting a job from your organization with some context. Start small if you’re not sure how.

What about connecting with strangers? Some people don't like it. I do it, and people do it with me all the time. I usually accept if we have a lot of mutual connections or if it’s clear to me why we might want to chat.

I spend a lot of time on LinkedIn because it helps with things I care about—events, side projects, my blog. So I post frequently, and I connect with people more than average. It's what I do instead of scrolling Reddit. If you don't have those goals, the time you spend will probably be less.

Why This Matters Now

If your job security assumptions have changed, or if you're in a field that's evolving faster than it used to, LinkedIn is now more valuable. For instance, for government folks—if you need to navigate the job market in ways you previously wouldn't have—this is a tool worth understanding.

Success isn't about becoming an influencer. It's about:

• Finding opportunities you'd otherwise miss

• Being visible when it matters

• Having an easy way to reach people—and be reached

You don't need Premium. You don't need to post every day. You don't need thousands of connections. Just being present and clear about what you're doing, and reading what other people are doing, can be helpful.