Your Coders Use LLMs

How should your organization adapt?

These days, almost all the code I "write" is actually written by an LLM. Overall, this is fine: I can iterate more quickly toward the correct solution, including trying approaches and discarding them when they don't work. And I'm definitely not vibe coding—I remain intentional about what I'm building and how.

But there are also tradeoffs. Since I can produce more code more quickly, I need to think harder about system design to keep everything under control, and be more deliberate about going back to refactor and delete code.

It’s also possible for me to write code that works without understanding its details. Sometimes that seems like a problem, like when I struggle to debug; sometimes it doesn't. And I've definitely lost some ability to code without LLMs.

These aren't just individual problems. As organizations shift toward more LLM-generated code, they'll benefit from being intentional in how to manage these issues.

Here's how I think about this: LLMs give organizations a productivity dividend. Some of this should be deliberately reinvested in quality control and professional development. By doing this, we don't just end up with code that's faster to produce—we end up with code that's more likely to do what we intend it to do than the code we were writing before.

My focus here is on analysis and data science teams, because that's where my experience lies and I think these teams have the most gains to be made. This is because they can shift not just to coding faster, but to adopting very different languages and practices.

Why Analytics Teams Have the Most to Gain

Data, research, and analytics teams vary widely in terms of coding practices, but more so than on engineering teams, you're likely to see the following:

• Jupyter notebooks with code blocks that get commented out as needed.

• Ad-hoc analyses that get copied rather than properly versioned.

• Long, multi-purpose functions, or no functions at all.

• Proprietary languages with high licensing costs, vendor lock-in, and restricted functionality compared to open-source alternatives.

Fixing these practices is a great application area for LLMs. Even if experienced software engineers legitimately debate how useful LLMs are for them, the benefits for data teams with these issues are clear. LLMs can explain git commands, help structure code more effectively, suggest tests, help transition from proprietary to open-source languages, and generally make software engineering best practices more accessible.

The barrier to adopting better practices has always been the learning curve—taking time away from analysis to learn principles that didn't feel central to the daily work. LLMs significantly flatten that curve.

But while the potential gains for these types of analytics teams are significant, so is the need for new practices. Engineering teams may already have robust code review, testing, and quality control processes in place that can accommodate LLM-generated code with minimal adjustment. Data teams that rely on notebooks, ad-hoc analyses, and proprietary languages are much less likely to have these established practices, meaning they'll need to build these systems largely from scratch as they adopt LLMs.

Questions To Ask Yourselves

To be intentional about LLM adoption at your organization, consider these questions:

1. How much code understanding is enough?

There's a spectrum between "I can explain every function and line" and "it works but I have no idea why." Where should your analysts be on that spectrum, and how does that vary by project or project phase?

2. What practices lead to the right level of understanding?

You could require every code change to be a pull request where the coder has a conversation with the person approving it and they go through line by line and explain what they did. You shouldn't do that. But since it's become so much easier to code without understanding what you're doing, you might want to create practices which add some healthy friction.

Maybe the answer is adding more code reviews. Alternatively, consider other ways to get your team thinking and talking about their code, like pair programming. The focus could also include making sure your analysts have particular skills or understand certain things conceptually that they use in their work, in addition to getting them talking about their code.

3. How much are you willing to slow down?

LLMs enable speed. Measures like code review and writing tests will slow you down. Some slowdown is worthwhile. How much? Figure out which parts of your development process benefit from being slower and which can safely move at LLM-accelerated speeds. You might even occasionally have team members sit in a room and code without LLMs, or even whiteboard, if you want to keep certain skills sharp.

4. What should your new coding standards be?

Since LLMs lower the cost of adopting software development practices, you should adopt more of them. But which ones? What's enough for your organization? If you have code, you should use git. Beyond that, maybe it depends what you're doing.

5. What do you need to explicitly teach?

To what extent should you manage professional development, either in terms of LLM adoption or engineering practices? What training should you provide on effective LLM usage? Which engineering practices should be taught directly versus learned independently, and how does this vary by job and level?

What to Avoid

I don't have great answers to the above questions, but here are some suggestions as you’re adapting to LLMs:

• Don’t distinguish between LLM and non-LLM code. Treating computer-generated code differently only encourages concealing tool usage. All code should meet the same standards, regardless of origin, meaning that if there’s a standard you want to apply to LLM-generated code, you should apply it to all code.

• Avoid unrealistic time pressure. When teams operate under too much pressure, they're more likely to accept LLM outputs without examination. Build in time for understanding.

• Don't use outdated hiring practices. Reconsider your assessment approaches. Test what you're going to need at the actual job—which has probably changed.

The Path Forward for Data Teams

Two years ago, "if you're not using git already, get an LLM to help you" was a pretty reasonable perspective on LLMs—and it was, in fact, mine at the time¹. But it's not an organizational strategy. And if you're running an analytics shop, at this point you should be thinking about how to incorporate these tools in a more systematic way. This is because LLMs create new ways for code to fail that your old processes weren't built to catch². Plus, if you want better engineering practices, you need to get specific about what that means and how to make it happen.

I think data teams in general can benefit greatly, but the biggest gains from intentional, directed LLM adoption exist for data teams with these attributes:

• Code-based workflows—as opposed to Excel-based ones—but implemented in proprietary languages or spaghetti code.

• Regular deliverables that would benefit from automation—for instance, monthly or annual reports where pulling the data, making the figures, and putting together the report are still heavily manual—and maybe the analysis part still is, as well.

• Senior leadership support for significant changes—providing LLM access, updating hiring practices, setting clear expectations around adaptation, and investing in career development.

But this transformation won't happen automatically. Simply saying "go use LLMs" or even "go use git and get an LLM to help you" won't drive the adoption you could achieve through intentional implementation.

Without organizational guidance, you're leaving individuals to navigate LLM challenges on their own, and they may not address them in alignment with organizational goals—partly because they might be gone by the time their code breaks, and partly because they lack the perspective to know what practices will benefit the team as a whole.

1. By the way, I've been glad to see another position of mine from 2023, Maybe You Should Have In-Person Tech Interviews, resonating with others!

2. You might ask: what if LLMs get so good that we don't need any of this structure or oversight? If you think that’s going to happen, you should probably make some different career plans! But what I think we're seeing, at least for the next few years, is LLMs moving towards being a new abstraction layer, not becoming autonomous agents that need no human guidance.