Three Questions to Ask About Your Job Listings
I read a lot of job listings, particularly for federal government and federal contractor jobs in technical areas. These are jobs which are frequently difficult to hire for: the federal government approves direct hiring authority for STEM and cyber occupations, in recognition of this fact. While some of the constraints to hiring that apply to the federal government don't apply to contractors, others do — such as the need for citizenship and frequently the ability to obtain a security clearance. Furthermore, both government and contractor positions face salary limits, whether due to government pay scales or reimbursement rates.
I’ve written and spoken about specific pitfalls in these listings – for instance, jobs where the duties are vague, the title doesn’t match up with the duties, or the tools listed aren’t the most effective ones for doing the job. But these are part of a wider set of issues, and so I wanted to go through a few more general questions that I think you should ask about your job listing and general hiring strategy:
The questions are:
Does anyone with these qualifications exist?
If so, are they the correct person for the job duties?
And finally, would they want this job?
In this article, we’ll explore each of these questions, discuss how to gather and analyze related data, and look at some ideas to try if you answer “no” to one or more of these questions.
Does Anyone with These Qualifications Exist?
In the software development world, the infamous example of nonexistent qualifications is requiring X years of experience in a tool or language that hasn’t even been out for that long.
For instance, the following tweet mentions a job listing looking for four years of experience in a framework for building APIs that’s less than two years old.
Another example going around the internet recently asked for “10+ years of experience building and deploying Generative AI models.” We can quibble about what constitutes a generative AI model, but the Attention is All You Need paper was published in 2017, followed by the BERT and GPT-2 models in 2018 and 2019.
Now, asking for flat-out impossible experience is not the only way to fail this question. A more common way is to ask for a combination of skills or experiences which individually do exist, but are rarely or never found together. I would distinguish this from looking for a senior role with a narrow set of expertise; in that case, many of the more niche or advanced skills you’re looking for will tend to come with other skills your candidate needs.
For instance, I saw a data science job ad requiring SAS or R experience and experience working with healthcare claims data. And as a preferred qualification, it listed the development and maintenance of large language model pipelines. I’m not going to say there are zero people with all three of those skills – but I don’t think there are very many. The “SAS or R healthcare data” group is going to mostly consist of researchers or data analysts with subject matter experience. Meanwhile, the group developing and maintaining LLM pipelines is going to consist mostly of data engineers, machine learning engineers, and software developers working in Python. And I don’t think listing this under “preferred qualifications” solves this problem: you’re still scaring ‘unqualified’ people away, and communicating confusion about the role and who you have in mind for it.
Do the Qualifications Correspond to the Job Duties?
Once you’ve determined that there’s a group of people who meet your desired qualifications, the next question is whether they’re the right people for the role’s duties. That is, are the qualifications actually helpful for performing the job, and are you leaving off any that are important.
For instance, there was a pooled data science hiring announcement last year – that is, an announcement designed to hire a number of people into data science roles for the government department in question. Like many government roles, they used a hiring questionnaire to screen potential employees. The way these questionnaires work is that if you say you don’t have the experiences they’re looking for, you’re marked as disqualified and your candidacy ends.
But of the eighteen questions listed for this mid-level, non-supervisory technical role, only one mentioned specific tools or programming languages – most focused on leadership or managerial experience. Additionally, the bar for technical experience in the one question was very low, asking just for “applied experience in a workplace setting” with a programming or statistical tool. Later parts of the process I’m sure introduced higher technical barriers – but candidates who decided they weren’t qualified based on the questionnaire, or who answered in the negative and were formally disqualified, were already gone.
Going back to the previous section, the use of years of experience with a particular programming language or tool is typically a bad proxy for level of proficiency, and therefore an example of a mismatch between a requirement and a job duty. I see this a lot in listings for government contractor roles, along with requiring experience in specific tools which are very similar to other tools — like a version control system within a particular environment that anyone who has used standard version control tools will be able to pick up quickly.
Finally, I frequently see this issue with roles which wear many hats – such as government data science roles which also include data engineering, data architecture, and data management. If the qualifications are written to focus narrowly on just one part of the role, then you pass the first question but fail the second.
Would the People I’m Describing Want This Job?
OK, so you’ve decided that there are people who have the qualifications you’re looking for and that they’re the people who are capable of executing the job duties. But you should also ask yourself: would they want to do this job?
Let’s go back to the “10+ years of experience building and deploying Generative AI models” job posting, and let’s say I was being unfair by defining ‘generative AI’ too narrowly. There was actually generative AI work being done in 2014; case in point, the eight authors of the 2014 Generative Adversarial Networks paper. And let’s ignore that you don’t need someone with that many years of experience for the role. Even if you want them, you’re not going to be able to hire them, because they have very good jobs somewhere else.
This is an extreme example, but most job seekers have some options – which often include staying at their current jobs – as well as opinions about the kinds of work they want to do! For instance, a position for a data scientist with a background in AI/ML is going to be less appealing if it includes maintaining legacy VBA products — the person you want to hire to build your Python models doesn’t want to maintain your legacy VBA heatmap, even if they can. Technical positions in general where the tech stack doesn’t include industry-standard tools are going to be less appealing to many candidates.
It's important to remember that for nearly any given job duty and set of tools, there is someone out there who would be happy to perform it and is capable of doing so. Plenty of people would be content maintaining your legacy VBA heatmaps or working with SAS or Stata instead of Python. However, if you've determined that you also need someone with significant ML experience and expertise in open-source tools and that these are legitimate requirements of the role, then you've necessarily narrowed your candidate pool to exclude most of those people.
The following questions can help you work out whether qualified people would want to perform the duties listed for your organization at the salary being offered:
Does this job let them use and grow their skills?
Would this job represent career growth for them?
Is our organization an appealing place to be for this role?
Is this a competitive salary, or are there other things we can offer?
And if you don’t have any ideas about how to answer these questions, you can do some ad hoc market research, which I’ll discuss in the next section.
How to Gather and Analyze Data
As you work on answering the questions above, here’s some data to consider:
Do the people we hire actually have the listed qualifications? If not, we’re listing requirements that aren’t necessary and likely dissuading potential applicants who we would have hired.
Do the duties listed correspond to what the people we hire are doing? If not, maybe the duties section is inaccurate – or the people we’re hiring don’t have the skills to perform the duties we were envisioning.
Are we happy with employee performance? If not, maybe our requirements are the wrong requirements for the job.
Are we regularly losing candidates we tried to hire? Alternatively, are people with backgrounds we’d like to hire not applying? If so, maybe our jobs are unappealing to the people we want.
For this last part, you can do some informal market research. Start by picturing the ideal candidate you have in mind for the role. Who are they? Where do they work now? What skills and experiences do they have? Then, go out and find people via your network who match this description. Reach out and ask for candid feedback on what’s appealing about your job listings and organization and what’s not.
While this approach may not be necessary for every position, it can be invaluable when hiring for high-value roles or addressing ongoing hiring challenges within your department or office. Talking to potential candidates directly can provide insights that you won't get any other way. They know more about their alternatives than you do, they may pick up on issues that you don’t — and they may even tell you that they’re not the right people for your job duties. And even in the context of government hiring, there are no regulations prohibiting these conversations.
Additionally, you can leverage other sources of information. Go on Glassdoor and read candidate and employee reviews. When you lose candidates to other jobs, ask them why. (You might also uncover other issues about your hiring process which are unrelated to the job postings, like long time-to-hire.)
Where This May Lead You
If you conclude this process with one or more “no” answers to the three questions we started with – do people exist who meet your requirements, are they capable of performing your duties, and do they want your job – there are a few possible actions you could take:
Split up positions. If one position calls for multiple distinct and rarely overlapping skill sets, it may be easier to create multiple positions than look for one person with all of the skills.
Train people. If you need one skill or experience that takes a lot of time to develop, and one that the people with that skill rarely have but can be quickly taught, teach them yourself!
Go for specificity: Eliminate overly broad requirements in favor of asking for what you actually need. For example, if you need someone who understands some basic math and statistics, don’t ask for a STEM degree; ask for coursework or experience in statistics and then validate that through your interview process.*
Raise the pay (or the prestige, or the career development, or the benefits): The earlier three questions involved changing what you’re looking for. But what if you’ve meditated on it and you really do need the hard-to-locate or hard-to-hire folks you’ve been seeking, but they’re not applying or they’re turning you down? Talk to them, find out what they want more of, and try to give it to them.
Now, you can answer one or more of these questions in the negative and still be making great hires. Maybe you have such a deep hiring pool that you can afford to lose candidates who opt out of applying because they think they’re not qualified. Maybe the people doing the hiring know enough to ignore the written requirements.
However, there’s one measure I think you should avoid using to determine that this is the case, and that’s the number of applicants. A lot of people apply scattershot to jobs, and that’s especially the case for the federal government. If you’re getting hundreds of applicants, that doesn’t mean you’re not missing out on large parts of your potential hiring pool.
Conclusion
I haven’t done a ton of hiring, but the things I’ve heard from potential candidates for technical jobs in government have included: “This job posting doesn’t make any sense/isn’t what I want to do,” “[another agency] is doing much more interesting work,” and, my favorite, “I already make much more than the top of the GS pay scale.” (This was from a software developer in his 20s.) It wasn’t awesome hearing these things, but they were useful for me to know.
I think more organizations, especially those in or around the government which struggle with filling positions or hiring the kinds of candidates they want, should show curiosity about potential candidates. When you write a job posting, who do you have in mind? Have you thought about whether that person is a good fit for the position you’re describing? Have you reached out to people like this and asked them what they think of your job posting? Are you using the data that’s out there, like reviews of your organization, and even just consistently asking people who accept other offers why they didn’t come work for you?
Even if what you hear in this process is disappointing, and even if you hear things that aren’t currently actionable, this is still good information for you to know – and the more you pull together data, the better the case you can make to your leadership (or hold onto until you are the leadership!) about what in your processes ought to change.
__________________
*A counterargument for a narrow skills focus is that you don’t just want to hire people who will succeed in their initial role but also who will have the skills and abilities to get promoted and move up. I don’t think that’s crazy! But that also doesn't explain what's going on with term-limited positions and positions, like with small contractors, where very few people move up. I also don’t think a qualification that’s as broad as a STEM degree is going to get you that. If you’re knee-deep in people analytics data for your firm and you can tell me that it actually does, I will, of course, concede the point.