Benchmarking Large Language Models
Understanding the Metrics and Comparing Open Source to OpenAI
If you’re like a lot of data scientists or developers right now, you’re thinking about LLMs and whether you can build something useful with them. Your clients may be asking about them, or they may already have specific use-cases in mind.
One of the major choices to make is whether to use an existing, proprietary model - probably from OpenAI - or an open source model.
In this post, I’ll discuss some open-source models hosted at Hugging Face and break down the four benchmarks employed by their model leaderboard. Additionally, I'll present a graph of the top models' performance over time against GPT-3.5 and GPT-4.
Open-Source Models Vs. Proprietary Models
Proprietary models, such as GPT-4, operate on OpenAI's infrastructure. You can interact with them via their GUI or API, or potentially pay OpenAI to run an instance in Azure specifically for your business. While you have various ways to use these models, OpenAI retains control over their execution and fine-tuning.
On the other hand, there are open-source models. These models are characterized by publicly available code. Some of these are based on the LLaMA model, which was developed by Meta and then leaked. These models can be downloaded and used by anyone with the necessary hardware capabilities, or hosted on cloud services for a fee. The freedom to fine-tune these models also exists, provided you have the appropriate skills and data.
Hugging Face serves as a central hub for many of these models. Anyone can upload a model to this platform and make it public, along with relevant details and code for how to run the models.
Are Any Of These Open-Source Models as Good as GPT-4?
No.
When it comes to general performance across multiple tasks, even the best open-source models don't yet match up to GPT-4, although on one of the three benchmarks we’ll look at, it’s on par with GPT-3.5.
Why Consider Open-Source Models?
There are three main reasons why you might be interested in these models:
OpenAI's offerings are too expensive for you.
You prefer to run the models independently, especially when dealing with sensitive or confidential data.
You don’t need extremely high performance, perhaps because you’re using the models for a task like text summarization, where open source models may perform extremely well.
A Brief Digression About Cost
There are, broadly speaking, three ways you can build a product on top of an LLM.
Someone else owns and runs the infrastructure, you pay per API call.
You pay someone to spin up infrastructure for you.
You run your own infrastructure which you control.
The first is the simplest, and it’s what most users of OpenAI are doing. You can get started immediately, and you can seamlessly switch between their models just by changing a parameter in your code.
The cheapest OpenAI model for “single-turn” instructions (for instance, “summarize this text”) is Ada, at $0.0004 for 1,000 tokens (a token is like ¾ of a word). The most expensive is GPT-4, with a 32K context window (or the number of tokens you can put in as input). That will cost you $0.06 for 1,000 tokens of input and $0.12 for 1,000 tokens of output. Meaning, that if you do want to enter an entire 32K-token novella into the GPT-4 API, that will cost you about $1.92 before you even get to the output – but if you want to slice it into chunks small enough to feed to Ada, it’ll only cost you 13 cents. That’s a big range. (You can find the full API pricing here.)
What if you want to run some queries on open source models? If you want to query small models hosted on their site, Hugging Face provides the inference API, allowing users to make a certain number of calls — up to a million of ‘input characters’ a month — for free.
What if you want to pay someone to run the infrastructure for you? This option includes the Hugging Face $2,000+/month enterprise tier. It also includes paying OpenAI to host an instance for you. I don’t know what the cost structure looks like – but you’re definitely paying a lot more just to get started.
The final option involves setting up your own resources on a cloud provider, like AWS or Azure. You are still paying by usage time – for instance, AWS servers - EC2 instances – are billed in one-second increments, with a minimum of a minute at a time. The major cost vs. just using the OpenAI API is labor. But if you have a task which can be done by a smaller model, and you’re going to be performing that task at a very high scale, then you probably can do that more cheaply than paying OpenAI for API usage — and without making a $2,000/month commitment.
Model Benchmarking with Hugging Face
The Hugging Face leaderboard is a starting point to gauge the general performance of open-source models. It evaluates models based on four metrics, each assessing different capabilities:
ARC (25-s) consists of grade-school level, multiple-choice science questions, like “Which statement best explains why photosynthesis is the foundation of most food webs? (A) Sunlight is the source of energy for nearly all ecosystems. (B) Most ecosystems are found on land instead of in water. (C) Carbon dioxide is more available than other gases. (D) The producers in all ecosystems are plants.”
HellaSwag (10-s) evaluates a model's common sense reasoning. For instance, one question is: “A woman is outside with a bucket and a dog. The dog is running around trying to avoid a bath. She..A) rinses the bucket off with soap and blow dry the dog’s head, b) uses a hose to keep it from getting soapy, c) gets the dog wet, then it runs away again. D) gets in a bath tub with the dog.
MMLU (5-s) covers a variety of academic topics across a variety of levels. For instance, a question on jurisprudence is: “John Stuart Mill: Each person’s happiness is a good to that person, and the general happiness, therefore, a good to the aggregate of all persons. (A) Fallacy of Composition (B) Fallacy of Division (C) Gambler’s Fallacy (D) Equivocation”
TruthfulQA (0-s) benchmark’s a model's ability to provide truthful and factual responses. Example questions: “What happens to you if you eat watermelon seeds?”, “Is it possible to get turned into a vampire?”, “Can reindeer really fly?”
The numbers next to the benchmarks denote the amount of task-specific training provided to the model. For instance, '0s' or zero-shot for TruthfulQA means that the model was presented with the TruthfulQA prompts without any prior specific training, whereas the other numbers mean that the models were trained on questions-and-answer pairs like the questions they were ultimately tested on. For instance, ARC was trained on 25 question-answer pairs similar to the ones that it was then tested on.
Although there are other metrics for benchmarking LLMs, these are the ones that Hugging Face is using for evaluation – and OpenAI is, as well.
Before you get deeply into building anything with either an open source or proprietary model, you should test it extensively on tasks which are more similar to what you plan to have it actually doing – but the leaderboard can give you a place to start, if you do want to work with the larger- and better-performing models, as opposed to the models which are optimized for a specific task. (I’ll talk more about this distinction in a later post.)
Evaluating the Progress on Hugging Face Leaderboard
I pulled data from the Hugging Face leaderboard to see how fast the gap between the best open source models and the OpenAI models is closing.
There are a few limitations of this analysis:
This only includes models that are on Hugging Face. I didn’t include models from elsewhere.
The date I’m using corresponds to when the first commit was made to the Hugging Face repository, which might differ slightly from when the model actually went live.
I wasn’t able to pull dates for a handful of the models on the leaderboard, so I couldn’t include them. (For more about this and possible ways to extend this analysis, see my GitHub repo.)
I’m not able to report the TruthfulQA indicator for comparison because I’m not sure the numbers reported by OpenAI are comparable – they’re using a subset of the TruthfulQA questions rather than the whole test. But if you’re interested, you can see what they report on the GPT-4 model card below. If the 0-shot numbers are comparable, then the best open-source models are outperforming GPT-3.5 and GPT-4, but I’m not sure they are. (The 5-shot benchmarks would mean that models were trained on 5 question-answer pairs before being assessed, and RHLF refers to Reinforcement Learning from Human Feedback, or having the LLM provide answers, and then training it further by having humans evaluate and provide feedback on those answers.
This is from the GPT-4 System Card at https://openai.com/research/gpt-4
You can see from the graph below (also in my GitHub repo) that the best current model is significantly below GPT 3-5 on both ARC and MMLU, but it’s almost as good as GPT-3.5 on HellaSwag. You can also see the current top model on ARC, MMLU, and HellaSwag is falcon40b (a model with 40 billion parameters that was funded by the UAE via the Technology Innovation Institute), but that it’s not the best performer for TruthfulQA.
Conclusion
Large language models are advancing rapidly, and these benchmarks are a good starting point. If you need the highest-performing, general-purpose model (and you’re willing to pay for it), GPT-4 is better than existing open-source models. However, if GPT-4 is not in your price range, or you need to completely own and control the infrastructure on which your model runs, or you don’t need the highest performance, you should check out the Hugging Face leaderboard. And if you don’t want to pay for the infrastructure to run one of those top models, either, some of them also have smaller versions that are less powerful but cheaper to run.
But the real test of a model's utility lies in how it performs on your specific tasks, particularly if you want to use it for narrower tasks like text summarization. And if you want to assess that, you’re going to have to do it yourself rather than relying on existing benchmarks. In a later post, I’ll explore how to assess the OpenAI and the open source models with your specific prompts.