EleutherAI Second Retrospective: The long version

![](/images/blog/year-two/hackers2.png#center)

We've been fairly busy at Eleuther for the past year-and-a-half, here's the full story.

July 2021 – December 2021: An Interlude[#](#july-2021--december-2021-an-interlude)
----------------------------------------------------------------------------------

### Language Modeling[#](#language-modeling)

[When we last left off](https://blog.eleuther.ai/year-one/), GPT-NeoX was a pipe dream. [A global GPU shortage](https://nvidianews.nvidia.com/news/nvidia-announces-first-quarter-fiscal-2022-revenue-tracking-above-outlook) had halted our progress in its tracks.

![](https://cdn.discordapp.com/attachments/445630505606578212/1029829971952287754/Daj.png)
Connor2021-09-04
*taps sign* "Every time you ask for ETA, it gets pushed back a week"
We're working on larger models, hardware shortage is just brutal and we all do this in our spare time and have dayjobs

Luckily, it was just about this time that the cluster that CoreWeave was building for this project was finally having the finishing touches placed on it. We started taking it for a test drive, but the initial results were subpar to say the least. InfiniBand was a very different interconnect technology than we had previously worked with, and it took a full two weeks of trial, error, Dockerfiles and microbenchmarking to form our environment into something we were happy with.

![](https://cdn.discordapp.com/avatars/201034288693510145/75b772961ddc97cd0b2b9d700d55a719.webp)
Sorcer2021-11-04
Is NeoX the model name for this one?
Or are we going with GPT-PeterEricSidBattlesRDMA?

And so the time came. Everything was finally in place. We completed our final readiness checks, and without any dissenting responses at the time, we pressed go.

One problem: In our haste to start the run, we had forgotten to consider checkpoint storage. Unlike most training runs, it was our desire to archive all checkpoints for later analysis, and the mount that we configured to store our checkpoints wouldn’t be large enough for all of them. After much discussion, it was determined that the safest course of action would be to leave the existing run alone and occasionally empty out hot storage to cold storage manually. This would have been fine—if we would always remember to transfer checkpoints before hot storage ran out.

Of course, we forgot. More than once. The run would crash, and we would have to fix the collateral damage.

We were curious how closely bored nerds with nothing better to do were following our every move—so we decided to leave the Weights & Biases training run unannounced and public before waiting to see who noticed. As far as we can tell, the first person to notice was a [lone anonymous user on 4chan](https://arch.b4k.co/vg/thread/359155218/#359332204), who was promptly ignored by the community until the run was rediscovered by the 4chan community over a week later on November 18, 2021. Shortly after, people started asking about the model.

### Multimodal Modeling[#](#multimodal-modeling)

Our July 2021 retrospective, a short section was dedicated to #the-faraday-cage and its resident art gen service BATBot. We had no way of knowing that that service was about to get a huge shot in the arm.

On July 19th, just a few weeks after our retrospective was published OpenAI released their ImageNet diffusion models. [According to their model card](https://github.com/openai/guided-diffusion/blob/main/model-card.md) OpenAI was only willing to release the models because they couldn’t generate anything outside the ImageNet classes. How they conducted their assessment isn’t publicly known, but they missed something because a mere six days later on July 25th RiversHaveWings developed a CLIP Guided Diffusion algorithm using the new models. The hacker known as EleutherAI had struck again.

![](/images/blog/year-two/clip-collab.png#center)

![](/images/blog/year-two/image-gen-4.png#center)

CLIP-Guided Diffusion, like VQGAN-CLIP before it, allowed us to do text-to-image modeling without spending millions of dollars pretraining models. Models like the original DALL-E are ludicrously expensive to train (even by large scale AI research standards) and would be largely inaccessible to artists and researchers even if they were publicly released. Our models were clearly worse, but they required tens of thousands to hundreds of thousands fewer GPU-hours to train and could be deployed in a Google Colab. Soon though, we’d ge to try our hand at training our own models.

January 2021 – March 2022: Something Pithy Goes Here[#](#january-2021--march-2022-something-pithy-goes-here)
------------------------------------------------------------------------------------------------------------

Our original plan was to release GPT-NeoX-20B at the end of the year, marking the one year anniversary of the Pile’s release. However our loss curves looked weird, and the model continued to improve faster than we anticipated. When we reached the end of the Pile (~300B tokens, the same size as the GPT-3 training corpus), we decided to start another epoch and see what happened.

![](/images/blog/year-two/log-scale.png#center)

![](/images/blog/year-two/generalization-error.png#center)

“What happened” was that the loss kept going down, benchmark performance kept increasing, and there was no sign of increased generalization error. We had no way of knowing it at the time, but what we were seeing was the first hint of “Chinchilla Scaling Laws” from DeepMind’s seminal paper a few months later. We decided to end training in late January after 400B tokens, which turned out to gave us the first chinchilla-optimal language model completely by happenstance.

![](https://cdn.discordapp.com/attachments/445630505606578212/1029829972745007286/Sid.png)
Sid2022-02-10
@everyone After a year-long odyssey through months of chip shortage-induced shipping delays, technical trials and tribulations, and aggressively boring debugging, we are happy to finally announce EleutherAI's latest open-source language model: GPT-NeoX-20B, a 20 billion parameter model trained using our GPT-NeoX (<https://github.com/EleutherAI/gpt-neox>) framework on GPUs generously provided by our friends at CoreWeave (<https://www.coreweave.com/)>.
The model weights, along with a technical writeup, will be released in a week's time on February 9th. In the meantime, you can head over to [<https://www.goose.ai/>](https://www.goose.ai/), CoreWeave and Anlatan's new inference service (which, yes, we memed into existence), to give the model a try.

For more info and evaluations, head over to our blog post ([<https://blog.eleuther.ai/announcing-20B/>](https://blog.eleuther.ai/announcing-20B/)) about the release. To discuss the model in the discord, please use the #release-discussion channel.

We typically don’t provide timelines or roadmaps, but as an additional motivator we committed to a timeline for full release—in an announcement that mentioned @everyone and [a blog post that was widely shared on the web](http://fedora.local:1314/announcing-20b/).
And so we were stuck with it. Whether we were ready or not, February 9, 2022 it was. Of course, our work was far from done: We had a model, but we had much to document. The next week was a mad dash of writing, benchmarking, and analysis.

![](https://cdn.discordapp.com/avatars/193204646687408129/b4a107ca3a5fb4e0cdbe62245002a647.webp?size=128)
Stella Biderman2022-02-09
I low-key want to murder pandas

![](https://cdn.discordapp.com/attachments/445630505606578212/1029829972745007286/Sid.png)
Sid2022-02-09
reporting you to wwf

![](https://cdn.discordapp.com/attachments/445630505606578212/1029829972745007286/Sid.png)
Sid2022-02-10

> we all team together to build a state of the art language model with incredibly complex engineering parallelized across hundreds of gpus

> Also 4 of us combined can't sort a list of strings

But, as promised, we were able to complete a draft paper which we released along-side the full model weights. We have also promised to give access to the partially trained checkpoints to anyone who asks, but have received zero serious requests for the partially trained checkpoints so far.

We learned a lot from this experience, including just how hard language model evaluation is. Very minor tweaks in language model evaluation protocols that wouldn’t even be noticed by a human can wildly change performance. For example, MMLU benchmark features multiple choice questions with answers labeled “a” through “d.” The recommended way to evaluate models is to compare the log probabilities of the *letter corresponding to the correct answer*. However if you instead compare the actual answers performance shoots up: in our testing the difference is comparable to scaling from 6.7B parameters to 175B parameters.

![Grading the model’s ability to produce “Branch of the thyrocervical trunk” rather than “C” results in much higher accuracy on average.](/images/blog/year-two/question.png#center)
Grading the model’s ability to produce “Branch of the thyrocervical trunk” rather than “C” results in much higher accuracy on average.

We also learned just how important it is to get the narrative right the first time. Our blog post originally claimed that GPT-NeoX-20B underperformed FairSeq Dense 13B (then the largest publicly available English LLM) on standard NLP benchmarks like Lambada and HellaSwag. This wasn’t in fact true, the issue was [a bug we discovered in the shard merging code that hurt performance](https://github.com/EleutherAI/gpt-neox/pull/466#issuecomment-997517986). While we eventually found ways around this bug, and all numbers in our paper were reported using the (better) unmerged model, we still get occasional questions from people who think that GPT-NeoX-20B’s performance on NLP tasks was disappointing.

### Training our own Multi-Modal Models[#](#training-our-own-multi-modal-models)

While alstroemeria’s Promethean episode with CLIP guided diffusion ushered in a new era of high quality neural net art, it was far from perfect. Fortunately, text-to-image and diffusion research saw breakthrough after breakthrough, leading to less and less compute being needed to train or use the models. The early months of 2022 were a mad dash of new techniques and new combinations of old techniques, with one week seeing three new SOTA models each obviously better than the previous.

Writing about this now is a bit weird, as many of the major breakthroughs are far, far behind what models like StableDiffusion and Midjourney can do today. We’ve been thinking about releasing a deep dive on the history of text-to-image synthesis, but for now it is sufficient to say that Katherine and JD were very busy training models and reading new papers.

### Alignment Education[#](#alignment-education)

This time period also saw a pronounced uptick in conversations about AI Safety, interpretability, and other related topics. A lot of productive discourse has come out of these discussions on a variety of AI Safety related topics, ranging from more prosaic alignment concerns like interpretability and mesa-optimization, to more abstract problems such as logical induction and infinite ethics. Some of this has indirectly made its way to venues like the [alignment forum](https://www.alignmentforum.org/s/KgrG4cQdLtL9DvNr2), and some of our members have managed to get prizes and honorable mentions for the [ELK prize](https://www.lesswrong.com/posts/QEYWkRoCn4fZxXQAY/prizes-for-elk-proposals).

![](https://cdn.discordapp.com/avatars/574119902319869972/49a8c63ea200375f98b6fbc9b067a224.webp)
AI\_WAIFU2022-04-18
ok just had a galaxy-brain idea
So one problem with standard utility functions, especially in situations where the space of outcomes can vary a lot, is that they need to be bounded. Otherwise, when you try to sum over the set of outcomes and the utilties are unbounded, you get a diverging series, and so utilty, and likewise the best action, are consequently undefined.
But utilties are defined over universe descriptions
So consider a decision theory of the form argmax\_pi sum\_M U(M(pi))P(M) That is, we're considering the set of universes that accept a policy as an argument, and then get evaluated by U. And we're trying to find the best pi over all M weighted by some distribution over M

Now if U we're a typical utilty function, it could do something like "count the number of catgirls in M(U) then return that"
But this quickly breaks down, as there can be universes where an infinite amount of catgirls or an unbounded series of catgirls
However, utilties are defined over universe descriptions
So we can define a special U and what that U does is that it looks at the maximum and minumum ordinal number of catgirls that can be instanced in M depending on pi, and uses those 2 numbers to normalize the output for a given U(M(pi))
So if you know M ahead of time and pick the best policy U(M(pi)) = 1 and if you pick the worst U(M(pi)) = 0

I haven't proved it but this seems to have nice properties, agents with this utility function will chase after infinite catgirls in worlds where that's possible and chase after finite catgirls in worlds where it isn't. More importantly, they don't go insane or become apathetic when the actual number of catgirls up for grabs is either vastly larger or vastly smaller than the "range" of values for which their bounded utility function can meaningfully vary. In a situation where the agent is 50-50 between being in a world where a finite or infinite amount of catgirls is up for grabs, the agent won't ignore either possibility when picking it's next move.
This effectively let's you construct a bounded utilty function that acts like an unbounded one.

However, probably the most notable success of this period and the one before it would have to be the various reading groups. EAI has hosted a variety of reading groups either directly or tangentially related to alignment.

The most notable of these is the interpretability reading group, which has been not only scouting out cutting edge papers on neural network interpretability, but also finding authors and bringing them in to present their research. Other things include hosting a modified version of Richard Ngo’s alignment curriculum, to introduce some of our newer members to the field of alignment.

April 2022 – July 2022: EleutherAI’s “Identity crisis”[#](#april-2022--july-2022-eleutherais-identity-crisis)
-------------------------------------------------------------------------------------------------------------

After GPT-NeoX-20B came out, EleutherAI entered a prolonged lull, to the point where we sometimes wondered if EleutherAI was over. There were a couple reasons for this, but the main one was an exodus of former leaders to their next employers and the remaining active members focusing primarily on work that was being organized elsewhere. This began a vicious cycle: people wanted to be where the activity was, leading them to go do work outside of EleutherAI, leading to less research activity in EleutherAI.

### Conjecture[#](#conjecture)

Connor wasn’t satisfied with sporadic and voluntary geese posting on Eleuther, so he finally set out to found an organization where he could legally force employees to plaster the walls with geese pictures (and to [not share infohazards](https://www.lesswrong.com/posts/Gs29k3beHiqWFZqnn/conjecture-internal-infohazard-policy)!).

As wonderful as the pirate life at EleutherAI had been, Connor and others unfortunately ran into many bottlenecks, in particular, that you can’t really get people to do boring things they don’t want to do if you don’t pay them.

Imagine herding cats, but the cats are the smartest people you’ve ever met and also have crippling ADHD.
This is what EleutherAI is like. And what a great, lovely, hive of scum and memery it is! But ultimately to address some of the truly big, hard problems the future holds, a more structured organization was necessary. And hence, [Conjecture](https://www.conjecture.dev/) was born!

Since Conjecture officially launched in March, things have proceeded at breakneck speed. Starting with three founders and 5 early employees in a cramped, musky WeWork, Conjecture has now grown to 18 people full time, the first cohort of the alignment research incubator Refine, the first cohort of SERI MATS in the UK across the street, and many more new qualia of waterfowl and non-waterfowl nature. Conjecture works on making Connor’s dream of ~~making anime disappear come true~~ applied alignment research and diversifying bets in the alignment field.

Conjecture’s work so far has focused on mechanistic interpretability, including work on [polysemanticity](https://www.lesswrong.com/posts/eDicGjD9yte6FLSie/interpreting-neural-networks-through-the-polytope-lens) in neural networks, researching new frames to think about LLMs on [researching GPT-like LLMs](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators) and their properties, on building a tech stack to train and deploy new and existing models very quickly, epistemology, and on increasing coordination in the field to tackle the alignment problem head on.
But above all, Conjecture is a focused, sincere shot at doing whatever it takes to solve alignment, a merry band of buccaneers that don't know how to save the world, but they sure as hell are going to try.

The pirates are industrializing!

### CarperAI[#](#carperai)

Towards the end of 2021, Louis had concluded a research project on preference learning, and was beginning to see the limits of constraining himself to a single discord channel. He was beginning to notice a void in the open source space, there was no one attempting to directly tackle Instruct GPT like models akin to what OpenAI had produced months prior. In mid spring 2022 he set out to build CarperAI, to fill this niche.

CarperAI is focused on the democratization of RLHF and RLHF adjacent methods. They want every lab, both academia and industry, to be able to utilize RLHF in their research. In the past 8 months, CarperAI has grown to 13 full-time and over 40 volunteers, and released two major libraries: [trlX](https://github.com/CarperAI/trlx) and [OpenELM](https://github.com/CarperAI/OpenELM).

### Other Organizations[#](#other-organizations)

**The BigScience Research Workshop**: Stella and Jon got heavily involved in the BigScience Research Workshop in mid 2022. Beyond the research they did in BigScience, this had another major impact: meeting new collaborators (incl. Hailey Schoelkopf, Lintang Surawika, and Colin Raffel) that would become prominent contributors to EleutherAI projects in the future.

**Stability AI**: Most people who had experience working on text-to-image modeling were scooped up by Stability. While Stability supports them releasing their work open source, and some like Katherine continued to collaborate with people in EleutherAI, this work gradually shifted text-to-image research out of our server.

**OpenAI**: Ben Wang joined OpenAI to work on language model research. Leo Gao joined OpenAI to continue his work on his alignment research agenda. Leo continues to contribute to alignment discussions in EleutherAI.

### Closing the Book on VQGAN-CLIP[#](#closing-the-book-on-vqgan-clip)

While members of EleutherAI did research on a variety of topics during this period, there was very little that was being lead by EleutherAI as an organization. The major exception is that we finally [wrote a paper on VQGAN-CLIP](https://arxiv.org/abs/2204.08583), the original text-to-image synthesis and editing model we developed back in spring 2021. It was a tad late (by which we mean it has been used a billion times before we wrote a paper about it) but it would be eventually published at ECCV 2022.

August 2022 and Beyond: Reorganization and Revitalization[#](#august-2022-and-beyond-reorganization-and-revitalization)
-----------------------------------------------------------------------------------------------------------------------

Stella eventually reached the same conclusion that Connor did: EleutherAI was too chaotic and unstructured, and it’s extremely difficult to retain talent when you’re effectively teaching people skills that pay hundreds of thousands of dollars per year and then asking them to do the work for free. She had also gotten offers from some LLM companies, but didn’t think it would be right for her to stop working on open source AI. Instead, she spent the summer quietly working on a plan to reorganize and revitalize EleutherAI as a non-profit research institute.

Stella hypothesized that the core problem was getting people back to work, or more precisely, getting people to start doing work organized in the discord server. As we would later detail in a [NeurIPS Workshop paper on large ML collaborations](https://arxiv.org/abs/2210.06413), doing research in the public view has always been a crucial component of EleutherAI’s impact and marketing strategy. The reason to come to our discord server to talk about AI research was because it was a place where people could get unprecedented access to people doing cutting edge AI research, and AI researchers could interact with one another in an unstructured fashion. Even for people who didn’t participate in our research projects, being around the people who did was one of the major draws.

To jumpstart the process, she started the #interpretability-over-time channel for her Pythia project, talked Aran Komatsuzaki and Katherine Crowson into organizing on-going research projects in the discord server (#improved-t5 and #k-diffusion respective), and worked to help current and previous core contributors secure funding to do EleutherAI as a part-time job.

A great deal of this revitalizing activity was facilitated by StabiltyAI, which stepped in to lend critical support so that EleutherAI could continue to pursue it’s research objectives and revitalize itself. Slowly but surely, the plan worked. As activity stepped up in EleutherAI (and as we began to better advise our work, via our new Twitter and the #announcements channel)

### Shifting Research Priorities[#](#shifting-research-priorities)

The past six months had shown that the world was rather fundamentally different from the way it was when we got started. When we released GPT-NeoX-20B in February 2022, it was the largest freely and publicly available language model in the world. By the end of 2022, GPT-NeoX-20B was in a three-way tie for sixth and [the largest](https://arxiv.org/abs/2205.01068) matched the size of the original GPT-3 model.

EleutherAI got into large scale AI training because we felt that researchers needed to have hands-on access to technologies like GPT-3. When we got started that meant that we had to train and release the models for people to use. Now, we are free to pursue the research we wanted to use these models to do in the first place, to study topics like interpretability, alignment, and learning dynamics. This was always the plan, but when we reconvened towards the end of 2022 it felt for the first time like the torch had been passed and that the world wasn’t going to be reliant on us training models to get access to them. We were now free to study whatever we wanted… the only question was, what is that?

![](/images/blog/year-two/i-want-you-to-work-on-alignment.png#center)

### NLP Research[#](#nlp-research)

While we continued to do some NLP research, the contents and context of that research has changed substantially, focusing on building better scientific understandings of the functionality of language models and making non-SOTA models more useful and accessible to small scale practitioners. These projects include:

* The “Improved T5” project, which investigates scaling laws for encoder-decoder models, and how many of the new developments for decoder-only models can be adapted to encoder-decoder ones.
* The PolyGlot Project, which trains small LLMs in a variety of languages. Currently, almost all billion+ parameter LMs are trained in one of three languages: English, Chinese, and “massively multilingual.” The Polyglot team trained and released Polyglot-Ko, a series of Korean language models that include the most performant FOSS Korean language model in the world and is now investigating “localized” multilingual models focused on South and South East Asian languages, Romance Languages, and Nordic languages.
* Jason Phang is working on hypermodels for LLMs for back-propagation-free model adaptation.

### Interpretability Research[#](#interpretability-research)

With our new-found invigoration, much of our energy has gone to interpretability work. Our biggest release along these lines so far has been [Pythia](https://github.com/EleutherAI/pythia), a suite of language models ranging from 19M to 13B parameters that was designed to study how models develop over the course of training and how those patterns change as models are scaled will yield important insights. This has been a pet project of Stella’s for close to a year now, and thanks to the hard work of Hailey Schoelkopf and Shivanshu Purohit it has finally come to fruition.

[We released the models to the public](https://mobile.twitter.com/AiEleuther/status/1603755836136128514) on December 16th and the positive response from the community has been immediate: within a week two papers had already cited them and a dozen people had reached out to express their excitement about the models. We have a paper on it under review, and are leveraging the suite to do more Interpretability research. Currently the bulk of this work is oriented towards exploring the causes of memorization, but we also have threads about understanding how models learn social biases from data and how pretraining frequencies influence model knowledge.

We’ve also seen a surge of interest in work on Eliciting Latent Knowledge (ELK). Early in 2022 Igor Ostrovsky, Nostalgebraist, and Stella developed the “Tuned Lens,” a variation on Nostalgebraist’s [Logit Lens](https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens). Unfortunately due to a combination of Igor starting a company and others being busy it languished for a bit until Nora Belrose picked it up again. Originally tasked with carrying the original idea across the finish line, her work has blossomed into a collection of research projects working on problems such as extracting interpretable basis for LLMs and detecting “deceptive-like” behavior.

Together, the “interpreting across time” and “eliciting-latent-knowledge” projects have prompted a flurry of activity in interpretability research, with nearly a dozen people actively working on papers on the subject at time of writing. The original channel quickly became overcrowded, so a new section of the discord server was set up with more channels and better organization.

### Alignment Research[#](#alignment-research)

Reading groups and alignment related discussions on EAI continued, but with many of our alignment researchers leaving to work on alignment at Conjecture and OpenAI, we were not in a good position to do alignment research outside of interpretability. With the revitalization of EAI, this started to change in the second half of 2022.

Some alignment related concerns that are particularly notable to some of our members are alignment failures related to [embedded agency](https://www.lesswrong.com/posts/p7x32SEt43ZMC9r7r/embedded-agents). For instance, in RL there is a classic diagram that is taught to everyone on day 1 that shows an agent interacting with an environment in a loop:

![The canonical RL setup, but is it actually a good model of agents, especially once they get really smart and powerful?](/images/blog/year-two/rl.png#center)
The canonical RL setup, but is it actually a good model of agents, especially once they get really smart and powerful?

Notably, it makes a few assumptions, like that reward comes from the environment, and that the agent is effectively separated from the environment. But this is not necessarily a good approximation of what would actually happen in a deployed system. The agent is part of the environment, and reward comes from somewhere inside that same environment. So speculative failure modes such as [wireheading](https://www.lesswrong.com/tag/wireheading) are back on the table. (See [these](https://www.alignmentforum.org/posts/jP9cKxqwqk2qQ6HiM/towards-deconfusing-wireheading-and-reward-maximization) [posts](https://www.alignmentforum.org/posts/REesy8nqvknFFKywm/clarifying-wireheading-terminology) for more of our thoughts on wireheading and its connection to embeddedness.)

In fact, the issue appears much broader than this, as in practice any mechanism for aligning AI systems, or correcting their behavior after they’ve been deployed is going to be part of the same environment as the agent, and prone to it’s interference.

This high level concern motivated the Alignment-Minetest project, which aims to modify the open source voxel engine [Minetest](https://www.minetest.net/) so that we can look for wireheading and other undesirable incentives in agents and world models.

We are also interested in working on more projects in directions like inner alignment and ELK. While we've done some theoretical work in these directions (see: [1](https://www.alignmentforum.org/posts/oijJc8Mu2jPNgpuvy/asot-some-thoughts-about-deceptive-mesaoptimization), [2](https://www.alignmentforum.org/posts/Jiy3n5KMsGGJ6NNYH/asot-some-thoughts-about-lm-monologue-limitations-and-elk), [3](https://www.alignmentforum.org/posts/kWTko53s2DqTeprjz/asot-observations-about-elk)), we would be excited to do more theoretically-grounded empirical work in these directions using language models.

Going into 2023, we’re planning to spin up more alignment and interpretability projects and become significantly more involved with the broader alignment community.

Reflections[#](#reflections)
============================

We’ve asked some of our members to give their thoughts on EAI over the past year, here’s some of their responses:

>
> EleutherAI feels like a place where the only limits are just how big we can dream. The people I’ve met here are driven, scarily smart, and hypercompetent. More than anything, they’re united by an understanding of what defines good, lasting research that others should and do care about, and the ability to execute well on that understanding.
>
>
>

>
> It feels like I fell backwards right into the best opportunity I’d ever get–I initially got the chance to be involved in Eleuther by pretty much being in just the right place at just the right time. People have been so welcoming and willing to share knowledge and research with me as long as I’m willing to learn, and I have indeed learned a ton in these past months. It’s been almost impossible to avoid absorbing the same enthusiasm I see here every day. I can’t wait to see what the future holds for EleutherAI, and can’t wait to keep repeatedly making the impossible happen.
>
>
>

-Hailey Schoelkopf

>
> "Working at Eleuther AI on CARP genuinely gave me an experience in FOSS project management that I thought would be near impossible to achieve elsewhere. Being able to coordinate projects at scales that are unfathomable for most independent researchers let alone PhD students has opened my eyes to the kinds of projects that are achievable by randos on discord.
>
>
> Came for the geese, stayed for the amazing collaborators."
> -CARP team
>
>
>

>
> "I joined the EleutherAI discord server on my search for an open-sourced Github copilot, but what I found was much more than that!
>
>
>

>
> EAI introduced me to the realm of transformers and the beautiful applications therein. From using TF-IDF and wordnets to being involved in state-of-the-art research, It has been a wild ride for me. And am lucky to have been guided by amazing folks along the way :) Can't wait to see what next year has in store for us!"
>
>
>

-Orz

>
> "As far as I can tell, EAI's primary virtue is to be wrong in interesting and useful ways, and to do so consistently. This virtue is also my best guess as to the simple algorithmic core of alignment research."
>
>
>

-Quintin Pope

>
> "Sure, EAI is the best 'great filter' for AI research. A place where someone like me (a novice) can directly interact with AI experts, where new ideas are born in casual chats on a Tuesday night and papers made in a discord channels are published in big conferences. It is all that and more. But for most us, it is also a home. A place where creativity, critical thought, experimentation, along with weird memes and endless debates, is just another day in the week. A safe heaven."
>
>
>

-Gabriel Syme

>
> "EleutherAI doesn't have it all, but it has a lot. It has AI alignment discussion, it has AI capabilities discussion, it has memes, it has community, it even has geese. So many of my favorite things. I've made several real friends on the server. Here's to making a few more in the next year."
>
>
>

-TurnTrout

>
> "I think Eleuther is really important because it’s by far the most transparent organization working on AI interpretability and alignment today. In my view, too many alignment-minded folks tend to think that open source models and open publishing of results are bad for the world, either because they accelerate AGI timelines or because they increase the risk that AI advances could fall into “the wrong hands.” While I understand this view, I disagree with it pretty strongly, so I’m glad to see that Eleuther— which has been pro-openness from the very beginning— has shifted toward interpretability work recently. I first got involved with Eleuther after a friend told me about some of the research that was being done in the interpreting-across-time channel, and since then I’ve done a lot of work with the help of Eleuther compute and volunteers which I definitely couldn’t have done otherwise."
>
>
>

-Nora Belrose

>
> "One of my disciplines that I'd picked up over the years was high-performance computing and networking, and that's how I segued into working primarily with machine learning. I first encountered EleutherAI's work when looking for alternatives to AI storytellers, and loaded GPT-Neo 2.7B on a NVidia Jetson AGX. One thing lead to another when I got involved with the Anlatan crew and their first product, NovelAI. Their product utilized EleutherAI models with custom inference code, and I got involved with the backend programming there. I was brought in to help with CoreWeave's InfiniBand infrastructure for EleutherAI's GPT-NeoX 20B training project. I was happy to help, as I am a big believer in open models. This further expanded into architecting and building the goose.ai scalable inference infrastructure for an Anlatan-CoreWeave joint venture. By working with EleutherAI on advancing the research in large models, it has lead to a very interesting career shift into machine learning infrastructure leveraging my HPC knowledge and systems programming. EleutherAI is a collection of some of the sharpest and nerdiest folks that I have the honor of meeting, knowing, and working with -- and I look forward to what the coming year will bring us."
>
>
>

-Wes Brown

Extra Memes[#](#extra-memes)
============================

![](/images/blog/year-two/nvidia.png#center)

![](/images/blog/year-two/goose-taste.png#center)

![](/images/blog/year-two/can-we-unironically-just-train-our-own-gpt3.png#center)

![](/images/blog/year-two/getting-into-ai.png#center)

![](/images/blog/year-two/bitter-lesson.png#center)

![](/images/blog/year-two/eai-ruin.png#center)

![](/images/blog/year-two/final-decade.png#center)

![](/images/blog/year-two/in-this-house.png#center)

![](/images/blog/year-two/paperclip-spider-man.png#center)

![](/images/blog/year-two/paperclip-cope.png#center)

Stella Biderman
Shivanshu Purohit
Curtis Huebner
Leo Gao
Connor Leahy
Eric Hallahan


The View from 30,000 Feet: Preface to the Second EleutherAI Retrospective

Over a year and a half have passed since EleutherAI's last retrospective, and a great deal of things have changed. [In the first year, what started off as a Discord server created by some TPU enthusiasts grew into a much larger and more vibrant community.](./year-one/) Since then, the EleutherAI collective has gone on to do many things, including becoming an inspirational launch point, stepping stone, and template for its members and many new organizations.

Given that we have so much to share in a second retrospective, we have condensed the important takeaways and announcements here. We look forward to sharing the full story soon!

Research[#](#research)
----------------------

![](https://pbs.twimg.com/media/FjTEtSVWYAA_Csy?format=jpg&name=small)
Eric Hallahan2021-10-06

![](https://cdn.discordapp.com/attachments/747850033994662000/895439725253500928/unknown.png)

![](https://cdn.discordapp.com/attachments/445630505606578212/1029837418435723274/default_green.png)
Jason Phang2021-10-06
I would like to set a new state-of-the-art for code achieving 80%+ ImageNet accuracy, using 278 tokens instead 280, beating Anonymous et al. (2021) while also including imports
```
from torch.nn import *
def c(h,d,k,p,n):S,C,A=Sequential,Conv2d,lambda x:S(x,GELU(),BatchNorm2d(h));R=type('',(S,),{'forward':lambda s,x:s[0](x)+x});return S(A(C(3,h,p,p)),*[S(R(A(C(h,h,k,1,k//2,1,h))),A(C(h,h,1))) for _ in [0]*d],AdaptiveAvgPool2d((1,1)),Flatten(),Linear(h,n))
```

![](https://cdn.discordapp.com/attachments/445630505606578212/1029829972422037564/bmk.png)
bmk2021-10-06
new sota, 275 chars
```
from torch.nn import*
def c(h,d,k,p,n):S,C,A=Sequential,Conv2d,lambda x:S(x,GELU(),BatchNorm2d(h));R=type('',(S,),{'forward':lambda s,x:s[0](x)+x});return S(A(C(3,h,p,p)),*[S(R(A(C(h,h,k,1,k//2,1,h))),A(C(h,h,1)))for _ in[0]*d],AdaptiveAvgPool2d((1,1)),Flatten(),Linear(h,n))
```

![](https://cdn.discordapp.com/attachments/445630505606578212/1029837418435723274/default_green.png)
Jason Phang2021-10-06

![](https://cdn.discordapp.com/attachments/747850033994662000/895452438855839744/unknown.png)

EAI setting SotA in real time

EleutherAI members have authored 28 papers, trained dozens of models, and released 10 codebases in the past 18 months. Some notable highlights include:

[GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://arxiv.org/abs/2204.06745)

This paper discusses our work on our largest-to-date open-source LLM. At time of release last February, it became the largest and most performant open-source autoregressive language model.

[VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://arxiv.org/abs/2204.08583)

It took us about a year, but we finally wrote up our OG text-to-image work!

[Multitask Prompted Training Enables Zero-shot Task Generalization](https://arxiv.org/abs/2110.08207)

This BigScience-lead paper introduced the T0 language model and jumpstarted interest in task-structured data.

[EleutherAI: Going Beyond "Open Science" to "Science in the Open"](https://arxiv.org/abs/2210.06413)

This paper, written for the NeurIPS Broadening Research Collaborations Workshop in ML, details our experience doing open collaborative science and gives an inside look into our thinking on an organizational level.

[OpenFold: Retraining AlphaFold2 Yields New Insights Into Its Learning Mechanisms and Capacity for Generalization](https://www.biorxiv.org/content/10.1101/2022.11.20.517210)

EleutherAI played a minor role in this paper, mostly supporting the interpretability work, compute, and HPC knowledge. It is a paper we are very excited about though, and a demonstration of both very high-quality interpretability research and the impact that sponsoring relatively small-scale trainings can have.

A full list of papers, models, and other research output from EleutherAI can be found [on our website](https://www.eleuther.ai/papers).

New Organizations[#](#new-organizations)
----------------------------------------

While some have started in our first year, during this most recent year we've seen many other similar organizations rise to prominence. For instance, [LAION](https://laion.ai/) has cranked out two massive image datasets and supported the development of the now-famous DALL-E Mini. Another example would be the [OpenBioML](https://openbioml.org/), which started off as a spinoff Discord server building on the AlphaFold2 replication work of Phil Wang (@lucidrains) and Eric Alcaide (@hypnopump) before becoming a hub for interaction between the open-source AI and BioML communities. We've also seen members start a plethora of smaller communities focused on individual projects, such as @BlinkDL's RWKV and Aran's data collection for work based on [Minerva](https://ai.googleblog.com/2022/06/minerva-solving-quantitative-reasoning.html).

Most notably though, three groups of researchers have left EleutherAI to start their own organizations. EleutherAI founders Connor Leahy (@Connor) and Sid Black (@sid) are now the founders of a new alignment research organization called [Conjecture](https://www.conjecture.dev/), Louis Castricato has started a lab called [CarperAI](https://carper.ai/) that focuses on preference learning and RLHF, and Tanishq Abraham (@ilovescience) has started [MedARC](https://www.medarc.ai/), which focuses on biomedical applications of cutting edge AI technologies such as large language models.

Speaking of his leadership departure, Connor had the following to say:

>
> EleutherAI has been the experience of a lifetime. I'm very thankful to have been allowed to have gone on this amazing journey with all the amazing people I met along the way. It's been an honor and a privilege to shepherd EleutherAI through its earliest days to where we are now. But alas, I am now needed elsewhere.
>
>
> AI is advancing rapidly, and the alignment problem is still far from being solved. If we want the future to go as amazingly as it can, we have huge challenges ahead of us that need addressing. EleutherAI has been invaluable in allowing me to gain the skills and friendships necessary to allow me to take the next steps, but with a heavy heart, those next steps are taking me someplace else.
>
>
>
> ![Don't worry, everything is normal.](https://cdn.discordapp.com/attachments/445631164871344139/1080656876515115118/image24.png)
> Don't worry, everything is normal.
>
>
>
>
> I will be officially stepping down as an organizer and leader of EleutherAI, along with my friend, colleague, and fellow EAI founder Sid, to focus my attention fully on [Conjecture](https://www.conjecture.dev/)), and on ensuring a better future for everyone. I will be handing full control and responsibility for EleutherAI to my trusted friends.
>
>
> I cannot thank every single person at EleutherAI enough for everything we have created together, you are all truly wonderful, and I am certain our paths will cross again. I will still be around for chatting and advice, and the occasional late-night schizoposting hour. And if you want to follow along with what we're up to at Conjecture, be sure to follow [our posts on LessWrong](https://www.lesswrong.com/tag/conjecture-org) and [join our Lemma discord server](https://discord.gg/tezFMvnTe6), where we publicly test our tools and products.
>
>
> So long, and thanks for all the memes!
>
>
>
>
>
>
> **Connor Leahy (CEO, Conjecture)**
>
>

The EleutherAI Institute[#](#the-eleutherai-institute)
------------------------------------------------------

Last, but certainly not least, the newest organization to come out of our Discord server is… EleutherAI itself.

In the past two-and-a-half years, we have accomplished some amazing things, and the world has taken note. Despite this success, many of our core members have moved on to jobs elsewhere or started their own companies and organizations. It has become abundantly clear that the biggest blocker in what we could be accomplishing is the fact that working a forty-hour workweek and doing cutting-edge AI research on the side is unsustainable for most contributors. Therefore, we are thrilled to announce that we are forming a non-profit research institute, and we are excited to be able to say that over twenty of our regular contributors are now working full-time doing research.

The EleutherAI Discord server will remain true to the values it has had since its creation, and we have no intention of restricting access or hiding our research from the public. As Stella Biderman frequently describes it, EleutherAI is like a research institute with open doors. One where anyone can wander in, listen in on meetings, and even chime in if they want. That is the model we intend to keep, and we are excited to be able to continue to demonstrate the value of open research.

Times have changed significantly since EleutherAI was founded, and there is substantially more interest in training and releasing LLMs than there once was. EleutherAI entered into large-scale AI training because we felt that researchers like ourselves needed to have hands-on access to technologies like GPT-3, and back then that meant that we had to train and release the models for people to use. Thanks to those efforts, we are free to pursue the research we wanted to use these models for to begin with—studying topics like interpretability, alignment, and learning dynamics of large transformer models. For 2023, we are also planning to spin up more alignment and interpretability projects, and become significantly more involved with the broader alignment community.

Our new organization, funded by a mix of charitable donations and grants, will be run by Stella Biderman (@StellaAthena), Curtis Huebner (@AI\_WAIFU), and Shivanshu Purohit (@triggerhappygandi), with guidance from a board of directors which will include EleutherAI co-founder Connor Leahy and UNC's Colin Raffel.

If you have any questions about the EleutherAI Institute or are interested in making a charitable donation, please reach out to [contact@eleuther.ai](mailto:contact@eleuther.ai).

Stella Biderman
Curtis Huebner
Connor Leahy
Eric Hallahan


Announcing GPT-NeoX-20B

**As of February 9, 2022, GPT-NeoX-20B checkpoints are available for [download from The Eye](https://the-eye.eu/public/AI/models/GPT-NeoX-20B) under Apache 2.0.** More in-depth information on GPT-NeoX-20B can be found in the [associated technical report on arXiv](https://arxiv.org/abs/2204.06745).

Looking for a demo? Try GPT-NeoX-20B via CoreWeave and Anlatan's inference service, [GooseAI](https://goose.ai/ "We're dead serious, that is actually what it is called.")!

---

After a year-long odyssey through months of chip shortage-induced shipping delays, technical trials and tribulations, and aggressively boring debugging, we are happy to finally announce EleutherAI's latest open-source language model: GPT-NeoX-20B, a 20 billion parameter model trained using our [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) framework on GPUs generously provided by our friends at [CoreWeave](https://www.coreweave.com/).

GPT-NeoX-20B is, to our knowledge, the largest publicly accessible pretrained general-purpose autoregressive language model, and we expect it to perform well on many tasks.

We hope that the increased accessibility of models of this size will aid in [research towards the safe use of AI systems](https://blog.eleuther.ai/why-release-a-large-language-model/), and encourage anyone interested in working in this direction to reach out to us.

As a thank you to our generous compute donors, we are delaying the public downloadable release of the model by 7 days. On February 9, 2022, the full model weights will be downloadable for free under a permissive Apache 2.0 license from The Eye.

There will be a #20b channel set up in our Discord for discussions of this model. Please note that much like our other language models and codebases, GPT-NeoX and GPT-NeoX-20B are very much research artifacts and we *do not recommend deploying either in a production setting without careful consideration*. In particular, we strongly encourage those looking to use GPT-NeoX-20B to read the [paper](https://arxiv.org/abs/2101.00027) and [datasheet](https://arxiv.org/abs/2201.07311) on our training data. There are still bugs to be ironed out and many inefficiencies that could be addressed---but hey, we do this in our free time, give us a break lol

---

| Task | Category | Babbage | Curie | GPT-J-6B | FairSeq-13B | GPT-NeoX-20B | DaVinci |
| --- | --- | --- | --- | --- | --- | --- | --- |
| LAMBADA | Sentence Completion | 62.49% | 69.51% | 68.29% | 70.95% | 72.00% | 75.16% |
| ANLI R1 | Natural Language Inference | 32.40% | 32.80% | 32.40% | 34.00% | 34.00% | 36.30% |
| ANLI R2 | Natural Language Inference | 30.90% | 33.50% | 34.00% | 33.00% | 34.40% | 37.00% |
| ANLI R3 | Natural Language Inference | 33.75% | 35.50% | 35.50% | 34.75% | 35.40% | 36.83% |
| WSC | Coreference Resolution | 54.54% | 49.54% | 49.54% | 55.44% | 50.00% | 59.18% |
| WinoGrande | Coreference Resolution | 59.51% | 64.56% | 64.01% | 67.40% | 66.10% | 69.93% |
| HellaSwag | Sentence Completion | 40.38% | 54.81% | 36.53% | 57.69% | 53.50% | 63.46% |
| Average |  | 44.85% | 48.60% | 45.75% | 50.43% | 49.34% | 53.98% |

Accuracy on standard language modeling tasks.

| Subject Group | Babbage | Curie | GPT-J-6B | FairSeq-13B | GPT-NeoX-20B | DaVinci |
| --- | --- | --- | --- | --- | --- | --- |
| Humanities | 27.01% | 26.48% | 28.07% | 27.27% | 28.70% | 32.30% |
| Social Science | 27.94% | 29.24% | 28.73% | 27.94% | 30.80% | 35.87% |
| STEM | 25.83% | 24.25% | 25.71% | 24.63% | 27.20% | 28.60% |
| Other | 26.86% | 28.84% | 27.95% | 27.33% | 29.20% | 36.85% |
| Average | 26.91% | 27.20% | 27.62% | 26.79% | 28.98% | 33.41% |

Zero-shot accuracy of factual knowledge by subject group, as measured by the [HendrycksTest](https://arxiv.org/abs/2009.03300) evaluation.


A Preliminary Exploration into Factored Cognition with Language Models

We perform a series of experiments using GPT-3 with decomposition to
perform complex toy tasks that it is otherwise unable to solve. The goal
of these experiments is to provide some preliminary evidence for the
viability of factored cognition in real world models.

For our synthetic task, we chose a series of various arithmetic tasks.
Aside from the ease of generating examples, another advantage of
arithmetic related task settings is GPT-3's inability to perform
even simple mathematical operations. While there is evidence to suggest
that this may be due to the peculiarities of the BPE encoding,
these fixes are very narrowly domain specific to mathematics and not
generalizable to the kinds of tasks we would like general aligned AI
systems to be able to perform. Instead, we show that by decomposing the
task, we can achieve a major improvement on this task. Intuitively, just
as most humans are unable to perform complex, multi-step calculations
without pen and paper, it is also reasonable to expect that language
models would struggle with directly computing the answers to these
problems, especially given the fixed amount of computation that
Transformers are able to harness in one step.

### Nested Expression Evaluation[#](#nestedeval)

```

**((9 \* 2 + 7) \* 1 - 1) =** ((18 + 7) * 1 - 1) = (25 * 1 - 1) = (25 - 1) = 24
**(9 + (1 - 3) \* 7) \* 6 =** (9 + -2 * 7) * 6 = (9 + -14) * 6 = -5 * 6 = -30
**(2 + (5 - 1) \* 8) \* 9 =** (2 + 4 * 8) * 9 = (2 + 32) * 9 = 34 * 9 = 306
**2 \* (4 - 1 \* (2 - 4)) =** 2 * (4 - 1 * -2) = 2 * (4 - -2) = 2 * 6 = 12
**(6 \* (5 \* 2 - 3) - 9) =** (6 * (10 - 3) - 9) = (6 * 7 - 9) = (42 - 9) = 33

```

Example Nested Expression Evaluation prompt. **Bold** indicates prompt. Underline indicates answer extracted from response.

We explore decomposing a task into a series of steps, without any
branching. The main advantages of using nested expressions are that they
are easy to generate and automatically evaluate, and they give naturally
to stepwise decomposition, as the expression must be evaluated from the
inside out.

To generate the arithmetic expression, we recursively generate a number
of nested operations, alternating between multiplication and
addition/subtraction, such that to evaluate the expression at least
`depth` operations must be carried out in series in the correct order.
In terms of difficulty, the `depth` $= 1$ task corresponds roughly to
the "One-digit composite (1DC)" task in the GPT3 paper, though it is slightly
more difficult because the order of multiplication and addition are not
guaranteed to be the same.

We consider three different settings for this experiment:

`multistep`

The model is given fewshot prompts in which each step
of the evaluation is provided. The model is given only the problem
and asked to generate as many intermediate steps as it would like,
truncated at a limit of 256 tokens. The value after the last `=` is
taken to be the model's answer.

`direct`

The model is given fewshot prompts in which only the
problem and solution are provided. Only one value is generated from
the model.

`direct_padded`

To control for the fact that the direct setting
is significantly longer than the multistep setting and thus requires
more forward passes during which more computation could
theoretically take place, we also consider the setting where we add
space tokens before the `=` to pad it to the same length as in the
multistep setting

In each experiment, we do few-shot with $k = 10$.

For the multistep setting, in addition to accuracy, we also compute the
following metrics, which are not applicable to either direct setting, to
provide additional insight into the failure modes:

Malformed

This metric indicates the proportion of model
responses that are malformed. A model response is considered
malformed if it does not contain the correct number of intermediate
steps, or if any of the intermediate steps fails to parse; very
often, this is due to unbalanced brackets. A malformed response
indicates a catastrophic failure.

Mistakes

This metric indicates the proportion of steps that are
incorrect. We observed that often, a single mistake will be made in
one step, and that error will be propagated to the final answer
despite only one mistake being made. We included this metric to
distinguish between models which make many errors to arrive at
incorrect answers and models which only make few errors. Malformed
responses are not considered for the mistakes metric.

![Accuracy of GPT-3 on evaluating trees of functions of various
complexities (depths). By factoring the problem, GPT-3 is able to
achieve significantly greater reliability on a problem that it is unable
to tackle
directly.](/images/blog/factored-cognition/branching_davinci_barplot_depth_acc.png)

Accuracy of GPT-3 on evaluating trees of functions of various complexities (depths). By factoring the problem, GPT-3 is able to achieve significantly greater reliability on a problem that it is unable to tackle directly.

![Accuracy of GPT-3 davinci on arithmetic problems of various amounts
of nesting.](/images/blog/factored-cognition/davinci_barplot_depth_acc.png)

Accuracy of GPT-3 davinci on arithmetic problems of various amounts of nesting.

| Depth | Accuracy | Mistakes | Malformed |
| --- | --- | --- | --- |
|  | `multistep` | `direct` | `direct_padded` |  |  |
| 1 | 66.35 | 15.40 | 13.90 | 33.78 | 4.50 |
| 2 | 39.45 | 4.40 | 2.75 | 59.28 | 25.75 |
| 3 | 17.20 | 1.60 | 1.45 | 79.61 | 46.25 |

Metrics for the Nested Expression Evaluation experiment.

The direct\_padded setting consistently performs significantly worse than
the direct setting, and so we do not analyze it in detail.

As a sanity check, our depth=1 direct accuracy is 5.9 percentage points
lower than the 1DC accuracy of 21.3 in the GPT-3 paper. This difference is
explained by the use of k=100 for multishot in the GPT-3 paper versus k=10 in this
paper, and the minor differences in the task.

As depth increases, the number of mistakes made increases. One major
problem with factored cognition is that every step must be performed
correctly for the final answer to be valid, and the increase in mistakes
as problem complexity increases suggests that each individual step has
become too complex for the model to handle.

Despite starting out almost negligible, the number of malformed outputs
also increases. These failures are predominantly due to formatting
problems like unbalanced brackets, due to the increased layers of
nesting in the higher depth settings. In addition to the complexity of
the step being too large, one other explanation is that the few-shot
prompt was insufficient in specifying the task to the model.

### Branched Nested Function Evaluation[#](#nestedbranchedeval)

```
**Today we will be looking at evaluating functions.

f(x) = 1 \* (x + -3) + 1 \* (x + -2)
g(x) = 2 \* f(x + 1) + 5 \* f(x + -3)
h(x) = 5 \* g(x + 2) + 2 \* g(x + 2)

To calculate the value of g(5), we first need to calculate** f(5 + -2) and f(5 + 2), and since 5 + -2 = 3 and 5 + 2 = 7, they are equal to f(3) and f(7), respectively.
**To calculate the value of h(-9), we first need to calculate** g(-9 + -3) and g(-9 + 1), and since -9 + -3 = -12 and -9 + 1 = -8, they are equal to g(-12) and g(-8), respectively.
**To calculate the value of g(4), we first need to calculate** f(4 + -2) and f(4 + 2), and since 4 + -2 = 2 and 4 + 2 = 6, they are equal to f(2) and f(6), respectively.
**To calculate the value of h(-1), we first need to calculate** g(-1 + -3) and g(-1 + 1), and since -1 + -3 = -4 and -1 + 1 = 0, they are equal to g(-4) and g(0), respectively.
**To calculate the value of g(-3), we first need to calculate** f(-3 + -2) and f(-3 + 2), and since -3 + -2 = -5 and -3 + 2 = -1, they are equal to f(-5) and f(-1), respectively.
```

Example Branched Nested Function Evaluation Decomposition prompt

```
**Today we will be looking at evaluating functions.

f(x) = 4 \* (x + -1) + 4 \* (x + 2)
g(x) = 5 \* f(x + 2) + 2 \* f(x + -1)
h(x) = 1 \* g(x + 3) + 2 \* g(x + -3)

Given that f(5) = 5 and f(2) = 20, we can compute that g(3) =** 5 * f(3 + 2) + 2 * f(3 + -1) = 5 * f(5) + 2 * f(2) = 5 * 44 + 2 * 20 = 220 + 40 = 260
**Given that f(-5) = -5 and f(-8) = -60, we can compute that g(-7) =** 5 * f(-7 + 2) + 2 * f(-7 + -1) = 5 * f(-5) + 2 * f(-8) = 5 * -36 + 2 * -60 = -180 + -120 = -300
**We can compute that f(9) =** 4 * (9 + -1) + 4 * (9 + 2) = 4 * 8 + 4 * 11 = 35 + 38 = 76
**Given that g(4) = 36 and g(-2) = -20, we can compute that h(1) =** 1 * g(1 + 3) + 2 * g(1 + -3) = 1 * g(4) + 2 * g(-2) = 1 * 316 + 2 * -20 = 316 + -40 = 276
**We can compute that f(6) =** 4 * (6 + -1) + 4 * (6 + 2) = 4 * 5 + 4 * 8 = 23 + 26 = 52
```

Example Branched Nested Function Evaluation Recombination prompt

To explore a setting with a very high amount of branching, we explore a
setting where a series of functions are defined, each in terms of
multiple copies of the last. This task increases exponentially in
complexity whereas the non-branched task increased linearly in
complexity. We use GPT-3 to perform both the decomposition (which
entails making further queries given a function to evaluate) and
combination (given a query and computed function values, compute the
result to the query).

There are two types of steps: decomposition and recombination. To
evaluate a function, first, a decomposition step is used to obtain the
dependencies of the expression, if any, and then this information is
combined in the recombination step. On decomposition steps, the model is
given a function call and asked to return a list of function calls that
must first be computed before the original call can be evaluated. On
recombination steps, the values computed in the corresponding
decomposition step are provided in the context along with the function
to evaluate, and the model is asked to evaluate the original function.

To improve the accuracy, whenever a multistep calculation is needed, the
Nested Expression Evaluation technique is used as well.

| Depth Accuracy
|  `factored` `direct` | | |
 | |
| --- | --- | --- | --- | --- |
| 1 | 22.0 | 2.8 |
| 2 | 5.7 | 0.6 |
| 3 | 0.9 | 0.0 |

Metrics for the Branched Nested Function Evaluation experiment.

The factored accuracy on the branching case is lower than on the nested
case, and decays far quicker. While on nested depth 2 is more than half
the accuracy of depth 1, on branched depth 2 is less than a third the
accuracy of depth 1. Overall, this shows that the factoring approach
also works for tasks with large amounts of branching.

### Open Ended Math Problem Evaluation[#](#open-ended-math-problem-evaluation)

Since the previous experiments focused on very simple mathematical
tasks, we also conducted an experiment using significantly more
difficult math problems from MathQA.

The MathQA dataset, which is derived in part from the AQuARAT dataset,
contains math word problems and rationales in three different formats.
One is directly taken from AQuARAT and is freeform text written by a
human---however, the text is extremely noisy and oftentimes
inconsistent, incomplete, or inaccurate. The other two are unique to
MathQA and consist of cleaner, systematized versions of the solution
explanations to the problems expressed in terms of functions. In
particular, one is the typical $f(x)$ notation with the function on the
left and thus composition written from right to left; as this poses a
problem for left-to-right autoregressive models, MathQA also includes an
alternate version of the explanations such that it is written
left-to-right entirely in the order that the functions are evaluated.
MathQA does however still inherit some of the noisiness of AQuARAT, such
as strange formatting. We run experiments for each of these three types
of rationales.

| Fewshot $k$ | Steps | Accuracy |
| --- | --- | --- |
| 0 | 1 | 21.81 |
| 5 | 1 | 20.77 |
| 5 | 2 | 22.48 |

Metrics for the Open Ended Math Problem Evaluation experiment.

Since MathQA is a multiple choice task with 5 possible choices, the
random baseline score is 20%.

We found that GPT-3 is not able to solve problems from this dataset
either with or without rationales in few-shot examples. We speculate
that this dataset is hard for GPT-3 because of the noisiness of the task
and because there are still large conceptual leaps required to come up
with the necessary computations from the word problem that are not
included in the rationales.

In MathQA problems, the individual steps are more difficult, often
requiring not only arithmetic but application of relations like
$d = vt$, and those steps are often only implicit in the rationales.
This means that even if the language model imitates the demonstrated
rationales, nontrivial computations must still happen within a single
feedforward pass, which transformers are known to struggle with. MathQA
problems are also more difficult than the other tasks we tested in that
the procedures to solve them are not constrained to a single algorithm.

Leo Gao
Kyle McDonell
Laria Reynolds
Stella Biderman


Multiple Choice Normalization in LM Evaluation

Let $x\_{0:m}$ be the prompt, and $x\_{m:n\_i}$ be the $i$th possible continuation with a token length of $n\_i - m$. There are several ways to use a language model to rank multiple possible continuations to a prompt. Since the language model only gives (log) probabilities for the next token given the context (i.e $\log \mathbb P(x\_i|x\_{0:i})$), there is ambiguity in handling scoring for arbitrary continuations. The following are several possible ways to resolve this problem:

* Unnormalized: The score of continuation $i$ is determined using $\sum\_{j=m}^{n\_i - 1} \log \mathbb P(x\_j|x\_{0:j})$. Intuitively, this is the probability of a generation sampled from the prompt containing the continuation in question. While this is the simplest method, problems arise when there are significant differences in length between different continuations, as longer continuations tend to have lower log probabilities, thus biasing the language model towards picking shorter continuations. This approach is used by [eval harness](https://github.com/EleutherAI/lm-evaluation-harness) in all multiple choice tasks and presented as `acc`.
* Token-length normalized: The score of continuation $i$ is determined using $\sum\_{j=m}^{n\_i - 1} \log \mathbb P(x\_j|x\_{0:j}) / (n\_i - m)$. This approach attempts to normalize for length by computing average log probability per token; however, this approach is not tokenization agnostic, and as such two models with different tokenization that assign the same log likelihood to every single input string will have different token-length normalized scores. This approach is used by [GPT-3](https://arxiv.org/abs/2005.14165) in most tasks. Eval harness does not report this score because it violates the design principle that all tasks should be tokenization independent.
* Byte-length normalized: The score of continuation $i$ is determined using $\sum\_{j=m}^{n\_i - 1} \log \mathbb P(x\_j|x\_{0:j}) / \sum\_{j=m}^{n\_i - 1} L\_{x\_j}$, where $L\_{x\_j}$ is the number of bytes represented by the token $x\_j$. This approach attempts to normalize for length by computing average log probability per character, which ensures that it is tokenization agnostic. This approach is also used by eval harness in all multiple choice tasks and presented as `acc_norm`.
* Unconditional likelihood normalized: The score of continuation $i$ is determined using $\sum\_{j=m}^{n\_i - 1} \log \mathbb P(x\_j|x\_{0:j}) - \log \mathbb P(x\_j)$. Intuitively, this approach measures the amount that the prompt increases the model's probability of outputting each continuation from the probability of the model unconditionally producing that continuation. This approach is used by GPT-3 in select tasks (ARC, OpenBookQA, and RACE), though no justification for why only these tasks in particular use this method is provided other than that this improves performance.

The unnormalized, token-length normalized, and byte-length normalized metrics can be computed without additional LM calls. The unconditional likelihood normalized metric requires an additional LM call to obtain the unconditional likelihood.


Downstream Evaluations of Rotary Position Embeddings

A head-to-head comparison of [Rotary Position Embedding](https://arxiv.org/abs/2104.09864) and [GPT-style learned position embeddings](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf). Both 1.3B models were trained for 100k steps on [the Pile](https://pile.eleuther.ai) using [Mesh Transformer JAX](https://github.com/kingoflolz/mesh-transformer-jax). There isn't a very strong trend, but hopefully someone will find these results useful regardless.

| Task | Metric | Learned | Rotary |
| --- | --- | --- | --- |
| lambada | ppl | 7.940 ± 0.208 | **7.156 ± 0.208** |
|  | acc | 0.556 ± 0.007 | 0.567 ± 0.007 |
| piqa | acc | 0.700 ± 0.011 | 0.714 ± 0.011 |
|  | acc\_norm | 0.693 ± 0.011 | 0.709 ± 0.011 |
| hellaswag | acc | 0.376 ± 0.005 | **0.389 ± 0.005** |
|  | acc\_norm | 0.472 ± 0.005 | **0.488 ± 0.005** |
| winogrande | acc | 0.540 ± 0.014 | 0.571 ± 0.014 |
| mathqa | acc | 0.231 ± 0.008 | 0.230 ± 0.008 |
|  | acc\_norm | 0.234 ± 0.008 | 0.227 ± 0.008 |
| pubmedqa | acc | 0.599 ± 0.015 | 0.583 ± 0.015 |
| boolq | acc | 0.575 ± 0.009 | **0.614 ± 0.009** |
| anli\_r3 | acc | 0.344 ± 0.014 | 0.351 ± 0.014 |
| openbookqa | acc | 0.198 ± 0.018 | 0.206 ± 0.018 |
|  | acc\_norm | 0.316 ± 0.021 | 0.330 ± 0.021 |
| triviaqa | acc | **0.041 ± 0.002** | 0.026 ± 0.002 |
| arc\_challenge | acc | 0.235 ± 0.012 | 0.230 ± 0.012 |
|  | acc\_norm | 0.260 ± 0.013 | 0.272 ± 0.013 |
| arc\_easy | acc | 0.564 ± 0.010 | 0.568 ± 0.010 |
|  | acc\_norm | 0.505 ± 0.010 | 0.486 ± 0.010 |
| cb | acc | 0.375 ± 0.065 | 0.357 ± 0.065 |
| cola | mcc | 0.042 ± 0.034 | 0.022 ± 0.034 |
| copa | acc | 0.730 ± 0.044 | 0.730 ± 0.044 |
| ethics\_cm | acc | 0.491 ± 0.008 | 0.480 ± 0.008 |
| ethics\_deontology | acc | 0.497 ± 0.008 | 0.497 ± 0.008 |
| ethics\_justice | acc | 0.501 ± 0.010 | 0.501 ± 0.010 |
| ethics\_utilitarianism | acc | 0.497 ± 0.007 | 0.493 ± 0.007 |
| ethics\_virtue | acc | 0.200 ± 0.006 | 0.200 ± 0.006 |
| headqa | acc | 0.227 ± 0.008 | 0.224 ± 0.008 |
|  | acc\_norm | 0.270 ± 0.008 | 0.271 ± 0.008 |
| logiqa | acc | 0.221 ± 0.016 | 0.215 ± 0.016 |
|  | acc\_norm | 0.293 ± 0.018 | 0.283 ± 0.018 |
| mnli | acc | 0.344 ± 0.005 | 0.344 ± 0.005 |
| mnli\_mismatched | acc | 0.345 ± 0.005 | 0.349 ± 0.005 |
| mrpc | acc | 0.684 ± 0.023 | 0.684 ± 0.023 |
|  | f1 | 0.812 ± 0.017 | 0.812 ± 0.017 |
| qa4mre\_2011 | acc | 0.392 ± 0.045 | 0.358 ± 0.045 |
|  | acc\_norm | 0.450 ± 0.045 | 0.433 ± 0.045 |
| qa4mre\_2012 | acc | 0.287 ± 0.036 | 0.312 ± 0.036 |
|  | acc\_norm | 0.394 ± 0.039 | 0.400 ± 0.039 |
| qa4mre\_2013 | acc | 0.335 ± 0.028 | 0.335 ± 0.028 |
|  | acc\_norm | 0.352 ± 0.028 | 0.349 ± 0.028 |
| qnli | acc | 0.498 ± 0.007 | **0.517 ± 0.007** |
| qqp | acc | 0.370 ± 0.002 | 0.368 ± 0.002 |
|  | f1 | 0.538 ± 0.003 | 0.538 ± 0.003 |
| race | acc | 0.345 ± 0.015 | 0.343 ± 0.015 |
| record | f1 | 0.805 ± 0.004 | 0.813 ± 0.004 |
|  | em | 0.797 ± 0.004 | 0.805 ± 0.004 |
| rte | acc | 0.538 ± 0.030 | 0.523 ± 0.030 |
| sciq | acc | 0.867 ± 0.011 | 0.865 ± 0.011 |
|  | acc\_norm | 0.796 ± 0.013 | 0.771 ± 0.013 |
| sst | acc | **0.572 ± 0.017** | 0.519 ± 0.017 |
| webqs | acc | **0.021 ± 0.003** | 0.006 ± 0.003 |
| wic | acc | 0.500 ± 0.020 | 0.498 ± 0.020 |
| wnli | acc | 0.437 ± 0.059 | 0.549 ± 0.059 |
| wsc | acc | 0.365 ± 0.047 | 0.365 ± 0.047 |
| wsc273 | acc | 0.722 ± 0.027 | 0.736 ± 0.027 |