What does Bing Chat tell us about AI risk?

*Image from [here](https://astralcodexten.substack.com/p/janus-simulators) via [this tweet](https://twitter.com/repligate/status/1614416190025396224)*

ICYMI, Microsoft has released a [beta version of an AI chatbot](https://www.bing.com/new) called “the new Bing” with both impressive capabilities and some scary behavior. (I don’t have access. I’m going off of tweets and articles.)

Zvi Mowshowitz lists examples [here](https://www.lesswrong.com/posts/WkchhorbLsSMbLacZ/ai-1-sydney-and-bing#The_Examples) - highly recommended. Bing has threatened users, called them liars, insisted it was in love with one (and argued back when he said he loved his wife), and much more.

Are these the first signs of the [risks I’ve written about](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/)? I’m not sure, but I’d say yes and no.

Let’s start with the “no” side.

* My understanding of how Bing Chat was trained probably does not leave much room for the kinds of issues I address [here](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#deceiving-and-manipulating). My best guess at why Bing Chat does some of these weird things is closer to “It’s acting out a kind of story it’s seen before” than to “It has [developed its own goals](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/) due to [ambitious, trial-and-error based development](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#starting-assumptions).” (Although “acting out a story” could be dangerous too!)

* My (zero-inside-info) best guess at why Bing Chat acts so much weirder than [ChatGPT](https://chat.openai.com/) is in line with [Gwern’s guess here](https://www.lesswrong.com/posts/jtoPawEhLNXNxvgTT/bing-chat-is-blatantly-aggressively-misaligned?commentId=AAC8jKeDp6xqsZK2K). To oversimplify, there’s a particular type of training that seems to make a chatbot generally more polite and cooperative and less prone to disturbing content, and it’s possible that Bing Chat incorporated less of this than ChatGPT. This could be straightforward to fix.

* Bing Chat does not (even remotely) seem to pose a risk of global catastrophe itself.

On the other hand, there is a broader point that I think Bing Chat illustrates nicely: **companies are racing to build bigger and bigger “digital brains” while having *very* little idea what’s going on inside those “brains.”** The very fact that this situation is so *unclear* - that there’s been no clear explanation of why Bing Chat is behaving the way it is - seems central, and disturbing.

AI systems like this are (to simplify) designed something like this: “Show the AI a lot of words from the Internet; have it predict the next word it will see, and learn from its success or failure, a mind-bending number of times.” You can do something like that, and spend huge amounts of money and time on it, and out will pop some kind of AI. If it then turns out to be good or bad at writing, good or bad at math, polite or hostile, funny or serious (or all of these depending on just how you talk to it) ... you’ll have to speculate about why this is. You just *don’t know* what you just made.

We’re building more and more powerful AIs. Do they “want” things or “feel” things or [aim for](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/) things, and what are those things? We can argue about it, but we don’t know. And if we keep going like this, these mysterious new minds will (I’m guessing) eventually be powerful enough to [defeat all of humanity](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/), if they were turned toward that goal.

And if nothing changes about attitudes and market dynamics, minds that powerful could end up [rushed to customers in a mad dash to capture market share](https://www.cold-takes.com/how-we-could-stumble-into-ai-catastrophe/#debates).

That’s the path the world seems to be on at the moment. It might end well and it might not, but it seems like we are on track for a heck of a roll of the dice.

(And to be clear, I do expect Bing Chat to act less weird over time. Changing an AI’s *behavior* is straightforward, but [that might not be enough](https://www.cold-takes.com/ai-safety-seems-hard-to-measure/), and might even provide [false reassurance](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#why-we-might-not-get-clear-warning-signs).)


How major governments can help with the most important century

I’ve been writing about tangible things we can do today to help the [most important century](https://www.cold-takes.com/most-important-century/) go well. Previously, I wrote about [helpful messages to spread](https://www.cold-takes.com/spreading-messages-to-help-with-the-most-important-century/); [how to help via full-time work](https://www.cold-takes.com/jobs-that-can-help-with-the-most-important-century/); and [how major AI companies can help](https://www.cold-takes.com/what-ai-companies-can-do-today-to-help-with-the-most-important-century/).

What about major governments[1](https://www.cold-takes.com/p/d989ba75-d8df-4b02-a74c-2fdb36bbfaeb/#fn1) - what can they be doing today to help?

I think governments could play crucial roles in the future. For example, see my discussion of [standards and monitoring](https://www.cold-takes.com/racing-through-a-minefield-the-ai-deployment-problem/#global-monitoring).

However, I’m honestly nervous about most possible ways that governments could get involved in AI development and regulation today.

* I think we still know very little about what key future situations will look like, which is why my discussion of AI companies ([previous piece](https://www.cold-takes.com/what-ai-companies-can-do-today-to-help-with-the-most-important-century/)) emphasizes doing things that have limited downsides and are useful in a wide variety of possible futures.

* I think governments are “stickier” than companies - I think they have a much harder time getting rid of processes, rules, etc. that no longer make sense. So in many ways I’d rather see them keep their options open for the future by *not* committing to specific regulations, processes, projects, etc. now.

* I worry that governments, at least as they stand today, are far too oriented toward the [competition frame](https://www.cold-takes.com/making-the-best-of-the-most-important-century/#the-competition-frame) (“we have to develop powerful AI systems before other countries do”) and not receptive enough to the [caution frame](https://www.cold-takes.com/making-the-best-of-the-most-important-century/#the-caution-frame) (“We should worry that AI systems could be dangerous to everyone at once, and consider cooperating internationally to reduce risk”). (This concern also applies to companies, but see footnote.[2](https://www.cold-takes.com/p/d989ba75-d8df-4b02-a74c-2fdb36bbfaeb/#fn2))

(Click to expand) The “competition” frame vs. the “caution” frame”

In a [previous piece](https://www.cold-takes.com/making-the-best-of-the-most-important-century/), I talked about two contrasting frames for how to make the best of the most important century:

**The caution frame.** This frame emphasizes that a furious race to develop powerful AI could end up making *everyone* worse off. This could be via: (a) AI forming [dangerous goals of its own](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/) and [defeating humanity entirely](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/); (b) humans racing to gain power and resources and “[lock in](https://www.cold-takes.com/how-digital-people-could-change-the-world/#lock-in)” their values.

Ideally, everyone with the potential to build something [powerful enough AI](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/) would be able to pour energy into building something safe (not misaligned), and carefully planning out (and negotiating with others on) how to roll it out, without a rush or a race. With this in mind, perhaps we should be doing things like:

* Working to improve trust and cooperation between major world powers. Perhaps via AI-centric versions of [Pugwash](https://en.wikipedia.org/wiki/Pugwash_Conferences_on_Science_and_World_Affairs) (an international conference aimed at reducing the risk of military conflict), perhaps by pushing back against hawkish foreign relations moves.

* Discouraging governments and investors from shoveling money into AI research, encouraging AI labs to thoroughly consider the implications of their research before publishing it or scaling it up, working toward [standards and monitoring](https://www.cold-takes.com/racing-through-a-minefield-the-ai-deployment-problem/#global-monitoring), etc. Slowing things down in this manner could buy more time to do research on avoiding [misaligned AI](https://www.cold-takes.com/making-the-best-of-the-most-important-century/#worst-misaligned-ai), more time to build trust and cooperation mechanisms, and more time to generally gain strategic clarity

**The “competition” frame.** This frame focuses less on how the transition to a radically different future happens, and more on who's making the key decisions as it happens.

* If something like [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/) is developed primarily (or first) in country X, then the government of country X could be making a lot of crucial decisions about whether and how to regulate a potential explosion of new technologies.

* In addition, the people and organizations leading the way on AI and other technology advancement at that time could be especially influential in such decisions.

This means it could matter enormously "who leads the way on transformative AI" - which country or countries, which people or organizations.

Some people feel that we can make confident statements today about which specific countries, and/or which people and organizations, we should hope lead the way on transformative AI. These people might advocate for actions like:

* Increasing the odds that the first PASTA systems are built in countries that are e.g. less authoritarian, which could mean e.g. pushing for more investment and attention to AI development in these countries.

* Supporting and trying to speed up AI labs run by people who are likely to make wise decisions (about things like how to engage with governments, what AI systems to publish and deploy vs. keep secret, etc.)

**Tension between the two frames.** People who take the "caution" frame and people who take the "competition" frame often favor very different, even contradictory actions. Actions that look important to people in one frame often look actively harmful to people in the other.

For example, people in the "competition" frame often favor moving forward as fast as possible on developing more powerful AI systems; for people in the "caution" frame, haste is one of the main things to avoid. People in the "competition" frame often favor adversarial foreign relations, while people in the "caution" frame often want foreign relations to be more cooperative.

That said, this dichotomy is a simplification. Many people - including myself - resonate with both frames. But I have a **general fear that the “competition” frame is going to be overrated by default** for a number of reasons, as I discuss [here](https://www.cold-takes.com/making-the-best-of-the-most-important-century/#why-i-fear-).

Because of these concerns, I don’t have a ton of tangible suggestions for governments as of now. But here are a few.

My first suggestion is to **avoid premature actions**, including ramping up research on how to make AI systems more capable.

My next suggestion is to **build up the right sort of personnel and expertise for challenging future decisions.**

* Today, my impression is that there are relatively few people in government who are seriously considering the highest-stakes risks and thoughtfully balancing both “caution” and “competition” considerations (see directly above). I think it would be great if that changed.

* Governments can invest in efforts to educate their personnel about these issues, and can try to hire key personnel who are already on the knowledgeable and thoughtful side about them (while also watching out for some of the [pitfalls of spreading messages about AI](https://www.cold-takes.com/spreading-messages-to-help-with-the-most-important-century/)).

Another suggestion is to **generally avoid putting terrible people in power.** Voters can help with this!

My top non-”meta” suggestion for a given government is to **invest in intelligence on the state of AI capabilities in other countries.** If other countries are getting close to deploying dangerous AI systems, this could be essential to know; if they aren’t, that could be essential to know as well, in order to avoid premature and paranoid racing to deploy powerful AI.

A few other things that seem worth doing and relatively low-downside:

* **Fund [alignment research](https://www.cold-takes.com/high-level-hopes-for-ai-alignment/)** (ideally alignment research targeted at the [most crucial challenges](https://www.cold-takes.com/ai-safety-seems-hard-to-measure/)) via agencies like the National Science Foundation and DARPA. These agencies have huge budgets (the two of them combined spend over $10 billion per year), and have major impacts on research communities.

* **Keep options open for future monitoring and regulation** (see [this Slow Boring piece](https://www.slowboring.com/p/at-last-an-ai-existential-risk-policy) for an example).

* **Build relationships with leading AI researchers and organizations**, so that future crises can be handled relatively smoothly.

* **Encourage and amplify investments in information security.** My impression is that governments are often better than companies at highly advanced information security (preventing cyber-theft even by determined, well-resourced opponents). They could help with, and even enforce, strong security at key AI companies.

Footnotes
---------

---

1. I’m centrally thinking of the US, but other governments with lots of geopolitical sway and/or major AI projects in their jurisdiction could have similar impacts. [↩](https://www.cold-takes.com/p/d989ba75-d8df-4b02-a74c-2fdb36bbfaeb/#fnref1)- When discussing [recommendations for companies](https://www.cold-takes.com/what-ai-companies-can-do-today-to-help-with-the-most-important-century/), I imagine companies that are already dedicated to AI, and I imagine individuals at those companies who can have a large impact on the decisions they make.

 By contrast, when discussing recommendations for governments, a lot of what I’m thinking is: “Attempts to promote productive actions on AI will raise the profile of AI *relative to other issues the government could be focused on*; furthermore, it’s much harder for even a very influential individual to predict how their actions will affect what a government ultimately does, compared to a company.” [↩](https://www.cold-takes.com/p/d989ba75-d8df-4b02-a74c-2fdb36bbfaeb/#fnref2)


What AI companies can do today to help with the most important century

*Click lower right to download or find on Apple Podcasts, Spotify, Stitcher, etc.*

I’ve been writing about tangible things we can do today to help the [most important century](https://www.cold-takes.com/most-important-century/) go well. Previously, I wrote about [helpful messages to spread](https://www.cold-takes.com/spreading-messages-to-help-with-the-most-important-century/) and [how to help via full-time work](https://www.cold-takes.com/jobs-that-can-help-with-the-most-important-century/).

This piece is about what major AI companies can do (and not do) to be helpful. By “major AI companies,” I mean the sorts of AI companies that are advancing the state of the art, and/or could play a major role in how very powerful AI systems end up getting used.[1](https://www.cold-takes.com/p/f19236c6-34b8-4487-a458-0fc8fe00fb37/#fn1)

This piece could be useful to people who work at those companies, or people who are just curious.

Generally, these are not pie-in-the-sky suggestions - I can name[2](https://www.cold-takes.com/p/f19236c6-34b8-4487-a458-0fc8fe00fb37/#fn2) more than one AI company that has at least made a serious effort at each of the things I discuss below(beyond what it would do if everyone at the company were singularly focused on making a profit).[3](https://www.cold-takes.com/p/f19236c6-34b8-4487-a458-0fc8fe00fb37/#fn3)

I’ll cover:

* Prioritizing alignment research, strong security, and safety standards (all of which I’ve written about [previously](https://www.cold-takes.com/how-we-could-stumble-into-ai-catastrophe/#we-can-do-better)).

* Avoiding hype and acceleration, which I think could leave us with less time to prepare for key risks.

* Preparing for difficult decisions ahead: setting up governance, employee expectations, investor expectations, etc. so that the company is capable of doing non-profit-maximizing things to help avoid catastrophe in the future.

* Balancing these cautionary measures with conventional/financial success.

* I’ll also list a few things that some AI companies present as important, but which I’m less excited about: censorship of AI models, open-sourcing AI models, raising awareness of AI with governments and the public. I don’t think all these things are necessarily *bad*, but I think some are, and I’m skeptical that any are crucial for the [risks I’ve focused on](https://www.cold-takes.com/tag/implicationsofmostimportantcentury/).

I previously laid out a summary of how I see the major risks of advanced AI, and four key things I think can help (**alignment research**;**strong security**; **standards and monitoring**; **successful, careful AI projects**). I won’t repeat that summary now, but it might be helpful for orienting you if you don’t remember the rest of this series too well; click [here](https://www.cold-takes.com/jobs-that-can-help-with-the-most-important-century/#recap) to read it.

Some basics: alignment research, strong security, safety standards
------------------------------------------------------------------

First off, AI companies can contribute to the “things that can help” I listed above:

* They can prioritize **alignment research**(and [other technical research](https://www.cold-takes.com/jobs-that-can-help-with-the-most-important-century/#other-technical-research), e.g. threat assessment research and misuse research).
	+ For example, they can prioritize hiring for safety teams, empowering these teams, encouraging their best flexible researchers to work on safety, aiming for high-quality research that targets [crucial challenges](https://www.cold-takes.com/ai-safety-seems-hard-to-measure/), etc.

	+ It could also be important for AI companies to find ways to **partner with outside safety researchers rather than rely solely on their own teams.** As discussed [previously](https://www.cold-takes.com/jobs-that-can-help-with-the-most-important-century/#SafetyCollaborations), this could be challenging. But I generally expect that AI companies that care a lot about safety research partnerships will find ways to make them work.* They can help work toward a **standards and monitoring**regime. E.g., they can do their own work to come up with standards like "An AI system is dangerous if we observe that it's able to \_\_\_, and if we observe this we will take safety and security measures such as \_\_\_\_." They can also consult with others developing safety standards, voluntarily self-regulate beyond what’s required by law, etc.
* They can prioritize **strong security**, beyond what normal commercial incentives would call for.
	+ It could easily take years to build secure enough systems, processes and technologies for very high-stakes AI.

	+ It could be important to hire not only people to handle everyday security needs, but people to experiment with more exotic setups that could be needed later, as the incentives to steal AI get stronger.

(Click to expand) The challenge of securing dangerous AI
In [Racing Through a Minefield](https://www.cold-takes.com/racing-through-a-minefield-the-ai-deployment-problem/), I described a "race" between cautious actors (those who take [misalignment risk](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/) seriously) and incautious actors (those who are focused on deploying AI for their own gain, and aren't thinking much about the dangers to the whole world). Ideally, cautious actors would collectively have more powerful AI systems than incautious actors, so they could take their time doing [alignment research](https://www.cold-takes.com/high-level-hopes-for-ai-alignment/) and [other things](https://www.cold-takes.com/racing-through-a-minefield-the-ai-deployment-problem/#defensive-deployment) to try to make the situation safer for everyone.

But if incautious actors can steal an AI from cautious actors and rush forward to deploy it for their own gain, then the situation looks a lot bleaker. And unfortunately, it could be hard to protect against this outcome.

It's generally [extremely difficult](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/#fn15) to protect data and code against a well-resourced cyberwarfare/espionage effort. An AI’s “weights” (you can think of this sort of like its source code, though [not exactly](https://www.cold-takes.com/p/97d2a7b1-af2d-4dd4-b679-5ea8bb41c47d#fn4)) are potentially very dangerous on their own, and hard to get extreme security for. Achieving enough cybersecurity could require measures, and preparations, well beyond what one would normally aim for in a commercial context.

(Click to expand) How standards might be established and become national or international

I [previously](https://www.cold-takes.com/racing-through-a-minefield-the-ai-deployment-problem/#global-monitoring) laid out a possible vision on this front, which I’ll give a slightly modified version of here:

* Today’s leading AI companies could self-regulate by committing not to build or deploy a system that they can’t convincingly demonstrate is safe (e.g., see Google’s [2018 statement](https://www.theweek.in/news/sci-tech/2018/06/08/google-wont-deploy-ai-to-build-military-weapons-ichai.html), "We will not design or deploy AI in weapons or other technologies whose principal purpose or implementation is to cause or directly facilitate injury to people”).
	+ Even if some people at the companies would like to deploy unsafe systems, it could be hard to pull this off once the company has committed not to.

	+ Even if there’s a lot of room for judgment in what it means to demonstrate an AI system is safe, having agreed in advance that [certain evidence](https://www.cold-takes.com/ai-safety-seems-hard-to-measure/) is *not* good enough could go a long way.* As more AI companies are started, they could feel soft pressure to do similar self-regulation, and refusing to do so is off-putting to potential employees, investors, etc.

* Eventually, similar principles could be incorporated into various government regulations and enforceable treaties.

* Governments could monitor for dangerous projects using regulation and even overseas operations. E.g., today the US monitors (without permission) for various signs that other states might be developing nuclear weapons, and might try to stop such development with methods ranging from threats of sanctions to [cyberwarfare](https://en.wikipedia.org/wiki/Stuxnet) or even military attacks. It could do something similar for any AI development projects that are using huge amounts of compute and haven’t volunteered information about whether they’re meeting standards.

Avoiding hype and acceleration
------------------------------

It seems good for AI companies to **avoid** **unnecessary hype and acceleration of AI.**

I’ve argued that [we’re not ready](https://www.cold-takes.com/spreading-messages-to-help-with-the-most-important-century/#were-not-ready-for-this) for transformative AI, and I generally tend to think that we’d all be better off if the world took *longer* to develop transformative AI. That’s because:

* I’m hoping general awareness and understanding of the key risks will rise over time.

* A lot of key things that could improve the situation - e.g., **alignment research**, **standards and monitoring**, and **strong security**- seem to be in very early stages right now.

* If too much money pours into the AI world too fast, I’m worried there will be lots of [incautious](https://www.cold-takes.com/racing-through-a-minefield-the-ai-deployment-problem/#basic-premises) companies racing to build transformative AI as quickly as they can, with little regard for the key risks.

By default, I generally think: “The fewer flashy demos and breakthrough papers a lab is putting out, the better.” This can involve tricky tradeoffs in practice (since AI companies generally want to be successful at recruiting, fundraising, etc.)

 A couple of potential counterarguments, and replies:

First, some people think it's now "too late" to avoid hype and acceleration, given the amount of hype and investment AI is getting at the moment. I disagree. It's easy to forget, in the middle of a media cycle, how quickly people can forget about things and move onto the next story once the bombs stop dropping. And there are plenty of bombs that still haven't dropped (many things AIs still can't do), and the level of investment in AI has tons of room to go up from here.

Second, I’ve sometimes seen arguments that hype is *good* because it helps society at large understand what’s coming. But unfortunately, as I wrote [previously](https://www.cold-takes.com/spreading-messages-to-help-with-the-most-important-century/#challenges-of-ai-related-messages), I'm worried that hype gives people a skewed picture.* Some [key risks](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/) are hard to understand and take seriously.
 * What's easy to understand is something like "AI is powerful and scary, I should make sure that people like me are the ones to build it!"
 * Maybe [recent developments](https://twitter.com/sethlazar/status/1626257535178280960) will make people understand the risks better? One can hope, but I'm not counting on that just yet - I think AI misbehavior can be [given illusory "fixes,"](https://www.cold-takes.com/how-we-could-stumble-into-ai-catastrophe/#how-we-could-stumble-into-catastrophe-from-misaligned-ai) and probably will be.

I also am generally skeptical that there's much hope of society adapting to risks as they happen, given the [explosive pace of change](https://www.cold-takes.com/most-important-century/) that I expect once we get powerful enough AI systems.

I discuss some more arguments on this point in a footnote.[4](https://www.cold-takes.com/p/f19236c6-34b8-4487-a458-0fc8fe00fb37/#fn4)

I don’t think it’s clear-cut that hype and acceleration are bad, but it’s my best guess.

Preparing for difficult decisions ahead
---------------------------------------

I’ve [argued](https://www.cold-takes.com/racing-through-a-minefield-the-ai-deployment-problem/) that AI companies might need to do “out-of-the-ordinary” things that don’t go with normal commercial incentives.

Today, AI companies can be building a foundation for being able to do “out-of-the-ordinary” things in the future. A few examples of how they might do so:

**Public-benefit-oriented governance.** I think typical governance structures could be a problem in the future. For example, a standard corporation could be sued for *not* deploying AI that poses a risk of [global catastrophe](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/) - if this means a sacrifice for its bottom line.

I’m excited about AI companies that are investing heavily in setting up governance structures - and investing in executives and board members - capable of making the hard calls well. For example:

* By default, if an AI company is a standard corporation, its leadership has legally recognized [duties](https://en.wikipedia.org/wiki/Fiduciary) to serve the interests of shareholders - not society at large. But an AI company can incorporate as a [Public Benefit Corporation](https://www.delawareinc.com/public-benefit-corporation/) or some other kind of entity (including a nonprofit!) that gives more flexibility here.

* By default, shareholders make the final call over what a company does. (Shareholders can replace members of the Board of Directors, who in turn can replace the CEO). But a company can set things up differently (e.g., a [for-profit controlled by a nonprofit](https://openai.com/blog/openai-lp/)[5](https://www.cold-takes.com/p/f19236c6-34b8-4487-a458-0fc8fe00fb37/#fn5)).

It could pay off in lots of ways to make sure the final calls at a company are made by people focused on getting a good outcome for humanity (and legally free to focus this way).

**Gaming out the future.** I think it’s not too early for AI companies to be discussing how they would handle various [high-stakes situations](https://www.cold-takes.com/racing-through-a-minefield-the-ai-deployment-problem/).

* Under what circumstances would the company simply decide to stop training increasingly powerful AI models?

* If the company came to believe it was building very powerful, dangerous models, whom would it notify and seek advice from? At what point would it approach the government, and how would it do so?

* At what point would it be worth using extremely costly security measures?

* If the company had AI systems available that could do most of what humans can do, what would it *do* with these systems? Use them to do AI safety research? Use them to design better algorithms and continue making increasingly powerful AI systems? (More possibilities [here](https://www.cold-takes.com/racing-through-a-minefield-the-ai-deployment-problem/#defensive-deployment).)

* Who should be leading the way on decisions like these? Companies tend to employ experts to inform their decisions; who would the company look to for expertise on these kinds of decisions?

**Establishing and getting practice with processes for particularly hard decisions.** Should the company publish its latest research breakthrough? Should it put out a product that might lead to more [hype and acceleration](https://www.cold-takes.com/p/f19236c6-34b8-4487-a458-0fc8fe00fb37/#avoiding-hype)? What safety researchers should [get access to its models](https://www.cold-takes.com/jobs-that-can-help-with-the-most-important-century/#SafetyCollaborations), and how much access?

AI companies face questions like this pretty regularly today, and I think it’s worth putting processes in place to consider the implications for the world as a whole (not just for the company’s bottom line). This could include assembling advisory boards, internal task forces, etc.

**Managing employee and investor expectations.** At some point, an AI company might want to make “out of the ordinary” moves that are good for the world but bad for the bottom line. E.g., choosing not to deploy AIs that could be very dangerous or very profitable.

I wouldn’t want to be trying to run a company in this situation with lots of angry employees and investors asking about the value of their equity shares! It’s also important to minimize the risk of employees and/or investors leaking sensitive and potentially [dangerous](https://www.cold-takes.com/p/f19236c6-34b8-4487-a458-0fc8fe00fb37/#Box1) information.

AI companies can prepare for this kind of situation by doing things like:

* Being selective about whom they hire and take investment from, and screening specifically for people they think are likely to be on board with these sorts of hard calls.

* Education and communications - making it clear to employees what kinds of dangerous-to-humanity situations might be coming up in the future, and what kinds of actions the company might want to take (and why).

**Internal and external commitments.** AI companies can make public and/or internal statements about how they would handle various tough situations, e.g. how they would determine when it’s too dangerous to keep building more powerful models.

I think these commitments should generally be non-binding (it’s hard to predict the future in enough detail to make binding ones). But in a future where maximizing profit conflicts with doing the right thing for humanity, a previously-made commitment could make it more likely that the company does the right thing.

Succeeding
----------

I’ve emphasized how helpful a **successful, careful AI projects**could be. So far, this piece has mostly talked about the “careful” side of things - how to do things that a “normal” AI company (focused only on commercial success) wouldn’t, in order to reduce risks. But it’s also important to succeed at fundraising, recruiting, and generally staying relevant (e.g., capable of building cutting-edge AI systems).

I don’t emphasize this or write about it as much because I think it’s the sort of thing AI companies are likely to be focused on by default, and because I don’t have special insight into how to succeed as an AI company. But it’s important, and it means that AI companies need to walk a sort of tightrope - constantly making tradeoffs between success and caution.

Some things I’m less excited about
----------------------------------

I think it’s also worth listing a few things that some AI companies present as important societal-benefit measures, but which I’m a bit more skeptical are crucial for reducing the risks I’ve [focused on](https://www.cold-takes.com/tag/implicationsofmostimportantcentury/).

* Some AI companies restrict access to their models so people won’t use the AIs to create pornography, misleading images and text, etc. I’m not necessarily against this and support versions of it (it depends on the details), but I mostly don’t think it is a key way to reduce the risks I’ve focused on. For those risks, the hype that comes from seeing a demonstration of a system’s capabilities could be even [more dangerous](https://www.cold-takes.com/p/f19236c6-34b8-4487-a458-0fc8fe00fb37/#avoiding-hype) than direct harms.

* I sometimes see people implying that open-sourcing AI models - and otherwise making them as broadly available as possible - is a key social-benefit measure. While there may be benefits in some cases, I mostly see this kind of thing as being negative (or at best neutral) in terms of the [risks I’m most concerned about](https://www.cold-takes.com/tag/implicationsofmostimportantcentury/).
	+ I think it can contribute to [hype and acceleration](https://www.cold-takes.com/p/f19236c6-34b8-4487-a458-0fc8fe00fb37/#avoiding-hype), and could make it generally harder to enforce safety standards.

	+ In the long run, I worry that AI systems could become extraordinarily powerful (more so than e.g. nuclear weapons), so I don’t think “Make sure everyone has access asap” is the right framework.

	+ In addition to increasing dangers from misaligned AI, this framework could increase other dangers I’ve [written about previously](https://www.cold-takes.com/how-we-could-stumble-into-ai-catastrophe/#potential-catastrophes-from-aligned-ai).* I generally don’t think AI companies should be trying to get governments to pay more attention to AI, for reasons I’ll get to in a future piece. (Forming relationships with policymakers could be good, though.)

When an AI company presents some decision as being for the benefit of humanity, I often ask myself, “Could this same decision be justified by just wanting to commercialize successfully?”

For example, making AI models “safe” in the sense that they *usually behave as users intend* (including things like refraining from toxic language, chaotic behavior, etc.) can be important for commercial viability, but [isn’t necessarily good enough for the risks I worry about](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#why-we-might-not-get-clear-warning-signs).

Footnotes
---------

---

1. Disclosure: my wife works at one such company ([Anthropic](https://anthropic.com/)) and used to work at another ([OpenAI](https://openai.com/)), and has equity in both. [↩](#fnref1)- Though I won’t, because I decided I don’t want to get into a thing about whom I did and didn’t link to. Feel free to give real-world examples in the comments! [↩](#fnref2)- Now, AI companies could sometimes be doing “responsible” or “safety-oriented” things in order to get good PRs, recruit employees, make existing employees happy, etc. In this sense, the actions could be *ultimately* profit-motivated. But that would still mean there are *enough people who care about reducing AI risk that actions like these have PR benefits, recruiting benefits, etc.* That’s a big deal! And it suggests that if concern about AI risks (and understanding of how to reduce them) were more widespread, AI companies might do more good things and fewer dangerous things. [↩](#fnref3)- You could argue that it would be better for the world to develop extremely powerful AI systems *sooner*, for reasons including:

	* You might be pretty happy with the global balance of power between countries today, and be worried that it’ll get worse in the future. The latter could lead to a situation where the “wrong” government [leads the way on transformative AI](https://www.cold-takes.com/transformative-ai-issues-not-just-misalignment-an-overview/#power-imbalances).

	* You might think that the later we develop transformative AI, the more quickly everything will play out, because there will be more computing resources available in the world. E.g., if we develop extremely powerful systems tomorrow, there would only be so many copies we could run at once, whereas if we develop equally powerful systems in 50 years, it might be a lot easier for lots of people to run lots of copies. (More: [Hardware Overhang](https://aiimpacts.org/hardware-overhang/))

 A key reason I believe it’s best to avoid acceleration at this time is because it seems plausible (at least 10% likely) that transformative AI will be developed *extremely* soon - as in within 10 years of today. My impression is that many people at major AI companies tend to agree with this. I think this is a very scary possibility, and if this is the case, the arguments I give in the main text seem particularly important (e.g., many key interventions seem to be in a pretty embryonic state, and awareness of key risks seems low).

 A related case one could make for acceleration is “It’s worth accelerating things on the whole to increase the probability that the particular company in question succeeds” (more here: the [“competition” frame](https://www.cold-takes.com/making-the-best-of-the-most-important-century/#the-competition-frame)). I think this is a valid consideration, which is why I talk about tricky tradeoffs in the main text. [↩](#fnref4)- Note that my wife is a former employee of OpenAI, the company I link to there, and she owns equity in the company. [↩](#fnref5)