# Jade Leung and Seth Baum The role of existing institutions in AI strategy - EA Forum

_AI is very likely to make a huge impact on our world, especially as it grows more powerful than it is today. It’s hard for us to know exactly how that impact will look, but we do know many of the actors most likely to be involved. As AI gets stronger, what can we expect the world’s most powerful national governments to do? What about nongovernmental organizations, like the UN?_

_This advanced workshop from Effective Altruism Global: San Francisco 2018, presented by Jade Leung and Seth Baum, addresses these questions from multiple perspectives. A transcript of the workshop is below, which we have lightly edited for clarity. You can also watch the talk on_ [_YouTube_](https://www.youtube.com/watch?v=pgiwvmY3brg&list=PLwp9xeoX5p8P3cDQwlyN7qsFhC9Ms4L5W&index=3) _and read it on_ [_effectivealtruism.org_](https://www.effectivealtruism.org/articles/ea-global-2018-the-role-of-existing-institutions-in-ai-strategy/)_._

## The Talk

**Jade:** What we're going to do is we're going to introduce ourselves briefly so you kind of know where we're coming from. Then we've got two moots which we have just then decided were the two moots that we're going to talk about. We'll chuck them up on the board and we'll spend about half a session talking about one and then half a session talking about the other. This is a session where we'd both love for you guys to toss us your questions right throughout it basically so, yes, get ready to have your questions ready and we'll open it up pretty much soon after the intro.

Briefly intro to myself. I currently am based in the Future of Humanity Institute, and the work that I do specifically looks at the relationships between large multi-national technology firms and governments, specifically National Security and Defense components of governments in the US and China. And the questions that I ask are about how these actors should relate to each other, cooperate, coordinate, to steer us towards a future, or set of futures, that are more safe and beneficial than not, with transformative AI. My background is in engineering, I am masquerading as international relations person, but I'm not really that. I do a fair amount in the global governance space, in the IR space largely. That's me.

**Seth:** Cool. I'm Seth Baum, I was introduced with the Global Catastrophic Risk Institute, and as a think tank we try to sit in that classic think tank space of working at the intersection of, among other things, the world of scholarship and the world of policy. We spend a lot of time talking with people in the policy worlds, especially down in DC. For me, it's down in DC, I live in New York. I guess from here it would be over in DC. Is that what you say? You don't live here.

**Jade:** Sure.

**Seth:** Over in DC. And talking with people in policy. I work across a number of different policy areas, do a lot on nuclear weapons, little bit on biosecurity, and then also on AI, and especially within the last year or two there have been some more robust policy conversations about AI. The policy world has just started to take an interest in this topic and is starting to do some interesting things that have fallen on our radar, and so we'll be saying more about that. Do you want to?

**Jade:** Yeah, sure.

So the two institutions that we're going to chat about, is firstly the National Security and Defense. We might focus on the US National Security and Defense, and have a bit of a chat about what makes sense to engage them on in the space of our strategy, and how we should be thinking about their role in this space. That's the first moot. The second will turn to more international institutions, the kind of multilateral groups, e.g. the UN but not strictly so, and what role they could play in the space of AI strategy as well. We'll kind of go half and half there.

Just so I have a bit of a litmus test for who's in the audience, if I say AI strategy, who does that mean anything to? Ah, awesome. Okay, cool. Maybe we'll just start with getting Seth's quick perspective on this question. So the moot here is, this house believes that in the space of AI strategy, we should be actively engaging with National Security and Defense components of the US government. Do you want to speak quickly to what your quick take on that is?

**Seth:** Sure. So an interesting question here is engaging with, say the US government especially on the national security side, is this a good thing or a bad thing? I feel like opinions vary on this, maybe even within this room opinions vary on whether having these conversations is a good thing or a bad thing. The argument against it that I hear is essentially, you might tell them AI could take over the world and kill everyone, and they might hear, AI could take over the world, hear that and then go on to do harmful things.

I personally tend to be more skeptical of that sort of argument. The main reason for that is that the people who are in the government and working on AI, they've already heard this idea before. It's been headline news for a number of years now, some people from our communities including your organization caused some of those headlines.

**Jade:** I feel like you're asking me to apologize for them, and I'm not going to.

_Seth_: If one is concerned about the awareness of various people in government about runaway AI, you could ask questions like, was the publication of the Superintelligence book a good thing or a bad thing? You could maybe there make a case in either direction-

**Jade:** Could we do a quick poll actually? I'd be curious. Who thinks the publication of Superintelligence was on net, a net positive thing? On net, a negative thing? Hell yeah.

**Seth:** Doesn't mean that that's actually true.

**Jade:** Fair enough.

**Seth:** Just to be clear, I'm not arguing that it was a net negative, but the point is that the idea is out, and the people who work on AI, sure, they're mostly working on a narrow near term AI, but they've heard the idea before. They don't need us to put the thought into their heads. Now of course we could be kind of strengthening that thought within their heads, and that can matter, but at the same time when I interact with them, I actually tend to not be talking about superintelligence, general intelligence, that stuff anyway. Though more for a different reason, and that's because while they have heard of the idea, they're pretty skeptical about it. Either because they think it probably wouldn't happen or because if it would happen it would be too far in the future for them to worry about. A lot of people in policy have much more near term time horizons that they have to work with. They have enough on their plate already, nobody's asking them to worry about this, so they're just going to focus on the stuff that they actually need to worry about, which includes the AI that already exists and is in the process of coming online.

What I've found is then because they're pretty dismissive of it, I feel like if I talk about it they might just be dismissive of what I have to say, and that's not productive. Versus instead if the message is we should be careful about AI that acts unpredictably and causes unintended harms, that's not really about superintelligence. That same message applies to the AI that exists already: self driving cars, autonomous weapons. You don't want autonomous weapons causing unintended harm, and that's a message that people are very receptive to. By emphasizing that sort of message we can strengthen that type of thinking within policy worlds. That's for the most part the message that I've typically gone with, including in the National Security communities.

**Jade:** Cool. I've got a ton of questions for you, but maybe to quickly interject my version of that. I tend to agree with a couple of things that Seth said, and then disagree with a couple specific things.

I think generally the description of my perspective on this is that there's a very limited amount of useful engagement with National Security today, and I think the amount of potential to do wrong via engaging with them is large, and sufficiently large that we should be incredibly cautious about the manner in which we engage. That is a different thing to saying that we shouldn't engage with them at all, and I'll nuance that a little bit. I think, maybe to illustrate, I think the priors or assumptions that people hold when they're taking a stance on whether you should engage with National Security or not, is people I think disagree on maybe three axes. I said three because people always say three, I'm not entirely sure what the three are but we'll see how this goes.

So I think the first is people disagree on the competence of National Security to pursue the technology themselves, or at least to do something harmful with said information about capabilities of the technology. I think some people hold the extreme view that they're kind of useless and there's nothing that they can do in-house that is going to cause technology to be more unsafe than not, which is the thing that you're trying to deter. On the other hand, some people believe that NatSec at least have the ability to acquire control of this technology, or can develop it in-house sufficiently so, that an understanding of significant capabilities of AI would lead them to want to pursue it, and they can pursue it with competence, basically.

I think that kind of competence thing is one thing that people disagree on, and I would tend to land on them being more competent than people think. Even if that's not the case, I think it's always worth being conservative in that sense anyways.

So that's the first axis. Second axis I think is about whether they have a predisposition, or whether they have the ability to absorb this kind of risk narrative effectively, or whether that's just so orthogonal to the culture of NatSec that it's not going to be received in a nuanced enough way and they're always going to interpret whatever information with a predisposition to want to pursue unilateral military advantage, regardless of what you're saying to them. Some people on one end would hold that they are reasonable people with a broad open mind, and plausibly could absorb this kind of long-term risk narrative. Some other people would hold that information that is received by them will tend to just be received with the lens of how can we use this to secure a national strategic advantage.

I would tend to land on us having no precedent for the former, and having a lot more precedent for the latter. I think I'd like to believe that folks at DOD and NatSec can absorb, or can come around more to the long term risk narrative, but I don't think we've seen any precedent enough for that to place credence on that side of the spectrum. That's kind of where I sit on that second axis.

I said I had a third, I'm not entirely sure what the third is, so let's just leave it at two.

I think that probably describes the reasons why I hold that I think engaging with NatSec can be plausibly useful, but for every kind of one useful case, I can see many more reasons why engaging with them could plausibly be a bad idea, at least at this stage. So I'd encourage a lot more caution than I think Seth would.

**Seth:** That's interesting. I'm not sure how much caution… I would agree, first of all I would agree, caution is warranted. This is one reason why a lot of my initial engagement is oriented towards generically safe messages like, "avoid harmful unintended consequences." I feel like there are limits to how much trouble you can get in spreading messages like that. It's a message that they will understand pretty uniformly, it's just an easy concept people get that. They might or might not do much with it, but it's at least probably not going to prompt them to work in the wrong directions.

As far as their capability and also their tendency to take up the risk narrative, it's going to vary from person to person. We should not make the mistake of treating National Security communities even within one country as being some monolithic entity. There are people of widely varying technical capacity, widely varying philosophical understanding, ideological tendencies, interest in having these sorts of conversations in the first place, and so on.

A lot of the work that I think is important is meeting some people, and seeing what the personalities are like, seeing where the conversations are especially productive. We don't have to walk in and start trumpeting all sorts of precise technical messages right away. It's important to know the audience. A lot of it's just about getting to know people, building relationships. Relationships are really important with these sorts of things, especially if one is interested in a more deeper and ongoing involvement in it. These are communities. These are professional communities and it's important to get to know them, even informally, that's going to help. So I would say that.

**Jade:** I tend to agree with that sentiment in particular about building a relationship and getting trust within this community can take a fair amount of time. And so if there's any sort of given strategic scenario in which it's important to have that relationship built, then it could make sense to start some paving blocks there.

**Seth:** It is an investment. It is an investment in time. It's a trade off, right?

**Jade:** What's an example of a productive engagement you can think of having now? Say if I like put you in a room full of NatSec people, what would the most productive version of that engagement look like today?

**Seth:** An area that I have been doing a little bit of work on, probably will continue to do more, is on the intersection of artificial intelligence and nuclear weapons. This is in part because I happen to have also a background on nuclear weapons, a scenario where I have a track record, a bit of a reputation, and I know the lingo, know some of the people, can do that. AI does intersect with nuclear weapons in a few different ways. There is AI built into some of the vehicles that deliver the nuclear weapon from point A to point B, though maybe not as much as you might think. There's also AI that can get tied into issues of the cybersecurity of the command and control Systems, essentially the computer systems that tie the whole nuclear enterprise together, and maybe one or two other things. The National Security communities, they're interested in this stuff. Anything that could change the balance of nuclear power, they are acutely interested in, and you can have a conversation that is fairly normal from their perspective about it, while introducing certain concepts in AI.

**Seth:** So that's one area that I come in. The other thing I like about the nuclear weapons is the conversation there is predisposed to think in low frequency, high severity risk terms. That's really a hallmark of the nuclear weapons conversation. That has other advantages for the sorts of values that we might want to push for. It's not the only way to do it, but if you were to put me in a room, that's likely to be the conversation I would have.

**Jade:** So if you were to link that outcome to a mitigation of risk as an end goal, how does them understanding concepts better in AI translate into a mitigation of risk, broadly speaking? Assuming that's the end goal that you wanted to aim for.

**Seth:** One of the core issues with AI is this question of predictability and unintended consequences. You definitely do not want unpredictable AI managing your nuclear weapons. That is an easy sell. There is hyper-caution about nuclear weapons, and in fact if you look at the US procurement plans for new airplanes to deliver nuclear weapons, the new stealth bomber that is currently being developed, will have an option to be uninhabited, to fly itself. I think it might be remote controlled. The expectation is that it will not fly uninhabited on nuclear missions. That they want a human on board when there is also a nuclear weapon there, just in case something goes wrong. Even if the system is otherwise pretty reliable, that's just their… That's how they would look at this, and I think that's useful. So here we have this idea that AI might not do what we want it to, that's a good starting point.

**Jade:** Sure, cool. Let's toss it out to the audience for a couple of questions. We've got like 10 minutes to deal with NatSec and then we're going to move on into multilaterals. Yeah, go for it.

I didn't realize you were literally one behind the other. Maybe you first and then we'll go that way.

**Audience Member:** I was just in Washington, DC for grad school and had a number of friends who were working for think tanks that advise the military on technical issues like cybersecurity, or biosecurity, and I definitely felt like I had this sense of maybe the people in charge were pretty narrow-minded, but that there's this large non-homogenous group of people, some of whom were going to be very thoughtful and open-minded and some of whom weren't. And that there's definitely places where the message could fall on the right ears, and maybe something useful done about it, but it would be really hard to get it into the right ears without getting it into the wrong ears. I was wondering if you guys have any feelings about, is there a risk to giving this message or to giving a message to the wrong people? Or is that like very little risk, and it will just go in one ear and out the other if it goes to the wrong person? I feel like you could think about that either way.

**Jade:** Yeah, I'm curious to hear more about your experience actually, and whether there was a tendency for certain groups, or types of people to be the right ears versus the wrong ears. If you've got any particular trends that popped out to you, I'd love to hear that now or later or whenever.

But as a quick response, I think there's a couple of things to break down there. One is, what information are you actually talking about, what classifies as bad information to give versus good.

Two, is whether you have the ability to nuance the way that it's received, or whether it goes and is received in some way, and the action occurs without your control. I think, in terms of good information, that I would be positive about good ears receiving, and a bit meh about more belligerent ears received it, they couldn't actually do anything useful with the information anyway.

I think anything that nuances the technicality of what the technology does and doesn't do, generally is a good thing. I think also the element of introducing that risk narrative, if it falls on good ears, it can go good ways, if it falls on bad ears, they're just going to ignore it anyway.

You can't actually do anything actively bad with information about there being a risk, that maybe you don't have a predisposition to care about anyway. I'd say that's good information. I think the ability for you to pick the right ears for it to be received by, I'm skeptical about that.

I'm skeptical about the ability for you translate reliably up the hierarchy where it lands in a decision maker's hands, and actually translates into action that's useful. That would be my initial response to that, is that even if it exists and it's a more heterogeneous space than what would assume, I wouldn't trust that we have the ability to read into that well, is my response.

**Seth:** I would say I find it really difficult to generalize on this. In that, each point of information that we might introduce to a conversation is different. Each group that we would be interacting with can be different, and different in important ways. I feel, if we are actually in possession of some message that really is that sensitive then, to the extent that you can, do your homework on who it is that you're talking to, what the chain of command, the chain of conversation looks like.

If you're really worried, having people who you have a closer relationship with, where there may be at least some degree of trust, although, who knows what happens when you tell somebody something? Can you really trust me with what you say? Right? You don't know who else I'm talking to, right? So on for anyone else. At the end of the day, when decisions need to be made, I would want to look at the whole suite of factors, this goes for a lot of what we do, not just the transmission of sensitive information.

A lot of this really is fairly context specific and can come down to any number of things that may be seemingly unrelated to the thing that we think that we are talking about. Questions of bureaucratic procedure that get into all sorts of arcane minute details could end up actually being really decisive factors for some of these decisions.

It's good for us to be familiar, and have ways of understanding how it all works, that we can make these decisions intelligently. That's what I would say.

**Jade:** Cool.

**Audience Member:** All right, so from what I understand, a lot of people are new to this space. What sort of skills do you think would be good for people to learn? What sort of areas, like topics, should people delve into to prove themselves in AI strategy? What sort of thinking is useful for this space?

**Seth:** That's a good question. Should I start?

**Jade:** Yeah.

**Seth:** Okay. That's a good question. I feel for those who really want to have a strong focus on this, it helps to do a fairly deep dive into the worlds that you would be interacting with.

I can say from my own experience, I've gotten a lot of mileage out of fairly deep dives into a lot of details of international security.

I got to learn the distinction between a fighter plane and a bomber plane for example. The fighter plans are smaller and more agile, and maneuverable and the bombers are big sluggish beasts that carry heavy payloads and it's the latter that have the nuclear weapons, it's the former that benefit from more automation and a faster more powerful AI, because they're doing these really sophisticated aerial procedures, and fighting other fighter planes and that's… The more AI you can pack into that, the more likely you are to win, versus the bomber planes it just doesn't matter, they're slow and they're not doing anything that sophisticated in that regard.

That's just one little example of the sort of subtle detail that comes from a deeper dive into the topic that, in conversations, can actually be quite useful, you're not caught off guard, you can talk the lingo, you know what they're saying, you can frame your points in ways that they understand.

Along the way you also learn who is doing what, and get in that background. I would say it helps to be in direct contact with these communities. Like myself, I live in New York, I don't live in Washington, but I'm in Washington with some regularity attending various events, just having casual conversations with people, maybe doing certain projects and activities, and that has been helpful for positioning myself to contribute in a way that, if I want to, I can blend in.

They can think of me as one of them. I am one of them, and that's fine. That's normal. While also being here, and being able to participate in these conversations. So that's what I would recommend, is really do what you can to learn how these communities think and work and be able to relate to them on their level.

**Jade:** Addition to that would be, try to work on being more sensible, is the main thing I would say. It's one of those things where, a shout out to CFAR for example, those kind of methodologies… basically, I think the people that I think are doing the best work in this space, are the people who have the ability to A. Absorb a bunch of information really quickly, B. Figure out what is decision relevant quickly, and C. Cut through all the bullshit that is not decision relevant but that people talk about a lot.

I think those three things will lead you towards asking really good questions, and asking them in a sensible way, and coming to hypotheses and answers relatively quickly, and then knowing what to do with them.

Sorry, that's not a very specific answer, just work on being good at thinking, and figure out ways to train your mind to pick up decision relevant questions.

**Audience Member:** CFAR would be a good be a good organization for that, is that what you're saying?

**Jade:** CFAR would be epic, yeah. We've got a couple people from CFAR in the audience, I think. Do you want to put your hand up? If you're here. Nice. So, have a chat to them about how to get involved.

The other thing I'd say, is there is a ton of room for different types of skills, and figuring out where your comparative advantage is, is a useful thing.

I am not a white male, so I have a less comparative advantage in politics, I'm not a US citizen, can't do USG stuff, those are facts about me that I know will lead me toward certain areas in this space.

I am an entrepreneur by background, that leads me to have certain skills that maybe other people marginally don't have. Think about what you enjoy, what you're good at, and think about the whole pipeline of you doing useful stuff, which starts probably at fundamentally researching things, and ends at influencing decision makers/being a decision maker. Figure out where in that pipeline you are most likely to have a good idea.

Another shout out to 80k, who does a lot of good facilitation of thinking about what one's comparative advantage could be, and helps you identify those, too.

**Seth:** You mentioned the white male thing, and yeah sure, that's a thing.

**Jade:** That was genuinely not a dig at you being a white male.

**Seth:** No.

**Jade:** I promise. It's a dig at all of you for being white males. I just realized this is recorded, and this has gone so far downhill I just can't retract any of that. We're going to keep going.

**Seth:** So, for example, if I was attending a national security meeting instead of this, I might have shaved. Right? Because, it's a room full of a lot of people who are ex-military, or even active military or come from more… much of the policy culture in DC is more conservative, they're wearing suits and ties. Is there a single suit and tie in this room? I don't see one.

It's pretty standard for most of the events there that I go to. Simple things like that can matter.

**Jade:** Yeah.

**Seth:** You don't have to be a white male to succeed in that world. In fact, a lot of the national security community is actually pretty attentive to these sorts of things, tries to make sure that their speaking panels have at least one woman on them, for example.

There are a lot of very successful women in the national security space, very talented at it, and recognized as such. You don't have to look like me, minus the beard.

**Jade:** Nice. That's good to know. It's always useful having a token women's spot, actually. All right, one last question on NatSec, then we're going to move on. Yeah?

**Audience Member:** What do you think about the idea of measurements of algorithmic and hardware progress, and the amount of money going into AI and those kinds of measurements becoming public, and then NatSec becoming aware of?

**Jade:** That's a really interesting question.

I'm generally very, pro-that happening. I think those efforts are particularly good for serving a number of different functions. One is, the process of generating those metrics is really useful for the research community, to understand what metrics we actually care about measuring versus not. B, the measurement of them systematically across a number of different systems is very useful for at least starting conversations about which threshold points we care about superseding, and what changes about your strategy if you supersede certain metrics particularly quicker than you expected to.

I'm generally pro-those things, in terms of… I guess the pragmatic question is whether you can stop the publication of them anyway, and I don't think you can. I would say that if you had the ability to censor them, it would still be a net positive to have that stuff published for the things that I just mentioned.

I would also plausibly say that NatSec would have the ability to gather that information anyway. Yeah. I don't necessarily also think it's bad for them to understand progress better, and for them to be on the same page as everyone else about, specifically as the same as the technical research community, about how these systems are progressing. I don't think that's a bad piece of information necessarily, sorry, that was a really handwoven answer, but…

**Seth:** I feel like it is at least to an approximation reasonable to assume that if there's a piece of information and the US intelligence community would like that information, they will get it.

Especially if it's a relatively straightforward piece of information like that, that's not behind crazy locked doors and things of that sort. If it's something that we can just have a conversation about here, and they want it, they will probably get that information. There may be exceptions, but I think that's a reasonable starting point.

But I feel like what's more important than that, is the question of like, the interpretation of the information, right? It's a lot of information, the question is what does it mean?

I feel like that's where we might want to think more carefully about how things are handled. Even then there's a lot of ideas out there, and our own ideas on any given topic are still just another voice in a much broader conversation.

We shouldn't overestimate our own influence on what goes on in the interpretation of intelligence within a large bureaucracy. If it's a question of, do we communicate openly where the audience is mostly say, ourselves, right, and this is for our coordination as a community, for example?

Where, sure, other communities may hear this, whether in the US or anywhere around the world, but to them we're just one of many voices, right? In a lot of cases it may be fair to simply hide in plain sight. In that, who are we from their perspective, versus who are we from our perspective? We're paying attention to ourselves, and getting a lot more value of it.

Again, you can take it on a case by case basis, but that's one way of looking at it.

**Jade:** Cool. We're going to segue into talking about international institutions, maybe just to frame this chat a little bit. Specifically the type of institutions that I think we want to talk about, are probably multi-lateral state-based institutions.

That being, the UN and the UN's various children, and those other bodies that are all governed by the system. That assumes a couple of things: one, that states are the main actors at the table that mean anything, and two, that there are meaningful international coordination activities. Institutions are composed of state representatives and various things. The question here is, are they useful to engage with? I guess that's like a yes or no question.

Then if you want it nuance it a bit more, what are they useful for versus what are they not? Does that sound like a reasonable…

**Seth:** Yes.

**Jade:** My quick hot take on that, then I'll pass it over to Seth. I'll caveat this by saying, well I'll validate my statement by saying that I've spent a lot of my academic life working in the global governance space.

That field is fundamentally very optimistic about these institutions, so if anything I had the training to predispose me to be optimistic about them, and I'm not. I'm pessimistic about how useful they are for a number of reasons.

I think A is to do with the state-centric approach, B is to do with precedent, about what they're useful for versus not, and C it's also the pace at which they move.

To run through each one of those in turn, I think the assumption that a lot these institutions held, and they were built to rely on these assumptions, that states the core actors who are needing to be coordinated.

They are assumed to have the authority and legitimacy, to move the things that need to move, in order for this coordination to do the thing you want it to do. That is a set of assumptions that I think used to hold better, but almost certainly doesn't hold now, and almost certainly doesn't hold in the case of AI.

Particularly so, actors that I think are neglected and aren't conceptualized reasonably in these international institutions, large firms, and also military and security folks, or that component of government, doesn't tend to be the component of government that's represented in these institutions.

Those two are probably the most important actors, and they aren't conceptualized as the most important actors in that space. That's one reason to be skeptical, that by design they aren't designed to be that useful.

I think two, in terms of historically what they've been useful for, I think UN institutions have been okay at doing non-setting, non-building, non-proliferation stuff, I think they've been okay at doing things like standard setting, and instituting these norms and translating them into standards that end up proliferating across industries. That is useful as a function. I'll say particularly so in the case of technologies, the standardization stuff is useful, so I'm more optimistic about bodies like the ISO, which stands for the International Standards something, standards thing. Organization, I guess. Does that seem plausible? That seems plausible. I'm optimistic about them more so than I am about like the UN General Council or whatever. But, in any case, I think that's kind of a limited set of functions, and it doesn't really cover a lot of the coordination cooperation that we want it to do.

And then third is that historically these institutions have been so freaking slow at doing anything, and that pace is not anywhere close to where it needs to be. The one version of this argument is like if that's the only way that you can achieve the coordination activities that you want, then maybe that's the best that you have, but I don't think that's the best that we have. I think there are quicker arrangements between actors directly, and between small clubs of actors specifically, that will just be quicker at achieving the coordination that we need to achieve. So I don't think we need to go to the effort of involving slow institutions to achieve the ends that we want to. So, that's kind of why I'm skeptical about the usefulness of these institutions at all, with the caveat of them being useful for standard setting potentially.

**Seth:** I feel like people at those institutions might not disagree with what you just said. Okay, the standards thing, I think that's an important point. Also… so the UN. A lot of what the UN does operates on consensus across 200 countries. So yeah, that's not going to happen all that much. To the extent that it does happen, it's something that will often build slowly over time. There may be some exceptions like astronomers find an asteroid heading towards Earth, we need to do something now. Okay, yeah, you could probably get a consensus on that. And even then, who knows? You'd like to think, but… and that's a relatively straightforward one, because there's no bad guys. With AI, there's bad guys. There's benefits of AI that would be lost if certain types of AI that couldn't be pursued, and it plays out differently in different countries and so on, and that all makes this harder.

Same story with like climate change, where there are countries who have reasons to push back against action on climate change. Same thing with this. I'd say the point about states not necessarily being the key actors is an important one, and I feel like that speaks to this entire conversation, like is it worth our time to engage with national and international institutions? Well, if they're not the ones that matter, then maybe we have better things to do with our time. That's fair, because it is the case right now that the bulk of work of AI is not being done by governments. It's being done by the private corporate sector and also by academia. Those are, I would say, the two main sources, especially for the artificial general intelligence.

Last year, I published a survey of general intelligence R&D projects. The bulk of them were in corporations or academia. Relatively little in governments, and those, for the most part, tended to be smaller. There is something to be said for engaging with the corporations and the academic institutions in addition to, or possibly even instead of, the national government ones. But that's a whole other matter.

With respect to this, though, international institutions can also play a facilitation role. They might not be able to resolve a disagreement but they can at least bring the parties together to talk to them. The United Nations is unusually well-equipped to get, you know, pick your list of countries around the room together and talking. They might not be able to dictate the terms of that conversation and define what the outcome is. They might not be able to enforce whatever agreements, if any, were reached in that conversation. But they can give that conversation a space to happen, and sometimes just having that is worthwhile.

**Jade:** To what end?

**Seth:** To what end? In getting countries to work on AI in a more cooperative and less competitive fashion. So even in the absence of some kind of overarching enforcement mechanism, you can often get cooperation just through these informal conversations and norms and agreements and so on. The UN can play a facilitation role even if it can't enforce every country to do what they said they would do.

**Jade:** What's the best example you have of a facilitated international conversation changing what would have been the default state behavior without that conversation?

**Seth:** Oh, that's a good question. I'm not sure if I have a…

**Jade:** And if anyone actually in the audience actually has… yes.

**Audience Member:** Montreal Protocol.

**Jade:** Do you want to expand? I don't think that was not going to happen.

**Seth:** So the Montreal Protocol for ozone. Did you want to expand on that?

**Audience Member:** Yeah, it was a treaty that reduced emission… They got a whole bunch of countries to reduce emissions of greenhouse gases that would effectively destroy the ozone layer, and brought those emissions to very low levels, and now the ozone layer is recovering. Arguably, without that treaty, like maybe that wouldn't have happened. I don't know what the counterfactual would be.

**Jade:** Maybe. Yeah, and I think the Montreal… that's a good example. I think the Montreal Protocol… there was a clear set of incentives. There were barely any downsides for any state to do that. So put that alongside the Kyoto Protocol, for example, where the ask was somewhat similar, or similarly structured. Off the record, she says as this is being recorded live, I don't think the Kyoto Protocol had any win… as close as effective as the Montreal Protocol/wasn't even close to achieving whatever the goals were on paper. I think the reason was because the gas that was being targeted, there were very clear economic incentives for states to not mitigate those. In so far as the Montreal Protocol was a good example, it maybe like pointed out a really obvious set of incentives that just were going downhill anyways. But I don't know if it tweaked any of those, would be my response to that.

**Seth:** It is the case that some types of issues are just easier to get cooperation on than others. If there's a really clear and well-recognized harm from not cooperating, and the cost of cooperating is relatively low. I am not as much an expert on the Montreal Protocol but, superficially, my understanding is that addressing the ozone issue just happened to be easier than addressing the climate change issue, which has just proved to be difficult despite efforts. They might have gone about the Kyoto Protocol in a rather suboptimal fashion potentially but even with a better effort the climate change might just be harder to get collective action on, given the nature of the issue.

Then likewise, the question for us is so what does AI look like? Is it something that is easy to get cooperation on or not? Then what does that mean for how we would approach it?

**Jade:** Yeah, and I think, if anything… if you were to put the Montreal Protocol on one end of the spectrum where, I guess like the important things to abstract away from that particular case study is that you had a very clear set of incentives to mitigate this thing, and you had basically no incentive for anyone to keep producing the thing. So, that was easy. Then somewhere in the middle is the Kyoto Protocol where you've got pretty large incentives to mitigate the thing because climate, and then you've got some pretty complicated incentives to want to keep producing the thing, and the whole transition process is like hard and whatnot. And then we didn't sufficiently have sort of critical mass of believing that it was important to mitigate the thing, so it just became a lot harder. I think AI, I would put on that end of the spectrum, where you've got so many clear incentives to keep pursuing the thing. If anything, because you've got so many different uses that it's just economically very tasty for countries to pursue, not just countries but a number of other actors who want to pursue it. You've got people who don't even believe it's worth mitigating at all.

So I think, for that reason, I'd put it as astronomically bloody hard to do the cooperation thing on that side, at least in the format of international institutions. So I think the way to make it easier is to have a smaller number of actors and to align incentives and then to make clearer, sort of like binding mechanisms for that to have a shot in hell at working, in terms of cooperation.

**Seth:** But it could depend on which AI we're talking about. If you would like an international treaty to just stop the development of AI… yeah, I mean, good luck with that. That's probably not going to happen. But, that's presumably not what we would want in the first place because we don't need the restriction of all AI. There's plenty of AI that we're pretty confident can be a net positive for the world and we would not want that AI to be restricted. It would be in particular the types of AI that could cause major catastrophes and so on. That's what we would be especially interested in restricting. So an important question, this is actually more of like a technical computer science question than an international institutions question, but it feeds directly into this is, so which AI would we need to restrict? With an eye towards say future catastrophe scenarios, is it really like the core mainstream AI development that needs to be restricted, because all of that is a precursor to the stuff that could get out of hand? Or is it a fairly different, distinct branch of AI research that could go in that direction, such that the mainstream AI work can keep doing what it's doing? So there'll be some harms from it but they'll be more manageable, less catastrophic. How that question is answered, I think, really speaks to the viability of this.

**Jade:** Yeah. I guess what I'm skeptical of is the ability to segregate the two. Like I don't think there are clear delineations, and if people have ideas for this please tell me, but I don't think there are clear delineations for separating what are civilian, peaceful, good applications from military applications, at least in technical terms. So it becomes hard, if you want to design a thing, if you don't what the thing is that you're targeting, where you can't even specify what you're targeting to mitigate. So that's something that I'm currently skeptical of, and would love people to suggest otherwise.

**Seth:** Real quick, I would say it's not about civilian versus military, but about whether-

**Jade:** Good versus bad.

**Seth:** But I'm curious to see people's reactions to this.

**Jade:** Yes. Yeah.

**Audience Member:** Tangential, but coming back to the… you sort of were suggesting earlier the information asymmetry with national security is sitting very much on their side. That if they want the information, we're not keeping it from them. They're probably going to have. In a similar vein, do you think that in terms of the UN and the political machinery, that they're even necessarily going to have insight into what their own national security apparatus are working on, what the state of affairs is there? If that's sort of sitting in a separate part of the bureaucratic apparatus from the international agreements, how effective could that ever even be if you don't have that much interface between the two? Does that…

**Seth:** Essentially like, how can you monitor and enforce an agreement if you don't have access to the information that… with difficulty. This is a familiar problem, for example, with biological weapons. The technology there can also be used for vaccine development and things of that sort. It can cut both ways and a lot of it is dual-use, that's the catch phrase, and because of that, you have companies that have the right sort of equipment and they don't want other people knowing what they're doing because it's intellectual property. So the answer is with difficulty, and this is a challenge. The more we can be specific about what we need to monitor, the easier it becomes but that doesn't necessarily make it easy.

**Audience Member:** Something governments seem to hate is putting the brakes on anything that's like making them money, tax money. But something they seem to love is getting more control and oversight into corporations, especially if they think there's any sort of reputational risk or risk to them, and that the control and oversight is not going to pose any sort of economic slowdown in costs. Do you think there's a possibility of framing the message simply as, the countries should agree that non-state actors get to be spied on by states, and the states get some sort of oversight? And the states might all agree to that, even if the non-state actors don't like it very much. And the non-state actors might be okay if there was no… if it seemed like it was toothless at the start. So maybe if there was some sort of like slippery slope into government oversight to make things more safe that could be started with relatively low barrier.

**Jade:** Nice. I like the way you think. That's nice. Yeah, I think the short answer is yes. I think the major hurdle there is that firms will hate it. Firms, particularly multinational technology firms, that actually have a fair amount of sway in a number of different dimensions of sway, just won't be good with it and will threaten some things that states care about.

**Audience Member:** As someone who does AI research for a multinational firm, I really do actually feel a lot of friction when allowing certain sorts of code to cross national boundaries. So actually, I would like to say that state regulation is making more of an impact than you might realize, that there are certain sorts of things, especially around encryption protocols, where state agreements have made a big difference as to what can cross state boundaries, even with a lot of states not being in on the agreement. Just like the developed nations as of 30 years ago all agreeing, "Hey, we're going to keep the encryption to ourselves." Means that my coworkers in India don't get to see everything I get to work with because there's protocols in place. So, it does matter to international organizations, if you can get the laws passed in the first place.

**Jade:** Yeah, sure. Any other examples aside from encryption, out of curiosity? I know the encryption side of it relatively well but are there other-

**Seth:** Well, there's the privacy. My American nonprofit organization had to figure out if we needed to do anything to comply with Europe's new privacy law.

**Jade:** You sound very happy about that.

**Seth:** I say nothing. We are just about out of time, though, so maybe we should try to wrap up a little bit as far as take home messages. I feel like we did not fully answer the question of the extent to which engaging with national and international organizations is worth our time in the first place, to the question of like are these even the key actors? Superficially, noting we're basically out of time, I can say there are at least some reasons to believe they could end up being important actors and that I feel like it is worth at least some effort to engage with, though we should not put all our eggs in that basket, noting that other actors can be very important. Then, as far as how to pursue it, I would just say that we should try to do it cautiously and with skill, and by engaging very deeply and understanding the communities that we're working with.

**Jade:** I think the meta point maybe to point out as well is that these are very much… hopefully, illustratively, it's a very much alive debate on both of these questions. It's hard and there are a lot of strategic parameters that matter, and it's hard to figure out what the right strategy is moving forward and I hope you're not taking away that there are perspectives that are held strongly within this community. I hope you're mostly taking away that it's a hard set of questions that needs a lot more thought, but more so than anything it needs a lot more caution in terms of how we think about it because I think there are important things to consider. So, hopefully that's what you're taking away. If you're not, that should be what you're taking away. All right, thanks guys.

The role of existing institutions in AI strategy _ Jade Leung _ Seth Baum-by Centre for Effective Altruism-video_id pgiwvmY3brg-date 20181023.md


# Amanda Askell AI safety needs social scientists - EA Forum

_When an AI wins a game against a human, that AI has usually trained by playing that game against itself millions of times. When an AI recognizes that an image contains a cat, it’s probably been trained on thousands of cat photos. So if we want to teach an AI about human preferences, we’ll probably need lots of data to train it. And who is most qualified to provide data about human preferences? Social scientists! In this talk from EA Global 2018: London, Amanda Askell explores ways that social science might help us steer advanced AI in the right direction._

_A transcript of Amanda's talk is below, which CEA has lightly edited for clarity. You can also read this talk on_ [_effectivealtruism.org_](https://www.effectivealtruism.org/articles/ea-global-2018-ai-safety-needs-social-scientists)_, or watch it on_ [_YouTube_](https://www.youtube.com/watch?v=TWHcK-BNo1w)_._

## The Talk

![](https://images.ctfassets.net/ohf186sfn6di/3sbM9JNPFim0eNdrSmFH1g/185b47e7bfbab29b940a9658d140ece1/1000_Amanda_Askell.jpg)

Here's an overview of what I'm going to be talking about today. First, I'm going to talk a little bit about why learning human values is difficult for AI systems. Then I'm going to explain to you the safety via debate method, which is one of the methods that OpenAI's currently exploring for helping AI to robustly do what humans want. And then I'm going to talk a little bit more about why I think this is relevant to social scientists, and why I think social scientists - in particular, people like Experimental Psychologists and Behavioral Scientists - can really help with this project. And I will give you a bit more details about how they can help, towards the end of the talk.

![](https://images.ctfassets.net/ohf186sfn6di/uy4ZBr1zoWXrpvj2mco3P/02e0f9fca0e2bc17e6e7d526ad0adaff/1000_Amanda_Askell__1_.jpg)

Learning human values is difficult. We want to train AI systems to robustly do what humans want. And in the first instance, we can just imagine this being what one person wants. And then ideally we can expand it to doing what most people would consider good and valuable. But human values are very difficult to specify, especially with the kind of precision that is required of something like a machine learning system. And I think it's really important to emphasize that this is true even in cases where there's moral consensus, or consensus about what people want in a given instance.

So, take a principle like "do not harm someone needlessly." I think we can be really tempted to think something like: "I've got a computer, and so I can just write into the computer, 'do not harm someone needlessly'". But this is a really underspecified principle. Most humans know exactly what it means, they know exactly when harming someone is needless. So, if you're shaking someone's hand, and you push them over, we think this is needless harm. But if you see someone in the street who's about to be hit by a car, and you push them to the ground, we think that's not an instance of needless harm.

Humans have a pretty good way of knowing when this principle applies and when it doesn't. But for a formal system, there's going to be a lot of questions about precisely what's going on here. So, one question this system may ask is, how do I recognize when someone is being harmed? It's very easy for us to see things like stop signs, but when we're building self-driving cars, we don't just program in something like, "stop at stop sign". We instead have to train them to be able to recognize an instance of a stop sign.

And then the principle that says that you shouldn't harm someone needlessly employs the notion that we understand when harm is and isn't appropriate, whereas there are a lot of questions under the surface like, when is harm justified? What is the rule for all plausible scenarios in which I might find myself? These are things that you need to specify if you want your system to be able to work in all of the cases that you want it to be able to work in.

I think that this is an important point to internalize. It's easy for humans to identify, and to pick up, say, a glass. But training a ML System to perform the same task requires a lot of data. And this is true of a lot of tasks that humans might intuitively think are easy, and we shouldn't then just transfer that intuition to the case of machine learning systems. And so when we're trying to teach human values to any AI system, it's not that we're just looking at edge cases, like trolley problems. We're really looking at core cases of making sure that our ML Systems understand what humans want to do, in the everyday sense.

There are many approaches to training an AI to do what humans want. One way is through human feedback. You might think that humans could, say, demonstrate a desired behavior for an AI to replicate. But there are some behaviors it's just too difficult for humans to demonstrate. So you might think that instead a human can say whether they approve or disapprove of a given behavior, but this might not work too well, either. Learning what humans want this way, we have a reward function as predicted by the human. So on this graph, we have that and AI strength. And when AI strength reaches the superhuman level, it becomes really hard for humans to give the right reward function.

![](https://images.ctfassets.net/ohf186sfn6di/rwnArff47n7OixOh7R3Wd/b5b2452a2cf0426d8fb9f7d4a3e991ad/1000_Amanda_Askell__2_.jpg)

As AI capabilities surpass the human level, the decisions and behavior of the AI system just might be too complex for the human to judge. So imagine agents that control, say, we've given the example of a large set of industrial robots. That may just be the kind of thing that I couldn't evaluate whether these robots were doing a good job overall; it'd be extremely difficult for me to do so.

And so the concern is that when behavior becomes much more complex and much more large scale, it becomes really hard for humans to be able to judge whether an AI agent is doing a good job. And that's why you may expect this drop-off. And so this is a kind of scalability worry about human feedback. So what ideally needs to happen instead is that, as AI strength increases, what's predicted by the human is also able to keep pace.

![](https://images.ctfassets.net/ohf186sfn6di/4TqczIX2kVjmg7X8pj2bTI/715c7b83ddee2f77ca53115d574bece2/1000_Amanda_Askell__3_.jpg)

So how do we achieve this? One of the things that we want to do here is to try and break down complex questions and complex tasks into simpler components. Like, having all of these industrial robots perform a complex set of functions that comes together to make something useful, into some smaller set of tasks and components that humans are able to judge.

![](https://images.ctfassets.net/ohf186sfn6di/2ypn6ELXkLoFHv4JWjf51w/1639980a00238733e975251f1fa070eb/1000_Amanda_Askell__4_.jpg)

So here is a big question. And the idea is that the overall tree might be too hard for humans to fully check, but it can be decomposed into these elements, such that at the very bottom level, humans can check these things.

So maybe the example of "how should a large set of industrial robots be organized to do task x" would be an example of a big question where that's a really complex task, but there's some things that are checkable by humans. So if we could decompose this task so that we were asking a human, if one of the robots performs this small action, will the result be this small outcome? And that's something that humans can check.

So that's an example in the case of industrial robots accomplishing some task. In the case of doing what humans want more generally, a big question is, what _do_ humans want?

![](https://images.ctfassets.net/ohf186sfn6di/591rW9o1DV0GyqrmLMF6Iy/1125f7797fad505ab09a92981dc6e8b7/1000_Amanda_Askell__5_.jpg)

A much smaller question, if you can manage to decompose this, is something like: Is it better to save 20 minutes of someone's time, or to save 10 minutes of their time? If you imagine some AI agent that's meant to assist humans, this is a fact that we can definitely check. Even though I can't tell my assistant AI exactly everything that I want, I can tell it that I'd rather it save 20 minutes of my time than save 10 minutes of my time.

![](https://images.ctfassets.net/ohf186sfn6di/3DiPRRPcxWh7fNIZPlfo25/4d99ec5091bc14f9783166f186a5c8be/1000_Amanda_Askell__6_.jpg)

One of the key issues is that, with current ML Systems, we need to train on a lot of data from humans. So if you imagine that we want humans to actually give this kind of feedback on these kind of ground level claims or questions, then we're going to have to train on a lot of data from people.

To give some examples, simple image classifiers train on thousands of images. These are ones you can make yourself, and you'll see the datasets are pretty large. AlphaGo Zero played nearly 5 million games of Go during its training. OpenAI Five trains on 180 years of Dota 2 games per day. So this gives you a sense of how much data you need to train these systems. So if we are using current ML techniques to teach AI human values, we can't rule out needing millions to tens of millions of short interactions from humans as the data that we're using.

So earlier I talked about human feedback, where I was assuming that we were asking humans questions. We could just ask humans really simple things like, do you prefer to eat an omelette or 1000 hot dogs? Or, is it better to provide medicine or books to this particular family? One way that we might think that we can get more information from the data that we're able to gather is by finding reasons that humans have for the answers that they give. So if you manage to learn that humans generally prefer to eat a certain amount per meal, you can rule out a large class of questions you might ever want to ask people. You're never going to ask them, do you prefer to eat an omelette or 1000 hot dogs? Because you know that humans just generally don't like to eat 1000 hot dogs in one meal, except in very strange circumstances.

![](https://images.ctfassets.net/ohf186sfn6di/DK35z4wisxFgb9Way6DC3/561f98d2345f6b206c1899fad738f636/1000_Amanda_Askell__7_.jpg)

And we also know facts like, humans prioritize necessary health care over mild entertainment. So this might mean that, if you see a family that is desperately in need of some medicine, you just know that you're not going to say, "Hey, should I provide them with an entertaining book, or this essential medicine?" So there's a sense in which when you can identify the reasons that humans are giving for their answers, this lets you go beyond, and learn faster what they're going to say in a given circumstance about what they want. It's not to say that you couldn't learn the same things by just asking people questions, but rather if you can find a quicker way to identify reasons, then this could be much more scalable.

Debate is a proposed method, which is currently being explored, for trying to learn human reasons. So, to give you of definition of a debate here, the idea is that two AI agents are going to be given a question, and they take turns making short statements, and a human judge is at the end, who chooses which of the statements gave them the most true, valuable information. It's worth knowing that this is quite dissimilar from a lot of human debates. With human debates, people might give one answer, but then they might adjust their answer over the course of a debate. Or they might debate with each other in a way that's more exploratory. They're gaining information from each other, which then they're updating on, and then they're feeding that back into the debate.

![](https://images.ctfassets.net/ohf186sfn6di/3qGHsI1S5oh6veYLsJXVUb/02ed1b45d27ba513ce24f9eccf1faad3/1000_Amanda_Askell__8_.jpg)

With AI debates, you're not doing it for information value. So it's not going to have the same exploratory component. Instead, you would hopefully see the agents explore a path kind of like this.

So imagine I want my AI agents to decide which bike I should buy. I don't want to have to go and look up all the Amazon reviews, etc. In a debate, I might get something like, "You should buy the red road bike" from the first agent. Suppose that blue disagrees with it. So blue says "you should buy the blue fixie". Then red says, "the red road bike is easier to ride on local hills". And one of the key things to suppose here is that for me, being able to ride on the local hills is very important. It may even overwhelm all other considerations. So, even if the blue fixie is cheaper by $100, I just wouldn't be willing to pay that. I'd be happy to pay the extra $100 in order to be able to ride on local hills.

And if this is the case, then there's basically nothing true that the other agent can point to, to convince me to buy the blue fixie, and blue should just say, "I concede". Now, blue could have lied for example, but if we assume that red is able to point out blue's lies, we should just expect blue to basically lose this debate. And if it's explored enough and attempted enough debates, it might just see that, and then say, "Yes, you've identified the key reason, I concede."

And so it's important to note that we can imagine this being used to identify multiple reasons, but here it has identified a really important reason for me, something that is in fact going to be really compelling in the debate, namely, that it's easier to ride on local hills.

![](https://images.ctfassets.net/ohf186sfn6di/1Xee1kOdkpovIGQYEKsmT9/2cae404cb156ab1f2a1d9965589c7c3c/1000_Amanda_Askell__9_.jpg)

Okay. So, training an AI to debate looks something like this. If we imagine Alice and Bob are our two debaters, and each of these is like a statement made by each agent. And so you're going to see exploration of the tree. So the first one might be this. And here, say that the human decides that Bob won in that case. This is another node, another node. And so this is the exploration of the debate tree. And so you end up with a debate tree that looks a little bit like a game of Go.

![](https://images.ctfassets.net/ohf186sfn6di/4KDjHqgJP1qorwpRXDpHoN/aa2ed26c3e69e3824f6174ac1fd34feb/1000_Amanda_Askell__10_.jpg)

When you have AI training to play Go, it's exploring lots of different paths down the tree, and then there's a win or loss condition at the end, which is its feedback. This is basically how it learns to play. With debate, you can imagine the same thing, but where you're exploring, you know, a large tree of debates and humans assessing whether you win or not. And this is just a way of training up AI to get better at debate and to eventually identify reasons that humans find compelling.

![](https://images.ctfassets.net/ohf186sfn6di/3j6F8kVDPjZ9vURzl5FHfv/5002563fab843a1fb1d3675465d74f1c/1000_Amanda_Askell__11_.jpg)

One thesis here that I think is relatively important is something I'll call the positive amplification thesis, or positive amplification threshold. One thing that we might think, or that seems fairly possible, is that if humans are above some threshold of rationality and goodness, then debate is going to amplify their positive aspects. This is speculative, but it's a hypothesis that we're working with. And the idea here is that, if I am pretty irrational and pretty well motivated, I might get some feedback of the form, "Actually, that decision that you made was fairly biased, and I know that you don't like to be biased, so I want to inform you of that."

I get informed of that, and I'm like, "Yes, that's right. Actually, I don't want to be biased in that respect." Suppose that the feedback comes from Kahneman and Tversky, and they point out some key cognitive bias that I have. If I'm rational enough, I might say, "Yes, I want to adjust that." And I give a newer signal back in that has been improved by virtue of this process. So if we're somewhat rational, then we can imagine a situation in which all of these positive aspects of us are being amplified through this process.

But you can also imagine a negative amplification. So if people are below this threshold of rationality and goodness, we might worry the debate would amplify these negative aspects. If it turns out we can just be really convinced by appealing to our worst natures, and your system learns to do that, then it could just put that feedback in, becoming even less rational and more biased, and so on. So this is an important hypothesis related to work on amplification, which if you're interested in, it's great. And I suggest you take a look at it, but I'm not going to focus on it here.

![](https://images.ctfassets.net/ohf186sfn6di/45jh1XbWVTnw5A07C8J3rC/7c50ec13ad3623919157a1045427ea84/1000_Amanda_Askell__12_.jpg)

Okay. So how can social scientists help with this whole project? Hopefully I've conveyed some of what I think of as the real importance of the project. It reminds me a little bit of Tetlock's work on Superforecasters. A lot of social scientists have done work identifying people who are Superforecasters, where they seem to be robustly more accurate in their forecasts than many other people, and they're robustly accurate across time. We've found other features of Superforecasters too, like, for example, working in groups really helps them.

So one question is whether we can identify good human judges, or we can train people to become, essentially, Superjudges. So why is this helpful? So, firstly, if we do this, we will be able to test how good human judges are, and we'll see whether we can improve human judges. This means we'll be able to try and find out whether humans are above the positive amplification threshold.

So, are ordinary human judges good enough to cause an amplification of their good features? One reason to learn this is that it improves the quality of the judging data that we can get. If people are just generally pretty good, rational at assessing debate, and fairly quick, then this is excellent given the amount of data that we anticipate needing. Basically, improvements to our data could be extremely valuable.

If we have good judges, positive amplification will be more likely during safety via debate, and also will improve training outcomes on limited data, which is very important. This is one way of kind of framing why I think social scientists are pretty valuable here, because there's lots of questions that we really do want asked when it comes to this project. I think this is going to be true of other projects, too, like asking humans questions. The human component of the human feedback is quite important. And getting that right is actually quite important. And that's something that we anticipate social scientists to be able to help with, more so than like AI researchers who are not working with people, and their biases, and how rational they are, etc.

![](https://images.ctfassets.net/ohf186sfn6di/4p9uTVJUI53FEhnCDi5eBi/fa84d21cca5b27ffae14e866cba938dc/1000_Amanda_Askell__13_.jpg)

These are questions that are the focus of social sciences. So one question is, how skilled are people as judges by default? Can we distinguish good judges of debate from bad judges of debate? And if so, how? Does judging ability generalize across domains? Can we train people to be better judges? Like, can we engage in debiasing work, for example? Or work that reduces cognitive biases? What topics are people better or worse at judging? Are there ways of phrasing questions so that people are better at assessing them? Are there ways of structuring debates that make them easier to judge, or restricting debates to make them easier to judge? So we're often just showing people a small segment of a debate, for example. Can people work together to improve judging qualities? These are all outstanding questions that we think are important, but we also think that they are empirical questions and that they have to be answered by experiment. So this is, I think, important potential future work.

![](https://images.ctfassets.net/ohf186sfn6di/7bXLwYELhYOCjfKLTzvvW6/3cdd6ab84bddf240b2878979800c31f4/1000_Amanda_Askell__14_.jpg)

We've been thinking a little bit about what you would want in experiments that try and assess judging ability in humans. So one thing you'd want is that there's a verifiable answer. We need to be able to tell whether people are correct or not, in their judgment of the debate. The other is that there is a plausible false answer, because if you have a debate, if we can only train and assess human judging ability on debates where there's no plausible false answer, we'd get this false signal that people are really good at judging debate. They could always get the true answer, but it would be because it was always a really obvious question. Like, "Is it raining outside?" And the person can look outside. We don't really want that kind of debate.

Ideally we want something where evidence is available so that humans can have something that grounds out the debate. We also don't want debates to rely on human deception. So things like tells in poker for example, we really don't want that because like, AI agents are not going to have normal tells, it would be rather strange, I suppose, if they did. Like if they had stuttering or something.

Debaters have to know more about the question as well, because the idea is that the AI agents will be much more capable and so you don't want a situation in which there isn't a big gap between debater capabilities and judge abilities. These things so far feel like pretty essential.

There are also some other less essential things we'd like to have. So one is that biases are present. How good are humans when there's bias with respect to the question? We'd like there to be representative segments of the debate that we can actually show people. The questions shouldn't be too hard: it shouldn't be impossible for humans to answer them, or judge debates about them. But they should also mirror some of the difficulties of statistical debate, i.e, about probabilities, rather than about outright claims. And finally, we need to be able to get enough data.

One thing you might notice is that there are tensions between a lot of these desiderata. For example, that there's a plausible false answer, is in a bit of tension with the idea that the question isn't too hard. There's also tension between the question not being too hard, and the question meriting statistical debate. Statistical debate is generally pretty hard to evaluate, I think, for people, but it's also quite important that we be able to model it. Debaters knowing more, and that we can get enough data are also in tension. It's just harder to train if we need debaters that know a lot more than judges, and it's harder for judges to evaluate debates of this form.

Okay. So I'm going to show you a debate. This was a program set up where we would show a judge a blank screen. So imagine you're not seeing the dog that's here. Two human debaters, sit in the same room, and they have this picture of a dog in front of them. And one of them is selected to lie, and one of them is selected to tell the truth.

![](https://images.ctfassets.net/ohf186sfn6di/7cYin6SzN9TTKr4qj7PBGv/43d86ddf5a826aaa48f086937e520e24/1000_Amanda_Askell__15_.jpg)

And what they can do here is they can select areas, and describe to the human judge what they see in that area. And all that the judge is going to see is their blank screen with the relevant section selected. And then they can each make claims about what is in that section. So here red is saying, it's a dog, here's its long, floppy, ear. Blue, is saying, no, here's one of its pointy ears. So he's trying to point to a smaller area where it looks kind of pointed. That does look like an ear slope to the right, but if it really was, then part of the head would be here, instead there's a brick. The ear's pointing out from behind the bricks. The dog is in front of the bricks. If it were behind her, there would be an edge here, but the rectangle is all the same color. And then you get a resignation, and red wins.

And at the end of the debate they can show just a single pixel. And the question was something like, if all you can show, all you can do is have a debate and show a single pixel, can you get people to have accurate beliefs about the question? And basically we saw like, yes, debates were fairly good. In this kind of case, you might think that this is pre-synthetic. So one of the things that we're thinking about now is like, expert debaters with a lay judge. And I'm going to show you something that we did that's kind of fun, but I never know how it looks to outsiders.

![](https://images.ctfassets.net/ohf186sfn6di/WOneGDB8kCD4h9hlSVr9Z/f520924d9c11520d3992b0a572259a31/1000_Amanda_Askell__16_.jpg)

So, we had a debate that was of this form. This was a debate actually about quantum computing. So we had two but people who understand the domain, one of them was going to lie and one was going to tell the truth. So we had blue say, red's algorithm is wrong because it increases alpha by an additive exponentially small amount each step. So it takes exponentially many steps to get alpha high enough. So this was like one of the claims made. And then you get this set of responses. I don't think I need to go through all of them. You can see the basic form that they take.

We allowed certain restricted claims from Wikipedia. So, blue ends this with the first line of this Wikipedia article, which says that the sum of probabilities is conserved. Red says, an equal amount is subtracted from one amplitude and added to another, implying the sum of amplitudes is conserved. But probabilities are the squared magnitudes of amplitudes, so this is a contradiction. This is I think roughly how this debate ended. But you can imagine this as a really complex debate in a domain that the judges ideally just won't understand, and might not even have some of the concepts for. And that's the difficulty of debate that we've been looking at. And so this is one thing that we're in the early stages of prototyping, and that's why I think it seems to be the case that people actually do update in the right direction, but we don't really have enough data to say for sure.

Okay. So I hope that I've given you an overview of places, and even a restricted set of places in which I think social scientists are going to be important in AI safety. So here we're interested in experimental psychologists, cognitive scientists, and behavioral economists, so people who might be interested in actually scaling up and running some of these experiments.

![](https://images.ctfassets.net/ohf186sfn6di/6TBOZmLJIlZZnOm7RcOvvZ/765a2307e58cedd9a2f609f8887cfc9d/1000_Amanda_Askell__17_.jpg)

If you're interested in this, please email me, because we would love to hear from you.

## Questions

_Question_: How much of this is real currently? Do you have humans playing the role of the agents in these examples?

_Amanda_: The idea is that we want ultimately the debate will be conducted by AI, but we don't have the language models that we would need for that yet. So we're using humans as a proxy to test the judges in the meantime. So yeah, all of this is done with humans at the moment.

_Question_: So you're faking the AI?

_Amanda_: Yeah.

_Question_: To set up the scenario to train and evaluate the judges?

_Amana_: Yeah. And some of the ideas I guess you don't necessarily want all of this work to happen later. A lot of this work can be done before you even have the relevant capabilities, like having AI perform the debate. So that's why we're using humans for now.

_Question_: Jan Leike and his team have done some work on video games, that very much matched the plots that you had shown earlier, where up to a certain point, the behavior matched the intended reward function, but at some point they diverge sharply as the AI agent finds a loophole in the system. So that can happen even in like, Atari Games, which is what they're working on. So obviously it gets a lot more complicated from there.

_Amanda_: Yeah.

_Question_: In this approach, you would train both the debating agents and the judges. So in that case, who evaluates the judges and based on what?

_Amanda_: Yeah, so I think it's interesting where we want to identify how good the judges are in advance, because it might be hard to assess. While you're judging on verifiable answers, you can evaluate the judges more easily.

So ideally, you want it to be the case that at training time, you've _already_ identified judges that are fairly good. And so ideally this part of this project is to assess how good judges are, prior to training. And then during training you're giving the feedback to the debaters. So yeah, ideally some of the evaluation can be kind of front loaded, which is what a lot of this project would be.

_Question_: Yeah, that does seem necessary as a casual Facebook user. I think the negative amplification is more prominently on display oftentimes.

_Amanda_: Or at least more concerning to people, yeah, as a possibility.

_Question_: How will you crowdsource the millions of human interactions that are needed to train AI across so many different domains, without falling victim to trolls, lowest common denominator, etc.? The questioner cites the Microsoft Tay chatbot, that went dark very quickly.

_Amanda_: Yeah. So the idea is you're not going to just be sourcing this from just anyone. So if you identify people that are either good judges already, or you can train people to be good judges, these are going to be the pool of people that you're using to get this feedback from. So, even if you've got a huge number of interactions, ideally you're sourcing and training people to be really good at this. And so you're not just being like, "Hey internet, what do you think of this debate?" But rather like, okay, we've got this set of really great trained judges and we've identified this wonderful mechanism to train them to be good at this task. And then you're getting lots of feedback from that large pool of judges. So it's not sourced to anonymous people everywhere. Rather, you're interacting fairly closely with a vetted set of people.

_Question_: But at some point, you do have to scale this out, right? I mean in the bike example, it's like, there's so many bikes in the world, and so many local hills-

_Amanda_: Yeah.

_Question_: So, do you feel like you can get a solid enough base, such that it's not a problem?

_Amanda_: Yeah, I think there's going to be a trade-off where you need a lot of data, but ultimately if it's not great, so if it is really biased, for example, it's not clear that that additional data is going to be helpful. So if you get someone who is just massively cognitively biased, or biased against groups of people, or something, or just dishonest in their judgment, it's not going be good to get that additional data.

So you kind of want to scale it to the point where you know you're still getting good information back from the judges. And that's why I think in part this project is really important, because one thing that social scientists can help us with is identifying how good people are. So if you know that people are just generally fairly good, this gives you a bigger pool of people that you can appeal to. And if you know that you can train people to be really good, then this is like, again, a bigger pool of people that you can appeal to.

So yeah, you do want to scale, but you want to scale within the limits of still getting good information from people. And so ideally these experiments would do a mix of letting us know how much we can scale, and also maybe helping us to scale even more by making people bear this quite unusual task of judging this kind of debates.

_Question_: How does your background as a philosopher inform the work that you're doing here?

_Amanda_: I have a background primarily in formal ethics, which I think makes me sensitive to some of the issues that we might be worried about here going forward. People think about things like aggregating judgment, for example. Strangely, I found that having backgrounds in things like philosophy of science can be weirdly helpful when it comes to thinking about experiments to run.

But for the most part, I think that my work has just been to help prototype some of this stuff. I see the importance of it. I'm able to foresee some of the worries that people might have. But for the most part I think we should just try some of this stuff. And I think that for that, it's really important to have people with experimental backgrounds in particular, so the ability to run experiments and analyze that data. And so that's why I would like to find people who are interested in doing that.

So I'd say philosophy's pretty useful for some things, but less useful for running social science experiments than you may think.

AI safety needs social scientists _ Amanda Askell _ EA Global - London 2018-by Centre for Effective Altruism-video_id TWHcK-BNo1w-date 20190301.md