# Baobao Zhang How social science research can inform AI governance - EA Forum

_Political scientist Baobao Zhang explains how social science research can help people in government, tech companies, and advocacy organizations make decisions regarding artificial intelligent (AI) governance. After explaining her work on public attitudes toward AI and automation, she explores other important topics of research. She also reflects on how researchers could make broad impacts outside of academia._

_We’ve lightly edited Baobao’s talk for clarity. You can also watch it on_ [_YouTube_](https://www.youtube.com/watch?v=eTkvtHymI9s) _and read it on_ [_effectivealtruism.org_](https://effectivealtruism.org/articles/baobao-zhang-how-social-science-research-can-inform-ai-governance)_._

## The Talk

**Melinda Wang (Moderator):** Hello, and welcome to this session on how social science research can inform AI governance, with Baobao Zhang. My name is Melinda Wang. I'll be your emcee. Thanks for tuning in. We'll first start with a 10-minute pre-recorded talk by Baobao, which will be followed by a live Q&A session.

Now I'd like to introduce you to the speaker for this session, Baobao Zhang. Baobao is a fellow at the [Berkman Klein Center for Internet and Society](https://cyber.harvard.edu/) at Harvard University and a research affiliate with the [Centre for the Governance of AI](https://www.fhi.ox.ac.uk/govai/) at the University of Oxford. Her current research focuses on the governance of artificial intelligence. In particular, she studies public and elite opinions toward AI, and how the American welfare state could adapt to the increasing automation of labor. Without further ado, here's Baobao.

**Baobao:** Hello, welcome to my virtual presentation. I hope you are safe and well during this difficult time. My name is Baobao Zhang. I'm a political scientist focusing on technology policy. I'm a research affiliate with the Centre for the Governance of AI at the Future of Humanity Institute in Oxford. I'm also a fellow with the Berkman Klein Center for Internet and Society at Harvard University.

Today I will talk about how social science research can inform AI governance.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/fb440c6dca78eaffcdd72fe55393333b640267fc6ec1cb69.png)

Advances in AI research, particularly in machine learning (ML), have grown rapidly in recent years. Machines can outperform the best human players in strategy games like [Go](https://en.wikipedia.org/wiki/Go_(game)) and poker. You can even generate synthetic videos and news articles that easily fool humans.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/0f845f3709e38afe2a292ab5b2e946e6e8d9215c84b91c7c.png)

Looking ahead, ML researchers believe that there's a 50% chance of AI outperforming humans in all tasks by 2061. This \[estimate\] is based on a survey that my team and I conducted in 2016.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/060864652249468dd881dd8d05b2f98a822b6a5355d2c684.png)

The EA community has recognized the potential risks, even existential risks, that unaligned AI systems pose to humans. Tech companies, governments, and civil society have started to take notice as well. Many organizations have published AI ethics principles to guide the development and deployment of the technology.

A [report by the Berkman Klein Center](https://cyber.harvard.edu/publication/2020/principled-ai) counted 36 prominent sets of AI principles.

Now we're entering a phase where tech companies and governments are starting to translate these principles into policy and practice.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/5822e0b2f9709ee135eaa0b886440c1311d886f434e3c5d0.png)

At the Centre for the Governance of AI (GovAI), we think that social science research — whether it’s in political science, international relations, law, economics, or psychology — can inform decision-making around AI governance.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/9a3f2cf2c41107d7b6629b938f562100c79b819cb3b5a2a6.png)

For more information about our research agenda, please see “[AI Governance: A Research Agenda](https://www.fhi.ox.ac.uk/wp-content/uploads/GovAI-Agenda.pdf)” by Allan Dafoe. It's also a good starting place if you're curious about the topic and are new to it.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/0934fcedb98c6f697a5d01d1afa7359faaefb082da4dd962.png)

Here's a roadmap for my talk. I’ll cover:

-   My research on public opinion toward AI
-   EA social science research highlights on AI governance
-   Research questions that I've been thinking about a lot lately
-   How one can be impactful as a social scientist in this space

### Why study the public’s opinion of AI?

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/e0106a04f566e148745c8c8dc5880e9ea1152ac5306427a8.png)

From a normative perspective, we need to consider the voices of those who will be impacted by AI. In addition, public opinion has shaped policy in many other domains, including climate change and immigration; therefore, studying public opinion could help us anticipate how electoral politics may impact AI governance.

The research I'm about to present comes from [this report](https://isps.yale.edu/sites/default/files/files/Zhang_us_public_opinion_report_jan_2019.pdf).

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/d63fd689a349045a7260a648497f5b9eb3ec024d894829ea.png)

It's based on a nationally representative survey of 2,000 Americans that Allan Dafoe and I conducted in the summer of 2018.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/d71f6b1b571a5c127b8e7808e157f6bc613de4701e111ebc.png)

Here are the main takeaways from the survey:

1\. An overwhelming majority of Americans think that AI should be carefully managed.
2\. They considered all 13 governance challenges that we presented to them to be important.
3\. However, they have only low-to-moderate levels of trust in the actors who are developing and managing AI.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/f64b1b9c399746f28f4b8bb98bc7ba062a7200d364e3306a.png)

Now, on to some results. Here's a graph of Americans' view of AI governance challenges. Each respondent was randomly assigned to consider five challenges randomly selected from 13. The x-axis shows the respondents’ perceived likelihood that the governance challenge would impact large numbers of people around the world. The y-axis shows the perceived importance of the issue.

Those perceived to be high in both dimensions include protecting data privacy, preventing AI-enhanced cyber attacks, preventing mass surveillance, and preventing digital manipulation — all of which are highly salient topics in the news.

I’ll point out that the respondents consider all of these AI governance challenges to be important for tech companies and governments to manage. But we do see some variations between respondents when we break them down by subgroups.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/eedca4761990c8164513480745457644cb21962ee78fec14.png)

Here we've broken it down by age, gender, race, level of education, partisanship, etc. We're looking at the issue’s perceived importance in these graphs. Purple means greater perceived issue importance. Green means lesser perceived issue importance.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/9660a291d9489697ffe72699b383e0e3830f8a2f014bcc21.png)

I'll highlight some differences that really popped out. In this slide, you see that older Americans, in contrast to younger Americans, perceive the governance challenges presented to them to be more important.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/4517087c727b1965c6872371b66055cdef7399f465882c53.png)

Interestingly, those who have CS \[computer science\] or engineering degrees, in contrast to those who don't, perceive all of the governance challenges to be less important. We also observed this techno-optimism among those with CS or engineering degrees in other survey questions.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/c256c6159734d52f2836096b8c18f17390cbaa4abab0ec69.png)

Despite Americans perceiving that these AI governance challenges are important, they have low to moderate levels of trust in the actors who are in a position to shape the development and deployment of AI systems.

There are a few interesting observations to point out in these slides:

-   While trust in institutions has declined across the board, the American public still seems to have relatively high levels of trust in the military. This is in contrast to the ML community, whose members would rather not work with the US military. I think we get this seemingly strange result because the public relies on heuristics when answering this question.
-   The American public seems to have great distrust of Facebook. Part of it could be the fallout from the [Cambridge Analytica scandal](https://en.wikipedia.org/wiki/Facebook%E2%80%93Cambridge_Analytica_data_scandal). But when we ran a previous survey before the scandal broke, we observed similarly low levels of trust in Facebook.

I'm sharing just some of the results from our report. I encourage you to read it. We're currently working on a new iteration of the survey. And we're hoping to launch it concurrently in the US, EU, and China, \[which will allow us\] to make some interesting cross-country comparisons.

I would like to highlight two works by my colleagues in the EA community who are also working in AI governance.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/8f91c170f28bfdb93332b1219f8a3606b422840d6f966aef.png)

First, there’s “[Toward Trustworthy AI Development](https://arxiv.org/abs/2004.07213).” This paper came out recently. It's a massive collaboration among researchers in different sectors and fields, \[with the goal of determining\] how to verify claims made by AI developers who say that their algorithms are safe, robust, and fair. This is not merely a technical question. Suggestions in the report include creating new institutional mechanisms like bounties for detecting bias and safety issues in AI systems, and creating AI incident reports.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/52527536eb1165dcbab706872defa17ddbb73c1ec298e298.png)

Second, there’s “[The Windfall Clause](https://www.fhi.ox.ac.uk/windfallclause/)” by my colleagues at GovAI. Here, the team considers this idea of a “windfall clause” as a way to redistribute the benefits from transformative AI. This is an ex-ante agreement, where tech companies — in the event that they make large profits from their AI systems — would donate a massive portion of those profits. The report combines a lot of research from economic history and legal analysis to come up with an inventive policy proposal.

### New research questions

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/28134c847ff756822ffa4845ea7fb0d3239afcd0a64b010c.png)

There are a lot of new and interesting research questions that keep me up at night. I’ll share a few of them with you, and let's definitely have a discussion about them during the Q&A.

-   **How do we build incentives for developing safe, robust, and fair AI systems — and avoid a race to the bottom?** I think a lot of us are rather concerned about the rhetoric of an AI arms race. But it's also true that even the EU is pushing for competitiveness in AI research and development. I think the “[Toward Trustworthy AI Development](https://arxiv.org/abs/2004.07213)” paper gives some good recommendations on the R&D \[research and development\] front. But what will the market and policy incentives be for businesses and the public sector to choose safer AI products? That's still a question that I and many of my colleagues are interested in.
-   **How can we transition to an economic system where AI can perform many of the tasks currently done by humans?** I've been studying perceptions of automation. Unfortunately, a lot of workers underestimate the likelihood that their jobs will be automated. They actually have an optimism bias. Even correcting workers’ \[false\] beliefs about the future of work in my studies has failed to make them more supportive of redistribution. And it doesn't seem to decrease their hostility toward globalization. So certainly there's a lot more work to be done on the political economy around the future of work.
-   **How do other geopolitical risks make AI governance more difficult?** I think about these a lot. In Toby Ord's book, [_The Precipice_](https://www.amazon.com/dp/B07V9GHKYP/), he talks about the risk factors that could increase the probability of existential risk. And one of these risks is great-power war. We're not at \[that point\], but there has certainly been a rise in aggressive nationalism from some of these great powers. Instead of coming together to combat the COVID pandemic, many countries are pointing fingers at each other. And I think these trends don't bode well for international governance. Therefore, thinking about how these trends might shape international cooperation around AI governance is definitely a topic that my colleagues and I are working on.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/ec30e6c2c3b8cc47b7f2c179a9067bc7f525eb4117acf089.png)

I’ll conclude this presentation by talking about how one can be impactful as a social scientist. I have the great luxury of working in academia, where I have plenty of time to think and carry out long-term research projects. At the same time, I have to constantly remind myself to engage with the world outside of academia — the tech industry and the policy world — by writing op eds, doing consulting, and communicating with the media.

Fortunately, social scientists with expertise in AI and AI policy are also in demand in other settings. Increasingly, tech companies have sought to hire people to conduct research on how individual humans interact with AI systems, or what the impact of AI systems may be on society. Geoffrey Irving and Amanda Askell have published a paper called “[AI Safety Needs Social Scientists](https://openai.com/blog/ai-safety-needs-social-scientists/).” I encourage you to read it if you're interested in this topic.

To give a more concrete example, some of my colleagues have worked with [OpenAI](https://openai.com/) to test whether their [GPT-2 language model](https://openai.com/blog/better-language-models/) can generate news articles that fool human readers.

Governments are also looking for social scientists with expertise in AI. Policymakers in both the civilian government and in the military have an AI literacy gap. They don't really have a clear understanding of the limits and the potentials of the technology. But advising policymakers does not necessarily mean that you have to work in government. Many of my colleagues have joined think tanks in Washington, DC, where they apply their research skills to generate policy reports, briefs, and expert testimony. I recommend checking out the [Center for the Security and Emerging Technology](https://cset.georgetown.edu/) or CSET, based at Georgetown University. They were founded about a year ago, but they have already put out a vast collection of research on AI and US international relations.

Thank you for listening to my presentation. I look forward to your questions during the Q&A session.

**Melinda:** Thank you for that talk, Baobao. The audience has already submitted a number of questions. We're going to get started with the first one: \[Generally speaking\], what concrete advice would you give to a fresh college graduate with a degree in a social science discipline?

**Baobao:** That's a very good question. Thank you for coming to my talk.

One of the unexpected general pieces of advice that I would give is to have strong writing skills. At the end of the day, you need to translate all of the research that you do for different audiences: readers of academic journals, policymakers, tech companies \[that want you to produce\] policy reports.

Besides that, I think \[you may need to learn certain skills depending on\] the particular area in which you specialize. For me, learning data science and statistics is really important for the type of research that I do. For other folks it might be game theory, or for folks doing qualitative research, it might be how to do elite interviews and ethnographies.

But overall, I think having strong writing skills is quite critical.

**Melinda:** Great. \[Audience member\] Ryan asks, “What are the current talent gaps in AI safety right now?”

**Baobao:** That's a good question. I must confess: I'm not an AI safety expert, although I did talk about a piece that folks at OpenAI wrote called “[AI Safety Needs Social Scientists.](https://openai.com/blog/ai-safety-needs-social-scientists/)” And I definitely agree with the sentiment, given that the people who are working on AI safety want to run experiments. You can think of them as psychology experiments. And a lot of computer scientists are not necessarily trained on how to do that. So if you have skills in running surveys or psych experiments, that’s a skill set that I hope tech companies will acknowledge and recognize as important.

**Melinda:** That's really interesting. Would you consider psychology to be within the realm of social sciences, \[in terms of how\] people generally perceive it? Or do you mean fields related to STEM \[science, technology, engineering, math\]?

**Baobao:** I think psychology is quite interesting. People who are more on the neuroscience side might be \[considered to be in\] STEM. I work with some experimental psychologists, particularly social psychologists, and they read a lot of the literature in economics, political science, and communications studies. I do think that there's a bit of overlap.

**Melinda:** How important do you consider interdisciplinary studies to be, whether that's constrained within the realm of social science, or social science within STEM, etc.?

**Baobao:** I think it's important to work with both other social scientists and with computer scientists.

This is more \[along the lines of\] career advice and not related to AI governance, but one of the realizations I had in doing recent work on several COVID projects is that it’s important to have more than just an “armchair” \[level of understanding\] of public health. I try to get my team to talk with those who are either in vaccine development, epidemiology, or public health. And I'd like to see more of that type of collaboration in the AI governance space. We do that quite well at that, I think, at GovAI, where we can talk to in-house computer scientists at the [Future of Humanity Institute](https://www.fhi.ox.ac.uk/).

**Melinda:** Yes. That's an interesting point and relates to one of the questions that just came in: How can we more effectively promote international cooperation? Do you have any concrete strategies that you can recommend?

**Baobao:** Yes. That's a really good question. I work a bit with folks in the European Union, and bring my AI governance expertise to the table. The team that I'm working with just submitted a consultation to the EU Commission. That type of work is definitely necessary.

I also think that collaboration with folks who work on AI policy in China is fruitful. I worry about the decoupling between the US and China. There's a bit of tension. But if you're in Europe and you want to collaborate with Chinese researchers, I encourage it. I think this is an area that more folks should look into.

**Melinda:** Yes, that's a really good point.

In relation to the last [EAGxVirtual talk on biosecurity](https://www.youtube.com/watch?v=BbOHQLrVSX4), can you think of any information hazards within the realm of AI governance?

**Baobao:** That's a good question. We do think a lot about all of our publications. At GovAI, we talk about being careful in our writing so that we don't necessarily escalate tensions between countries. I think that's definitely something that we think about.

At the same time, there is the [open science movement](https://en.wikipedia.org/wiki/Open_science) in the social sciences. It’s tricky to \[find a good\] balance. But we certainly want to make sure our work is accurate and speaks to our overall mission at GovAI of promoting beneficial AI and not doing harm.

**Melinda:** Yes. A more specific question someone has asked is “Do you think this concept of info hazards — if it's a big problem in AI governance — would prohibit one from spreading ideas within the \[discipline\]?”

**Baobao:** That's a good question. I think the EA community is quite careful about not spreading info hazards. We're quite deliberate in our communication. But I do worry about a lot of the rhetoric that other folks who are in this AI governance space use. There are people who want to drum up a potential AI arms race — people who say, “Competition is the only thing that matters.” And I think that's the type of dangerous rhetoric that we want to avoid. We don't want a race to the bottom where, whether it's the US, China, or the EU, researchers only care about \[winning, and fail to consider\] the potential risks of deploying AI systems that are not safe.

**Melinda:** Yes. Great. So we're going to shift gears a little bit into more of the nitty-gritty of your talk. One question that an audience member asked is “How do you expect public attitudes toward AI to differ by nationality?”

**Baobao:** That's a good question. In the talk, I mentioned that at GovAI, we're hoping to do a big survey in the future that we run concurrently in different countries.

Judging from what I've seen of the literature, [Eurobarometer](https://ec.europa.eu/commfrontoffice/publicopinion/index.cfm) has done a lot of good surveys in the EU. As you would expect, folks living in Europe — where there are tougher privacy laws — tend to be more concerned about privacy. But it’s not necessarily so that Chinese respondents are totally okay with a lack of privacy.

That’s why we're hoping to do a survey in which we ask \[people in different countries\] the same questions, around the same time. I think it’s really important to make these cross-national comparisons. With many questions, you get different responses because of how the questions are framed or worded, so \[ensuring a\] rigorous approach will yield a better answer to this question.

**Melinda:** Yes. I'd like you to unpack that a bit more. In what concrete ways can we be more culturally sensitive in stratifying these risks when it comes to international collaboration?

**Baobao:** I think speaking the \[appropriate\] language is really important. I can provide a concrete example. As my team worked on the EU consultation that I mentioned, we talked to folks who work at the European Commission in order to understand their particular concerns and “speak their language.”

They're concerned about AI competition and the potential for an arms race, but they don't want to use that language. They also care a lot about human rights and privacy. And so when we make recommendations, since we’ve read the \[relevant\] reports and spoken to the people involved in the decision-making, those are the two things that we try to balance.

And in terms of the survey research that we're hoping to do, we're consulting with folks on the ground so that our translation work is localized and \[reflects\] cultural nuances.

**Melinda:** Yes, that's a really good point about using the right language. It also \[reminds\] me of how people in the social sciences often think differently, and may use different \[terminology\] than people from the STEM realm, for example. The clash of those two cultures can sometimes result in conflicts. Do you have any advice on how to mitigate those kinds of conflicts?

**Baobao:** That's a good question. Recently, GovAI published a guide to \[writing the\] impact statements required for the [NeurIPS](https://nips.cc/) conference. One of the suggestions was that computer scientists trying to \[identify\] the societal impact of their research talk with social scientists who can help with this translational work. Again, I think it’s quite important to take an interdisciplinary approach when you conduct research.

**Melinda:** Yes. Do you think there's a general literacy gap between these different domains? And if so, how should these gaps be filled?

**Baobao:** Yes, good question. My colleagues, Mike Horowitz and \[Lauren\] Kahn at the University of Pennsylvania, have written a [piece about the AI literacy gap in government](https://warontherocks.com/2020/01/the-ai-literacy-gap-hobbling-american-officialdom/). They acknowledge that it's a real problem. Offering crash courses to train policymakers or social scientists who are interested in advising policymakers is one way to \[address this issue\]. But I do think that if you want to work in this space, doing a deep dive — not just taking a crash course — can be really valuable. Then, you can be the person offering the instruction. You can be the one writing the guides.

So yes, I do think there's a need to increase the average level of AI literacy, but it’s also important for the social science master's degree and PhD programs to train people.

**Melinda:** Yes, that's interesting. It brings up a question from \[audience member\] Chase: “Has there been more research done on the source of technical optimism from computer scientists and engineers? \[Is this related to\] overconfidence in their own education or in fellow developers?”

**Baobao:** That's a good question. We \[work with\] machine learning researchers at GovAI, and \[they’ve done\] research that we hope to share later this year. I can't directly address that research, but there may be a U-shape curve. If the x-axis represents your level of expertise in AI, and the y-axis represents your level of concern about AI safety or risks from AI, those who don't have a lot of expertise are kind of concerned. But those who have CS or engineering degrees are perhaps not very worried.

But then, if you talk to the machine learning researchers themselves, many are concerned. I think they're recognizing that what they work on can have huge societal impacts. You have folks who work in AI safety who are very concerned about this. But in general, I think that the machine learning field is waking up to these potential risks, given the proliferation of AI ethics principles coming from a lot of different organizations.

**Melinda:** Yes, it’s interesting how the public is, for once, aligned with the experts in ML. I suppose in this instance, the public is pretty enlightened.

I'm not sure if this question was asked already, but are organizations working on AI governance more funding-constrained or talent-constrained?

**Baobao:** That's a good question. May I \[mention\] that at GovAI, we're looking to hire folks in the upcoming months? We're looking for someone to help us on survey projects. And we're also looking for a project manager and other researchers. So, in some sense, we have the funding — and that's great. And now we just need folks who can do the research. So that's my plug.

I can't speak for all organizations, but I do think that there is a gap in terms of training people to do this type of work. I'm going to be a faculty member at a public policy school, and they're just beginning to offer AI governance as a course. Certainly, a lot of self-study is helpful. But hopefully, we'll be able to get these courses into the classrooms at law schools, public policy schools, and different PhD or master's degree programs.

**Melinda:** Yes. I think perhaps the motivation behind that question was more about the current general trend around a lot of organizations in EA being extremely competitive to get into — especially EA-specific job posts. I'm wondering whether you can provide a realistic figure for how talent-constrained AI governance is, as opposed to being funding-constrained.

**Baobao:** I can't speak to the funding side. But in terms of the research and the human-resource side, I think we're beginning to finally \[arrive at\] some concrete research questions. At the same time, we're building out new questions. And it's hard to predict what type of skill set you will need.

As I mentioned in my talk, GovAI has realized that social science skills are really important — whether you can do survey research, legal analysis, elite interviews on the sociology side, or translational work. So, although I don't have a good answer to that question, I think getting some sort of expertise in one of the social sciences and having expertise in AI policy or AI safety are the types of skills that we're looking for.

**Melinda:** Okay. Given that we have six minutes left, I'm going to shift gears a little bit and try to get through as many questions as possible. Here’s one: “What insights from nuclear weapons governance can inform AI governance?”

**Baobao:** I think that's a really good question. And I think you've caught me here. I have some insights into the dual-use nature of nuclear weapons. People talk about AI as a “general-purpose technology.” It could be both beneficial and harmful. And speaking from my own expertise in public opinion research, one of the interesting findings in this space is that people tend to reject nuclear energy because of its association with nuclear weapons. There's a wasted opportunity, because nuclear energy is quite cheap and not as harmful as, say, burning fossil fuels. But because of this negative association, we’ve sort of rejected nuclear energy.

\[Turning to\] trust in AI systems, we don't want a situation where people's association of AI systems is so negative that they reject applications that could be beneficial to society. Does that make sense?

**Melinda:** Yes.

**Baobao:** I recommend \[following the work of\] some of my colleagues in international relations, at organizations like [CSET](https://cset.georgetown.edu/) who have written about this. I'm sorry — it's not my area of expertise.

**Melinda:** Okay. The next question is “Is there a way we can compare public trust in different institutions, specifically regarding AI, compared to a general baseline trust in that institution — for example, in the case of the public generally having greater trust in the US military?”

**Baobao:** That's a good question. I think that, because AI is so new, the public relies on their heuristics rather than what they know. They're just going to rely on what they think of as a trustworthy institution. And one thing that I've noticed, and that I wrote about in my [Brookings report](https://www.brookings.edu/research/public-opinion-lessons-for-ai-regulation/), is how some areas of political polarization around AI governance map to what you would expect to find in other domains. It’s concerning, in the context of the US at least, that if we can't agree on the right policy solution, we might just \[rely on\] partisan rhetoric about AI governance.

For example, acceptance of facial recognition software maps to race and partisanship. African Americans really distrust facial recognition. Democrats tend to distrust facial recognition. Republicans, on the other hand, tend to have a greater level of acceptance. So you see this attitude towards policing being mapped to facial recognition.

You also see it in terms of regulating algorithmically curated social media content. There seems to be a bipartisan backlash against tech, but when you dig more deeply into it, as we've seen recently, Republicans tend to think about content moderation as censorship and \[taking away\] a right, whereas Democrats tend to see it as combating misinformation. So unfortunately, I do think that partisanship will creep into the AI governance space. And that's something that I'm actively studying.

**Melinda:** Yes. We only have one minute left. I'm going to go through the last questions really quickly. With regards to the specific data that you presented, you mentioned that the public considers all AI governance challenges important. Did you consider including a made-up governance challenge to check response bias?

**Baobao:** Oh, that's a really good suggestion. We haven't, but we certainly can do that in the next round.

**Melinda:** Okay, wonderful. And that concludes the Q&A part of this session. \[...\] Thanks for watching.

How social science research can inform AI governance _ Baobao Zhang _ EAGxVirtual 2020-by Centre for Effective Altruism-video_id eTkvtHymI9s-date 20200615.md


# Rohin Shah on Effective altruism, AI safety, and learning human preferences from the world’s state by Jeremie Harris on the Towards Data Science Podcast

If you walked into a room filled with objects that were scattered around somewhat randomly, how important or expensive would you assume those objects were?

What if you walked into the same room, and instead found those objects carefully arranged in a very specific configuration that was unlikely to happen by chance?

These two scenarios hint at something important: human beings have shaped our environments in ways that reflect what we value. You might just learn more about what I value by taking a 10 minute stroll through my apartment than by spending 30 minutes talking to me as I try to put my life philosophy into words.

And that’s a pretty important idea, because as it turns out, one of the most important challenges in advanced AI today is finding ways to communicate our values to machines. If our environments implicitly encode part of our value system, then we might be able to teach machines to observe it, and learn about our preferences without our having to express them explicitly.

The idea of leveraging deriving human values from the state of an human-inhabited environment was first developed in a paper co-authored by Berkeley PhD and incoming DeepMind researcher Rohin Shah. Rohin has spent the last several years working on AI safety, and publishes the widely read AI alignment newsletter — and he was kind enough to join us for this episode of the Towards Data Science podcast, where we discussed his approach to AI safety, and his thoughts on risk mitigation strategies for advanced AI systems.

Here were some of my favourite take-homes from our conversation:

-   Like many, Rohin was driven towards AI alignment and AI safety work in part through his exposure to members of the Effective Altruism community. Effective Altruism is a philosophical movement focused on determining how people can make the maximum positive impact on the world, either through charitable giving or their professional career. It’s focused on asking questions like: what is the move I could make, that would maximize the expected value of my contribution to the world? For Rohin, the possible harm that could be done by powerful AI systems in the future — and the possible benefits that these systems could have if they’re developed safely — made AI alignment appealing.
-   Rohin discussed two important potential failure modes of advanced AI systems, which already appear in different forms in current systems.
-   First, he highlighted the risk of bad generalization: AIs learning the wrong lessons from their training data, leading them to generalize in ways that humans might not anticipate or want. As an example, he cited an initial concern with OpenAI’s GPT-3 model, which surfaced when a developer prompted the model with a nonsense question (like, “how many glubuxes are in a woolrop?”). Rather than answering this query “honestly”, by saying something like, “I don’t know — I’m not familiar with those words,” GPT-3 tried to answer with a best guess, like “there are 3 glubuxes in a woolrop.” You could argue that this actually wasn’t a bad generalization: GPT-3 basically answered like a student taking a test, who wants to hide the fact that they don’t know the answer to a question by taking a guess based purely on the context of the question. But if our hope was to make an honest language model — one that admits its ignorance when appropriate — GPT-3 in its non-fine-tuned form seemed to fail the test. As AI systems get more powerful, this kind of behavior could get more harmful, so Rohin thinks it’s worth paying attention to.
-   Second, Rohin discussed the challenge of communicating human preferences to AIs. This is a hard problem: most humans don’t actually know what they want out of life, and are even less capable of communicating those desires and values to other humans — let alone to machines, which are currently less context-aware and flexible in their reasoning. That’s where Rohin’s work on teaching machines to infer human preferences from their environment comes in: he thinks the strategy shows promise as an additional source of data on human preferences that machines might be able to use to decipher human values without needing us to be able to articulate them explicitly. We talked about many interesting strengths and weaknesses of this strategy.

## Chapters:

-   0:00 Intro
-   1:44 Effective altruism
-   6:50 Rohin’s introduction to AI safety work
-   11:18 Why AI risk is so serious
-   18:33 Child rearing analogy
-   22:15 Statistical learning theory
-   25:09 What is preference learning?
-   32:23 Application to higher levels of abstraction
-   34:45 Revealed preferences and the state of the world
-   36:26 Broken vase metaphor
-   44:10 Rule of time horizons
-   49:03 Wrap-up

## Please find below the transcript for Season 2 Episode 4:

Jeremie (00:00):
Hey, everyone. Welcome to another episode of the Towards Data Science podcast. My name is Jeremie and, apart from hosting the podcast, I’m also on the team at the SharpestMinds data science mentorship program. I’m really excited about today’s episode because I’ve been thinking about getting today’s guest on the podcast for a very long time. I’m so glad we finally made it happen.

Jeremie (00:16):
He’s actually in between a transition right now from Berkley where he’s wrapping up his PhD. He’s working at the Center for Human Compatible AI, and he’s transitioning over into DeepMind where he’ll be doing some alignment work. His name is Rohin Shah. And, apart from being a very prolific researcher in AI and AI alignment in particular, he is also the publisher of the AI alignment newsletter which is a really, really great resource if you’re looking to get oriented in the space to learn about some of the open problems and open questions in AI alignment. I really recommend checking it out.

Jeremie (00:47):
We’re going to talk about a whole bunch of different things, including the philosophy of AI, the philosophy of machine learning and AI alignment, ways in which it can be accomplished, some of the challenges that exist, and we’re going to explore one of the most interesting proposals, I think, that Rohin’s come up with, which is an idea about extracting human preferences from the state of the environment. So, basically, the idea here is that humans, through their activity, have encoded their preferences implicitly in their environments, we do a whole bunch of different things, different actions, that reveal our preferences. And it would be great if we could have AIs look at the world and figure out what our preferences are implicitly based on the state of that world. And that might be a great way to bootstrap AI alignment efforts.

Jeremie (01:30):
We’ll be talking about that proposal in depth, along with a whole bunch of other things. I’m really looking forward to the episode, so I hope you enjoy. Without further ado, here we go.

Jeremie (01:39):
Hello, and thank you so much for joining us for the podcast.

Rohin (01:41):
Yeah, thanks for having me here. I’m excited.

Jeremie (01:44):
Well, I’m very excited to have you. There’s so many interesting things that you’ve been working on in the alignment space in general. But, before we tackle some of the more technical questions, there’s an observation that I think anybody who’s spent any time working on alignment or talking to alignment researchers is going to end up making at some point, which is that the vast majority of people in the space seem to come from the effective altruism community. And I’d love to get your take on, number one, what the effective altruism community is, what effective altruism is, and number two, why you think there’s this deep connection between EA, effective altruism, and AI alignment and AI safety research.

Rohin (02:20):
Yeah, sure. The overarching idea effective altruism, the very easy to defend one that doesn’t get into specifics, is with whatever money, time, resources, whatever you’re willing to spend altruistically, you should try to do the most good you can with it, rather than… And you should think about that. It’s pretty hard to argue against this. I don’t think I’ve really seen people disagree with that part.

Rohin (02:56):
Now, in practice, effective altruism the movement has whole bunch of additional premises that are meant to be in support of the skill, but are more controversial. I think the really fundamental big idea of effective altruism is that of cause prioritization. Many people will say, “Okay, I want to have, say, clean water in Africa. I will work towards that.” And they’ll think about different ways in which you can get clean water in Africa, maybe you could try sanitizing the water that people already get, or you could try building some new treatment plants in order to provide fresh, flowing water that’s drinkable to everyone. And they’ll think about how best can they achieve their goal of delivering clean water.

Rohin (03:47):
It’s much, much, much less common for people to think, “Okay, well, should I work on getting clean water to people in Africa, or combating racism in the US? Which of these should I put my effort into? Or my money into?” The main premise of effective altruism is you can, in fact, do this. There are actually significant differences between causes and, by thinking about it, have much more impact by selecting the right cause to work on in the first place.

Rohin (04:20):
It’s very focused on this take weird… Take ideas seriously, actually evaluate them, figure out whether or not they are true and not whether or not they sound crazy. Whether or not they sound crazy does have some relation to whether or not they are true, but they are not necessarily the same. I think that ties into why it’s also such a hotbed for AI safety research. The EA case for AI safety, for work on AI safety, is that AI has a good chance of being extremely influential in the next century, let’s say.

Rohin (05:00):
There is some argument that is debatable, but it doesn’t seem like you can rule it out. It seems like at least moderately likely that if we don’t take care of how we do this, the AI system might “take over” in the sense of all of the important decisions about the world are made by the AI system and not by humans. And one possible consequence of this is humans go extinct. I’ll go into this argument later, I’m sure, but-

Jeremie (05:36):
So \[crosstalk 00:05:37\]-

Rohin (05:37):
\[crosstalk 00:05:37\] believe this argument somewhat, and so then it becomes extremely important and impactful to work on. It sounds crazy, but one of the EA’s, to me, strengths is that it separates what sounds crazy from what is true.

Jeremie (05:52):
It seems like, really, the focus is there’s this extra missing step that a lot of people don’t apply to their thinking when deciding what causes to contribute to, what to work on, what to spend their lives on, and that is the step of going, “What areas are going to give outsized returns on my time?”

Rohin (06:09):
Yep, exactly.

Jeremie (06:12):
I can really think back to most of the conversations I ever had with people about causes, about charity, and it’s usually focused on stuff like what are the administrative fees associated with this charity? Oh, I want to donate to a place where all my dollars go to the cause, rather than asking the more fundamental question, is this cause actually going to give the best ROI from the standpoint of benefiting everyone or benefiting humanity? It’s interesting that that kind of thinking, a more first principles approach, leads a lot of people to the area of AI alignment and AI safety. As you said, it makes sense, you’ve got this super high risk high reward profile.

Jeremie (06:50):
What was it that drew you, for example, to AI alignment, AI safety work in particular, rather than any of the other, I could imagine, bio terrorism, I could imagine all kinds of horrible things that could happen to us, but why AI alignment in particular?

Rohin (07:05):
Yeah, so, my story is kind of weird. It may be a classic AI story in that convinced by weird very weird arguments. I got into effective altruism in 2014. I heard the arguments for AI risk within a year of that, probably. I was deeply unconvinced by them. I just did not buy them.

Rohin (07:37):
And, so, until 2017, I basically didn’t engage very much with AI safety. I was also unconvinced of, basically, there’s this field of ethics called population ethics which tries to deal with the question of how do you compare how good different worlds are when they have different populations of people in them? We don’t need to go into the details, but it’s a very confusing area. Lots of impossibility results that say you might want these six very intuitive properties, but, no, you can’t actually have all of them at the same time, stuff like this. So you’re \[crosstalk 00:08:21\]-

Jeremie (08:20):
Would the idea here be like is a world with 100 decently happy people better than a world with 1,000 decent minus epsilon happy people? Is that the kind of calculation?

Rohin (08:31):
Yes. That’s an example of the question that I would deal with, yeah.

Rohin (08:35):
So, anyway, I was thinking about this question a lot in the summer of 2017. And, eventually, I was like, “Okay, I think I should actually put a fair amount of weight,” not certainty, certainly, but a fair amount of weight on the view that more happy people is, in fact, just means a better world, even if they’re in the future. Once you put a decent probability on that, it starts looking overwhelmingly important to ensure that the future continues to exist and have happy people because it’s just so big relevant to the present.

Rohin (09:21):
And so, then, I wanted to do something that was more future oriented and I had a ton of skills in computer science and math and, basically, everything you would want to work in AI alignment. I still was not very convinced of AI risk but I was like, “Okay, a bunch of smart people have thought about this, maybe I should work on it for a while, see whether or not it makes sense.” That’s what caused me to actually switch, and a year later I actually started believing the arguments.

Jeremie (09:57):
That’s so interesting.

Rohin (09:58):
I also saw different arguments.

Jeremie (10:00):
You were led by… Is it the quality of the people who were drawn to the problem more so than necessarily the initial arguments themselves? Do you remember an ah-ha moment as you were working on this stuff where you’re like, “Well, wait a minute. This is actually for real.” I can now see why Nick Bostrom, and maybe Eliezer Yudkowsky, and whoever else is talking about it back then was on to something?

Rohin (10:21):
I never really had an ah-ha moment. I remember, at one point, I was like, “I guess I now believe these arguments,” but it wasn’t like I… I guess I now believe that AI risk is substantial and real. I can’t point to a specific point in time where yes, now I believe it. I just, one day, was reflecting on it and noticed, “Oh, yeah. I used to not believe this. And now I do.”

Jeremie (10:52):
That’s so interesting. It seems like there’s a bifurcation between people who they read Superintelligence, or they read less wrong, and they get really excited about the problem and really scared of it right off the bat because, for whatever reason, they’re wired in such ways to have that happen. And then people, yeah, who are like you. It’s like a slow burn and you ease into it. I guess this is part of the problem, almost, of articulating the problem if it takes that long to get people to think of this as a really important thing.

Jeremie (11:18):
Do you have a strategy that you use when you try to explain to people why is AI risk so serious? Why is the probability nontrivial that you think might’ve worked on you back then to accelerate the process?

Rohin (11:32):
Yeah, I should note that I still… I’m not super happy with the arguments in like Superintelligence, for example. I would say that it’s slightly different arguments that are motivating it for me with still a fair amount of emphasis on things that were in Superintelligence.

Jeremie (11:50):
I think a lot of people won’t have heard of Superintelligence, by the way.

Rohin (11:54):
Oh, yes.

Jeremie (11:55):
If you want to address any of the arguments that you raise, please feel free to give that background, too.

Rohin (12:00):
Yeah, maybe I’ll just talk about the arguments I personally like since I can explain them better. But, just for context, Superintelligence is a book by a professor at Oxford named Nick Bostrom. It was published in 2014 and it was the first \[inaudible 00:12:19\] treatment of why AI risk is a thing that might occur and why we should think it might be plausible, what solutions might seem like they should be investigated and stuff like that.

Rohin (12:33):
And then, for me, personally, the argument I would give… So, A, we’re going to \[inaudible 00:12:45\] as a premise that we build AI systems that are intelligent, like as intelligent as a human, let’s say. We can talk about that later, but that’s a whole other discussion. I’ll just say that I think it is not… I think it is \[inaudible 00:13:03\] reasonably likely to happen in the next century. But, for now, take it as an assumption.

Rohin (13:10):
One thing about intelligence is it means that you can adapt to new situations, you’re presented with a new situation, you learn about it and you do something and that something is coherent. It makes sense. One example I give of this, where we see this even with current neural nets, is a specific example from GPT-3. I believe viewers will be familiar with GPT-3… Listeners, not viewers. But if not, GPT-3 is a \[inaudible 00:13:47\] language generation \[inaudible 00:13:48\] that OpenAI developed and released recently.

Rohin (13:51):
I think I like one particular example which ones from the the post giving GPT-3 a Turing test. Where the context to GPT-3 was a bunch of questions and answers. And GPT-3 would pose a question, how may bonks are in a quoit. These are nonsense words, you did not mishear me. GPT-3, in some sense, this is outside of its training distribution. It has never seen this sentence in its training corpus, presumably. It may not have even seen the words bonk and quoit ever. It’s actual distribution shift and you’re relying on some sort of generalization out of distribution.

Rohin (14:43):
Nonetheless, I think we can all predict that GPT-3 is not going to output some random string of characters. It’s going to probably say something sensible. In fact, the thing that it says is that, “You know, there are three bonks in a quoit.” Why three? I have no idea. But, you know, it’s sensible in some sense. It produced an answer that sounds like English.

Jeremie (15:10):
And we’ve all been there to some degree, if we write exams or whatever, we’re asked how many bonks are in a quoit, we haven’t done our studying, and, hey, there are three bonks in a quoit. There we go.

Rohin (15:18):
Exactly, right? In some sense, GPT-3 did generalize, it generalized the way a student taking a test would. In the original post, this was taken as evidence of GPT-3 not actually being reasonable because it doesn’t know how to say, “This question is nonsense.”

Rohin (15:39):
But then a followup post was like, “Actually, you totally can get GPT-3 to do that!” If you tell GPT-3 that… If in the context, you say whenever it sees a nonsense question the AI responds, “Yo, be real.” Then, when it’s asked how many bonks are in a quoit? It says, “Yo, be real.” So you know it’s got the ability to tell that this is nonsense, it just turned out that it generalized in a way where it was more like a test taker and less like somebody in conversation. Did we know that ahead of time? No, we did not. We had to actually run GPT-3 in order to figure this out.

Rohin (16:23):
I think AI risk is basically like this, but supercharged where your AI system, if it is human level intelligent, it’s definitely going to be deployed in new areas, in new situations, that we haven’t seen before. We just don’t really have a compelling reason to believe that it will continue to do the thing that we were training it to do as opposed to something else. In GPT-3, what were we training it to do? Well, on the training data set, at least, we were training it to do whatever a human would write in that context.

Rohin (17:05):
When you see there are three bonks in a… Sorry, how many bonks are in a quoit? What would a human do in that circumstance? I don’t know. It’s not really well defined, and GPT-3 did something sensible. I don’t think you could reasonably say it wasn’t doing what we trained it to do, it just did something that was coherent. And, similarly, if you’ve got AI systems that are human level intelligent or more, taking super impactful actions upon the world and they are put in these new situations where there’s not really a fact of the matter about how they will generalize then they might take actions that have a high impact on the world that aren’t what we want.

Rohin (17:49):
And then, as maybe intuition \[inaudible 00:17:52\] for why this could be really, really bad, like human extinction level bad. One particular distribution shift is you go from the training setting where humans have more power than the AI and can turn off the AI to the setting where the AI is sufficiently intelligent and sufficiently widely deployed, but no human can… Or humanity as a whole cannot turn it off. In that situation, that’s a new situation. AI has never been in a situation where it had this sort of power before. Will it use it in some way that… Will it generalize in some way that was different than what we expected during training? We don’t really have a reason to say no, it won’t do that.

Jeremie (18:33):
Is there an analogy you think here with child rearing? I’m just thinking of here intergenerational human propagation where our ancestors in the 1600s, at least in the west, I’m sure would be absolutely disgusted by our vile ways today the way that we deal with sex, the way that we communicate to our elders, the way that we manage our institutions and so on, all our hierarchies are just completely different. And, in many ways, we’re \[inaudible 00:19:00\] to the moral frameworks that were applied in the middle ages or the early renaissance.

Jeremie (19:07):
I guess there is a difference here in the sense that at least we are still running on the same fundamental hardware, or something very similar. Maybe that ensures a minimum level of alignment, but does this analogy break apart in some way?

Rohin (19:18):
I think that’s a pretty good intuition. There are some ways that the analogy breaks, like for example, well… The analogy doesn’t break so much as I would say put a little bit less weight on it for these reasons. One is, in child rearing, you have some influence over children, but you don’t get to do a full training process where you give them gradients for every single time step of action that they ever do. You might hope that, given that you can have way, way, way more selection pressure on AI systems, you would be able to avoid this problem.

Rohin (20:00):
But, yes, I think that that is the same fundamental dynamic that I’m pointing at. You have some amount of influence over these agents, but those agents that encounter new situations and they do something in those situations and you didn’t think about those situations ahead of time and you didn’t train them to do the right thing.

Jeremie (20:23):
I definitely buy the idea here that this AI risk, this is really significant risk. The stakes are very high. When it comes to the solutions or the strategies that you think are most promising, you yourself are specialized, obviously, in one category, everyone has to be, in one subspace within the alignment problem domain. What is it that the area that you’ve decided to focus on and why do you think that is most deserving of the attention at this point?

Rohin (20:50):
The story that I’ve told you so far is one of generalization. The main issue is we don’t know how to generalize and, plausibly, you could get AI systems that are single mindedly pursuing power and that’s similar to the Superintelligence story and those could cause human extinction. The fundamental mechanism is bad generalization, or generalization that’s like your capabilities generalize. You do something coherent and high impact, but the thing you’re trying to do doesn’t generalize, relative to what humans wanted.

Rohin (21:31):
A lot of the things I’m most excited about are somehow generalization related. One thing that I’m interested in is can we get a better understanding of empirically how do neural nets tend to generalize? Can we say anything about this? There’s a lot of theory that tries to explain why neural nets have as good generalization power as they do. It can’t be explained by statistical learning theory because neural nets can memorize random noise, but nonetheless seem to generalize reasonably well on when the labels are not random noise.

Jeremie (22:15):
And do you mind explaining statistical learning theory as a reference? I’m actually not so sure that I can to the connection.

Rohin (22:23):
Statistical learning theory is like a branch of machine learning theory that tries to do several things. But, among other things, try to prove that if we train a machine learning model on such and such training data with such and such training properties, then we know that it will generalize in such and such way and it proves theorems about this.

Rohin (22:50):
Importantly, most approaches right now focus on making assumptions about your model, your hypothesis class. These assumptions usually preclude the ability to overfit to an arbitrary sized data set because if you could, then you can’t really say anything about generalization. But the fact of the matter is neural nets really can actually overfit to any data set. They can memorize labels that are literal random noise. And, so, these assumptions just don’t apply to neural nets.

Rohin (23:28):
The thing I’m excited about is can we talk about assumptions on the data set, rather than just the model? And using, if we think about assumptions on the data set and assumptions on the model, then can we say something about how neural nets tend to generalize? This is like a super vague not fleshed out hope that I have not really started working on, nor to my knowledge has anyone else.

Rohin (23:55):
There’s just so many empirical things about neural nets that are so deeply confusing to me, like deep double descent. I don’t get it. It’s an empirical phenomenon. If you don’t know, you can look it up. It’s probably not that worth me going into, just so confusing. I don’t know why it happens. It makes no sense to me. And I will want to know why, and I think that if we understood things like this, we might be able to start making statements about how neural nets tend to generalize that maybe that translates into things we can say about safety.

Jeremie (24:27):
That’s interesting, because the generalization story seems to be one ingredient of the problem, of course, and then the other ingredient, which, I mean, there’s some overlap, but it does seem like they have distinct components. Is this challenge of telling machines what human preferences even are, our ability to tell each other what we want out of life is already so limited, and it’s something that, I mean, at least I personally find somewhat jarring as a prospect, having to actually not only express our preference, but quantify them and etch them into some kind of loss function that we then feed to a model. You’ve done a lot of interesting work on this.

Jeremie (25:09):
And, actually, there’s one of your papers that I wanted to talk about. We discussed this before we started recording and I was so glad to hear that it was also the one that you thought was the most interesting. We have compatible views, at least on that. It was this idea of… Well, the paper’s title is Preferences Implicit in the State of the World. I guess, first, I wanted to ask a question to set the scene a little bit. What is preference learning? What is that concept?

Rohin (25:34):
This is actually the next thing I was going to say I was excited by.

Jeremie (25:38):
Oh, great.

Rohin (25:39):
Which is I’ve talked about generalization, but before you get to generalization, you want to train on the right thing in the first place. That seems like a good starting point for AI system. If you don’t have that, you’re probably toast. Lots of ink has been spilled on how it’s actually very typical to specify what you want by writing a program or an equation that captures it in a number which, as you know, how deep reinforcement learning, or any deep learning system, works. But it’s most commonly associating with deep reinforcement learning.

Rohin (26:21):
The idea of preferenced learning is rather than having to specify what you want by writing down an equation, you specify it by some easier method. For example, you could look at two trajectories in a reinforcement setting, you can look at two behaviors that the agent took, and you can say, “Ah, yes, the left one. That one was better.” That’s giving the agent some feedback about what it should be doing. It’s not trying to write down an equation that captures the ideal behavior in every possible scenario. It’s just saying, out of these two, which one is better? You would imagine that that’s easier for humans to do and more likely to be correct.

Rohin (27:08):
This preferenced learning field, I think of it as the field of how do we design mechanisms for humans to give feedback to an AI system, such that we can actually give feedback that incentivizes the behavior we actually want and we don’t make as many mistakes in specification as we do with reward functions.

Jeremie (27:34):
So, what I find really exciting about that aspect, too, is there’s this well known difference in humans between expressed desires and revealed desires, or expressed intent and revealed intent. I’ll say I want to work out for three hours today, I want to do a bunch of coding, and I want to have a bunch of vegan meals for the next month. And then if you check in on me next month, I will have not have done all those things, I wouldn’t have done nearly all those things. And the question is well, which me is me? Am I the aspirational self that said hey, I would love to be that person. Or am I the jackass who actually sat on his couch and watched Netflix the entire time?

Jeremie (28:16):
This seems to really scratch that itch in the sense that if probes are revealed preferences, for better or for worse, I guess that could also be a failure mode. Is that something that you see as valuable in this approach?

Rohin (28:29):
Yeah, I think you want to use both sources of information and not do either one. Actually, let me take a step back and distinguish between two different things you could be trying to do. There’s one thing where you’re trying to learn what humans value, which is the sort of thing that you’re talking about, and there’s another framing where you’re just like, “I want my AI system to do such and such task and I want to train it to do that, but I can’t write down a reward function for that task.”

Rohin (29:01):
I’m actually more interested in the latter, honestly, but the former is also something I’ve spent a lot of time on and I’m excited by. Right now, we’re talking about the former.

Jeremie (29:13):
Can I ask a naïve question? I think I understand what the difference is, but I just want to put it to you to tackle it explicitly. What is the difference between those two things?

Rohin (29:24):
One thing is maybe I want my AI system to vacuum my floors, or something. The task of vacuuming my floors is not well specified just by that sentence. Anyone who has a Roomba will tell you stories of the Roomba being super dumb. Some of those are just the Roomba being not intelligent enough, but some are also the task is not super well specified.

Rohin (29:57):
Should the agent vacuum underneath a Christmas tree where there’s a bunch of needles that might ruin their vacuum? Who knows. If there’s some random shiny button on the floor, should it be vacuumed up or left alone? Because maybe that button’s important. What sorts of things, should the cat ever be vacuumed? The cat has a lot of hair that gets everywhere. If you vacuum the cat, that seems like it would make your house cleaner.

Rohin (30:31):
There’s lots of ambiguity here. I wouldn’t really say that these are human values, like teaching your Roomba how to vacuum does not seem to be the same thing as teaching the Roomba about human values. For one thing, you can’t really talk that much about revealed preferences here because I don’t vacuum my house very often. If an AI system were going to queue vacuum, I might have it vacuum more often.

Jeremie (31:06):
Would you say this is a narrow application of human preferences? It almost seems like the distinction between narrow AI and AGI somehow maps onto this.

Rohin (31:16):
Yeah, and I think I agree with that. I would say, but in this sense, everything is narrow AI. You just get narrow AI that becomes more and more general, and at some point we decide to stop calling it narrow AI and start to call it AGI because of how broad it has become.

Rohin (31:34):
I like the idea of you start with something that can be applied to systems today and you just scale it up. It becomes more and more capable, more and more general, but it’s always the same technique. Eventually, the systems that we create with it, we would label them as AGI or human level intelligence or super intelligent. It’s the same technique, it’s the same general principle. That’s why I’m more excited by this framing of the problem, rather than the human value spamming.

Rohin (32:07):
As you get to more general systems, it merges in with the human values. Once you get AI systems that are designing government policies or something, whatever feedback you’re giving them, it better teach them about human values.

Jeremie (32:23):
Yeah, and hopefully, I guess, we start to do that at higher and higher levels of abstraction as you say as we climb that ladder. We fill in the convolutional filters in a sense as we go up.

Rohin (32:34):
Yes, exactly. You had asked a question about revealed preferences versus spoken preferences, or express preferences. I think, yes, this is an important distinction. I definitely want any method that we propose to not be dependent on one or the other but to instead be using both, and there will be conflicts, I’m mostly hoping that we can just have AI systems that set aside the conflicts and do things that are robustly good according to either set. Probably, you’ll have to have some amount of conflict resolution mechanism, but humans already have to do this in some sense. It seems plausible that we could do it.

Rohin (33:28):
I think it is a good, very nice aspect of this is that you don’t have to commit yourself to finding the behavior up front in every possible situation. We just don’t know this. Our values are not well defined enough, honestly, for that to be right. Our values are constantly in the process of being updated as we encounter new situations. Right now, we talk about democracy, one vote per person. If, someday, in the transhumanist future, if someday it becomes possible to just make copies of people, I think we would pretty quickly no longer want to have one vote per person. Because otherwise you can just pay to have anyone elected if you are sufficiently rich.

Jeremie (34:19):
Yeah. Or, I guess, just in the limited better information about brain states, we could say well, sure, this policy makes the majority of people happier, but the people it makes more unhappy, I mean, look at that horrible dopamine cycle. Those people are really taking a big hit and you wake those responses.

Rohin (34:37):
Yep, yeah, you could definitely optimize better for social welfare potentially, and maybe then you don’t want to just have one vote per person.

Jeremie (34:45):
Right, now, I guess this brings us back to preferences implicit in the state of the world, there are things, presumably, about the structure of the world that reveal our, I guess this is revealed preferences, mostly, right?

Rohin (34:57):
Yes.

Jeremie (34:57):
What we’ve actually done.

Rohin (34:58):
Yep, this is definitely a revealed preferences method. I think an important aspect of this is people will… I think one of the reasons I’m especially excited about this, which I want to say as a prelude, is that it’s not trying to do the hard things. When people think about value learning, they think about should a self driving car if it has a choice between running into two passengers, or killing the driver, what should it do? Those are hard, ethical questions. I’m less interested in them. I want to start with can we get an AI system that reliably knows that it should not kill humans. If there are two options where, yeah…

Rohin (35:50):
Anyway, the basic stuff that we all agree on or nearly all agree on, and so I think looking at the state of the world is a good way to see this and the basic intuition here is that we’ve been acting in the world for so long. We have preferences, we’ve been rearranging the world to fit the way that we want the world to be. As a result, you can invert that process to figure out what things we probably want.

Rohin (36:26):
There’s this nice toy example that illustrates this. Suppose there’s a room, and in the middle of the room there is this breakable vase. And vases, once they’re broken, they can never be repaired. We assume that the AI knows this. We’re going to assume that the AI knows all empirical facts. It knows how the world works, it knows what actions the human can take, it knows what actions the human can take, it knows what actions it itself can take, it knows what states of the world are possible, but it doesn’t know anything about the \[inaudible 00:36:58\] function which is the equivalent of human values.

Rohin (37:02):
It knows empirical facts. It knows that this vase, once broken, cannot be fixed. We’re going to leave aside glue and things like that. It then looks at the fact that it sees, it is deployed in this room, and it sees that its human, who I’ll call Alice, is in the room, the vase is unbroken. Now you can pose hypothetical questions like all right, well, what would I have expected to see if Alice wanted to break the vase? Well, I would have seen a broken vase. What would I have expected if Alice didn’t care about the vase? Well, probably, at some point, the most efficient way would have been to just walk through the room while knocking over the vase. So, probably in that case, also I would have seen the broken vase.

Rohin (37:58):
What would I expect to see if Alice did not want the vase to be, or actively wanted the vase to not be broken? In that case, I actually see an unbroken vase, probably. Since I actually see an unbroken vase, that tells me that of those three situations, only the last one seems consistent with my observations. So, probably, Alice did not want to break the vase. You can infer this fact about that Alice doesn’t want to break the vase just by looking at the state of the world and seeing that the vase is not broken.

Jeremie (38:33):
It seems like there’s a very deep connection here to the second law of thermodynamics and the universe has there’s so many more ways to end up in a situation where you have a broken vase, but the fact that there isn’t a broken vase is a huge piece of information.

Rohin (38:52):
Yeah, that’s exactly right.

Jeremie (38:57):
Basically, to the extent \[crosstalk 00:38:59\]-

Rohin (38:58):
I don’t think I have anything to add.

Jeremie (38:59):
Well, it just strikes me, the physicist instinct in me, but to the extent that the world looks any different from what we would expect with pure thermodynamic randomness, the assumption here is those differences come from human preferences. Would that be a fair way to characterize the…

Rohin (39:17):
Yep, that’s right.

Jeremie (39:19):
And does that imply certain failure modes then? Because I guess we encode information in our environment, I guess this is \[inaudible 00:39:25\] revealed preferences thing, but implicitly, I’ve hard coded my brain state into my apartment, every arrangement of things, any misogyny, any racism, any foot fetishes, the whole laundry list of weird quirks that may or may not be part of my personality are implicitly encoded in the room. Is this part of the risk of applying a technique like this?

Rohin (39:54):
Yeah, so, in theory, if you \[inaudible 00:39:59\] this method, it would… Is this going to get everything that is a revealed preference? And there are, well, I don’t know that it gets everything. But to a first approximation, it gets your revealed preferences. I’m sure there is some that it does not get. Sometimes, you just don’t like your revealed preferences and you think they should be different.

Rohin (40:27):
You have a revealed preference, many people have a revealed preference, to procrastinate that they probably do not in fact endorse and they wouldn’t want their AI system giving them more and more addictive material so that they can procrastinate better which is plausibly something that could happen. I would have to think significantly harder about how exactly concretely that could happen, but I could believe that that would be an effect.

Rohin (41:03):
Similarly, the technique as I’ve explained it so far it seems that there is only one human in the world and things get a lot more complicated if there are multiple humans and I have just ignored that case so far.

Jeremie (41:20):
It’s what you need to get the thing off the ground, right?

Rohin (41:22):
Yeah.

Jeremie (41:26):
In this context, I imagine at least, there’s another risk mode which is if, by pure chance, let’s say, in the example with the vase, let’s say that the human actually doesn’t care about the vase but just happens, in her demonstration, to have avoided the vase. Is there the risk that, I guess this is always a risk in machine learning, it sounds like just a case of added distribution sampling, like you would learn-

Rohin (41:57):
Yep.

Jeremie (41:57):
Okay.

Rohin (41:58):
Yeah, that’s right. If the vase is kept in an inconspicuous out of the way location where it’s not actually that likely that Alice would have broken the vase on the course of moving around the room, we actually have this on the paper, we show that in that environment you actually don’t learn anything significant about the vase. You’re just like, “Eh, she probably didn’t want it broken.” You infer that she did not deeply desire for the vase to be broken, but you don’t infer anything stronger than that. You’re uncertain between whether it’s bad to break vases rather versus yeah, it doesn’t really matter.

Jeremie (42:44):
Interesting.

Rohin (42:45):
It’s more like if there were efficiency gains to be had by breaking the vase, then you infer and you observe that actually the vase wasn’t broken, then you infer that it’s bad to break vases. It’s still possible that humans, we’re not perfect optimal people, we might not pick up an efficiency gain and so we might go around a vase even though it would be faster to go to the vase even if we didn’t care about the vase. And, yeah, this method would make an incorrect inference. In general, in preference learning, there’s a big tension between you’re assuming that humans do the things that the humans do to reflect what they want. And not always true.

Jeremie (43:37):
Right, and sometimes just for reasons of pure stupidity as well, I guess. We may want a thing, just not know how to make it happen.

Rohin (43:45):
Yup, exactly. That is a big challenge in preference learning and people have tried to tackle it, including me, actually. But I wouldn’t say that there has been super substantial progress on separating stupidity from the things you actually wanted.

Jeremie (44:10):
I think we’d end up solving a lot of other problems instead \[crosstalk 00:44:12\] if we do that. Actually, there’s one more aspect I wanted to ask you about, with respect to the paper. The rule of time horizons, or the rule that time horizons play in the paper is, I think, just really interesting because there’s certain assumptions that the robot makes or the AI makes about what is the time horizon that the human has in mind for this action that if there’s assumptions about that time horizon shift, you start to see different behavior. I’d love to hear you expand on that and describe that setting a little bit.

Rohin (44:43):
I think the main takeaway from that I think I would have about time horizons is if you assume a short time horizon, then cases where the state hasn’t been fully optimized are much more excusable because the human just hasn’t had enough time to fully optimize the state towards what would be optimal for them and so you can-

Jeremie (45:11):
And so maybe I should fill in that gap, I realize it was a little ambiguous, but by time horizon, I guess we’re talking about the amount of time the human would have to, say, go from one point in the room to a desired end point, right? \[crosstalk 00:45:24\]

Rohin (45:23):
Yeah, it’s like the amount of time that the robot assumes the human has been acting in the environment before the robot showed up. In the room case, it’s robot is deployed and sees an intact vase and it’s like, “Ah, yes, the human has been walking around this room for an hour,” or something like that.

Jeremie (45:40):
Right, if you’ve been walking around the room for a full hour and the vase is still there, you can then assume that the vase is probably pretty important.

Rohin (45:48):
Yeah, something along those lines, exactly. The actual setting in the paper is slightly different, but that’s the right intuition. Yeah, and so this isn’t illustrated best with the vase example, but imagine you’re trying to build a house of cards. This is another example where the state of the world is really very informative about your preferences. House of cards are super, super not entropic. You can infer a lot.

Jeremie (46:21):
Yeah, the more specific the arrangement, I guess the more… Which is interesting because that’s exactly what’s so challenging about preserving humanity in general, there’s almost a philosophically conservative streak to this approach in the sense that we’re assuming that we’ve gotten to somewhere that’s worth preserving because we’ve encoded so much of ourselves, so much of what’s good about us already in the environment and it almost seems like what I love about this time horizon stuff is the political philosophy behind it, it almost gives you a dial that you can tune from the progressive to the conservative end of the spectrum just by assuming different time horizons. If you assume that we just got here and it’s sort of blank slate, then, hey, we can try anything. We’re not really certain about what humans want in this environment, conversely…

Rohin (47:12):
Yeah.

Jeremie (47:12):
Right? Yeah.

Rohin (47:13):
That’s true, I’ve never actually thought about it that way, but you’re right. That is basically what it is. Another way of thinking about it is the way I actually got to this point, to the point of writing this paper, was asking myself why is it that we privilege the do nothing action? We’ll say that the safe action is to do nothing. Why? It’s just an action. This is an answer, we’ve already optimized the environment, random actions are probably, so… The current state is high on our preference ranking. Random actions take us out of that state into some random other state, so probably an expectation to go lower in our ranking whereas the do nothing action preserves it and so it’s good. The longer the time horizon, the stronger you want to do nothing as a default.

Jeremie (48:07):
Yeah, yeah, I remember reading that in the paper, actually, as almost a derivation of that intuition which is it’s so beautiful when you can see it laid out like that.

Rohin (48:15):
I know, so good.

Jeremie (48:18):
Yeah, yeah. In a way, it makes me think of so many arguments among and between people of different political stripes could be so much easier if we applied toy models like this where you can say, well, hey, there is value to the conservative. There is value to the progressive. We end up in a dystopia either way, and here’s the parameter we can tune to see how dystopic things get one way or the other, depending on how much we value things.

Jeremie (48:43):
Yeah, anyway I love the work and I thought it was… Anyway, again, one of these ah-ha moments, for anybody who’s interested in the intersection between philosophy, moral philosophy, and then an AI, anyway, it’s just such a cool piece of work.

Rohin (48:58):
Yeah, thanks. I like it for basically the same reasons.

Jeremie (49:03):
Sweet. Well, I’m glad we have compatible \[inaudible 00:49:06\], then. Awesome. Well, I think we’ve covered a lot of the bases here, but was there anything else you wanted to talk about? One thing I do want to make sure we get to is a reference to the alignment newsletter that you put out. I think everybody should check that out, especially if you’re looking to get into the space. Rohin puts out this amazing newsletter and anyway, we’ll link to it on the blog post that will come with the podcast.

Jeremie (49:30):
Did you have any social media links or things like that that you want to share?

Rohin (49:35):
I think the alignment newsletter is the best way to get my current thinking on things. If you’re new to the space, I’d probably recommend other things. Specific ranking of mine that I like as more introductory… It’s not exactly introductory, but more timeless material, on the alignment forum there is a sequence of blog posts called the value learning sequence that I wrote. I like that as a good introduction, there are two other recommended sequences on that forum that I also recommend, that I also think are pretty great.

Rohin (50:21):
In terms of social media, I have a Twitter. It’s @RohinMShah, but mostly it just sends out the alignment newsletter links. People can also feel free to e-mail me. My e-mail’s on my website, can’t guarantee that I will send you a response because I do actually get a lot of e-mail but I think I have a fairly high response rate.

Jeremie (50:50):
Yeah, well, I can vouch for that at my end. Thanks for making the time, really appreciate it, and I’m really looking forward actually to putting this out and also good luck with DeepMind because you’re heading over there in a couple of days, really, right?

Rohin (51:04):
Yeah, I’m heading there on Monday. It’s two more business days.

Jeremie (51:09):
All right, yeah, enjoy the long weekend such as it is.

Rohin (51:13):
Cool. Thanks.

Jeremie (51:13):
Awesome, thanks so much, Rohin.

Rohin Shah - Effective altruism, AI safety, and learning human preferences from the world_s state-by Towards Data Science-video_id uHiL6GNXHvw-date 20201028.md