Word Document

Table of Contents

Table of Contents 1

Interview Information 2

  Individually-selected 2

  NeurIPS-or-ICML 2

Intended Script (all interviews) 3

Post-Interview Resources Sent To Interviewees 5

  Master list of resources 5

Informal Interview Notes 10

  Thoughts from listening to myself doing these interviews 10

  Content analysis 11

Interview Information

These interviews are associated with the LessWrong Post: Transcripts of interviews with AI researchers.

(Please do not try to identify any interviewees from any remaining peripheral information.)

Individually-selected

“Five of the interviews were with researchers who were informally categorized as ‘particularly useful to talk to about their opinions about safety’ (generally more senior researchers at specific organizations).”

-   7ujun

-   zlzai

-   92iem

-   84py7

-   w5cb5

NeurIPS-or-ICML

“Six of the interviews were with researchers who had papers accepted at NeurIPS or ICML in 2021.”

-   a0nsf (this is the interview in which I most straightforwardly get through my questions)

-   q243b

-   7oalk

-   lgu5f

-   cvgig

-   bj9ne (language barriers, young)

Intended Script (all interviews)

There was a fixed set of questions that I was attempting to walk people through, across all of the interviews. It’s a sequence, so I generally didn’t move onto the next core question until I had buy-in for the previous core question. The core questions were: “do you think we’ll get AGI” (if yes, I moved on; if not I interacted with the beliefs there, sometimes for the entire interview), “[alignment problem]”, and “[instrumental incentives]”. I was reacting to the researchers’ mental models in all cases. I was trying to get to all of the core questions during the allotted time, but early disagreements often reappeared if the interviewee and I didn’t manage to reach initial agreement. I prioritized the core questions, and brought other questions up if they seemed relevant.

The questions (core questions are highlighted):

-   “What are you most excited about in AI, and what are you most worried about? (What are the biggest benefits or risks of AI?)”

-   “In at least 50 years, what does the world look like?”

-   “When do you think we’ll get AGI / capable / generalizable AI / have the cognitive capacities to have a CEO AI if we do?”

    -   Example dialogue: “All right, now I'm going to give a spiel. So, people talk about the promise of AI, which can mean many things, but one of them is getting very general capable systems, perhaps with the cognitive capabilities to replace all current human jobs so you could have a CEO AI or a scientist AI, etcetera. And I usually think about this in the frame of the 2012: we have the deep learning revolution, we've got AlexNet, GPUs. 10 years later, here we are, and we've got systems like GPT-3 which have kind of weirdly emergent capabilities. They can do some text generation and some language translation and some code and some math. And one could imagine that if we continue pouring in all the human investment that we're pouring into this like money, competition between nations, human talent, so much talent and training all the young people up, and if we continue to have algorithmic improvements at the rate we've seen and continue to have hardware improvements, so maybe we get optical computing or quantum computing, then one could imagine that eventually this scales to more of quite general systems, or maybe we hit a limit and we have to do a paradigm shift in order to get to the highly capable AI stage. Regardless of how we get there, my question is, do you think this will ever happen, and if so when?”

-   “What do you think of the argument ‘highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous’?”

    -   Example dialogue: “Alright, so these next questions are about these highly intelligent systems. So imagine we have a CEO AI, and I'm like, "Alright, CEO AI, I wish for you to maximize profit, and try not to exploit people, and don't run out of money, and try to avoid side effects." And this might be problematic, because currently we're finding it technically challenging to translate human values preferences and intentions into mathematical formulations that can be optimized by systems, and this might continue to be a problem in the future. So what do you think of the argument "Highly intelligent systems will fail to optimize exactly what their designers intended them to and this is dangerous"?

-   “What do you think about the argument: ‘highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals, and this is dangerous’?”

    -   Example dialogue: “Alright, next question is, so we have a CEO AI and it's like optimizing for whatever I told it to, and it notices that at some point some of its plans are failing and it's like, "Well, hmm, I noticed my plans are failing because I'm getting shut down. How about I make sure I don't get shut down? So if my loss function is something that needs human approval and then the humans want a one-page memo, then I can just give them a memo that doesn't have all the information, and that way I'm going to be better able to achieve my goal." So not positing that the AI has a survival function in it, but as an instrumental incentive to being an agent that is optimizing for goals that are maybe not perfectly aligned, it would develop these instrumental incentives. So what do you think of the argument, "Highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals and this is dangerous"?”

-   “Have you heard of the term “AI safety”? And if you have or have not, what does that term mean for you?”

-   “Have you heard of AI alignment?”

-   “What would motivate you to work on alignment questions?”

-   “If you could change your colleagues’ perception of AI, what attitudes/beliefs of theirs would you like to change?”

-   “What are your opinions about policy oriented around AI?”

I also had content prepared if we got to the end of the interview, based on Clarke et al. (2022), RAAPs, some of Critch’s content on pollution, and my general understanding of the space. My notes: “Scenarios here are about loss of control + correlated failures… can also think about misuse, or AI-assisted war. Also a scenario where the AI does recursive-self-improvement, and ends up actually able to kill humans via e.g. synthetic biology or nanotechnology or whatever, pollution.”

Post-Interview Resources Sent To Interviewees

I sent most interviewees resources after the interviews.

-   I usually floated the idea of sending them resources during the interview, and depending on their response, would send different amounts of resources.

-   I did not send resources if the interviewee seemed like they would be annoyed by them.

-   I only sent a couple of resources if they seemed not very open to the idea.

-   For people who were very interested, I often sent them different content that was more specific to them getting involved. These were the people who I sometimes sent the EA / Rationalist material at the end– I very rarely included EA/Rationalist-specific content in emails, only if they seemed like they’d be very receptive.

Here’s my master list of notes, which I selected from for each person based on their interests. I sometimes sent along copies of Human Compatible, the Alignment Problem, or the Precipice.

Master list of resources

Hello X,

Very nice to speak to you! As promised, some resources on AI alignment. I tried to include a bunch of stuff so you could look at whatever you found interesting. Happy to chat more about anything, and thanks again!

Introduction to the ideas:

-   The Most Important Century and specifically "Forecasting Transformative AI" by Holden Karnofsky, blog series and podcast. Most recommended for description of AI timelines

-   Introduction piece by Kelsey Piper (Vox)

-   A short interview from Prof. Stuart Russell (UC Berkeley) about his book, Human-Compatible (the other main book in the space is The Alignment Problem, by Brian Christian, which I actually like more!)

Technical work on AI alignment:

-   Some empirical work by DeepMind's Safety team about the alignment problem

-   Empirical work by an organization called Anthropic (mostly OpenAI's old Safety team) on alignment solutions

-   Podcast (and transcript) by Rohin Shah, describing the state of AI value alignment (probably want the first half or so)

-   Talk (and transcript) by Paul Christiano describing the AI alignment landscape in 2020

-   Alignment Newsletter for alignment-related work

-   A much more hands-on approach to ML safety, focused on current systems

-   Interpretability work aimed at long-term alignment: Elhage (2021), by Anthropic and Olah (2020)

-   Ah, and one last report, which outlines one small research organization's (Alignment Research Center) research direction and offers prize money for solving it: https://www.lesswrong.com/posts/QEYWkRoCn4fZxXQAY/prizes-for-elk-proposals

Introduction to large-scale, long-term risks from humanity-- including "existential risks" that would lead to the extinction of humanity:

-   The first third of this book summary, or the book The Precipice, by Toby Ord (not about AI particularly, more about long-term risks)

  Chapter 3 is on natural risks, including risks of asteroid and comet impacts, supervolcanic eruptions, and stellar explosions. Ord argues that we can appeal to the fact that we have already survived for 2,000 centuries as evidence that the total existential risk posed by these threats from nature is relatively low (less than one in 2,000 per century).

  Chapter 4 is on anthropogenic risks, including risks from nuclear war, climate change, and environmental damage. Ord estimates these risks as significantly higher, each posing about a one in 1,000 chance of existential catastrophe within the next 100 years. However, the odds are much higher that climate change will result in non-existential catastrophes, which could in turn make us more vulnerable to other existential risks.

  Chapter 5 is on future risks, including engineered pandemics and artificial intelligence. Worryingly, Ord puts the risk of engineered pandemics causing an existential catastrophe within the next 100 years at roughly one in thirty. With any luck the COVID-19 pandemic will serve as a "warning shot," making us better able to deal with future pandemics, whether engineered or not. Ord's discussion of artificial intelligence is more worrying still. The risk here stems from the possibility of developing an AI system that both exceeds every aspect of human intelligence and has goals that do not coincide with our flourishing. Drawing upon views held by many AI researchers, Ord estimates that the existential risk posed by AI over the next 100 years is an alarming one in ten.

  Chapter 6 turns to questions of quantifying particular existential risks (some of the probabilities cited above do not appear until this chapter) and of combining these into a single estimate of the total existential risk we face over the next 100 years. Ord's estimate of the latter is one in six.

-   How to Reduce Existential Risk by 80,000 Hours or "Our current list of pressing world problems" blog post

Governance:

-   AI Governance: Opportunity and Theory of Impact, by Allan Dafoe and GovAI generally

-   AI Governance: A Research Agenda, by Allan Dafoe and GovAI

-   The longtermist AI governance landscape: a basic overview if you're interested in getting involved, also more personal posts of how to get involved including Locke_USA - EA Forum

-   The case for building expertise to work on US AI policy, and how to do it by 80,000 Hours

How AI could be an existential risk:

-   AI alignment researchers disagree a weirdly high amount about how AI could constitute an existential risk, so I hardly think the question is settled. Some plausible ones people are considering (from the paper)

-   "Superintelligence"

    -   A single AI system with goals that are hostile to humanity quickly becomes sufficiently capable for complete world domination, and causes the future to contain very little of what we value, as described in “Superintelligence". (Note from Vael: Where the AI has an instrumental incentive to destroy humans and uses its planning capabilities to do so, for example via synthetic biology or nanotechnology.)

-   Part 2 of “What failure looks like”

    -   This involves multiple AIs accidentally being trained to seek influence, and then failing catastrophically once they are sufficiently capable, causing humans to become extinct or otherwise permanently lose all influence over the future. (Note from Vael: I think we might have to pair this with something like "and in loss of control, the environment then becomes uninhabitable to humans through pollution or consumption of important resources for humans to survive)

-   Part 1 of “What failure looks like”

    -   This involves AIs pursuing easy-to-measure goals, rather than the goals humans actually care about, causing us to permanently lose some influence over the future. (Note from Vael: I think we might have to pair this with something like "and in loss of control, the environment then becomes uninhabitable to humans through pollution or consumption of important resources for humans to survive)

-   War

    -   Some kind of war between humans, exacerbated by developments in AI, causes an existential catastrophe. AI is a significant risk factor in the catastrophe, such that no catastrophe would be occurred without the developments in AI. The proximate cause of the catastrophe is the deliberate actions of humans, such as the use of AI-enabled, nuclear or other weapons. See Dafoe (2018) for more detail. (Note from Vael: Though there's a recent argument that it may be unlikely for nuclear weapons to cause an extinction event, and instead it would just be catastrophically bad. One could still do it with synthetic biology though, probably, to get all of the remote people.)

-   Misuse

    -   Intentional misuse of AI by one or more actors causes an existential catastrophe (excluding cases where the catastrophe was caused by misuse in a war that would not have occurred without developments in AI). See Karnofsky (2016) for more detail.

-   Other

Off-switch game and corrigibility

-   Off-switch game and corrigibility paper, about incentives for AI to be shut down. This article from DeepMind about "specification gaming" isn't about off-switches, but also makes me feel like there's currently maybe a tradeoff in task specification, where more building more generalizability into a system will result in novel solutions but less control. Their follow-up paper where they outline a possible research to this problem makes me feel like encoding human preferences is going to be quite hard, and all of the other discussion in AI alignment, though we don't know how hard the alignment problem will be.

There's also a growing community working on AI alignment

-   The strongest academic center is probably UC Berkeley's Center for Human-Compatible AI. Mostly there are researchers distributed at different institutions e.g. Dylan Hadfield-Menell at MIT, Jaime Fisac at Princeton, David Krueger in Oxford, Sam Bowman at NYU, Alex Turner at Oregon, etc. Also, a good portion of the work is done by industry / nonprofits: Anthropic, Redwood Research, OpenAI's safety team, DeepMind's Safety team, ARC, independent researchers in various places.

-   There is money in the space! If you want to do AI alignment research, you can be funded by either Open Philanthropy (students, faculty-- one can also just email them directly instead of going through their grant programs) or LTFF or FTX-- this is somewhat competitive and you do have to show good work, but it's less competitive than a lot of sources of funding in academia.

-   If you wanted to rapidly learn more about the theoretical technical AI alignment space, walking through this curriculum is one of the best resources. A lot of the interesting theoretical stuff is happening online, at LessWrong / Alignment Forum (Introductory Content), since this field is still pretty pre-paradigmatic and people are still working through a lot of the ideas.

There's also two related communities who care about these issues, who you might find interesting:

-   Effective Altruism community, whose strong internet presence is on the EA Forum. Longtermism is a concept they care a lot about, and you can schedule a one-on-one coaching call here.

-   Rationalist community-- the best blog from this community is from Scott Alexander (first blog, second blog), and they're present on LessWrong. Amusingly, they also write fantastic fanfiction (e.g. Harry Potter and the Methods of Rationality) and I think some of their nonfiction is fantastic.

Happy to chat more about anything, and good to speak to you!

Best,

Vael

Informal Interview Notes

Thoughts from listening to myself doing these interviews

-   There’s obviously strong differences in what content gets covered, based on the interviewees’ opinions and where they’re at. I didn’t realize that another important factor in what content gets covered is the interviewee’s general attitude towards people / me. Are they generally agreeable? Do they take time to think over my statement, find examples that match what I’m saying? Do they try to speak over me? Are they curious about my opinions? How much time do they have? Separate from rapport (and my rapport differs with different interviewees), there’s a strong sense of spaciousness in some interviews, while many feel like they’re more rapid-fire exchanges of ideas. I often end up talking more in the agreeable / spacious interviews.

-   Participants differ in how much they want to talk in response to one question. I tended to not interrupt my interviewees, though I think that’s a good thing to do. (“I’m sorry, this is very interesting, but we really need to get to the next question.”) That meant that for participants who tended to deliver long answers, I had fewer chances to ask questions, which meant I often engaged less with their previous responses and tried to move them on to new topics more abruptly.

-   I make a lot of agreeable sounds, and try to rephrase what people say. People differ with how many agreeable sounds they make during my speech as well, and how much they’re looking at the camera and looking for cues.

-   I tended to adjust my talking speed to the interviewee somewhat, but usually ended up talking substantially more quickly. This made my speech harder to parse because of all the “like”s that get inserted while I’m thinking and talking at the same time. (I don’t think I realized this at the time; it’s more obvious when listening back through the interviews. I’ve removed a fair amount of the “like”s in the transcripts because it’s harder to read than hear.) Generally, I found it useful to try to insert technical vocabulary and understanding as early as possible, so researchers would explain more complicated concepts and be calibrated on my level of understanding. I did somewhat reduce speaking speed and vocabulary when speaking with interviewees whose grasp of English was obviously weaker, though in those cases I think it’s maybe not worth having the interview, since I found it quite hard to communicate across a concept gap under time and communication constraints. (These concepts are complicated, and hard enough to cover in 40m-60m even without being substantially limited by language.)

-   When I’m listening to these interviews, I’m often like: Vael, how did you completely fail to remember something that the interviewee said one paragraph up, what’s up with your working memory? And I think that’s mostly because there’s a lot to track during interviews, so my working memory gets occupied. I often found my attention on several things:

    -   Trying to take on the framework of their answer, and fit it into the framework of how I think about these issues. Some people had substantially different frames, so this took a lot of mental energy.

    -   Trying to figure out what counterpoint to respond with when I disagreed with something, so – fitting their answer into a mental category, flitting through my usual replies to that category, and then holding my usual replies in mind for when they were done, if there were multiple replies lined up.

    -   Trying to figure out whether I should reply to their answer, or move on. One factor here was whether they tended to take up a lot of talking space, so I needed to be very careful with what I used my conversational turn for. Another factor was how much agreement I had with my previous question, so that I could move on to the second. A third factor was tracking time– I spent a lot of time tracking time in the interview, and holding in mind where we were in the argument tree, and where I thought we could get to.

    -   If they’d said something that was actually surprising to me, and seemed true, rather than something I’d heard before and needed to reformulate my answer to, this often substantially derailed a lot of the above processing. I then needed to do original thinking while on a call, trying to evaluate whether something said in a different frame was true in my frames. In those cases I usually just got the interviewee to elaborate on their point, while throwing out unsophisticated, gut-level “but what if…” replies and seeing how they responded, which shifted the conversation towards more equality. I think thinking about these points afterwards (and many more things were new to me in the beginning of the interviews, compared to the end) was what made my later interviews better than my earlier interviews.

    -   Trying to build rapport / be responsive / engage with their points well / make eye contact with the camera / watch my body language / remember what was previously said and integrate it. This was mostly more of a background process.

-   Conversations are quite different if you’re both fighting for talking time than if you’re not. Be ready for both, I think? I felt the need to think and talk substantially faster the more interruptions there were in a conversation. I expected my interviewees to find the faster-paced conversations aversive, but many seemed not to and seemed to enjoy it. In conversations where the interviewee and I substantially disagreed, I actually often found faster-pace conversations more enjoyable than slower-paced conversations. This was because it felt more like an energetic dialogue in the faster conversations, and I often had kind of slow, sinking feeling that “we both know we disagree with each other but we’re being restrained on purpose” feel in the slower conversations.

-   My skill as an interviewer at this point seems quite related to how well I know the arguments, which like… I could definitely be better on that front. I do think this process is helpful for my own thinking, especially when I get stuck and ask people about points post-interview. But I do read these interviews and think: okay, but wouldn’t this have been better if I had had a different or fuller understanding? How good is my thinking? It feels hard to tell.

Content analysis

I have a lot to say about typical content in these types of interviews, but I think the above set of interviews is somewhat indicative of the spread. Hoping to have more information on these eventually once I finish sorting through more of my data.