# Interview with AI Researchers NeurIPSorICML_bj9ne by Vael Gates

**Interview with bj9ne, on 3/24/22**

\[Note: This transcript has been less edited than other due to language barriers. The interviewee is also younger than typical interviewees.\]

**0:00:00.0 Vael:** Alright. My first question is, can you tell me about what area of AI you work on in a few sentences?

\[Discusses research in detail\]

**0:03:31.9 Vael:** Got it. Great. Yeah, so thinking about the future, so thinking about what will happen in the future, maybe you think AI is important or maybe you don\'t think AI is important, but people talk about the ability to generate an AI that is a very capable general system, so one can imagine that in 2012, we had AlexNet and the deep learning revolution, and here we are 10 years later, and we\'ve got systems like GPT-3, which have a lot of different capabilities that you wouldn\'t expect it to, like it can do some text generation and language translation and coding and math, and one might expect that if we continue pouring in all of the investment with nations competing and companies competing and algorithmic improvements or software improvements and hardware improvements, that eventually we might reach a very powerful system that could, for example, replace all current human jobs, and we could have CEO AIs, and we could have scientist AIs, so do you think this will ever happen, and if so, when?

**0:04:43.6 Interviewee:** Honestly speaking, I don\'t think this is realistic in the future, like 10 years or 20 years. So basically, I agree that AI will replace more jobs, but in my mind, this replacement, I don\'t say that\... For example, in a factory, there were 100 employees, and in the future, we may replace that 100 employees with AI, and five employees, but this five may be advanced engineers, rather than regular people. Because I have gone to some factories that manufacture something we use in our daily life, and my impression is that there were very, very few people, but the factory is very large, and mostly those people are just sitting in a room with air-conditioning. But those robotics, which is guided by AI and processing maybe those request from customers very efficiently, but they tell me that, for example, for some security issues, they need to watch those machines, so that that won\'t happen something unexpected. So also as an example, cars, and automated cars, auto drive is very, very \[inaudible\] in these years, but this is a issue that there are a lot of security issues.

**0:06:39.3 Interviewee:** So basically, just, I think the manufacturer and auto drive has a similarity, that is in many scenarios, we do want to guarantee a high level of security, but maybe for scenarios like recommendations, so making mistakes, that doesn\'t matter, and also I think one issue is very important, that is interpretability. Because for many, many models we designed or we have designed, and maybe less, there are many popular or famous papers in top conferences, but these models, many of these models lack interpretability.

**0:07:30.0 Interviewee:** So this is not a\... Sometimes it is unacceptable to deploy these models into some scenarios that requires us to \[inaudible\] something, especially in the scenarios like in hospitals, but we can\'t accept this because sometimes, an error is\... For example, when we do salaries, and some errors is not acceptable. So I think this is a very good question, so for my mind, I think this should be discussed by classifications, so for scenarios like recommendations, where errors are acceptable, then AI will replace more and more work of human, and this work was done by maybe some data scientists, because with our models become more and more intelligent and our systems become more and more efficient, also our data become more and more efficient, so the work of data scientists will be reduced by a great margin. And I expect this growth to\... I think this is sustainable.

**0:08:49.9 Vael:** Yeah\--

**0:08:50.2 Interviewee:** For example, in probably one or two decades, this will keep growing, but for some scenarios like auto drive, then on my mind, maybe this is controversial, but I don\'t think that auto drive is really something so realistic that we can expect like L4 or L5 auto drive to be realistic, to be used by us in a decade.

**0:09:20.0 Vael:** Yes\--

**0:09:20.9 Interviewee:** So in these scenarios, I think AI is a very important system for us. It can reduce our work, reduce our burden, but human is needed.

**0:09:30.0 Vael:** Yeah. Great. Cool, so my question is, do you think we\'ll ever get very, very capable AI, regardless of whether we deploy it or not? Do you ever think that we\'ll have systems that could be a CEO AI or a scientist AI, and if so, when? Like it could be like 50 years from now, it could be like 1,000 years from now, when do you think that would happen?

**0:10:00.3 Interviewee:** Oh, this is a tough question. Let\'s see. At least, based on the method we have taken, I think the way we develop AI now will not lead us to that future, but maybe the human will find some different ways to develop AI. But through my mind, I guess is to first we talk about maybe 50 years or a century, I think that\'s not very possible, but in the future, this may be a question about some knowledge about our brains. So maybe human at the moment, we are not\... I mean the investigation or research into our brain is not very clear.

**0:11:01.0 Interviewee:** So it\'s quite hard to imagine if the machine can evolve into some stages, that is the machine can be as complex, as powerful like human brains. So maybe in a century or even in two centuries, I tend not to believe that this will happen. but in the very long run, it\'s very hard to tell because I think that the way we think about something and the way the machine difference or train themselves are totally different. They work in different ways. For example, machines may require very large amount of data to find some internal principles. But with human, we are very good at generalizations, so I think for this point, if then maybe we will achieve that after 1,000 years, but I\'m not very optimistic about this. I tend not to accept this, but I can\'t deny that entirely.

**0:12:13.3 Vael:** Yeah. There are some people who argue that the current way of scaling may eventually produce AI, like very advanced AI, like OpenAI and DeepMind. No? You don\'t think so? Cool.

**0:12:29.1 Interviewee:** I don\'t think so, because based on our current trend, the development of the models comes from the development of computing power. But computing power, essentially that comes from the increasing density of changes account. So according to a research by Princeton, we know that for computer, computing power per transistor hasn\'t changed a lot during the last few decades, and also, you know the end of dinosaurs and maybe in the future, we\'ll see the end of Moore\'s Law. And I know, I have solid GPT, GPT 2022 \[inaudible\] Nvidia has powered the AI and speed up by a million x in the last decade. But I don\'t think that this development is sustainable because you see, over here I spent GPT-3 with maybe \$4 million, if I don\'t get it wrong. This scale is not good because not only we need a larger cluster but also we need more money and more time to train that model. So I remember that OpenAI has forged some arcs\[?\] in this way, but we are not able to train that again, and I have some\... There are some articles about the scaling of deep-learning models based on the parameter count of the models and they found that, I think maybe before 2022\... No, before 2020, the growth is 75X per 18 months, but after that the growth has slowed down, has greatly slowed down because, lots of issues, like the scaling of GPU or the scaling of clusters to deploy those models in very large cluster that is not narrow anything.

**0:14:53.0 Interviewee:** So I think so, this is partly\... This is the main reason why I shift my interest from data to system, because I believe that AI has a bright future, but to make that future brighter, we need to make our system run faster. For example, can we achieve the same result using less parameters or using less power so that we can\... We can\'t have these hardware resources but can we make better use of it? So I think the future of AI largely depends on system people. That is, can we improve the system for AI? I mean, when I talk about system, I actually, from the perspective \[inaudible\] AI, I talk about three things, the hardware, software system, and the platforms. In my mind, those models are just like the top layer, that is the application. I think the lower three levels are the key to the future success of AI, but indeed I think AI is\... And also \[inaudible\] so, I think the future of AI also means we need more application scenarios, like dark\[?\] intervention or something like that, and also robotics, and I think this is very promising and it can bring us a lot of fortune or something like that.

**0:16:25.8 Vael:** Yeah, great. I\'m concerned about when we eventually\... I think that we may get AI, AGI a little faster than that. And I\'m concerned that the optimization functions that we\'ll be optimizing are not going to reflect human values well because humans aren\'t able to put their values and goals and preferences perfectly into AI, and I worry that problem will get solved less fast than the problem of how we get more and more capable AI. What do you think of that?

**0:17:00.6 Interviewee:** When you talk about values, it\'s something. Yeah, so although it\'s not a technical issue, but this is really what the technicians should care about. So this is a very important issue in open drives\[?\], when the cars make you a hero\[?\], a person first\[?\]. So maybe we should find some method or some way to plug our rules, plug our values to guide the models, guide AI, but I think it\'s also an issue of interpretability because now we don\'t have\... Sometimes we have no idea why some models can work so well. For example, when\... Because I have worked on GNN, and this year GCN and its variants, it\'s very popular, but many of that, when the researcher comes up with that model, they don\'t know why this is good.

**0:18:12.5 Interviewee:** So maybe similarly it\'s quite hard to guide AI to follow the rules, so I think this is also an important issue and important obstacle for the applications of AI. For example, we cannot put some non-CV skills to recognize human faces because sometimes it may violate the law. Yeah, I think this is an important issues, but will this stop AI? I think for my mind, this may be an obstacle for AI in some scenarios, but in many scenarios, this is not a\... This will generate some issues for us to think about, but in the end I think we will deal with this. For example, some people may use federated learning to deal with privacy and there are some techniques to deal with these issues. So yeah, I think we should put more emphasis on the values and the rules or even the assets about AI so that this community will grow faster. Yes, this is an important aspect. Although I don\'t put much emphasis on this.

**0:19:44.1 Vael:** Why not?

**0:19:49.8 Interviewee:** That\'s because, I guess, because my research and my internship mainly focus on recommendation and there is not much issues about this except for privacy, and because when we got those data, we don\'t know the meaning of data. When I get a little data intention, I just send out important numbers, or maybe sometimes this is a one \[inaudible\] issue, and I don\'t know much about that, so let\'s say that these privacy issues has been\... Maybe this has been dealt with by those data scientists, not by people like us, so this is because of my research interest, but I think for those people who do\...

**0:20:34.1 Interviewee:** Yes, I have taken some courses about AI and then teachers say that they have developed some robotics to help the elderly. But let\'s say that sometimes you cannot use a camera because using a camera will generate some privacy issues, so maybe sometimes we can just use something to catch its audio rather than video or something like this, but because\... I guess that\'s most of students\... Most of my schoolmates don\'t put\... Most of my classmates haven\'t paid much attention on these issues, but this is a very\... I think this issue will become more and more important in the future if we want to generate AI to more and more scenarios, so thank you for raising these points. I will think more about it in the future.

**0:21:37.0 Vael:** Yeah. I\'m happy to send you resources. One extra thing, one other additional thing. So I think probably, I think by default the systems we train will not do what we intend them to do, and instead will do what we tell them to do, so we\'ll have trouble putting all of our preferences and goals into mathematical formulations that we can optimize over. I think this will get even harder as time goes on. I\'m not sure if this is true, but I think it might get harder as the AI is optimizing over more and more complex things, state spaces.

**0:22:13.6 Interviewee:** So you mean that because in the future we will have more and more requirements, and that\'s so\...

**0:22:22.6 Vael:** No, no, the AI will just be operating under larger state spaces, so I will be like, \"Now I want you to be a CEO AI,\" or, \"Now I want you to be a manager AI.\"

**0:22:34.3 Interviewee:** Oh, did you say that we need to encode those requirements into optimization function so that AI will operate like what we want them to do? Did I get it wrong? Oh, that\'s a quite good question. I have discussed that with my roommates. Yeah, so yes, it is the optimization, the loss function that guides the model to do something that we want, and sometimes it\'s hard to find an appropriate function, especially for newbies. Sometimes we chose a wrong loss function, and the way the model is totally unusable.

**0:23:16.8 Vael:** Yeah, and I have one more worry about that scenario, so\...

**0:23:20.5 Interviewee:** Yeah, yeah, yeah. I think this is also a very important issues, and I\'m not very optimistic about this because it\'s really hard. And lots of things, because like a CEO AI, we not only need to care about the revenue of this company, but also learn maybe the reputation, and we may also want them to abide by the laws, and maybe when there is new business and we want to inject new rules into that loss function core business. Great.

**0:24:05.6 Vael:** Yeah, and I have one more twist. Alright, so imagine that we had a CEO AI, and it takes human feedback because we\'ve decided that that\'s a good thing for it to do, and it needs to write a memo to the humans so that they make sure that its decision is okay, and the AI is trying to optimize a goal. It\'s trying to get profit and trying not to hurt people and stuff, and it notices that when it writes this\... That when it writes, sometimes humans will shut it down. And it doesn\'t want that to happen because if humans shut it down, then it can\'t achieve its goal, so it may lie on this memo or omit or not include information on this memo, such that it is more likely to be able to pursue its goal. So then we may have an AI that is trying to keep itself alive, not because that was programmed into it, but because it is an agent optimizing a goal, and anything that interfered with its goal is like not achieving the goal. What do you think of this?

**0:25:16.3 Interviewee:** Oh, very good question, but it\'s very hard. It basically mean that we need to establish enough rules so that when\... Sometimes it\'s very hard to come up with some common cases that AI may\... Yes, it is optimizing towards its goal, but there may be something that we want it to do, so maybe we need to have a mechanism so that we can switch from the AI mode, from manual mode, that we can take control of AI or take control of\... For example, the company in the last second was guided by an AI, and for the next second, we want to guide in, we want to lead the company manually, so I think if we ask \[inaudible\] enough and we may establish maybe a thorough mechanism so that we can guarantee that, it is possible to take control back, and the AI will not lie to us. And yes, theoretically, this is possible and this should be the case. But correctly speaking, I think this can be done, but it\'s quite hard to estimate the cost. The engineering cost for us to make such a AI that is complete enough, that is secure enough to help us achieve the goal. So maybe in this point, if we want to do something like recommendations, we can be very radical, we can develop or deploy radical models. But maybe in the scenarios like a CEO AI, I think we should be conservative because some internal principles of AIs are not known entirely by the humans, so I think sometimes we need to be conservative to prevent some bad things from happening.

**0:27:45.1 Vael:** Yeah. I think that the problem\... I think it\'s not just an engineering problem. I think it\'s also a research problem, where people don\'t know how to construct optimization functions such that AI will be responsive to humans and won\'t be incentivizing against humans. And there\'s a population of people who are working on this, who are worried that if we have very intelligent AIs that are optimizing against us, then humans may be wiped out, and so you really don\'t want\... You really wanna make sure the loss function is such that it isn\'t optimizing against us. Have you heard of the AI safety community or AI alignment community?

**0:28:25.9 Interviewee:** No. I don\'t know much about that, but let me say the scenarios you have mentioned, that AI may optimize against us and even wipe out the human, I have seen some films, some movies about this, and yes, this is possible if we are too careless, I think this is possible.

**0:28:48.8 Vael:** Yeah.

**0:28:49.0 Interviewee:** But at least it\'s right at\... I haven\'t paid much attention on these issues, but I think this is an important question.

**0:28:58.9 Vael:** Yeah, I personally think that advanced AI may happen in the next 50 years, and this is just from looking at some surveys of experts. I\'m very unsure about it. But if that does happen, then I think that currently we have groups like\... China and the US are going to be competing, I expect, and we\'ll have lots of different corporations competing, and maybe DeepMind and OpenAI are not competing, but maybe they\'re just going really hard for the goal. And I worry that we\'re not going to spend enough effort on safety and we\'re going to be spending much more effort on trying to make the AI do the cool things, and if the safety problems are hard, then we may end up with a very unsafe, powerful AI.

**0:29:55.2 Interviewee:** Or this then may come up with the competition on nuclear weapons.

**0:30:01.1 Vael:** Yeah.

**0:30:01.4 Interviewee:** How likely \[inaudible\] just like nuclear weapons. Yes, the power of AI in the future may be something like the power of nuclear weapons, that is quite hard to control if the real war or something\... Maybe not so severe like a war, but it is possible, so maybe I think we need some international association about the use of AI. But both in China and the US, the government has established more and more rules about, for example, privacy, security, and what you can do and what you can\'t do. So yes, if we don\'t place enough emphasis on this, this may be a question, but well, most\--

**0:30:51.3 Vael:** I think they\'re placing emphasis on issues like privacy and fairness, but they\'re not placing emphasis on trying to develop loss functions that do what humans want. I think that is a more different type of research that is not being invested in.

**0:31:08.8 Interviewee:** Yes, you\'re right. So maybe the community should do more things about this because you can\'t count on those people in the government to realize that this is really a case, they don\'t know much about this, so yes, yes, the community should, like the conference or a workshop, we should talk more about this, yes, before that is too late. Yes, I agree. Before that is too late, or everything will be a disaster.

**0:31:38.5 Vael:** Amazing. Yeah, so there is actually a community of people working on this. I don\'t actually know\... I know fewer of the people who are working on it in \[country\], although I do think there are people working on it. I\'m curious, what would cause you to work on these sort of issues if you felt like it?

**0:32:00.0 Interviewee:** You say issues that we have just mentioned?

**0:32:01.9 Vael:** Yeah, long-term. Well, like trying to make the AI aligned with a human, trying to make sure that the things that the AI does is what the humans wanted to do, long-term issues from AI, anything like that.

**0:32:16.4 Interviewee:** So I try to, although from the bottom of my heart, I think I believe that this is an important issue, frankly speaking as a student, a student researcher, or as an engineer, I don\'t have much resources about this. So I think this is the most important issue why most of my schoolmates just like improving the models and then don\'t care about if the AI may optimize against us because\... I know this sound not good, but most of student just care about, \"So if I can graduate with a PhD degree or so.\" Yeah, so maybe\... I think for me, maybe\... because I guess I will be an engineer in the future, so maybe if I have enough resources or I have enough influence in the community, I\'m willing to spend my time on this, but if I just a low-level coder and I don\'t have much power to ask my superintendent that we should place more emphasis on this, they just take\...

**0:33:35.2 Interviewee:** For example, if I intend and they just say, \"Oh, this model, the accuracy is good enough, but the speed is not, so optimize the model so that it can run fast enough as maybe the customers\' requirement.\" So yeah, this is basically the entire ecosystem, both in the academia and the industry that force the researchers and the employees in the company that they will not put much emphasis on this, and also, most of the time they just focus on short-term issues, short-term profits, or in the universities, student just care about, \"Oh, can I have some publications so that\... \" I really don\'t know any of my class schoolmates who have publication on these issues.

**0:34:32.4 Interviewee:** So I don\'t know whether there are lots of the researchers who cares about this and will spend maybe several months on these issues where they will have some publications about this. I know the top conference has asked about some ethical issues, but yeah, we really don\'t pay enough attention on this. This is a very good point. I think we need some more incentives on the future of AI. For example, like environmental issues, some factories, they will not care about environmental issues if the government doesn\'t force them to do so. For example, now we have the trade on the carbon dioxide budget. That is, the government tell the factories that you shouldn\'t emit more carbon dioxide than maybe more than this threshold or you will be fined. Maybe we need some, yes, we need some incentives to force us to think about these issues or otherwise I think this is not optimistic because not many people will be guided to do this because maybe those on the other levels they don\'t care about this.

**0:36:05.7 Vael:** Yeah, yeah. That seems right. Yep. It does seem like it\'s not currently as popular in the top journals as it could be, seems like a pretty small community right now. I will look around to see if I can find any researchers in \[country\] who are working on this sort of thing, because I know a bunch of them in \[country\] and I know some of them in \[country\], but not as many in \[country\], and I\'ll send you some resources if you\'re interested. There is a group of people who pay attention to this a lot, and they\'re called the Effective Altruism community, and right now, one of the things they care about is trying to make sure that existential risks don\'t happen so that humans are okay. Some other things they\'re worried about are pandemics, nuclear stuff, climate change, stuff like this, and also many other things. Interesting. Alright, cool. I think my last question is, have you changed your mind on anything during this interview, and how was this interview for you?

**0:37:16.9 Interviewee:** Oh. Yeah, I think maybe the greatest change to my mind is, you say that if we want a CEO AI, we need to, maybe we need to encode those requirements into the optimization function, and maybe someday an advanced AI will optimize against us. Yes, basically, in the past, I think this may be an ethical issue, and now I\'ve realized that it\'s both an ethical and a social, as well as a technical issues and we have\...

**0:37:53.4 Vael:** Yeah.

**0:37:55.3 Interviewee:** Yes. I know we haven\'t paid enough emphasis on this, but yeah, now I think that it\'s time for us to do more things, and this is a very wonderful, wonderful idea for me to think about. Thank you very much.

**0:38:14.3 Vael:** Yeah. Well, I mean, it\'s super cool that you\'re interested in this, so I\'m very enthused by you being like, \"This does seem like a problem, this does seem like a technical\... \" Yeah, I\'m very excited about that. Cool. Alright. Well, I will send you some resources then and I\'ll see if I can find anyone who is doing anything like this and send anything I find your way. But thank you so much, and feel free to reach out if you have any questions or if there\'s anything I can help you with.

**0:38:41.8 Interviewee:** Oh, okay. Also, thank you. So no, I don\'t have much questions. I will read more about this in the future. I think this is very important and also very interesting. Thank you. Thank you.

**0:38:54.8 Vael:** Yeah, I\'ll send you some resources. Alright, email you soon. Bye.

**0:39:01.3 Interviewee:** Okay.

NeurIPSorICML_bj9ne-by Vael Gates-date 20220324.md


# Jaime Sevilla on Projecting AI progress from compute trends by Jeremie Harris on the Towards Data Science Podcast

## Jaime Sevilla on timelines for transformative AI and general intelligence

There’s an idea in machine learning that most of the progress we see in AI doesn’t come from new algorithms of model architectures. instead, some argue, progress almost entirely comes from scaling up compute power, datasets and model sizes — and besides those three ingredients, nothing else really matters.

Through that lens the history of AI becomes the history f processing power and compute budgets. And if that turns out to be true, then we might be able to do a decent job of predicting AI progress by studying trends in compute power and their impact on AI development.

And that’s why I wanted to talk to Jaime Sevilla, an independent researcher and AI forecaster, and affiliate researcher at Cambridge University’s Centre for the Study of Existential Risk, where he works on technological forecasting and understanding trends in AI in particular. His work’s been cited in a lot of cool places, including Our World In Data, who used his team’s data to put together [an exposé on trends in compute](https://ourworldindata.org/grapher/ai-training-computation). Jaime joined me to talk about compute trends and AI forecasting on this episode of the TDS podcast.

Here were some of my favourite take-homes from the conversation:

-   Jaime’s work involves projecting trends in AI to estimate when we might reach a level of AI capability known as transformative AI (TAI). TAI is roughly the stage at which AI is so powerful and generally capable that it triggers a process analogous to the Industrial Revolution, whose effects on human civilization are profound and entirely unpredictable. Interestingly, it probably doesn’t matter how exactly TAI is defined, because the general consensus among researchers and forecasters is that by the time a stage like TAI is reached, technological progress will be happening so fast that even fairly major differences in capability will be separated by short periods of time. As a result, disagreements over how precisely to define TAI are likely to result in estimates that differ only by a few years, rather than decades or centuries, as was the case with the Industrial Revolution.
-   Jaime’s work identified three distinct eras in the history of AI. In the first, AI progress is driven by academic labs with relatively fixed budgets. Still, thanks to Moore’s Law, compute power was getting twice as cheap every 20 months during this time, so labs tended to double their compute usage every two years. But around 2010, things started to change: deep learning was showing real potential to solve a variety of problems in vision and natural language processing — but deep learning is highly compute-intensive. As a result, companies began throwing an accelerating amount of resources at AI development, and therefore, at compute. Notably, Jaime argues that this actually started to happen slightly before AlexNet (the canonical and widely-celebrated “kick-off” moment for deep learning). This deep learning era left us with impressive systems for tasks as diverse as vision and game-playing, but they still couldn’t quite crack language generation. As GPT-3 dramatically showed, the path to better language models would be a third scaling phase — a trend that Jaime argues began sometime in 2015/2016, half a decade before GPT-3 itself.
-   There’s quite a bit of debate about how compute investments will translate into new AI capabilities, and therefore on when we can expect to reach TAI. One method is to look at biological intelligence, and try to estimate how much computational power has gone into creating human brains. There are several ways to do that: for example, you could imagine estimating the number of computations that a human needs to carry out to reach adulthood from birth, or the compute expended by nature as it evolved human brains from inanimate matter billions of years ago. These are known as “biological anchors” for AI.
-   Jaime identifies two main drivers of compute trends. The first is essentially Moore’s Law: compute power is getting exponentially cheaper, reducing in cost by a factor of two every 20 months as humans come up with new ways to squeeze more processing power or of less matter. The second driver is corporate investment in compute: as companies recognize the value of AI, and the crucial role that processing power plays in its development, they’ve been ploughing more and more resources into compute at an accelerating rate.
-   This second driver is currently the most significant, sure to the violent expansion of corporate AI budgets in recent years, but it’s unclear if those investments can continue to grow at their current rate indefinitely. Jaime estimates that this budget growth phase may fizzle out around the end of the decade, as AI related compute becomes a prohibitively large fraction of corporate budgets. That is, unless AI-related revenues grow fast enough to keep it going, which is also a distinct possibility.

Jaime Sevilla - Projecting AI progress from compute┬átrends-by Towards Data Science-video_id 2NXagVA3yzg-date 20220413.md


# Jade Leung Why companies should be leading on AI governance - EA Forum

_Are companies better-suited than governments to solve collective action problems around artificial intelligence? Do they have the right incentives to do so in a prosocial way? In this talk, Jade Leung argues that the answer to both questions is "yes"._

_A transcript of Jade's talk is below, which CEA has lightly edited for clarity. You can also watch this talk on_ [_YouTube_](https://www.youtube.com/watch?v=AVDIQvJVhso)_, or read its transcript on_ [_effectivealtruism.org_](https://www.effectivealtruism.org/articles/ea-global-2018-why-companies-should-be-leading-on-ai-governance)_._

## The Talk

In the year 2018, you can walk into a conference room and see something like this: There's a group of people milling around. They all look kind of frantic, a little bit lost, a little bit stressed. Over here, you've got some people talking about GDPR and data. Over there, you've got people with their heads in their hands being like, "What do we do about China?" Over there, you've got people throwing shade at Zuckerberg. And that's when you know you're in an AI governance conference.

The thing with governance is that it's the kind of word that you throw around, and it feels kind of warm and fuzzy because it's the thing that will help us navigate through all of these kind of big and confusing questions. There's just a little bit of a problem with the word "governance," in that a lot of us don't really know what we mean when we say we want to be governing artificial intelligence.

So what do we actually mean when we say "governance?" I asked Google. Google gave me a lot of aesthetically pleasing, symmetrical, meaningless management consulting infographics, which wasn't very helpful.

![](https://images.ctfassets.net/ohf186sfn6di/0QF4aqkkDrIDQKTo4vMJo/fa5db8b93b528ab2f29a5bb9934e6e07/jade_slide_1.PNG)

And then I asked it, "What is AI governance?" and then all the humans became bright blue glowing humans, which didn't help. And then the cats started to appear. This is actually what comes up when you search for AI governance. And that's when I just gave up, and I was like, "I'm done. I need a career change. This is not good."

![](https://images.ctfassets.net/ohf186sfn6di/2Ivakk5wkUquB12FarVQEe/5a5ecbe6542781095b2fa6c3de9581e2/jade_slide_2.PNG)

So it seems like no one really knows what we mean when we say, "AI governance." So I'm going to spend a really quick minute laying some groundwork for what we mean, and then I'll move on into the main substantive argument, which was, "Who should actually be leading on this thing called governance?"

![](https://images.ctfassets.net/ohf186sfn6di/6TgqeZBUAbBNm8LCWXTAY6/d19b7c98fa82625e8d33442c72373e23/jade_slide_3.PNG)

So governance, global governance, is a set of norms, processes, and institutions that channel the behavior of a set of actors towards solving a collective action problem at a global or transnational scale. And you normally want your governance regime to steer you towards a set of outcomes. When we talk about AI governance, our outcome is something like the robust, safe, and beneficial development and deployment of advanced artificial intelligence systems.

![](https://images.ctfassets.net/ohf186sfn6di/4S3XPxYuB52keZpivYY1jF/ce68e1ee2a08d3159ce21a23f672e0fe/jade_slide_4.PNG)

Now that outcome needs a lot of work, because we don't actually really know what that means either. We don't really know how safe is safe. We don't really know what benefits we're talking about, and how they should be distributed. And us answering those questions and adding granularity to that outcome is going to take us a while.

So you can also put in something like a placeholder governance outcome, which is like the intermediate outcome that you want. So the process of us getting to the point of answering these questions can include things like being able to avoid preemptive locking, so that we don't have a rigid governance regime that can't adapt to new information. It could also include things like ensuring that there are enough stakeholder voices around the table so that you are getting all of your opinions in. So those are examples of intermediate governance outcomes that your regime can lead you towards.

![](https://images.ctfassets.net/ohf186sfn6di/6yurMnugW4fN34drvdAeXF/01b8c15c094c42b00b49259b744cf72a/jade_slide_5.PNG)

And then in governance you also have a set of functions. So these are the things that you want your regime to do so that you get to the set of outcomes that you want. So common sets of functions that you talk about would be things like setting rules. What do we do? How do we operate in this governance regime? Setting context, creating common information and knowledge, doing common benchmarks and measurements. You also have implementation, which is both issuing and soliciting commitments from actors to do certain things. And it's also about allocating resources so that people can actually do the things. And then finally, you've got enforcement and compliance, which is something like making sure that people are actually doing the thing that they said that they would do.

So these are examples of functions. And the governance regime is something like these norms, processes, and institutions that get you towards that outcome by doing some of these functions.

So the critical question today is something like, how do we think about who should be taking the lead on doing this thing called AI governance?

I have three propositions for you.

![](https://images.ctfassets.net/ohf186sfn6di/3aBi5cF4bmBdAMJGBP3Aek/257d867d0d6911ddb2f6e30fc7c6557e/jade_slide_6.PNG)

One: states are ill-equipped to lead in the formative stages of developing an AI governance regime. Two: private AI labs are better, if not best-placed, to lead in AI governance. And three: private AI labs can and already, to some extent, are incentivized to do this AI governance thing in a prosocial way.

I'll spend some time making a case for each one of these propositions.

### States are Ill-Equipped

When we normally think about governance, you consider states as the main actors sitting around the table. You think about something like the UN: everyone sitting under a flag, and there are state heads who are doing this governance thing.

![](https://images.ctfassets.net/ohf186sfn6di/14y9bJyzhoLTk3tTnNjE0j/961bf1667043952d4ddda5072c32c28e/jade_slide_7.PNG)

You normally think that because of three different reasons. One is the conceptual argument that states are the only legitimate political authorities that we have in this world, so they're the only ones who should be doing this governance thing. Two is you've got this kind of functional argument: states are the only ones who can pass legislation, design regulation, and if you're going to think about governance as regulation and legislation, then states have to be the ones doing that function. And three, you've got something like the incentives argument, which is that states are set up, surely, to deliver on these public goods that no one else is going to care about as a result. So states are the only ones that have the explicit mandate and the explicit incentive structure to deliver on these collective action problems. Otherwise, none of this mess would get cleaned up.

Now all of those things are true. But there are trends and certain characteristics about a technology governance problem, which means that states are particularly increasingly undermined in their ability to do governance effectively, despite all of those things being true.

![](https://images.ctfassets.net/ohf186sfn6di/2mCyYhctogvxzVWjItUS6t/e433c20b26e360acdd5514fd57104211/jade_slide_8.PNG)

Now the first is that states are no longer the sole source of governance capacity. And this is a general statement that isn't specifically about technology governance. You've got elements like globalization, for example, creating the situation where these collective action problems are at a scale which states have no mandate or control over. And so states are increasingly unable to do this governance thing effectively within the scope of the jurisdiction that they have.

You also have non-state actors emerging on the scene, most notably civil society and multi-national corporations are at this scale that supersedes states. And they also are increasingly demonstrating that they have some authority, and some control, and some capacity to exercise governance functions. Now their authority doesn't come from being voted in. The authority of a company, for example, plausibly comes from something like their market power and the influence on public opinion. And you can argue about how legitimate that authority is, but it is exercised and it does actually influence action. So states are no longer the sole source of where this governance stuff can come from.

Specifically for technology problems, you have this problem that technology moves really fast, and states don't move very fast. States use regulatory and legislative frameworks that hold technology static as a concept. And technology is anything but static: it progresses rapidly and often discontinuously, and that means that your regulatory and legislative frameworks get out of date very quickly. And so if states are going to use that as the main mechanism for governance, then they are using irrelevant mechanisms very often.

Now the third is that you have emerging technologies specifically being a challenge. Emerging technologies have huge bars of uncertainty around the way that they're going to go. And to be able to effectively govern things that are uncertain, you need to understand the nature of that uncertainty. In the case of AI, for example, you need deep in-house expertise to understand the nature of these technology trajectories. And I don't know how to say this kindly, but governments are not the most technology-literate institutions that are around, which means that they don't have the ability to grapple with that uncertainty in a nuanced way, which means you see one of two things: you either see preemptive clampdown out of fear, or you see too little too late.

So states are no longer the sole source of governance capacity. And for technology problems that move fast and are uncertain, states are particularly ill-equipped.

## Private Labs are Better Placed

Which leads me to proposition two, which is that instead of states, private AI labs are far better-placed, if not the best-placed, actors to do this governance thing, or at least form the initial stages of a governance regime.

![](https://images.ctfassets.net/ohf186sfn6di/47lFfvgscAaYFssV338Lkm/83ec08c55b572a5b944ce88d317301a0/jade_slide_9.PNG)

Now this proposition is premised on an understanding that private AI labs are the ones at the forefront of developing this technology. Major AI breakthroughs have come from private companies, privately funded nonprofits, or even academic AI labs that have very strong industrial links.

Why does that make them well-equipped to do this governance thing? Very simply, it means that they don't face the same problems that states do. They don't face this pacing problem. They have in-house expertise and access to information in real time, which means that they have the ability to garner unique insights very quickly about the way that this technology is going to go.

![](https://images.ctfassets.net/ohf186sfn6di/MRIgUjKZM1xoDLtukUlVj/a6f8834edec27bb3557bfdc392b16787/jade_slide_10.PNG)

So of all the actors, they are most likely to be able to slightly preemptively, at least, see the trajectories that are most plausible and be able to design governance mechanisms that are nuanced and adaptive to those trajectories. No other actor in this space has the ability to do that except those at the forefront of leading this technology development.

![](https://images.ctfassets.net/ohf186sfn6di/dliu1XeCIrLxZgDbWP80e/ad791977f72ac0087d93d6458bf8d02b/jade_slide_11.PNG)

Now secondly, they also don't face the scale mismatch problem. This is where you've got a massive global collective action problem, and you have states which are very nationally scaled. What we see is multinational corporations which from the get-go are forced to be designed globally because they have global supply chains, global talent pools, global markets. The technology they are developing is proliferated globally. And so, necessarily, they both have to operate at the scale of global markets, and they also have experience, and they attribute resources to navigating at multiple scales in order to make their operations work. So you see a lot of companies scale at local, national, regional, transnational levels, and they navigate those scales somewhat effortlessly, and certainly effortlessly compared to a lot of other actors in this space. And so, for that reason, they don't face the same scale mismatch problem that a lot of states have.

So you've got private companies that both have the expertise and also the scale to be able to do this governance thing.

Now you're probably sitting there thinking, "This chick has drunk some private sector Kool-Aid if she thinks that private sector, just because they have the capacity, means that they're going to do this governance thing. Both in terms of wanting to do it, but also being able to do it well, in a way that we would actually want to see it pan out."

### Private Labs are Incentivized to Lead

Which leads me to proposition three, which is that private labs are already and can be more incentivized to lead on AI governance in a way that is prosocial. And when I say "prosocial" I mean good: the way that we want it to go, generally, as an altruistic community.

Now I'm not going to stand up here and make a case for why companies are actually a lot kinder than you think they are. I don't think that. I think companies are what companies are: they're structured to be incentivized by the bottom line, and they're structured to care about profit.

![](https://images.ctfassets.net/ohf186sfn6di/75u4VpesxfLDLY3HSvdDwA/3cbf76dbeb3c4822e9fdfa347da62178/jade_slide_12.PNG)

All that you need to believe in order for my third proposition to fly is that companies optimize for their bottom line. And what I'm going to claim is that that can be synonymous with them driving towards prosocial outcomes.

Why do I think that? Firstly, it's quite evidently in a firm's self-interest to lead on shaping the governance regime that is going to govern the way that their products and their services are going to be developed and deployed, because it costs a lot if they don't.

![](https://images.ctfassets.net/ohf186sfn6di/1IGpA6dC3YtDm6Uw3S2DX7/523fd902a9d58e6e9bd42487e3954a56/jade_slide_13.PNG)

How does that cost them? Poor regulation, and when I say "poor", I mean poor in terms of costly for firms to engage with, is something where you see a lot of costs incurred to firms when that happens across a lot of technology domains. And the history of technology policy showcases a lot of examples where firms haven't been successful in preemptively engaging with regulation and preemptively engaging with the governance, and so they end up facing a lot of costs. In the U.S, and I'll point to the U.S. because the U.S. is not worst example of it, but they have a lot of poor regulation in place particularly when it comes to things like biotechnology. In biotechnology, you've got blanket bans on certain types of products, and you also have things like export controls, which have caused a lot of loss of profit for these firms. You also have a lot of examples of litigation across a number of different technology domains where firms have had to battle with regulation that has been put in place.

Now it wasn't in the firms' interests to incur those costs. And so the most cost-effective way, in hindsight, would be for these firms to engage with the governance as they were shaping regulation, shaping governance, and doing what that would be.

Now just because it's costly doesn't mean that it's going to go in a good way. What are the reasons why them preemptively engaging is likely to lead to prosocial regulation? Two reasons why. One: the rationale for a firm would be something like, "We should be doing the thing that governance will want us to do, so that they don't then go in and put in regulation that is not good for us." And if you assume that governance has that incentive structure to deliver on public goods, then firms, at the very least, will converge on the idea that they should be mitigating their externalities and delivering on prosocial outcomes in the same way that the state regulation probably would.

The more salient one in the case of AI is that public opinion actually plays a fairly large role in dictating what firms think are prosocial. You've seen a lot of examples of this in recent months where you've had Google, Amazon, and Microsoft face backlash from the public and from employees where they've developed and deployed AI technologies that grate against public values. And you've seen reactions from these firms respond to those actions as well. It's concrete because it actually affects their bottom line: they lose consumers, users, employees. And that, again, ties back to their incentive structure. And so if we can shore up the power of something like the public opinion that translates into incentive structures, then there are reasons to believe that firms will engage preemptively in shaping things that will go more in line with what public opinion would be on these issues.

So the second reason is that firms already do a lot of governance stuff. We just don't really see it, or we don't really think about it as governance. And so I'm not making a wacky case here in that business as usual currently is already that firms do some governance activity.

Now I'll give you a couple of examples, because I think when we think about governance, we maybe hone in on the idea that that's regulation. And there are a lot of other forms of governance that are private sector-led, which perform governance functions, but aren't really called "governance" by the traditional term.

So here are some examples. When you think about the function of governance of implementing some of these commitments, you can have two different ways of thinking about private sector leading on governance. One is establishing practices along the technology supply chain that govern for outcomes like safety.

Again, in biotechnology, you've got an example of this where DNA synthesis companies voluntarily self-initiated schemes for screening customer orders so that they were screening for whether customers were ordering for malicious use purposes. The state eventually caught up. And a couple of years after most DNA synthesis companies had been doing this in the U.S., it became U.S. state policy. But that was a private sector-led initiative.

Product standards are another really good example where private firms have consistently led at the start for figuring out what a good product looks like when it's on the market.

Cryptographic products, the first wave of them, is a really good example of this. You had firms like IBM and a firm called RSI Security Inc., in particular, do a lot of early-stage R&D to ensure that strong encryption protocols made it onto the market and took up a fair amount of global market share. And for the large part, that ended up becoming American standards for cryptographic products, which ended up scaling across the global markets.

So those are two examples of many examples of ways in which private firms can lead on the implementation of governance mechanisms.

![](https://images.ctfassets.net/ohf186sfn6di/Wx52C5oyqcoMO9371kRPE/12cbb7572f38f2f1b080827ef1eaebef/jade_slide_14.PNG)

The second really salient function that they play is in compliance. So making sure that companies are doing what they do. There are a lot of examples in this space of climate change, in particular where firms have either sponsored or have directly started initiatives that are about disclosing the things that they're doing to ensure that they are in line with commitments that are made on the international scale. Whether that's things like divestment, or disclosing climate risk, or carbon footprints, or various rating and standards agencies, there is a long list of ways in which the private sector is delivering on this compliance governance function voluntarily, without necessarily needing regulation or legislation.

So firms already do this governance thing. And all that we have to think of is how can they lead on that and shape it in a more preemptive way.

And the third reason to think that firms could do this voluntarily is that, at the end of the day, particularly for transformative artificial intelligence scenarios, firms rely on the world existing. They rely on markets functioning. They rely on stable sociopolitical systems. And if those don't end up being what we get because we didn't put in robust governance mechanisms, then firms have all the more reason to want us to not get to those futures. And so, for an aspirationally long-term thinking firm, this would be the kind of incentive that would lead them to want to lead preemptively on some of these things.

So these are all reasons to be hopeful, or to think at least, that firms can do and can be incentivized to lead on AI governance.

![](https://images.ctfassets.net/ohf186sfn6di/3aBi5cF4bmBdAMJGBP3Aek/257d867d0d6911ddb2f6e30fc7c6557e/jade_slide_6.PNG)

So here are the three propositions again. You've got states who are ill-equipped to lead on AI governance. You've got private AI labs who have the capacity to lead. And finally, you've got reasons to believe that private AI labs can lead in a way that is prosocial.

Now am I saying that private actors are all that is necessary and sufficient? It wouldn't be an academic talk if I didn't give you a caveat, and the caveat is that I'm not saying that. It's only that they need to lead. There are very many reasons why the private sector is not sufficient, and where their incentive structures can diverge from what prosocial outcomes are.

More than that, there are some governance functions which you actually need non-private sector actors to play. They can't pass legislation, and then you often need like a third party civil society organization to do things like monitoring compliance very well. And the list goes on of a number of things that private sector can't do on their own.

So they are insufficient, but they don't need to be sufficient. The clarion call here is for private sector to recognize that they are in a position to lead on demonstrating what governing artificial intelligence can look like if it tracks technological progress in a nuanced, adaptive, flexible way, if it happens at a global scale and scales across jurisdictions easily, and finally avoids costly conflict between states and firms, which tends to precede a lot of costly governance mechanisms that are ineffective being put in place.

So firms and private AI labs can demonstrate how you can lead on artificial intelligence governance in a way that achieves these kinds of outcomes. The argument is that others will follow. And what we can look forward to is shaping the formative stages of an AI governance regime that is private sector-led, but publicly engaged and publicly accountable.

![](https://images.ctfassets.net/ohf186sfn6di/2CMgEPWgeJ0HFSPewNQYKr/ed9612653ca8344e0bb054b32c29cd3d/jade_slide_15.PNG)

Thank you.

## Questions

_Question_: Last time you spoke at EA Global, which was just a few months ago, it was just after the Google engineers' open letter came out saying, "We don't want to sell AI to the government". Something along those lines. Since then, Google has said they won't do it. Microsoft has said they will do it. It's a little weird that rank and file engineers are sort of setting so much of this policy, and also that two of the Big Five tech companies have gone so differently so quickly. So how do you think about that?

_Jade_: Yeah. It's so unclear to me how optimistic to be about these very few data points that we have. I think also last time when we discussed it, I was pretty skeptical about how effective research communities can be and technical researchers within companies can be in terms of affecting company strategy.

I think it's not surprising that different companies are making different decisions with respect to how to engage with the government. You've historically seen this a lot where you've got some technology companies that are slightly more sensitive to the way that the public thinks about them, and so they make certain decisions. You've got other companies that go entirely under the radar, and they engage with things like defense and security contracts all the time, and it's part of their business model, and they operate in the same sector.

So I think the idea that you can have the private sector operate in one fashion, with respect to how they engage with some of these more difficult questions around safety and ethics, isn't the way it pans out. And I think the case here is that you have some companies that can plausibly care a lot about this stuff, and some companies that really just don't. And they can get away with it, is the point.

And so I think, assuming that there are going to be some leading companies and some that just kind of ride the wave if it becomes necessary is probably the way to think about it, or how I would interpret some of these events.

_Question_: So that relates directly, I think, to a question about the role of small companies. Facebook, obviously, is under a microscope, and has a pretty bright spotlight on it all the time, and they've made plenty of missteps. But they generally have a lot of the incentives that you're talking about. In contrast, Cambridge Analytica just folded when their activity came to light. How do you think about small companies in this framework?

_Jade_: Yeah. That's a really, really good point.

I think small companies are in a difficult but plausibly really influential position. As you said, I think they don't have the same lobbying power, basically. And if you characterize a firm as having power as a result of their size, and their influence on the public, and their influence on the government, then small companies, by definition, just have far less of that power.

There's this dynamic where you can point to a subset of really promising, for example, startups or up-and-coming small companies that can form some kind of critical mass that will influence larger actors who, for example, in a functional, transactional sense, would be the ones that would be acquiring them. E.g., like DeepMind had a pretty significant influence on the way that safety was perceived within Google as a result of being a very lucrative acquisition opportunity, in a very cynical framing.

And so I think there are ways in which you can get really important smaller companies using their bargaining chips with respect to larger firms to exercise their influence. I would be far more skeptical of small companies being influential on government and policy makers. I think historically it's always been large industry alliances or large big companies that get summoned to congressional hearings and get the kind of voice that they want. But I think certainly, like within the remit of private sector, I think small companies, or at least medium-size companies, can be pretty important, particularly in verticals where you don't have such dominant actors.

_Question_: There have been a lot of pretty well-publicized cases of various biases that are creeping into algorithmic systems that sort of can create essentially racist or otherwise discriminatory algorithms based on data sets that nobody really fully understood as they were feeding it into a system. That problem seems to be far from solved, far from corrected. Given that, how much confidence should we have that these companies are going to get these even more challenging macro questions right?

_Jade_: Yeah. Whoever you are in the audience, I'm not sure if you meant that these questions are not naturally incentivized to be solved within firms. Hence, why can we hope that they're going to get solved at the macro level? I'm going to assume that's what the question was.

Yeah, that's a very good observation that within... unless you have the right configuration of pressure points on a company, there are some problems which maybe haven't had the right configuration and so aren't currently being solved. So put aside the fact that maybe that's a technically challenging problem to solve, and that you may not have the data sets available, etc. And if you assume that they have the capacity to solve that problem internally but they're not solving it, why is that the case? And then why does that mean that they would solve bigger problems?

The model of private sector-led governance requires, and as I alluded to, pressure points that are public-facing that the company faces. And with the right exertion of those pressure points, and with enough of those pressure points translating into effects on their bottom line, then that would hopefully incentivize things like this problem and things like larger problems to be solved.

In this particular case, in terms of why algorithmic bias in particular hasn't faced enough pressure points, I'm not certain what the answer is to that. Although, I think you do see a fair amount more like things like civil society action and whatnot popping up around that, and a lot more explicit critique about that.

I think one comment I'll say is that it's pretty hard to define and measure when it's gone wrong. So there's a lot of debate in the academic community, for example, and the ProPublica debate comes to mind too, where you've got debates literally about what it means for this thing to have gone fairly or not. And so that points to the importance of a thing like governance where you've got to have common context, and common knowledge, and common information about your definitions simply, and your benchmarks and your metrics for what it means for a thing to be prosocial in order for then you to converge on making sure that these pressure points are exercised well.

And so I think a lot of work ahead of us is going to be something like getting more granularity around what prosocial behavior looks like, for firms to take action on that. And then if you know basically what you're aiming for, then you can start to actually converge more on the kind of pressure points that you want to exercise.

_Question_: I think that connects very directly to another question from somebody who said, basically, they agree with everything that you said, but still have a very deep concern that AI labs are not democratic institutions, they're not representative institutions. And so will their sense of what is right and wrong match the broader public's or society's?

_Jade_: I don't know, people. I don't know. It's a hard one.

There are different ways of answering this question. One is that it's consistently a trade off game in terms of figuring out how governance is going to pan out or get started in the right way. And so one version of how you can interpret my argument is something like, look, companies aren't democratic and you can't vote for the decisions that they make. But there are many other reasons why they are better. And so if you were to trade off the set of characteristics that you would want in an ideal leading governance institution, then you could plausibly choose to trade off, as I have made the case for trading off, that they are just going to move faster and design better mechanisms. And so you could plausibly be able to trade off some of the democratic elements of what you would want in an institution. That's one way of answering that question.

In terms of ways of making... yeah, in terms of ways of aligning some of these companies or AI labs: so aside from the external pressure point argument... which if I were critiquing myself on that argument, there are many ways in which pressure points don't work sometimes and it kind of relies on them caring enough about it and those pressure points actually concretizing into kind of bottom line effects that actually makes that whole argument work.

But particularly in the case of AI, there are a handful of AI labs that I think are very, very important. And then there are many, many more companies that I think are not critically important. And so the fact that you can identify a small group of AI labs makes it an easier task to both kind of identify at almost like an individual founder level where some of these common views about what good decisions are can be lobbied to.

And I think it's also the case that there are a number of AI labs... we're not entirely sure how founders think or how certain decision makers think. But there are a couple who have been very public and have gone on record about, and have been pretty consistent actually, about articulating the way that they think about some of these issues. And I think there is some hope that at least some of the most important labs are thinking in quite aligned ways.

Doesn't quite answer the question about how do you design some way of recourse if they don't go the way that you want. And that's a problem that I haven't figured out how to solve. And if you've got a solution, please come tell me.

Yeah, I think as a starting point, there's a small set of actors that you need to be able to pin down and get them to articulate what the kind of mindset is around that. And also that there are an identifiable set of people that really need to buy in, particularly to get transformative AI scenarios right.

Why companies should be leading on AI governance _ Jade Leung _ EA Global - London 2018-by Centre for Effective Altruism-video_id AVDIQvJVhso-date 20190207.md