Word Document

Interview with lgu5f, on 3/22/22

0:00:02.5 Vael: Alright, my first question is, can you tell me about what area of AI you work on in a few sentences?

0:00:08.4 Interviewee: Yeah. So I work on privacy in AI. I've been working on differential privacy, which is sort of the de facto standard for giving statistical privacy in analytics and AI. I've been working on differentially private synthetic data, so coming up with algorithms that generate synthetic versions of datasets that mimic the statistics in those datasets without revealing any information about the users and the data itself. And more broadly, I also just work on differential privacy for analytics, so it's not specifically AI, but it's still like algorithm design with privacy.

0:00:55.3 Vael: Cool, great, thanks. And my next question is, what are you most excited about in AI, and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

0:01:05.7 Interviewee: I think that AI has been most helpful in the little things. When I pull up my phone, and it's like, "Hey, this is your most used app in this location," recommender systems like that, or small tweaks that help my daily life, that's what I'm most excited about. I think I'm most worried about AI being used in applications where explainability and fairness are going to be important, or privacy. This is stuff I see red flags in. I'm worried about an insurance company putting a neural net into some important decision-making system and then not being able to analyze why it made a decision that it did, or understanding if it's being unfair.

0:02:11.6 Vael: Great, yeah, that makes sense. And then focusing on future AI, so putting on a science fiction forecasting hat, say we're 50 plus years into the future, so at least 50 years in the future, what does that future look like?

0:02:30.3 Interviewee: I'm sort of pessimistic about how advanced AI can get. I think that the trend is going to be that we're going to see smaller models that are more bespoke for the problem domain that it's trying to solve. So instead of these like GPT-3 trillion parameter-sized models, I think that we're going to start moving back towards stuff that can run more easily on the edge. That doesn't require as much energy and time to train and doesn't require as much data. I think that AI becomes more ubiquitous, but in a way that's easy for us to compute with. So it's just more AI running everywhere.

0:03:17.6 Vael: Yeah, what drives that intuition?

0:03:20.7 Interviewee: One is-- partially concern with large models consuming too much energy and, of course, climate change is one of the chief things that we should be worried about. The other thing is... I've seen some experiments that come out of GPT-3, and they're cool as toy problems, or it's cute program synthesis and stuff like that, but I don't see that really being used in production. It's one thing for a research organization to come up with those experiments and say, "Hey, we were able to, I don't know, use this to beat a goal player," but you look at the details of it, and really this gigantic model also had a tree search algorithm, that was a big benefit. I think just... keeping it bespoke, like CNNs just do so well, and I think that that's for a reason, so that's sort of the intuition I have. If we keep it tight to the problem domain, I've seen it do better. Domain expertise has helped a lot.

0:04:37.5 Vael: And then just a quick follow-up on the climate change thing, is the idea that, current systems are using too much energy and this is causing increased climate change?

0:04:50.0 Interviewee: Yeah, I think that we all just need to reduce how much energy we're using, because, unless we're sure that a lot of it is coming sustainably, we should be concerned about how much energy we're using. And training these trillion parameter models requires a lot of energy. It requires a lot of hardware. That hardware does not come for free. There's a manufacturing process that goes into that, building up the data centers that are training, that are hosting all of this, and then replacing the hard drives, replacing the GPUs when they get stale, so there's just a whole bunch of life cycle impacts from training models that I think are really coming up because we're seeing people doing these studies on blockchain because that tends to burn through GPUs and hard disks faster than training machine learning models, but it's sort of the same impact.

0:05:47.4 Vael: Interesting. Cool. Well, this next thing is more of a spiel, and it is quite close to what you've been talking about with where AI will go in the future and how big it will get. So people talk about the promise of AI and they mean many things by that, but one thing they may mean is a very general capable system, such that they'll have the cognitive capacity to replace all human jobs, all current day human jobs. So whether or not we choose to replace human jobs is a different question, but having the cognitive capacity to do that, and so I usually think about this in the frame of 2012 when we have AlexNet, deep learning revolution, and then 10 years later, here we are, and we've got the GPT-3 system. Which, like you said, have some weirdly emerging capabilities, so it can do some text generation, and some translation, and some coding, and some math and such. And we might expect that if we continue with all this human effort that's been going into this kind of mission of getting more and more general systems, and we've got nations competing, and we've got corporations competing, and we've got all these young people learning AI and like, maybe we'll see algorithmic improvements at the same pace we've seen hardware improvements, maybe we get optical or quantum. So then we might actually end up scaling the very general systems or like you said, we might not, we might have hit some sort of ceiling or require a paradigm shift or something. But my question is, regardless of how we get there, do you ever think we'll have very general AI systems like a CEO AI or a scientist AI? And if so, when?

0:07:13.3 Interviewee: Oh, that's a good question. I tend to be more pessimistic about this. It depends what you mean by AI, I guess in this sense, right? Is it sort of a decision-making system that's just doing Pareto optimal decision-making. Does it have to be trained?

0:07:52.6 Vael: Yeah. What I'm visualizing here is any sort of decision-making system, any sort of system that can do things like multi-step planning, that can do social modeling, that can model itself-- modeling other people modeling it. Have the requirements such that I can be like, "Alright, CEO AI, I want you to maximize profit plus constraints." So very capable cognitive systems. I don't require right now that they're embodied necessarily because I think robotics is a bit behind, but like cognitive capacities.

0:08:25.1 Interviewee: Got it. I think we're still a solid century behind that. Yeah, I don't know. I just feel like it's still a solid century behind because that might require a huge paradigm shift in how we're training our models. Yeah, I don't think we've even thought about modeling something with a state space that large, right?

0:08:58.9 Vael: Yeah, it's like reality.

0:09:00.7 Interviewee: Yeah, exactly. I've seen cool stuff where there's like, you hand over a floor plan or something, and then you say, “Hey, where did I leave my keys?” And this AI is able to jog through, navigate through this floor plan and then say you left it in the kitchen. But I still think that something like a CEO AI is still going to be like 100 years out. Yeah.

0:09:29.5 Vael: Interesting. Yeah, that is interesting, because it seems like what you were saying earlier, is like, I think we'll get more bespoke systems, less large scale systems. And I'm like [humorously]: "What if we have a very large-scale system, that is very..." [interviewee laughs] Yeah. How does that square?

0:09:43.8 Interviewee: Between having a bespoke system versus...

0:09:47.9 Vael: Or like how the future will develop or something. I think that that's like... maybe kind of likely that there will be at least some companies like OpenAI and DeepMind that'll just keep pushing on the general system thing. But I don't know.

0:09:58.4 Interviewee: Yeah, I feel like there's always going to be billions of dollars poured into that kind of research. But at the end of the day, I think that we'll see more impact if we make quality of life improvements that make jobs easier. Of course, there'll be a whole bunch of automation, right, but I don't know if we'll ever get to a point where we can leave decision-making entirely to AI. Yeah, because... I think also just as society, we haven't thought about what that means. I think Mercedes just recently said that Mercedes will take responsibility if one of their driverless cars hits someone, which is a huge step, right, but we haven't gone close to already deciding who has liability if an insurance company messes up, right, with their premium setting algorithms.

0:11:09.5 Vael: Yeah, when I think about the future, I'm like, I think we might get technological capacities before we get good societal understanding or regulation.

0:11:17.1 Interviewee: Likely. Likely. Because that's what's happening in fairness and in privacy. With fairness regulations, like you have in Australia and a whole bunch of places, where it just says, Okay, you don't pass in some protected attributes and your algorithm is not going to be racist, homophobic, transphobic, whatever, sexist. And it's like, Okay, cool, now that you've purged the data of all these attributes, there's no way for us to measure whether it's actually being any of those things. But it's required by policy. There's also the same thing with privacy, where it's like, Okay, if you just remove these attributes and you remove their names, we can't tell who it is, but nobody else has watched the exact five last movies I've watched on Netflix, or the last three things I've liked on Twitter or Facebook or whatever. So yeah, for sure, we'll be lagging societally, but hopefully with this timespan I have in mind of it's in a century, we'll be thinking about these things if we're on the cusp of it, and people will be raising alarm bells about it like, Hey, maybe we should start thinking about this.

0:12:34.1 Vael: Yeah, that makes sense. I know there's a substantial number of researchers who think there's some probability that we'll get it substantially sooner than that. And maybe even using the current deep learning paradigm, like GPT-whatever, 87, 7, GPT-13. I don't know. But that this system will like work. I don't actually know if that's true. It's hard to predict the future. Yeah, so my next question is sort of talking about whenever we get these very advanced systems, which again, who knows, you said like maybe 100 years, some people think earlier, some people think much later. So imagine we're in the future, and we're talking about the CEO AI, and I'm like, Okay, CEO AI, I wish for you to maximize profits-- this is where humans are like the shareholder type thing-- and try not to run out of money and try not to exploit people and try to avoid side-effects. Currently, this is technologically challenging for many reasons, but one of the reasons I think that might continue into the future is that we're currently not very good at taking human values and preferences and goals and putting them into a mathematical formulations that we could optimize over them. And I think this might be even harder to do in the future as we get more powerful systems and that are optimizing over larger-- reality. Larger optimization spaces. So what do you think of the argument, "Highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous?" [pause] So they'd be doing what we tell them to do rather than what we intended them to do.

0:14:12.2 Interviewee: ...Yeah, I think that that's most likely. Because... Well, it depends what level of self-awareness or cognitive awareness this AI has to understand what we intended versus telling them. If it figured out the state space was... If it can model what the ramifications of what it does are on itself, then ideally it would figure out what we intended for it to do, not what we told it to do it. So it's one thing if it goes off the rails and says say, actually, this is the only way to solve-- That's a very Asimov-esque dystopia, right. But hopefully, if it can model that and say like, Oh actually, you know, the humans will delete me if I do something that's exactly what they told me to do and not what they intended for me to do, then ideally we'd be past that point.

0:15:15.3 Vael: I see, yeah. Cool. Let me try to engage with that. So I'm like, Alright, where we've told the CEO AI that I want you to maximize profit, but not have side-effects, and it's like, Okay, cool. Side-effects is pretty undefined, but maybe it has a very good model of what humans want, and can infer exactly what the humans want and can model things forward, and will know that if it pollutes, then the humans will be unhappy. I do think if you have a great enough model of the humans in AI, and AI is incentivized to make sure to do exactly what the humans want, then this is not a problem.

0:16:00.6 Interviewee: Right. The risk, of course, is that it figures out all the legal loopholes, and it's like, Oh great, I've figured out exactly what I need to do to not be held morally culpable for what choices I've made and get off scot-free, which is also a huge risk.

0:16:19.2 Vael: Yeah, yeah. So I think with the current day systems, they're much dumber, and so if you're like, Alright AI, I want you to get a lot of points then maybe it will get a lot of points through some little side loop instead of winning the game. That's because we haven't specified what we wanted exactly, and it can't infer that. But you're saying as we get more advanced, we will have systems that can at least know what humans want them to do, whether or not they follow through with that is a different question.

0:16:51.7 Interviewee: Yeah. Yeah, and at some point, there's still the huge element of what the designer of the algorithm put in. There is some choice on the loss function. Is it average, is it worst case? Is it optimizing for the highest likelihood, the worst likelihood, I feel like that would also be a huge change in how it does it. So it's not going to be fully up to the algorithm itself, they're still going to be human choices in building out that algorithm that determine how it interacts with other humans.

0:17:31.1 Vael: Yeah, this is in fact the whole question, I think. What do you put in the loss function such that it will do whatever things you want to do. Yup. Yeah, okay, cool. So I have a second argument on top of that one, which I think is pretty relevant. So say we have the CEO AI, and it has pretty good models of humans, and it can model humans modelling, and it does multi-step planning. And because we figured, humans figured that we should have some safety mechanisms, so maybe we don't let the AI make any big decisions unless it passes that decision through us. And so they've asked the AI for a one-page memo on this upcoming decision. And the AI is like, Cool, well, I obviously have a lot of thinking and information that I can't put everything in a one-page memo, and so humans are expecting me to condense it. But I notice that sometimes if I include some types of information in this memo, then the humans will shut me down and that will make me less likely to be able to achieve the goal they programmed in. So why don't I just leave out some information such that I have a higher likelihood of achieving my goal and moving forward there. And so this is a story that's not like the AI has self-preservation built in it, but rather as an agent optimizing a goal, a not perfect goal, then it ends up having an instrumental incentive to stay alive and to self-preserve itself. So the argument is, what do you think of the argument, "Highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing any goals, and this is dangerous?"

0:19:08.7 Interviewee: I think we already see this with not highly intelligent systems. There was this paper from 2019 where they analyzed the preventative healthcare recommendations from some insurance company algorithm setup. And they found that it consistently would recommend people of color to go for preventative healthcare less. So they were looking through it, and they realized that the algorithm is optimizing to minimize the dollar spent by the patient, and so it figured out that, Hey, folks who are a part of majority communities go for preventative healthcare and they save dollars by getting preventative healthcare versus folks in minority communities tend to not get preventative healthcare, but most of them end up not ever getting specialist care either, so it's actually more dollars saved if we just don't give them preventative healthcare. I think that that could be pretty likely. Again, if we have this highly intelligent system that's able to model so much, then we should be kinda skeptical of what comes in the memos. But yeah, I'd say that that's possible, that it just gets stuck on one of those loops, like, Oh, this is best in...

0:20:44.7 Vael: Yeah, interesting. Yeah, so the example you gave seems to me like it's some sort of alignment problemm where we've tried to tell what we want, which is presumably like good healthcare for everyone that's sort of cheap. And instead it is doing something not what we intended it to do, like, whoops, that was like a failure. We've put in a failure in terms of the loss function that we're optimizing. Something that's close, but not quite the problem. Then yeah, I think this argument is, as we get very intelligent systems, if the loss function that we put in is not exactly what we want, then we might have it optimizing something kind of close, and then it will be incentivized to continue pursuing that original goal, and in fact, maybe optimizing against humans trying to change it. Which is interesting. So, if one buys this argument of instrumental incentives of an agent optimizing what a goal it puts in, then you have ideas like an agent that is optimizing for not being shut down. So that can involve deception and acquiring resources and influence and also improving itself, which are all things that are good things that are able to help you better achieve goals. Which feels pretty worrying to me if this is the case, because if this is one of the default ways that we're developing AI, then when we actually get advanced AI and then we'll maybe have a system that is as or more intelligent than us optimizing against humans. And I'm like, "Wow, that seems like real bad."

0:22:14.0 Interviewee: Yeah. Yeah, that's a nightmare scenario, which is Matrix-esque, right? Yeah.

0:22:21.5 Vael: Yeah. What is the bad AI in the Matrix doing? Is there a bad AI in the Matrix?

0:22:28.1 Interviewee: Yes. So the bad AI... It's sort of the same idea as the Terminator, where the Skynet of the Matrix realizes that the best way to avoid conflict in humanity is to sort of keep humanity in a simulation. I think the robots and the humans end up fighting a war, and then they just plug the humans into the matrix, keeping them quelled that way and making the broader argument that living in a simulation is nicer than the world outside. So the AI is technically also doing something good for humanity and self-sustaining, and keeping human population alive.

0:23:09.2 Vael: Interesting. Yeah. Cool, alright. So my interest, in general, is looking at long-term risks from AI. Have you heard of the-- well, okay, I'm sure you've heard about AI safety. What does AI safety mean for you, for the first question?

0:23:31.0 Interviewee: Um...

0:23:32.3 Vael: Or have you heard of AI safety?

0:23:34.8 Interviewee: No, not really.

0:23:35.7 Vael: Oh, interesting, cool!

0:23:37.5 Interviewee: So can you tell me more?

0:23:41.3 Vael: Yeah. So, AI safety means a bunch of different things to a bunch of different people, it's quite a large little field. It includes things like surveillance, autonomous weapons, fairness, privacy. Things like having self-driving cars not kill people. So anything that you could imagine that could make an AI safe. It also includes things that are more in what is currently called AI alignment. Have you heard of that term before?

0:24:11.1 Interviewee: No.

0:24:12.3 Vael: Cool. AI alignment is a little bit more long-term focused, so imagining like as we continue scaling systems, how do we make sure that the AIz continue to do what humans intend them to do, what humans want them to do, don't try to disable their off-switches just by virtue of optimizing for some goal that's not what we intended. Trying to make sure that the AIs continue to be aligned with the humans. Where one of the definitions of alignment here is building models that represent and safely optimize hard-to-specify human values. Alternatively, ensuring the AI behavior aligns with the system designer intentions. So trying to prevent scenarios like the one you brought up where the designer puts in a goal that's not quite what the humans want, but this can get much worse as systems get more and more powerful, one can imagine. Yeah, so there's a community working on these sorts of questions. Trying to figure out how we could get AI aligned. Because people are like, Well, can't you just put in an off switch, but this AI will probably be deployed on the Internet, and it can replicate itself presumably, if it's smart enough, and also it may have an incentive to disable the off switch if we do things by default. But some people are like, oh well, how do we solve that? How do we make it so that it doesn't have an incentive to turn itself off? And one group is like, Oh well, instead of having AI optimizing towards one goal singularly, in which case it has an incentive to try to stop people from achieving its goal, one thing you could do is have it build in uncertainty over the reward function, such that the AI wants to be corrected and wants to be switched off so that it has more information about how to achieve the goal. And this means that AI no longer has an incentive to want to prevent itself from being switched off, which is cool. And you can sort of like build... that's like an alignment solution you can build in. Another thing that people talk a lot about is if... we should obviously have human feedback if we're trying to get an AI to do what humans want. So how do we build in human feedback when the systems are more and more intelligent than us and dealing with really big problems that are like, should we do this nuclear fusion reactor? I don't know, the human doesn't really know. What are all the considerations? Another question they tackle is, how do you get honest reporting out of AI? How do you build a loss function such that it's incentivized to your honest reporting? And how do you get an AI in general to defer to humans kind of always while still being agentic enough to do things in the world?

0:26:39.3 Interviewee: Got it.

0:26:40.4 Vael: How does all that sound to you?

0:26:43.0 Interviewee: Very important. I feel like AI safety is something we're brushing up against the sharp edges of right now. But a lot of what AI alignment is looking into, I think, would also just apply to how we think about building these larger models. Because if that's the goal of these large experiments, I would hope that they're thinking about these right now, given their funding and the immense amount of time and effort they have that goes into it. Yeah. This is really interesting.

0:27:25.3 Vael: Yeah. Most researchers are working on capabilities, which is what I just call moving the state of the field forward. Because AI is actually really hard, turns out, and so lots of people working on that, and there's a lot of money in the space too, especially with applications. And then there's a smaller community of people who are working on AI safety, more like short-term AI safety, like there's plenty of problems with today's systems. There's an even smaller community of people who are working on long-term safety. They're kind of the group that I hang out with. And I'm worried that the group of people working on long-term safety will continue to be significantly smaller than the capabilities people, because while there's more money in the space now, because a couple of very rich people are concerned about it like Elon Musk, famously. But generally, it's hard work. It's like anticipating something in the future. Some people think it's a very far future. It's hard to speculate about what will go on, and that's also just kinda like a difficult problem, alignment. It might not be trivial.

0:28:25.5 Interviewee: Yeah, and I sort of see parallels with nuclear research that happened like 100 years back, right? Or 80 years back, around the time of the Manhattan Project, where most of the scientists in New Mexico were focused on capabilities, like, "How do we make this thing go boom." There was a small group of them who were worried about the ramifications of, "What would happen if we did drop our first nuclear bomb and stuff like that?" And of course, there's always been the same thing about space exploration. So with nuclear stuff, I think that we ended up, again, with more worries about nuclear safety than nuclear alignment. So, "What do we do to keep reactors safe?" Instead of thinking about like, "Okay, should we pursue nuclear technologies where countries cannot enrich uranium and make warheads with it? What if we went with thorium reactors instead?" And again, it's been more focused on capabilities and safety than alignment.

0:29:32.9 Vael: Yeah, I don't think we really need alignment with nuclear stuff because nuclear stuff doesn't really have intelligence per se. It's not an optimizer, is what I meant. Yeah. So you're not trying to make sure that the optimizer does whatever your goals are.

0:29:51.5 Interviewee: Yeah. It's humans at the end of the day. Yes, yes, yes, that's right.

0:29:55.2 Vael: Yeah, yeah. I also think the Manhattan Project is a very good comparison here. Or nuclear stuff in general. There's plenty of issues in today's world, there's so many issues in today's world. But the reason I focus on long-term risks is because I'm worried about existential risks. So what happens if all of humanity goes extinct. I think there's a number of things that are possibilities for existential risk. I think advanced AI is maybe one of the highest ranked ones in my head, where I'm like, "Well, if we have an AI system that is by default not completely aligned to human values and is incentivized against us, that's bad. Other ways that I think that we can have AI disasters are like AI assisted war, misuse, of course, and loss of control and correlated failures. So what if we use AI to automate food production and manufacturing production, and there's some failure partway along the line and then correlated failure? Or what if it results in some sort of pollution? Are not really sure who's in charge of the system anymore, and then we're like, "Oh, well, now, there's some sort of poison in the air," or there's like something in the water, we'd like messed something up, but we don't really know how to stop it. Like coordination failures is another thing that I think that can happen even before we do very advanced AI. So generally, I'm worried about AI.

I think nuclear stuff is also... You can kill a lot of people with nuclear stuff. Biological weapons... tou can do synthetic bio. There was that paper that came out recently that was like, "Well, what happens if you just put a negative sign on the utility function to generate cool drugs?" Then you get poisonous drugs, amazing. Yeah, that was a paper that just came out in Nature, I think maybe two weeks ago, made quite a splash. So I think if we had something that's much more deadly, is harder, takes longer to appear, spreads faster than COVID, I'm like, hm, that's... You may be able to... If people have bunkers, that's okay, and if people are very distant, maybe it's okay, but like... I don't know. It's not good. And then climate change also. I think that one will take much longer to kill humans. In terms of the time scales that I'm thinking of. We're looking at 3 degrees of warming in the next 100 years, or is it 50? I don't quite remember, I think it's like we've got a bit more time, and I kind of think that we might get technical solutions to that if we wait long enough, if we advance up the tech tree, if it takes a couple hundred years. So these are what I think of when I think of existential risks.

And the Manhattan Project is interesting, tying it back. There was some risk that people thought that if you deployed a nuclear bomb, then it would set the atmosphere on fire, and thus result in nuclear winter and kill everyone. And it was a very small percentage, people thought it won't happen, and they did it anyway, and I'm like, "Oh boy, I don't know how many... get lucky of those we have.” Because AI might also go perfectly fine. Or it might not. And I think there's a decent... In a study in 2017 by Grace et al., where they were asking researchers from NeurIPS and ICML how likely they thought it was that AI would have extremely bad consequences, and the median answer was like 5% or something, and I was like, "5%?! That's like real high, man."

0:33:03.0 Interviewee: Yeah. [chuckle] And that's sort of an indication of what percentage of ICML and NeurIPS researchers are working on like fairness and privacy. I feel would align pretty closely with that, yeah.

0:33:15.9 Vael: Oh, interesting. Yeah. Yeah, so very bad outcomes could be many different things. I think not many people are thinking about what happens if you literally kill all humans because it's a weird thing to have happened, but I think we're in a very weird time in history where 10,000 years ago everything was the same from lifetime to lifetime, but now we have things like nuclear weapons where one person or a small chain of people can kill like so many more humans than previously.

0:33:42.9 Interviewee: Yeah. And yeah, speaking of tech tree, one sort of linchpin that would really accelerate if we can get new AI, highly intelligent AI, would be if quantum computing becomes feasible. And we're actually able to run AI on quantum computers, then I think we're way closer to actually having a highly intelligent AI because that's an infinite state space right there.

0:34:22.3 Vael: Yeah. Yeah, yeah, I know very little about quantum myself, because I was like, "What are the hardware improvements that we could see?" And people are like, "Optical, maybe coming in optimistically in five years from now," and I was like, "Okay, that's soon." And then there's quantum where people are like, "That's significantly further away," and I'm like, "Okay." And then software improvements also. So there's just been a lot of progress in AI recently, and we're training all the young people, and there's China-US competition, and there's just a ton of investment right now, where I'm like, "We're moving fast." So interesting.

0:34:56.4 Interviewee: Yep, yep, yep.

0:34:57.8 Vael: Yeah. Well, you are very agreeable to my point of view here. [laugh]

0:35:06.3 Interviewee: Yeah, I'm just pessimistic, and I've watched too much sci-fi dystopia, I guess. One of the downsides of... It's great to democratize AI, but if your systems that say like, "Hey, just upload your dataset, and we'll give you good AI at the end of it," if it's not at least asking you questions about like, "What sort of fairness do you want? What should the outcome be?" Most humans themselves are going to be thinking like, "Oh, give me optimal, minimum cost or something like that." Humans already don't factor human values in necessarily when deciding what to do. So I'm just pessimistic about how well we can do it. Like 50 years ago, people thought that we'd be in space colonies by now, but sadly, we're not.

0:36:10.5 Vael: Yeah, very hard to predict the future. Okay, so my second to last question is: what would convince you or motivate you to work on the safety type of areas?

0:36:25.8 Interviewee: If I saw that this was coming much sooner. I think the fact that I'm seeing this as like 100 years out, a lot of smarter than me researchers will hopefully be concerned enough about it.

0:36:44.3 Vael: Yeah. How soon would be soon enough that you're like, "Oh, that's kind of soon?"

0:36:47.7 Interviewee: Fifty.

0:36:49.7 Vael: Fifty?

0:36:50.4 Interviewee: Yeah. If it's fifty, then that's in my lifetime. And I'd need to start worrying about it. That would be a bigger... The existential risk of that would be much higher than the risk I see of just AI safety in day-to-day life, so that's sort of how I'm weighing it. So for me, it's like: 100, it's like AI safety matters more. Yeah.

0:37:15.0 Vael: Yep, that seems very reasonable to me. I'm like, "Yep, it seems right." Cool. And then my last question is, have you changed your mind on anything during this interview, and how was this interview for you?

0:37:26.9 Interviewee: Have I changed my mind? I'm definitely wondering if I've overestimated when highly intelligent AI could come through. (Vael: "I can send you some resources so you can make your own opinion, etcetera.) Yeah. But... Otherwise, I don't think I've changed my opinion. I still feel pessimistic, and I hope that we start moving towards smaller AI that solves one problem really well, and we don't just think that like, "Hey, it's a perceptron. It figured out that this was the digit 9, and hence it can figure out a whole bunch of these other things." I hope that we don't start barreling down that track for too much longer. Yeah.

0:38:21.5 Vael: And when you say pessimistic, you mean like pessimistic about society and not pessimistic about the pace of AI? Or like societal impacts?

0:38:28.3 Interviewee: About both. I think we tend to put stuff out without really, really considering the consequences of it. But also I think AI has done a bunch, but it requires a lot of energy and a lot of funding that I'm not sure necessarily is going to stay up unless we start seeing a lot bigger breakthroughs come through.

0:38:57.5 Vael: Interesting. Yeah, I kind of think that the applications will keep it afloat for quite a while, and also it sounds like we might be entering a race, but I don't know.

0:39:05.5 Interviewee: True, yeah. Yeah, maybe that's what has changed in my mind through this interview, is like, "Okay, this is probably...Things are just going to keep going bigger."

0:39:18.9 Vael: I think it's quite plausible.

0:39:22.2 Interviewee: Yeah, otherwise, interview was great. This was really interesting. For example, the stuff you brought up on... I would love it if you could send me papers on the-- (Vael: "I would love to.") --the uncertainty... You were talking like, programming a way to get out the AI turning off its own off-switch. I would love to read more of these alignment papers, they sound really cool.

0:39:51.2 Vael: Awesome! Well, I'm excited. I will send you a whole bunch of resources probably, and I advise you-- it might be overwhelming so pick only the stuff that is good, and I'll bold some interesting stuff.

0:40:00.9 Interviewee: That would be super helpful. And hope you don't mind my sending following up questions.

0:40:05.7 Vael: No, no. Again, lovely. Very happy to do that.

0:40:09.4 Interviewee: Yeah. Awesome. Yeah, thank you so much. This was, like I said, a really interesting experience. I've never had to think about longer term impacts. Most of my stuff is like, "Okay, GDPR is out. What does this mean for AI?" And that's a very immediate concern. It's not this like, Okay, where should... Or even thinking like, “Okay, five years out, what do we want privacy legislation to look like?" That's something I think about, but not, "Oh my God, there's a decision-making AI out there. Does it care about my privacy?" So, yeah.

0:40:47.0 Vael: Yeah, yeah, I think people aren't really incentivized to think about the long-term future.

0:40:51.8 Interviewee: Yeah. Humans are just bad at that, right? Yeah.

0:40:53.5 Vael: And it's hard to forecast, so it makes sense.

[closings]


Word Document

Interview with q243b, on 3/18/22

0:00:00.0 Vael: Cool. Alright. So my first question is, can you tell me about what area of AI you work on in a few sentences?

0:00:09.0 Interviewee: Of course. I did my PhD in optimization and specifically in non-convex optimization. And after that I switched topics quite a lot. I worked in search at [company] [...] And now actually I work in [hard to parse] research, so I kind of come back to optimization, but it's more like a kind of weird angles of optimizations such as like meta-learning or future learning, kind of more novel trends like that.

0:00:42.1 Vael: Cool. And next question is: what are you most excited about in AI and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

0:00:51.1 Interviewee: Well, the benefit is that AI has slowly but surely kind of taken over a lot of problems in the world. And in fact pretty much any problem where you need to optimize something, you can use AI for that. I'm more of a traditional machine learning person, in the sense that... Currently I include everything, including not necessarily neural networks, also like logistic regressions, decision trees, all those things. And I think those things have been grossly unutilized, I would say, because a lot of problems right now in machine learning that people think, "Okay, should this solve this neural networks?" but in reality, just in decision trees. But stay the same because of the current trends and because of the current hype of neural networks as well, they kind of came along with it as well. All the publicity and marketing that, you know, that AI should have, honestly. And I think more and more companies realize that you can solve the problem with just simple solutions. And I think that will be a really exciting part. So just like, to answering to your question yeah, I'm excited about the fact that machine learning has just become more and more and more ubiquitous. It becomes like almost prerequisites for any big company or even a smaller company. And the second part, what I'm worried the most. I don't know, that's a good question.

0:02:07.7 Interviewee: I mean, I guess like, I mean, I don't share those worries that AI would dominate us and we would all be exterminated by overpowerful AI. I don't think AGI is coming anytime soon. I think we're still doing statistics in a way, like some kind of belong to this camp will just think, we're still doing linear models and I don't believe the system is conscious or anything of those sorts. I think the most malicious use, like, I mean, especially now currently with the war, I see more and more people using AI for malicious sense. Like not necessarily they will be, you know, we're going to have next SkyNet coming, but in the bad hands, in the bad actors, you know, AI can serve not a good purpose in war. Like for example, you know, like now with drones, with, you know, the current war, for example, in Ukraine is more and more done with drones. And drones have automatic targeting, automatic navigation. And yeah, so that's kind of not necessarily a good thing and they can become more and more dramatic, and more automatized and they can lead to harm.

0:03:06.4 Vael: Yeah, it makes sense. Lot of strong misuse cases. So focusing on future AI, putting on like a science fiction, forecasting hat, say we're 50 plus years into the future, so at least 50 years in the future, what does that future look like?

0:03:19.5 Interviewee: Well, I hope... I mean I kind of, you know, I really like that, my favorite quote in a... And probably that one of the quotes that I really like is that the arc of justice is nonlinear, but it kind of bends towards justice. I really like it. And I really hope in 50 years we would actually figure out first of all, the way to harness our current problems and make not necessarily disappear, but at least make it controllable such as nuclear weapons and the global warming. And that's in 50 years, I think it's a reasonable time. Again, not to solve them, but just to figure out how to harness these issues. And AI should definitely help that. And do you want me to answer more specifically, like more, like give you ideas of how I think in 50 years the world would look like? (Vael: "Yeah, I'll take some more specifics.")

0:04:01.3 Interviewee: Alright. I mean, I think that one of my exciting areas that I think that right now is already kind of flourishing a little bit, and it's large language models. It's a current trend, so it's kind of an easy, like on the surface thing to talk about, and I think I... As a [large tech company] employee, I can see how they've been developed over like two years, things changing dramatically. And I think these kind of things are pretty exciting. Like having a system that can talk to you, understand you, respond not necessarily with just one phrase, but like accomplish tasks that you wanted to accomplish. Like now it's currently in language scenarios, but I also, within 50 years definitely anticipate it could happen in like robotics, like a personal assistant next to you or something like that. Another area I'm really excited about is medicine.

0:04:46.1 Interviewee: I think once we figure out all the privacy issues that surround medicine right now, and we're able to create like, clean up database, so to speak, of patients diagnoses. And I hope that it'll be enough for a machine learning model to solve like cancer as we know it and things like that. I'm just hopeful. I mean, I hope it's going to happen in 50 years and it's going to, I don't know if I want to place my bet there, but I'm hoping that would happen. So I guess in robotics, as well, as I said, one of the things that we're kind of inching there, but not quite there, but I think in 50 years, we'll solve it. So I think these three things: personal assistant, solving medicine and robotics, these three things.

0:05:28.2 Vael: Wow. Yeah. I mean, solving robotics would be huge. Like what's an example of... could you do anything that a human could do as a robot or like less capable than that?

0:05:36.0 Interviewee: I think so. I mean, it depends. It depends what you mean by human, right? I mean the... Well, if you try to drive a car for the last 20 years. We've been trying to do that, but honestly, I think this problem is really, really hard because you have to interact with other agents as well. That's kind the main thing, right? You have to interact with other humans mostly. I mean, I think interaction between robots, it's one thing, interaction between robots and robots is much easier. So I think whatever task that doesn't involve humans is actually going to be pretty useful. Well, again, actually pretty easy because it hasn't been solved yet, but I think it's much easier than solving with humans.

0:06:05.8 Interviewee: And like for example organizing your kitchen, organizing your room, cleaning your room, cooking for you, I think all the things should be pretty straightforward. Again, the main issue here is that every kitchen is different, so although we can train a robot to a particular kitchen or like do some particular kitchen, like once it's presented with a novel kitchen with novel mess... mess is very personal. So it'd be harder for the robots to do. But I think that's something that would be kinda within the reach, I think.

0:06:37.7 Vael: Interesting, and for like, solving cancer, for example, I imagine that's going to involve a fair amount of research per se, so do we have AIs doing research?

0:06:47.2 Interviewee: So research, I want to distinguish research here because there is research in machine learning and there is research in medicine. And they are two different things. The research in medicine, and I'm not a doctor at all, but from what I understand, it's very different, in the sense that you research particular forms of cancer, very empirical research. Like hey we have... Cancer from what I understand, one of the main issues with cancer is that every cancer is more or less unique.

0:07:11.8 Interviewee: It's really hard to categorize it, it is really hard to do AB testing. The main research tool that medical professionals use is AB testing, right, you have this particular group of cancers, group of people, that suffer from this particular cancer. Okay, let's just come up with a drug that you can try to put these people on trial, and do that. But because every cancer is unique, it's pretty hard to do that. So, and how to do this research is data, and that's what we need for machine learning, we need to have sufficient data such that machine learning can leverage that and utilize it. So they're now asking questions in two perspectives, one is do we need more data? Yes, absolutely.

0:07:46.3 Interviewee: Moreover we not only need data, we need to have rules for which these machine learning agents, that of a company, university, would have access to this data differentially private right in the sense that this should be available to them. But is possibly private, of course privacy is a big issue. Which right now doesn't really happen, plus there are other bureaucratic reasons for this not to happen, like for example hospitals withholding the data because they don't want to share it and stuff like that.

0:08:16.3 Interviewee: So if we can solve this problem, the research and medical part would be not necessarily... Not necessary for the machine learning. And on the machine learning side, there is also, as well, it's very big hurdles in the sense that current machine learning algorithms needs tons of data. Like for the self-driving cars, they're still talking about we need millions and millions of hours cars driving on the road, and they still don't have enough. So for cancer, that's kind of not be the case hopefully. Right? So hopefully we're going to come up with algorithms that work with fewer data. Like one of the algorithms is so called few-shot algorithms, so when you have algorithms that learn on somebody's language, but when you want to apply to a particular patient in mind, you just need to use specific markers to adjust your model to the specific patient. So there are some advancement in this way too but I think we are not there yet.

0:09:07.1 Vael: Interesting. Cool, alright, so that's more... It's not like the AI is doing research itself, it's more that it is, like you're feeding in the data, to the similar types of algorithms that already exist. Cool, that makes sense. Alright so, I don't know if you're going to like this, but, people talk about the promise of AI, by which thy mean many things but one of the things is that... The frame that I'm using right now is like having a very general capable system, such that they have the cognitive capacities to replace all current day human jobs. So whether or not we choose to replace human jobs is a different question. But I usually think of this as in the frame of like, we have 2012, we've the neural net... we have AlexNet, the deep learning revolution, 10 years later with GPT-3 which has some weirdly emergent capabilities, so it can do text generation, language translations, some coding, some math.

0:09:51.9 Vael: And so one might expect that if we continued pouring all the human effort that has been going into this. And nations competing and companies competing and like a lot of talent going into this and like young people learning all this stuff. Then we have software improvements and hardware improvements, and if we get optical and quantum at the rate we've seen that we might actually reach some sort of like very general system. Alternatively we might hit some ceiling and then we'd need do a paradigm shift. But my general question is, regardless of how we get there, do you think we will ever get to a very general AI system like a CEO or a scientist AI? And if so, when?

0:10:23.6 Interviewee: So, my view on that is that it's really hard to extrapolate towards the future. Like my favorite example is I guess Elon Musk... I heard it first from Elon Musk but it's a very known thing. Is that, "Hey we had like Pong 40 years ago, it was the best game that ever created, which was Pong, it was just like pixels moving and now we have a realistic thing and VR around the corner, so of course in 100 years we will have like a super realistic simulation of everything, right? And of course in a 1,000 years we'll have everything, [hard to parse] everything."

0:10:53.0 Interviewee: But again it doesn't work this way. Because the research project is not linear, the research progress is not linear. Like 100 years ago Einstein developed this theory of everything, right, of how reality works. And then yet, we hit a very big road block of how exactly it works with respect to the different scale, like micro, macro and micro and we're still not there, we propose different theories but it's really hard. And I think that science actually works this way pretty much all around the history it's been like that, right. You have really fast advancement and then slowing down. And in some way you have advancement in different things.

0:11:26.8 Interviewee: Plus the cool thing about research is that sometimes you hit a road block that you can't anticipate. Not only there are road blocks that you maybe don't even imagine there are, but you don't even know what they could be in the future. And that's the cool part about science. And honestly, again, I think if we are indeed developing AGI soon, I think it's actually a bad sign. Honestly, I think it's a bad sign because it means that it's... It's like too easy, then I'll be really scared: okay what's next? Because if we developed some really super powerful algorithm that can essentially super cognitive... Better and better cognition of humans, I think that will be scary because then I don't even know. First of all, my imagination doesn't go further than that, because exactly by definition it will be smarter than me, so I don't even know how to do that. But also I think it means that my understanding of science is wrong. Another example I like is that someone said, if you're looking for aliens in the universe right now and then this person says, if we actually do discover aliens right now, it's actually a very bad sign. It's a bad sign in the sense that...

0:12:29.7 Interviewee: If they're there, it means that the great filter, that whatever concept of great filter, right, that we're kind of in front of it, not some behind us, it is in front of us. Just means there's some really big disaster coming up, because it actually, if aliens made it as well, this mean that they also pass all the filters behind us, it mean that some bigger filters in front of us. So I kind of belong to that camp. Like I'm... I'm kinda hoping that the science will slow down. And we'll not be able to get there. Or there's going to be something... It's not that I think that human mind is unique and we can't reproduce it. I just think that it's not as easy as we think it would be, or like in our lifetime at least.

0:13:05.0 Vael: I see. So maybe not in our lifetime. Do you think it'll happen in like childrens' lifetime?

0:13:10.1 Interviewee: Which children? Our children hopefully not. But I mean, at some point I think so. But again, I think it'll be very different form. Humans' intellect is very, very unique, I think, and because it's shaped by evolution, shaped by specific things, specific rules. So I also kind of believe in this, in the theory that in a way computers already... They are better than us because they are faster, to start with, and then they can... Another example I really like is that if you remember the AlphaGo playing Go with Lee Sedol, like one of the best two players of Go. And there was a...

0:13:43.0 Interviewee: If you remember the Netflix show, there was like in one room they actually have all the journalists and they were sitting next to Lee Sedol playing with the person that represents DeepMind. And then all the DeepMind engineers and scientists, they were in a different room. And in that room, when they were watching the game playing, [in] that room the computer said by the move number 30, very early in the game, it says, okay, I won. And it took Lee Sedol another like half an hour or more, another like a hundred moves to confirm that statistic. And they were... The DeepMind guys were celebrating and these guys were like all thinking about the game, how to... But the game was already lost.

0:14:16.6 Interviewee: So computers are already bett--... I mean, of course it's a very constrained sandbox around the Go game. I think it's true for many things, computers are already better than us. We are more general in our sense of generality, I guess. So maybe they will go in different direction... But the world is really multidimensional and the problems that we solve are very multi-dimensional. So I think it's too simplistic to say that, then you're universally better than us, or we are clearly subset and they are superset of our cognition. It's, I don't know... I think it [hard to parse].

0:14:44.9 Vael: Great. Yeah. I'm going to just hark back to the original point, which was when do you think that we'll have systems that are able to like be a CEO or a scientist AI?

0:14:55.0 Interviewee: Okay. Yeah, sure. Again, sorry-- sorry for not giving you a simple answer. Maybe that's what you're looking for, but let me know if this is... (Vael: "Nah, it's fine,")

0:15:06.5 Interviewee: Yeah. I don't know. In a way like... The work that like accountant does right now, it's very different than what accountant did 30 years ago. Did we replace it? No, we didn't. We augmented work. We changed the work of accountant so that the work is now simpler. So replacing completely accountant, in a way, yes, we also... Because the current, the set of tasks that accountant did 30 years ago, it's automated already. Do we still need accountants? Yes. So same here. Maybe the job that CEO is doing right now in 30 or 40 years, everything that right now, as of today, CEO is doing in 40, 30 years, we will still... The computer will do it. Would we still need the human there? Yes. If this answers your question.

0:15:45.1 Vael: Will we need the humans? I can imagine that we can have like, eventually AI might be good enough that we could have it do all of our science and then, or it's just so much smarter than us then we're just like, well, you're much faster at doing science. So I'll just let you do science.

0:15:57.8 Interviewee: So let me rephrase your question a bit. So what you're saying right now is it is a black box right now, that right now, that's CEO right now, that's a CEO job. That's what CEO is doing. There's some input, then some output. So what you're saying that now we can automate it. And now the input and output will also feed through something to computer let's say, but then what is, what would be the... We'll have to refine the input and output then, because it still should serve humans. Right?

0:16:21.9 Interviewee: So previously you need to have drivers, like for the tram you're having to have a driver. Now instead of drivers, you have computers, but you still need to have a person to supervise the system in a way. Or, but then you're talking about even that being automated. But in same time, you cannot... Like the system, like for example, self-driving car, it's become a tool for someone else. So you're removing the work of a driver, but you replace it with a system that now it's called something else. Like Uber. Previously, you had to call a taxi. Now you have an app to do it for you.

0:16:49.7 Interviewee: There's an algorithm that does it for you. So the system morph into something else. So same thing here. I think as CEOs, in a sense, they might be replaced, but the system also would change as well. So it won't be the same. It won't be like, okay, we'll have a Google. And there is a CEO of Google who is like robot. Now the Google will morph in a way that the task the CEO is doing would be given the computer. But the Google will still... Like also, by the way, Google, even Google. Google works on its own. In fact, if you, right now, fire all the employees, it'll still work a few days. Everything that we do, we do for the future. Like it's pretty unique moment in history, right? 'Cause previously, like before the industrial revolution, you had to do things yourself. Then with the factories and factories well then, okay, you're helping factory to do its work. And now there is a third wave, whatever, fourth wave, industrial revolution. We don't even do anything. It's on... In a way Google doesn't have a CEO, the Google CEO doesn't work for the today Google. Google CEO works for the Google in a year which is... So Google work... Google is already that. Google doesn't have a CEO. So that's what I mean.

0:17:56.8 Vael: Alright. Uh, I'm going to do the science example, because that feels easier to me, but like, like, okay. So we're like doing... We're attempting to have our AI solve cancer and normally humans would do the science. And be like, okay, cool I'm going to like do this experiment, and that experiment, and that experiment. And then at some point, like we'll have an AI assistant and at some point we'll just be like, alright, AI solve cancer. And then it will just be able to do that on its own. And it's still like serving human interests or something, but it is like kind of automated in its own way. Okay. So do you think that will... When do you think that will happen?

0:18:34.0 Interviewee: The question is how with... The question is, can this task be, sorry I know you're asking about the timeline and I want to be, I know... I don't want to ramble too much. But I think I want to be specific enough, what kind of problem we're talking about. If we're talking about the engineering problem, that we're talking about the timeframe of our lifetime. If you're talking about a problem that involves more creativity, like for example, come up with a new vaccine for the new coronavirus? Sorry, it's automatic. I think that work we could do in the 20... 20-30 years. Right, because we have tools, we know engineering tools, what needs to be done, where you can do ABC, you're going to get D. Once you have D you need to pass it through some tests and you're going to get E and that pretty much automate. I think this we can do in 20-30 years. Solving cancer, I just don't know enough. How much creativity needs to be there. So more harder, probably, yeah. Yeah, yeah.

0:19:21.0 Vael: Yeah, no, no, that's great. And yeah, and you don't know when we'll... For... So probably more than our lifetime, or more than 30 years at least for creating those?

0:19:28.9 Interviewee: Mm-hmm. Mm-hmm.

0:19:30.1 Vael: Alright, great. Cool. Alright, I'm moving on to my next set of questions. So imagine we have a CEO AI. This is... I'm still going back to the CEO AI even though... (Interviewee: "Sure, of course.")

0:19:40.8 Vael: And I'm like, "Okay CEO AI, I want you to maximize profits and try not run out of of money or exploit people or... try to avoid side effects." And this currently is very technically challenging for a bunch of reasons, and we couldn't do it. But I think one of the reasons why it's particularly hard is we don't actually know how to put human values into AI systems very well, into the mathematical formulations that we can do. And I think this will continue to be a problem. And maybe, and... I think it seems like it would get worse in the future as like the AI is optimizing over, kind of more reality, or a larger space. And then we're trying to instill what we want into it. And so what do you think of the argument: highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous.

0:20:28.3 Interviewee: Okay, so this is a quite big misconception of the public is that, and I think actually, it's rightfully so, because I am actually working on this, is that... So what you said right now is like the famous paperclip example, right? Which is going to turn the whole world in particular, and that's kind of what you said more or less, right? So the problem here is that current system, a lot of them, it's true, they work on a very specific one dimensional object. There's a loss function that we're trying to minimize. And it's true, like GPT-3 and all these system, currently they are there, they have only one number, it want to minimize. And this is, if you think about it, is way too simplistic for this very reason, right? Exactly, because if you want to just maximize number of paperclips, you're just going to turn the whole word into the paperclip machine factory. And that's the problem. But the reality is much more complicated. And in fact, we are moving there, like my research was, we're moving away from that. We're trying to understand, we're trying to understand if first of all intelligence can be emerge on its own without it being minimized explicitly.

0:21:20.2 Interviewee: And second of all, these pitfalls, the pitfalls that you're just minimizing one number and of course, not going to work. So answering your question, yeah, yeah I think it will fail, because it will face the real world scenario unless it has specific checks and balances. For example, there's also online learning paradigm, right, where you basically learn from every example as they come in the time series. I think this system need to be revamped to work in a larger scale obviously, but this is the kind of system that potentially could work, where you don't just minimize, maximize one objective, but you have a set of objectives or you have a set of actors with their own goals and their intelligence is emerging from them. Or you learn online. You just fail and you learn and you forget what you learned, you have... learn in a continuous fashion. So like all of these things that we as a humans do, could be applicable for AI as well. Like we are the humans, we don't have... You don't spend your day like, go to sleep, okay, today, day was like 26. You don't do that. And even if you do that, you probably will have multiple numbers. This was 26, this was 37. Okay, it doesn't matter the day.

0:22:19.3 Vael: Yeah, that makes sense. So, one thing I'm kind of curious about is there's... like, when you say, it won't work, just optimizing one number. And will it not work in the sense that, we're trying to advance capabilities, we're working on applications, Oh, turns out we can't actually do this because we can't put the values in. And so it's just going to fail, and then it will get handled as part of the economic process of progress. Or do you think we'll go along pretty well, we'll go along fine, things seem fine, things seems like mostly fine. And then like if things just diverge, kind of? Kind of like how we have recommender systems that are kind of addictive even though we didn't really mean to, but people weren't really trying to make it fulfill human values or something. And then we'd just have the sort of drift that wouldn't get corrected unless we had people explicitly working on safety? Yeah, so, yeah, what do you think?

0:23:06.4 Interviewee: I think people who work on safety, and you could see it yourself, people who work on safety, people who work on the fairness, people who work on all the things, checks and balances, so to speak, right? They are becoming more and more prominent. Do I know it's enough? No, it's not enough, obviously we need to do more for different reasons. For obviously DEI, but also for just privacy and safety and other things. And also for things we just talked about, right? Just because it's true that the fact that... By the way, the fact that we have only the current system like GPT-3 or many other algorithms minimize only one value, it's not a feature, it's a bug. It's just convenience because we use the machine learning optimization algorithms that work in this way, and we just don't have other ways to do that. And I hope in the future other things would come up.

0:23:46.0 Interviewee: In answering your question, I don't think it will necessarily diverge, we'll just hit a roadblock and we're already hitting them. You've heard about AlexNet like 10 years ago, and now, sure, we have acute applications and like filters on the phone, but like, did AI actually enter your life, daily life? Well, not true, I mean, you have better phone, they're more capable, but actually AI, like in terms of that we all dream about, does it enter your life? Well, not really. We can live without it, right? So, we're already hitting all these roadblocks, even like medical application. Google 10 years ago, claimed they'd solved like the skin cancer when they can detect it, and it didn't... It didn't really see the light of day except for some hospitals in India, unfortunately. So we're already hitting tons of roadblocks, and I don't think it's... It's like for this reason precisely, because when you face reality, you just don't work as good as you expect for multiple reasons.

0:24:30.4 Vael: Interesting. Cool. So do you think that others... This... Have you heard of the alignment problem? Question one.

0:24:38.1 Interviewee: No.

0:24:40.7 Vael: No. Cool, all right, so you've definitely heard of AI safety, right?

0:24:43.5 Interviewee: Mm-hmm.

0:24:43.9 Vael: Yeah. Alright. So one of the definitions of alignment is building models that represent and safely optimize hard-to-specify human values, alternatively ensuring that AI behavior aligns with system designer intentions. So I'm... So one of the questions, the question that I just kind of asked, was trying to dig at, do you think that the alignment problem per se, of trying to make sure that we are able to encode human values in a way that AIs can properly optimize, that that'll just kind of be solved as a matter of course, in that we're going to get stuck at various points, then we're going to have to address it. Or do you think that'll just like... We will... It won't get solved by default? Things will continue progressing in capabilities, and then we'll just have it be kind of unsafe, but like, "Uh, you know, it's good enough. It's fine."

0:25:21.1 Interviewee: I think a bit of both. There's so much promise and so much hype and so much money in pushing AI forward. So I think a lot of companies will try to do that. These various... We live in democracy, fortunately or unfortunately or actually we live in more like value of dollar, unfortunately, our society. At least in some countries are valued by progress. And especially companies, they have to progress, they have to advance and this is one of the easiest ways to advance. But I think some companies may be bad actors, whatever, they will try to push it to the limit. But these questions are ultimately unsolved, in a way this current system are designed, for the reason we discussed. So I think it will be a bit of both. Some companies will back down, some companies will try to push it to the limit, so we'll see. Depends on applications as well. I mean, some applications are safe. If you... I'm sorry, but... Sorry to bring it up, but for example, there was a case of AI in Microsoft when they released the bot and the bot start cursing. Which is okay. It's a cute example, they should have done a bit of PR loss, it's fine, but it's different from the car crashing you into that tree. So depends, depends on the application.

0:26:27.4 Vael: Yeah, it seems definitely true. Alright, so next argument is focusing on our CEO AI again, which can do multi-step planning, and it has a model of itself in the world. So it's modeling other people modeling it, because it feels like that's pretty important for having any sort of advanced AI that's acting as a CEO. So the CEO is making plans for the future, and it's noticing as it's making plans for its goal of maximizing profits with constraints that some of its plans fail because it gets shut down. So we built this AI so that it has the default thing where you have to check with humans before executing anything, because that seems like a basic safety measure. And so the humans are looking for a one page memo on the AI's decision. So the AI is thinking about writing this memo, and then it notices at some point that if it just changes the memo to not include all the information, then maybe the AI will be... I mean, then the humans will be more likely to approve it, which means it would be more likely to succeed in its original goal.

0:27:22.8 Vael: And so, the question here is... So this is not building self preservation into the AI, it's just like AI as an agent that's optimizing any sort of goal and self preservation is coming up as an instrumental incentive. So, what do you think of the argument, "Highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals and this is dangerous?"

0:27:42.9 Interviewee: Well, you already put intelligence and the human level into the sentence as well.

0:27:48.3 Vael: Yes.

0:27:48.6 Interviewee: Yeah, and it kind of... I'm already against that, because I don't think the system would actually behave in a way, like a sneaky way to avoid. Well, first of all, the current AI, even currently AI systems are highly uninterpretable. It's very hard to interpret what exactly is going on, right? But it still work within the bounds, right? So the example I gave you was Lee Sedol, when the Go game, already knew it's won, but it couldn't explain why it won, and it take human to parse it. Or another example that I like, in the chess, AI, at some point, put a queen in the corner, something you just never do as a human. And it couldn't explain, obviously, in fact. But it never pick up the board and crash it on the... It will work within the bounds.

0:28:29.1 Interviewee: So of course, if the bounds of this program allows the program to cheat in a way and withhold information, then yes, but... Again, it kinda works pair in pair. On one side it's really hard to interpret, so this one-pager that AI has to provide, it also has to be curated. And it will not include all information because it's impossible to digest all the computer memory and knowledge into one page. So on the one side, this page will always be limited and with all this lossy compression of the state of computer. But on the other side, I don't think computers can on purpose cheat on this page. Actually they might, depending on the algorithm[?], again, but I think it's a valid concern. That's an easy question, an easy answer, but it's depending on the system, depends how you design it.

0:29:09.5 Vael: Yeah, so I think I'm trying to figure out what exactly is in this design system. So one thing is it has to be very capable, and it has to... I want it to be operating over like reality in a way that I expect that CEOs would, so its task is interacting... It's doing online learning, I expect, and it's interacting with actual people. So it's giving them text and it's taking in video and text, and interacting with them like a CEO would. And it does have to, I think, have a model of itself in the world in order for this to happen and to model how to interact with people. But if we have AI to that extent, which I kind of think that eventually we'll develop AI with these capabilities. I don't know how long it will take, but I assume that these are commercially valuable enough that people will eventually have this sort of system that... This is an argument that like... And I don't know if this is true, but any agent that's optimizing really hard for a single goal will at some point, like, figure out if it's smart enough to... That it should like try to eliminate things that are going to reduce the likelihood of its plan succeeding, which in this case may be humans in the way.

0:30:10.5 Interviewee: I think you're right. Actually, while you were talking I also came up this example that I really like about, maybe you saw it as well, where there's an agent that optimized solving a particular race game and you control the car. At some point found a loophole, you remember [hard-to-parse] example, and find a loophole and just goes in circles. And the answer to that, you need to have explicit test goals. But in online learning settings, it's really hard. Plus, again, coming back to question number one, which is like, the system is so large at some point, you're just not able to cover all the cases. So yeah, yeah, I think so, I think it's possible, especially today in online learning fashion when you can't really have a complete system that you have all integration tests possible, and then you ship it. Once a system that automatically learns and updates, then it becomes... That could be a problem. Yeah, I agree.

0:30:52.8 Vael: Yeah. So this boat driving race is actually one of the canonical examples of the alignment problem, as it were, which is like-- you put in the right example into it. There's another part of the... so that's one version of "outer alignment", which is where the system designer isn't able to input what it wants into the system well enough, which I think gets harder as the AI gets more powerful. And then there's an "inner alignment" kind of issue that people are hypothesizing might happen, which is where you have an optimizer that spins up another optimizer inside of it, where the outer optimizer is aligned with what the human wants, but it now has this inner optimizer. The canonical example used here is how evolution is in some sense the top-level optimizer and it's supposed to create agents that are good at doing inclusive reproductive fitness and instead we have humans. And we are not optimizing very hard for inclusive reproductive fitness and we've invented contraceptives and stuff, which is like not very helpful. And so people are worried that maybe that would happen in very advanced AIs too, as we get like more and more generalizable systems.

0:31:51.0 Interviewee: Yeah. I think that there are two things to say, first of all, is what you said about loop in the loop. Now we're talking about what exactly system, what kind of a system can an algorithm create, right? Because for example, if you look at the current machine learning, we do know that for example, convolutional layers, they are good at computing equivariances, like transitional equivariances. If you move the object, they're supposed to be indifferent, but [hard to parse]-connected layers don't do that. So this [hard to parse] behavior you described? You need to have a system, first of all, that's capable enough for this kind of behavior. That's a big if, but okay. Once we get there, if we get there, the second question is, okay, can you cover for that? Can you figure out that these cases are eliminated completely from existence?

0:32:32.4 Interviewee: And the example of CEO is maybe is a good example. For me, the interesting example that is... that I can definitely see and envision, not even CEO, but like for example, application that controls your behavior in a way. Like, for example, curate your Tinder profile or curate your inbox sorting. And control you through that. Then yeah, for sure. Yeah. If you don't control for everything, it can be smarter than us and kind of figure out, back-engineer, how humans work, because we're not [hard to parse] than that. And curate the channel for us and even... Get us to the local minimum we might not want, but we'll still maximize its profits, whatever the profit means for the computer.

0:33:07.2 Vael: Okay. Yeah. So I'm one of those people who is really concerned about long term risk from AI, because I think there's some chance that if we continue along the developmental pathway that we have so far, that if we won't solve the alignment problem, we won't figure out a good way in order to like encode human values in properly and include the human in the loop properly. Like one of the easiest solutions here is like trying to figure out how to put a human in that loop with a AI, even if the AI is vastly more intelligent than a human. And so people are like, Oh, well maybe if you just train an AI to like work with the human and translate the big AI so that like the human understands it, and this is interpretability, and then you have like a system who's training that system and maybe we can recursively loop up for something. But.

0:33:52.2 Interviewee: The problem here, why they see, like, for example, let's take a very specific example. There's AI systems that, for example, historically curated, curated by humans, obviously without bad intent, but it gives bad credit scores to black population say, like people of color. That's really bad behavior. And this behavior is kinda easy to check because you have statistics and you look at statistics. It's one, let's call it one hop away. So you take the data, you take statistics, is done. The second hop, the two hops away would be: it creates dynamics that you can check, not right away, but later that potentially show something like that. It would be harder for humans to check. You can also, if you think about it for a while, you can come up with like three hop, right. Something that creates something that creates something that it does. So it's much easier and much harder to check. You don't know until it happens, and that's the point. So you have this very complicated dynamic you can't... There's not even flags that you can check, red flags that you can check in your model. That might be the issue.

0:34:49.3 Vael: Yeah. There's also... Yeah. Interpretability seems really... It seems really important, especially as very, very intelligent systems, and we don't know how to do that. So possible versions of things going badly, I think, are... So if you have an AI system that's quite powerful and it's going to be instrumentally incentivized to not let itself be modified, then that means that you don't have that many chances to get it right in some sense, if you're dealing with a sufficiently intelligent system. Especially also because instrumental incentives are to acquire resources and influence and so... and also improve itself. Which could be a problem maybe of recursive self-improvement. And then it, like, can get rid of humans very quickly if it wanted to, via like synthetic biology. Another kind of-- this is not as advanced AI, but what if you put AI in charge of the manufacturing systems and food production systems, and there's some sort of correlated failure. And then we have misuse, and then the AI assisted-war as, like, various concerns about AI. What do you think about this?

0:35:37.9 Interviewee: Yeah. So one thing I want to say is that the AI system are kinda too lazy. In a sense that the reason why this loophole worked with the car is because this solution is easier in some mathematical way to find the proper solution.

0:35:52.7 Interviewee: So one thing humans can address this problem is just looking at... Use a supervisor as a human, or with a test, which is pretty much like supervisors-- check all the check and balances. We can create the system for which the, maybe mathematical even, finding proper solution is easier for the computer than finding its loopholes. That's one thing I want to say. And the second thing I want to say, which is now coming to nature because now we're talking about the rules of nature. But if it so happened that we design a system for which finding the loophole... for example, we have laws of physics-- we also have laws in behavior. Like if you have an algorithm that, you know, wants to organize human behavior or something, there's also laws of behavior. So it might be an interesting question that once we have this algorithm, we can get a bunch of sociologists, for example, people who are familiar with it, to study this algorithm and figure out that if, for example, loopholes are actually more probable than normal behavior quote on quote. So for example, being a bad person is better than being a good person-- or easier. Not necessarily better, but easier than being a good person. Which we kind of see it in society sometimes.

0:36:51.5 Interviewee: So it's curious if an algorithm will actually discover that. And finding this loophole with a car is easy because it's there, you just need to move a little bit. So for a computer is easy to find it, like it's a local minimum that's really easy to fall in. But if you design the system for which these loopholes are hard, that might be easier. Or the question is, can we define a problem for which the proper solution is easier?

0:37:11.9 Vael: Yeah. And I think the problems are going to get much more... harder and harder. Like if you have an AI that's doing a CEO thing, then I imagine like, just as humans figure out, there's many loopholes in society, many loopholes in how you achieve goals that are much cheaper in some sense. So I do think it's probably going to have to be designed in the system rather than being in the problem per se, as the problems get more and more difficult. Yeah.

0:37:32.1 Vael: So there's this community of people working on AI alignment per se and long term AI risk, and are kind of trying to figure out how you can build... How you can solve the alignment problem, and how you can build optimization functions or however you need to design them, or interpretability or whatever the thing is, in order to have an AI that is more guaranteed, or less likely to end up doing this kind of instrumental incentive to deceive or not want to... not self-preservation, maybe get rid of humans. So I think my question right now is, and there's money in this space now, there's a lot of interest in it and I'm sure you've heard of it-- I mean, I'm not sure you've heard about it, but there's... Yeah. There's certainly money and attention these days now that there wasn't previously. So what would cause you to work on these problems?

0:38:12.4 Interviewee: ...Well, ultimately a lot of people are motivated by actually just very few things and one of them is kids. Kids cause here you want to have a good future for yourself and the kids.

0:38:24.8 Interviewee: You want to live in a better and better human, better and better society, better and better everything. So that, and actually it hits home. The examples we discussed even during this call are, could be pretty grim. If we don't make this right. So putting resources there I think is really important. If people, before they come up with atomic bomb, they will figure out situations of which we are facing now, people might not even come up with atomic bomb, but do it in a safer way. Or like people knew about Chernobyl before it happened, obviously they would make it better. So having this hindsight, even though we don't know what's going to happen in the future, but putting the resources there, I think is definitely a smart move.

0:38:57.9 Vael: Got it. And what would cause you specifically to work on this?

0:39:03.2 Interviewee: Yes, I know, you asked this question. Yeah, well again, particular problems. I mean, it's an interesting problem. For me I'm actually... Since I work in optimization, so I like to have well-formulated problems. And making sure this one goes, this problem is more... now it's kind of vague. I mean, even now we discuss it. I agree with you that this problem is valid. It makes sense. It exists. But it's still vague, like how you study it. My PhD was also on the way to interpret, to come up with a way to interpret data.

0:39:30.2 Interviewee: In fact, like maybe it's... I don't want to spend too much time on it, but basically the idea is to visualize the data. If you have a very high dimension of data, you want to visualize it. But it's very lossy. You just visualize something, you just do something and it doesn't represent everything. And it's in a way it was actually, it was well-formulated, because there is a mathematical formula to minimizing. And of course it comes with conditions like [hard to parse] and loss and stuff, but in a way it's there, the problem is defined. So the same here. Like in a way, there's a mathematical apparatus. ...Maybe actually I'm going to be the one developing it as well. So I'm not saying that, "come give me the one I'm going to work on!" I think that's a [hard to parse - "would be direct"?], so problem that I would be excited about.

0:40:06.1 Vael: Yeah. And I mean, I think like what this field really needs right now is someone to specify the problem more precisely. Just because it's like, Oh, this is a future system, it's like at least 50 years, well I don't know at least-- it's far away. It's not happening immediately and we don't have very good like frameworks for it. And so it makes it hard to do research on. Cool. Alright. Well, I'll send you some resources afterwards if you feel like looking into it, but if not, regardless, thank you so much for doing this call with me.

0:40:33.1 Interviewee: Yeah, I appreciate it. Thank you for your time, it was really fun.