AI Alignment Podcast: On Becoming a Moral Realist

There's a fair amount of complexity in this podcast, and I'm not an expert on moral philosophy, but here's an *oversimplified* summary anyway. First, in the same way that we can reach mathematical truths through reason, we can also arrive at moral truths through reason, which suggests that they are true facts about the universe (a moral realist view). Second, preference utilitarianism has the problem of figuring out which preferences you want to respect, which isn't a problem with hedonic utilitarianism. Before and after the interview, Lucas argues that moral philosophy is important for AI alignment. Any strategic research "smuggles" in some values, and many technical safety problems, such as preference aggregation, would benefit from a knowledge of moral philosophy. Most importantly, given our current lack of consensus on moral philosophy, we should be very wary of locking in our values when we build powerful AI.

I'm not convinced that we should be thinking a lot more about moral philosophy. While I agree that locking in a set of values would likely be quite bad, I think this means that researchers should not hardcode a set of values, or create an AI that infers some values and then can never change them. It's not clear to me why studying more moral philosophy helps us with this goal. For the other points, it seems not too important to get preference aggregation or particular strategic approaches exactly perfect as long as we don't lock in values -- as an analogy, we typically don't argue that politicians should be experts on moral philosophy, even though they aggregate preferences and have large impacts on society.


How artificial intelligence is changing science

AI is being used in many different projects across many different fields at Stanford. This post has a list of a whole bunch of scientific projects that AI is helping with.


Talk to Books


Winner's Curse?

D. Sculley
Jasper Snoek
Ali Rahimi
Alex Wiltschko

A short paper arguing that we need more empirical rigor in ML, identifying some structural incentives that push against this and suggesting solutions.

While this isn't very relevant to technical alignment, it does seem important to have more rigor in ML, since ML researchers are likely to be the ones building advanced AI.


Generally capable agents emerge from open-ended play

Open-Ended Learning Team
Adam Stooke
Anuj Mahajan
Catarina Barros
Charlie Deck
Jakob Bauer
Jakub Sygnowski
Maja Trebacz
Max Jaderberg
Michael Mathieu
Nat McAleese
Nathalie Bradley-Schmieg
Nathaniel Wong
Nicolas Porcel
Roberta Raileanu
Steph Hughes-Fitt
Valentin Dalibard
Wojciech Marian Czarnecki

Artificial intelligence agents have become successful at games when trained for each game separately. However, it has proven challenging to build agents that can play _previously unseen_ games. This paper makes progress on this challenge in three primary areas: creating rich simulated environments and tasks, training agents with attention mechanisms over internal states, and evaluating agents over a variety of games. The authors show that agents trained with goal-based attention in their proposed environment (XLand) succeed at a range of novel, unseen tasks with no additional training required. Moreover, such agents appear to use general tactics such as decision-making, tool use, and experimentation during game-play episodes.

The authors argue that training-data generation is a central challenge to training general RL agents (an argument we’ve seen before with <@POET@>(@Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions@) and <@PAIRED@>(@Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design@)). They propose the training environment XLand to address this. XLand includes many multiplayer games within consistent, human-relatable 3D worlds and allows for dynamic agent learning through the procedural generation of tasks which are split into three components: world, agents, and goals. The inclusion of other agents makes this a partially observable environment. Goals are defined with Boolean formulas. Each goal is a combination of options and every option is a combination of atomic predicates. For example, in hide-and-seek one player has the goal see(me,opponent) and the other player not(see(opponent,me)). The space of worlds and games are shown to be both vast and smooth, which supports training.

The agents themselves are trained using deep-RL combined with a goal-attention module (GOAT). The per-timestep observations of the agent are ego-centric RGB images, proprioception values indicating forces, and the goal of the agent. The GOAT works by processing this information with a recurrent network and then using a goal-attention module to select hidden states that are most relevant to achieving a high return. This is determined by estimating the expected return if the agent focused on an option until the end of the episode.

As with many other major deep-RL projects, it is important to have a good curriculum, where more and more challenging tasks are introduced over time. The obvious method of choosing tasks with the lowest reward doesn’t work, because the returns from different games are non-comparable. To address this, an iterative notion of improvement is proposed, and scores are given as percentiles relative to a population. This is similar in spirit to the <@AlphaStar League@>(@AlphaStar: Mastering the Real-Time Strategy Game StarCraft II@). Following this, game-theoretic notions such as Pareto dominance can be used to compare agents and to determine how challenging a task is, which can then be used to create a curriculum.

Five generations of agents are trained, each of which is used in the next generation to create opponents and relative comparisons for defining the curriculum. Early in training, the authors find that adding an intrinsic reward based on self-play is important to achieve good performance. This encourages agents to achieve non-zero rewards in as many games as possible, which the authors call “participation”. The authors also conduct an ablation study and find that dynamic task generation, population-based methods, and the GOAT module have a significant positive impact on performance.

The agents produced during training have desirable generalization capability. They can compete in games that were not seen before in training. Moreover, fine-tuning dramatically improves the performance of agents in tasks where training from scratch completely fails. A number of case studies are also presented to explore emergent agent behavior. In one experiment, an agent is asked to match a colored shape and another environment feature such as a shape or floor panel. At the start of the episode, the agent decides to carry a black pyramid to an orange floor, but then after seeing a yellow sphere changes options and places the two shapes together. This shows that the agent has robust option evaluation capability. In other experiments, the agents show the capacity to create ramps to move to higher levels in the world environment. Additionally, agents seem capable of experimentation. In one instance, the agent is tasked with producing a specific configuration of differently colored cube objects. The agent demonstrates trial-and-error and goes through several different configurations until it finds one it evaluates highly.

There are limitations to the agent capabilities. While agents can use ramps in certain situations they fail to use ramps more generally. For example, they frequently fail to use ramps to cross gaps. Additionally, agents generally fail to create more than a single ramp. Agents also struggle to play cooperative games involving following not seen during training. This suggests that experimentation does not extend to co-player behavior. More broadly, whether or not co-player agents decide to cooperate is dependent on the population the agents interacted with during training. In general, the authors find that agents are more likely to cooperate when both agents have roughly equal performance or capability.

This is a fairly complicated paper, but the authors do a reasonable job of organizing the presentation of results. In particular, the analysis of agent behavior and their neural representations is well done. At a higher level, I found it interesting that the authors partially reject the idea of evaluating agents with just expected returns. I broadly agree with the authors that the evaluation of agents across multi-player tasks is an open problem without an immediate solution. With respect to agent capability, I found the section on experimentation to be most interesting. In particular, I look forward to seeing more research on how attention mechanisms catalyze such behavior.

**Rohin's opinion:** One of my models about deep learning is “Diversity is all you need”. Suppose you’re training for some task for which there’s a relevant feature F (such as the color of the goal pyramid). If F only ever takes on a single value in your training data (you only ever go to yellow pyramids), then the learned model can be specialized to that particular value of F, rather than learning a more general computation that works for arbitrary values of F. Instead, you need F to vary a lot during training (consider pyramids that are yellow, blue, green, red, orange, black, etc) if you want your model to generalize to new values of F at test time. That is, your model will be zero-shot robust to changes in a feature F if and only if your training data was diverse along the axis of feature F.  (To be clear, this isn’t literally true, it is more like a first-order main effect.)

Some evidence supporting this model:
- The approach in this paper explicitly has diversity in the objective and the world, and so the resulting model works zero-shot on new objectives of a similar type and can be finetuned quickly.
- In contrast, the similar <@hide and seek project@>(@Emergent Tool Use from Multi-Agent Interaction@) did not have diversity in the objective, had distinctly less diversity in the world, and instead got diversity from emergent strategies for multiagent interaction (but there were fewer than 10 such strategies). Correspondingly, the resulting agents could not be quickly finetuned.
- My understanding is that in image recognition, models trained on larger, more diverse datasets become significantly more robust.

Based on this model, I would make the following predictions about agents in XLand:
- They will not generalize to objectives that can’t be expressed in the predicate language used at training time, such as “move all the pyramids near each other”. (In some sense this is obvious, since the agents have never seen the word “all” and so can’t know what it means.)
- They will not work in any environment outside of XLand (unless that environment looks very very similar to XLand).
In particular, I reject the idea that these agents have learned “general strategies for problem solving” or something like that, such that we should expect them to work in other contexts as well, perhaps with a little finetuning. I think they have learned general strategies for solving a specific class of games in XLand.

You might get the impression that I don’t like this research. That’s not the case at all — it is interesting and impressive, and it suggests that we could take the same techniques and apply them in broader, more realistic domains where the resulting agents could be economically useful. Rather, I expect my readership to overupdate on this result and think that we’ve now reached agents that can do “general planning” or some such, and I want to push against that.


Announcement: AI alignment prize round 2 winners and next round

The winners of the second round of the AI alignment prize have been announced! All of the winners have already been sent out in this newsletter, except for the first place winner, "[The Alignment Problem for History-Based Bayesian Reinforcement Learners](http://www.tomeveritt.se/papers/alignment.pdf)". The deadline for the next iteration of the AI alignment prize is June 30, 2018.


Request for proposals for projects in AI alignment that work with deep learning systems

Open Philanthropy is seeking proposals for AI safety work in four major areas related to deep learning, each of which I summarize below. Proposals are due January 10, and can seek up to $1M covering up to 2 years. Grantees may later be invited to apply for larger and longer grants.

Overall, I like these four directions and am excited to see what comes out of them! I'll comment on specific directions below.


AI Safety Papers

AI Safety Papers (announced [here](https://www.alignmentforum.org/posts/GgusnG2tiPEa4aYFS/ai-safety-papers-an-app-for-the-tai-safety-database)) is an app to interactively explore a previously collected <@database of AI safety work@>(@TAI Safety Bibliographic Database@). I believe it contains every article in this newsletter (at least up to a certain date; it doesn’t automatically update) along with their summaries, so you may prefer to use that to search past issues of the newsletter instead of the [spreadsheet I maintain](https://docs.google.com/spreadsheets/d/1PwWbWZ6FPqAgZWOoOcXM8N_tUCuxpEyMbN1NYYC02aM/edit#gid=0).


DeepMind hiring Research Scientist, Safety


BERI seeking new university collaborators

[BERI](http://existence.org/faq) is expanding its offerings to provide free services to a wider set of university-affiliated groups and projects, and they’re now accepting applications from groups and individuals interested in receiving their support. If you’re a member of a research group, or an individual researcher, working on long-termist projects, you can [apply here](http://existence.org/apply).


FHI Summer Research Fellowship

This six week summer fellowship allows fellows to take the lead on a project relevant to the long-term future, working with an FHI Research Scholar. Application deadline is March 22.


Microsoft invests in and partners with OpenAI to support us building beneficial AGI

After moving to a <@capped-profit investment model@>(@OpenAI LP@), Microsoft has invested $1 billion in OpenAI. This allows OpenAI to keep their focus on developing and sharing beneficial AGI: instead of having to create a product to cover costs, they can license their pre-AGI technologies, likely through Microsoft.


Funding for Study and Training Related to AI Policy Careers

The Open Philanthropy Project has launched an AI policy scholarships program; the deadline for the first round is October 15.


SafeML Workshop: Accepted Papers

The camera-ready papers from the SafeML workshop are now available! There are a lot of good papers on robustness, adversarial examples, and more that will likely never make it into this newsletter (there's only so much I can read and summarize), so I encourage you to browse through it yourself.


OpenAI LP

OpenAI is transitioning to a new structure, consisting of a capped-profit company (OpenAI LP) controlled by the original OpenAI nonprofit organisation. The nonprofit is still dedicated to its charter, which OpenAI LP has a legal duty to prioritise. All investors must agree that generating profits for them is a secondary goal, and that their overall returns will be capped at 100x their investment (with any excess going back to the nonprofit).

Given the high cost of salaries and compute for machine learning research, I don't find this a particularly surprising development. I'd also note that, in the context of investing in a startup, a 100x return over a timeframe of decades is not actually that high.


Q&A with Jason Matheny, Founding Director of CSET

The [Center for Security and Emerging Technology](https://cset.georgetown.edu/) has been announced, with a [$55 million grant from the Open Philanthropy Project](https://www.openphilanthropy.org/giving/grants/georgetown-university-center-security-and-emerging-technology), and is [hiring](https://cset.georgetown.edu/careers/). While the center will work on emerging technologies generally, it will initially focus on AI, since demand for AI policy analysis has far outpaced supply.

One area of focus is the implications of AI on national and international security. Current AI systems are brittle and can easily be fooled, implying several safety and security challenges. What are these challenges, and how important are they? How can we make systems that are more robust and mitigate these problems?

Another area is how to enable effective competition on AI in a global environment, while also cooperating on issues of safety, security and ethics? This will likely require measurement of investment flows, publications, data and hardware across countries, as well as management of talent and knowledge workflows.

See also [Import AI](https://jack-clark.net/2019/03/04/import-ai-136-what-machine-learning-power-infrastructure-means-for-humanity-new-gca-benchmarkdataset-challenges-image-captioning-systems-and-google-uses-frankenrl-to-create-more-mobile-robot/).

It's great to see a center for AI policy that's run by a person who has wanted to consume AI policy analysis in the past (Jason Matheny was previously the director of IARPA). It's interesting to see the areas he focuses on in this Q&A -- it's not what I would have expected given my very little knowledge of AI policy.


Governance of AI Fellowship

The Center for the Governance of AI is looking for a few fellows to work for around 3 months on AI governance research. They expect that fellows will be at the level of PhD students or postdocs, though there are no strict requirements. The first round application deadline is Feb 28, and the second round application deadline is Mar 28.


SafeML ICLR 2019 Call for Papers

The SafeML workshop has a paper submission deadline of Feb 22, and is looking for papers on specification, robustness and assurance (based on [Building safe artificial intelligence: specification, robustness, and assurance](https://medium.com/@deepmindsafetyresearch/building-safe-artificial-intelligence-52f5f75058f1) ([AN #26](https://mailchi.mp/1ecd1b775703/alignment-newsletter-26))).


Olsson to Join the Open Philanthropy Project

Catherine Olsson, a researcher at Google Brain who was previously at OpenAI, will be joining the Open Philanthropy Project to focus on grant making for reducing x-risk from advanced AI. Given her first-hand research experience, she has knowledge of the dynamics of research groups and a nuanced understanding of various safety subproblems. Congratulations to both her and OpenPhil.


GovAI Summer 2022 Fellowships

Applications are now open for the GovAI 2022 Summer Fellowship! This is an opportunity for early-career individuals to spend three months working on an AI governance research project, learning about the field, and making connections with other researchers and practitioners. Application deadline is Jan 1.


Foundations of Cooperative AI Lab

This new lab at CMU aims to create foundations of game theory appropriate for advanced, autonomous AI agents -- think of work on agent foundations and <@cooperative AI@>(@Open Problems in Cooperative AI@). Apply for a PhD [here](https://csd.cmu.edu/academics/doctoral/admissions) (deadline Dec 9) or for a postdoc [here](https://apply.interfolio.com/98450).


OpenAI hiring Software Engineer, Alignment

Exactly what it sounds like: OpenAI is hiring a software engineer to work with the Alignment team.


CHAI Internships 2022

CHAI internships are open once again! Typically, an intern will execute on an AI safety research project proposed by their mentor, resulting in a first-author publication at a workshop. The early deadline is November 23rd and the regular deadline is December 13th.


Announcing the Vitalik Buterin Fellowships in AI Existential Safety

FLI is launching a fellowship for incoming PhD students and postdocs who are focused on AI existential safety. The application deadline is October 29 for the PhD fellowship, and November 5 for the postdoc fellowship.


The Open Phil AI Fellowship (Year 5)

Applications are now open for the fifth cohort of the <@Open Phil AI Fellowship@>! They are also due October 29.


Research Scientist, Long-term Strategy & Governance

DeepMind (my employer) is hiring for several Research Scientist positions on the Long-term Strategy and Governance Team, across a wide range of backgrounds and skills. (Though note that you do need a PhD, or equivalent experience.) See also this [EA Forum post](https://forum.effectivealtruism.org/posts/atbonGDAFegfeDbTF/deepmind-is-hiring-long-term-strategy-and-governance).


Cooperative AI Workshop 2021

The <@Cooperative AI@>(@Open Problems in Cooperative AI@) <@NeurIPS workshop@>(@Cooperative AI Workshop@) is running again this year! The paper submission deadline is September 25.


You can now apply to EA Funds anytime! (LTFF & EAIF only)

The Long-Term Future Fund (LTFF) has funding available for people working on AI alignment. I’m told that the LTFF is constrained by high-quality applications, and that applying only takes a few hours, so it is probably best to err on the side of applying. The LTFF has removed its previous round-based system and now accepts applications anytime.


Stanford Existential Risks Conference

This conference on existential risks will run April 17-18. Applications to attend close April 12. There will be no charge to attend the conference.


Research Engineer, Safety (OpenAI)

The Applied Safety team at OpenAI is looking to hire a research engineer, and explicitly states that the job is about safety of general-purpose AI systems (as opposed to narrow AI systems like autonomous vehicles).


Chinese translation of Human Compatible

The Chinese translation of <@Human Compatible@>(@Human Compatible: Artificial Intelligence and the Problem of Control@) came out in October and the first chapter is [here](https://cread.jd.com/read/startRead.action?bookId=30675029&readType=1).


DPhil Scholarships Applications Open

FHI will be awarding up to six scholarships for the 2021/22 academic year for DPhil students starting at the University of Oxford whose research aims to answer crucial questions for improving the long-term prospects of humanity. Applications are due Feb 14.


CHAI Internship

<@CHAI internships@>(@CHAI 2020 Internships@) are open once again! The deadline for applications is December 13.


The Open Phil AI Fellowship

We’re now at the fourth cohort of the <@Open Phil AI Fellowship@>! Applications are due October 22.


Navigating the Broader Impacts of AI Research

This is a workshop at NeurIPS; the title tells you exactly what it's about. The deadline to submit is October 12.


FHI is hiring Researchers, Research Fellows, and Senior Research Fellows

FHI is hiring for researchers across a wide variety of topics, including technical AI safety research and AI governance. The application deadline is October 19.


Cooperative AI Workshop

This NeurIPS workshop has the goal of improving the _cooperation_ skills of AI systems (whether with humans or other machines), which encompasses a _very_ wide range of research topics. The deadline to submit is September 18.


Senior Systems Safety Engineer

OpenAI is hiring for a senior systems safety engineer. From my read of the job description, it seems like the goal is to apply the principles from <@Engineering a Safer World@> to AI development.


Early-career funding for individuals interested in improving the long-term future

This Open Philanthropy program aims to provide support for people who want to focus on improving the long-term future. The primary form of support would be funding for graduate school, though other one-off activities that build career capital also count. They explicitly say that people interested in working on AI policy or risks from transformative AI should apply to this program (possibly in addition to their <@AI fellowship@>(@Open Phil AI Fellowship@)). The stage 1 deadline is January 1, but if you submit earlier they aim to respond within 10 working days.


OpenAI API

OpenAI has released a commercial API that gives access to natural language completions via <@GPT-3@>(@Language Models are Few-Shot Learners@), allowing users to specify tasks in English that GPT-3 can then (hopefully) solve.

This is notable since this is (to my knowledge) OpenAI’s first commercial application.


Worldbuilding Contest

FLI invites individuals and teams to compete for a prize purse worth $100,000+ by designing visions of a plausible, aspirational future including artificial general intelligence. The deadline for submissions is April 15.

FLI launches Worldbuilding Contest with $100,000 in prizes


CLR Open Positions: Researchers and Summer Research Fellows

The Center on Long-Term Risk is looking for researchers and summer research fellows to work on high-quality research relevant to s-risks, including on (among other areas) multiagent systems. The application deadline is May 13.


Careers at the Joint AI Center

The Joint AI Center is searching for ML experts for a variety of roles.

You might be wondering why I've included these jobs in the newsletter, given that I don't do very many promotions. I think that it is reasonably likely that the US government (and the military in particular) will be a key player in the future of AI, and that there could be a lot to learn from their testing, evaluation, validation & verification (TEV&V) framework (which often seems more risk-averse to me than many alignment schemes are). As a result, I would be excited if readers of this newsletter interested in how the military thinks about AI filled these positions: it seems great to have a flow of ideas between the two communities (so that the government learns about alignment concerns, and so that we learn about TEV&V).


TAISU - Technical AI Safety Unconference

This unconference on technical AI safety will be held May 14th-17th; application deadline is February 23.


AI Alignment Visiting Fellowship

This fellowship would support 2-3 applicants to visit FHI for three or more months to work on human-aligned AI. The application deadline is Feb 28.


AI Safety Unconference 2019

David Krueger
Orpheus Lummis
and Gretchen Krueger

Like last year, there will be an AI safety unconference alongside NeurIPS, on Monday Dec 9 from 10am to 6pm. While the website suggests a registration deadline of Nov 25, the organizers have told me it's a soft deadline, but you probably should [register](https://docs.google.com/forms/d/e/1FAIpQLSfOXyo2P0Wv6bxyNyzzdMnzSL8_wGa4pMIDTh1tQwYeZmMebw/viewform) now to secure a place.


CHAI 2020 Internships

CHAI (the lab where I work) is currently accepting applications for its 2020 internship program. The deadline to apply is **Dec 15**.


FHI DPhil Scholarships

The Future of Humanity Institute will be awarding up to two DPhil scholarships for the 2020/21 academic year, open to students beginning a DPhil at the University of Oxford whose research aims to answer crucial questions for improving the long-term prospects of humanity. Applications will open around January or February, and decisions will be made in April.


Open Phil AI Fellowship

The Open Phil AI Fellowship is seeking applications for its third cohort. Applications are due by October 25. The fellowship is open to current and incoming PhD students, including those with pre-existing funding sources. It provides up to 5 years of support with a stipend of $40,000 and a travel allocation of $10,000.


Join our rapidly growing research teams

The Future of Humanity Institute is hiring researchers across a wide range of topics, including AI safety and strategy. The deadline to apply is midday August 16.


Offer of collaboration and/or mentorship

This is exactly what it sounds like. You can find out more about Vanessa's research agenda from <@The Learning-Theoretic AI Alignment Research Agenda@>, and I've summarized two of her recent posts in this newsletter.


Human-aligned AI Summer School

Jan Kulveit
Tomáš Gavenčiak
Jan Romportl

The second Human-aligned AI Summer School will be held in Prague from July 25-28, with a focus on "optimization and decision-making". Applications are due June 15.


AI Safety workshop at IJCAI 2019

Huáscar Espinoza
Han Yu
Xiaowei Huang
Freddy Lecue
Cynthia Chen
José Hernández-Orallo
Seán Ó hÉigeartaigh
and Richard Mallah

There will be a workshop on AI safety at IJCAI 2019 in Macao, China; the paper submission deadline is April 12. In addition to the standard submissions (technical papers, proposals for technical talks, and position papers), they are seeking papers for their "AI safety landscape" initiative, which aims to build a single document identifying the core knowledge and needs of the AI safety community.


MIRI Summer Fellows Program

CFAR and MIRI are running the MIRI Summer Fellows Program from August 9-24. Applications are due March 31.


FHI DPhil Scholarships

The Future of Humanity Institute is accepting applications for scholarships for candidates beginning a DPhil programme.


PAI Fellowship Program Call For Applications

The Partnership on AI is opening applications for Research Fellows who will "conduct groundbreaking multi-disciplinary research".


Summit on Machine Learning meets Formal Methods

This is a one-day summit on July 13 that is part of the Federated Logic Conference. This seems like an unusually good venue to think about how to apply formal methods to AI systems -- in particular I'm impressed by the list of speakers, which includes a variety of experts in both fields.


The Open Philanthropy Project AI Fellows Program

The second Open Phil AI Fellowship has been announced, open to AI and ML PhD students, and people applying to PhD programs starting in 2019. Even if you aren't looking for a fellowship, you may want to read through their example research topics, which are split into three main categories -- reward learning, reliability, and interpretability.

I like the breakdown of research topics, though personally I would have made them broader. I think I would want the "reward learning" category to include anything that aims to provide a specification for the AIs behavior, such as natural language instructions (that are mapped directly to policies with no reward in between). The "reliability" section is then about successfully meeting the specification, while the third section would include anything that allows the operator to empirically verify whether and enforce that the previous two sections are working correctly, including "interpretability". Actually, having written this, it's pretty similar to the three categories in the DeepMind blog post covered in the highlights.


80,000 Hours Job Board: AI/ML safety research

80,000 Hours recently updated their job board, including the section on technical safety research. The [AI strategy and governance](https://80000hours.org/job-board/ai-strategy-governance/) section is probably also of interest.


DeepMind job: Science Writer

According to the job listing, the role would involve creating content for the blog, videos, presentations, events, etc. and would require a reasonably technical background and strong writing skills. Vishal Maini at DeepMind notes that this person would likely have a significant impact on how AI research is communicated to various key strategic audiences around the world -- from the technical community to the broader public -- and would spend some of their time engaging with AI alignment research, among other areas.


Internship: The Future Society

An internship which will focus on AI policy research as well as support to organize two large AI governance events. To apply, send a CV and a short letter explaining ‘why you?’ to caroline.jeanmaire@thefuturesociety.org.


Public reports are now optional for EA Funds grantees

This is your regular reminder that you can apply to the Long-Term Future Fund (and the broader EA Funds) for funding for a wide variety of projects. They have now removed the requirement for public reporting of your grant. They encourage you to apply if you have a preference for private funding.


$2 Million Donated to Keep Artificial General Intelligence Beneficial and Robust

The next round of FLI grants have been announced! There are fewer grants than in their [first round](https://futureoflife.org/first-ai-grant-recipients/) and the topics seem more focused on AGI safety.


AI Safety Camp Virtual 2022

Applications are open for this remote research program, where people from various disciplines come together to research an open problem under the mentorship of an established AI-alignment researcher. Deadline to apply is December 1st.


Q&A Panel on Applying for Grad School

In this event run by AI Safety Support on November 7, current PhD students will share their experiences navigating the application process and AI Safety research in academia. RSVP [here](evt.to/dhaimisw).


SafeAI Workshop 2022

The SafeAI workshop at AAAI is now accepting paper submissions, with a deadline of Nov 12.


FLI's $25M Grants Program for Existential Risk Reduction

This podcast talks about FLI's recent grants program for x-risk reduction. I've previously mentioned the <@fellowships@>(@Announcing the Vitalik Buterin Fellowships in AI Existential Safety@) they are running as part of this program. As a reminder, the application deadline is October 29 for the PhD fellowship, and November 5 for the postdoc fellowship.


[Job ad] Research important longtermist topics at Rethink Priorities!

Of particular interest to readers, there are roles available in AI governance and strategy. The application deadline is Oct 24.


2022 IEEE Conference on Assured Autonomy

The ICAA conference seeks contributions on all aspects of AI safety, security, and privacy in autonomous systems. The paper submission deadline is October 18 and the conference itself will take place March 22-24.


CSER Job Posting: Academic Programme Manager

CSER is searching for a candidate for a relatively senior role that combines academic, management and administrative responsibilities. The application deadline is September 20.


NIST AI Risk Management Framework

The National Institute of Standards and Technology (NIST) has put out a formal Request For Information (RFI) in the process of developing an AI Risk Management Framework that is intended for voluntary use in order to improve trustworthiness and mitigate risks of AI systems. According to the [legislative mandate](https://www.congress.gov/116/bills/hr6395/BILLS-116hr6395enr.pdf#page=1151), aspects of trustworthiness include “explainability, transparency, safety, privacy, security, robustness, fairness, bias, ethics, validation, verification, interpretability, and other properties related to artificial intelligence systems that are common across all sectors”. Multiple AI safety organizations are submitting responses to the RFI and would like additional AI safety researchers to engage with it. Responses are due September 15; if you'd like to help out, email Tony Barrett at tbambarrett@gmail.com.


Introducing the AI Objectives Institute

For years people have been talking about corporations and capitalism as an example of superintelligence that we have failed to align so far. This new institute plans to take this correspondence seriously and transfer insights between the two. In particular, we can (a) examine how proposed problems with AI are already taking place with capitalism, (b) use tools and ideas from AI safety to improve upon capitalism, and (c) use lessons from capitalism to assist in the project of building a safely aligned AI.


ML Engineer Position at Preamble

[Preamble](https://www.preamble.com/) is a seed-stage company aiming to build middleware for AI ethics and safety, with a current focus on recommender systems. They have an early prototype for Twitter users, implemented as a browser extension. They are currently trying to hire an ML engineer to push forward their work.


Ought's Progress Update July 2018

A lot of organizational updates that I won't summarize here. There's a retrospective about the Predicting Slow Judgments project, and some updates on the Factored Cognition project. Two particularly interesting points -- first, they have not yet run into questions where it seemed impossible to make progress by decomposing the problem, making them slightly more optimistic; and second, they are now more confident that decomposition will take a large amount of work, such that experiments will require some amount of automation using ML in order to be feasible.


Political Economy of Reinforcement Learning (PERLS) Workshop

Stuart Russell
Thomas Gilbert
Tom Zick
Aaron Snoswell
Michael Dennis

The deadline for submissions to this NeurIPS 2021 workshop is Sep 18. From the website: "The aim of this workshop will be to establish a common language around the state of the art of RL across key societal domains. From this examination, we hope to identify specific interpretive gaps that can be elaborated or filled by members of our community. Our ultimate goal will be to map near-term societal concerns and indicate possible cross-disciplinary avenues towards addressing them."


Building and Evaluating Ethical Robotic Systems

Justin Svegliato*
Samer Nashed*
Alan Winfield
Dylan Hadfield-Menell
Louise Dennis
Paul Bello

This workshop at IROS 2021 asks for work on ethical robotic systems, including value alignment as a subtopic. Notably, they also welcome researchers from disciplines beyond robotics, including philosophy, psychology, sociology, and law. The paper submission deadline is August 13.


Open Philanthropy Technology Policy Fellowship

Open Philanthropy is seeking applicants for a US policy fellowship program focused on high-priority emerging technologies, especially AI and biotechnology. Application deadline is September 15.


Hypermind forecasting contest on AI

Hypermind is running a forecasting contest on the evolution of artificial intelligence with a $30,000 prize over four years. The questions ask both about the growth of compute and about performance on specific benchmarks such as the <@MATH suite@>(@Measuring Mathematical Problem Solving With the MATH Dataset@).


Research Fellow- AI TEV&V

CSET is currently seeking a Research Fellow to focus on the safety and risk of deployed AI systems.


Deputy Director (CSER)

The Centre for the Study of Existential Risk (CSER) is looking to hire a Deputy Director.


Open Call for Advisees and Collaborators, May 2021

GCRI is open to inquiries from potential collaborators or advisees, regardless of background, career point, or geographic location, about any aspect of global catastrophic risk. Participation can consist of a short email exchange to more extensive project work.


BERI Seeking New University Collaborators

[BERI](https://existence.org/faq) is seeking applications for new collaborators. They offer free services to university groups. If you’re a member of a research group, or an individual researcher, working on long-termist projects, you can [apply here](http://existence.org/apply). Applications are due June 20th.


FLI Job Postings

The [Future of Life Institute](https://futureoflife.org/) has 3 new job postings for full-time equivalent remote policy focused positions. They're looking for a Director of European Policy, a Policy Advocate, and a Policy Researcher, all primarily focused on AI policy and governance. Additional policy areas of interest may include lethal autonomous weapons, synthetic biology, nuclear weapons policy, and the management of existential and global catastrophic risk. Applications are accepted on a rolling basis.


Postdoc role at CHAI

The Center for Human-Compatible AI (where I did my PhD) is looking for postdocs. Apply [here](https://forms.gle/8w9Jfjr3X86osAvTA).


Apply to EA Funds now

EA Funds applications are open until the deadline of March 7. This includes the Long-Term Future Fund (LTFF), which often provides grants to people working on AI alignment. I’m told that LTFF is constrained by high-quality applications, and that applying only takes a few hours, so it is probably best to err on the side of applying.


AISU 2021

The third AI safety unconference will take place online from April 23rd to April 28th, 2021. The registration deadline is April 13th.


MIT Postdoc Role

[Neil Thompson](http://www.neil-t.com/), who works on forecasting progress in AI (see for example [The Computational Limits of Deep Learning](https://arxiv.org/abs/2007.05558)), is looking for a postdoc in economics and computer science to (1) understand the key innovation trends in computing and artificial intelligence, and (2) analyze the economic and policy implications of these trends. The application deadline is Jan 3.


Metaculus AI Progress Tournament

Metaculus is running an AI forecasting tournament, with up to $50,000 in prizes. The tournament starts December 14, and will continue till around mid-June, and will involve forecasting targets on a 6-24 month timescale. You can pre-register to forecast now.


AI Safety Camp virtual edition 2021

Remmelt Ellen
Rebecca Baron
Richard Möhn
Max Chiswick
Nicholas Goldowsky-Dill

The second virtual AI Safety Camp will take place over the first half of 2021. Applications will close on December 15.


PhD Studentships in Safe and Trusted Artificial Intelligence

The UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence is offering 12 fully funded PhD Studentships. They focus on the use of symbolic AI techniques for ensuring the safety and trustworthiness of AI systems. There are multiple application periods; the application deadline for the first round is November 22.


OpenAI Licenses GPT-3 Technology to Microsoft

In the <@initial announcement of Microsoft’s investment in OpenAI@>(@Microsoft invests in and partners with OpenAI to support us building beneficial AGI@), OpenAI suggested that they would likely license pre-AGI technologies to Microsoft in order to get enough capital to run high-compute experiments. This has now happened with the <@GPT-3 API@>(@OpenAI API@).


AI Governance Project Manager

The Centre for the Governance of AI is hiring for a project manager role. The deadline to apply is September 30.


FHI Research Scholars Programme -- Applications Open

The Future of Humanity Institute’s Research Scholars Programme is hiring a second cohort of research scholars, likely to start in Spring 2021. The application deadline is September 14.


Research Scholars Programme

From the website: "The Future of Humanity Institute is launching a Research Scholars Programme, likely to start in October 2018. It is a selective, two-year research programme, with lots of latitude for exploration as well as significant training and support elements. We will offer around six salaried positions to early-career researchers who aim to answer questions that shed light on the big-picture questions critical to humanity’s wellbeing. We are collecting formal applications to the programme from now until 11 July, 2018."


Announcing the 2018 AI Fellows

The Open Philanthropy Project has chosen seven out of 180 applicants as the first class of AI fellows.


OpenAI Fellows—Fall 2018

Larissa Schiavo
Igor Mordatch
and John Schulman

The OpenAI Fellows program is accepting applications until July 8 for positions starting in September. The program is aimed at people who want to transition into doing AI research, but they do want evidence of interest in AI, either through past projects or self-study.


New Seminar Series and Call For Proposals On Cooperative AI

The Cooperative AI Foundation (CAIF) will be hosting a new fortnightly seminar series in which leading thinkers offer their vision for research on Cooperative AI. The first talk, 'AI Agents May Cooperate Better If They Don’t Resemble Us’, was given on Thursday (Jan 20) by Vincent Conitzer (Duke University, University of Oxford). You can find more details and submit a proposal for the seminar series [here](https://www.cooperativeai.com/seminars).


AI Risk Management Framework Concept Paper

After their <@Request For Information last year@>(@NIST AI Risk Management Framework@), NIST has now posted a concept paper detailing their current thinking around the AI Risk Management Framework that they are creating, and are soliciting comments by Jan 25. As before, if you're interested in helping with a response, email Tony Barrett at anthony.barrett@berkeley.edu.


Junior Research Assistant and Project Manager role at GCRI

This job is available immediately, and could be full-time or part-time. GCRI also currently has a [call](http://gcrinstitute.org/call-for-advisees-and-collaborators-for-select-ai-projects-january-2020/) for advisees and collaborators.


AI Safety Camp Toronto

The next <@AI safety camp@>(@The first AI Safety Camp and onwards@) will be held in early May, in Toronto. Apply [here](https://docs.google.com/forms/d/e/1FAIpQLSfZdu--EII061-KwWDSK6hZ5rtLCpBarKszw9btMs1dO1NOFA/viewform) by Jan 5.


Post-Doctoral Fellowship on Ethically Aligned Artificial Intelligence

Mila is looking for a postdoctoral fellow starting in Fall 2020 who would work on ethically aligned learning machines, towards building machines which can achieve specific goals while acting in a way consistent with human values and social norms. Applications are already being processed, and will continue to be processed until the position is filled.


Research Associate in Paradigms of Artificial General Intelligence and Their Associated Risk

CSER is hiring a post-doctoral research assistant to inform the AGI safety agenda by looking at existing and possible kinds of agents; the deadline is August 26.


Research Scholars Project Coordinator

FHI is looking to hire a coordinator for the Research Scholars Programme. Application deadline is July 10.


Open Phil AI Fellowship — 2019 Class

The Open Phil AI Fellows for this year have been announced! Congratulations to all of the fellows :)


AAAS Policy Fellowship

The AAAS Science & Technology Fellowship is open to Americans with a science PhD or 3 years of industry experience and a CS Masters. 80,000 Hours thinks this is one of the best ways into US Government AI policy careers. Application deadline is Nov 1.


BERI hiring ML Software Engineer

BERI is hiring a remote ML Engineer as part of their collaboration with the [Autonomous Learning Lab](https://all.cs.umass.edu/) at UMass Amherst. The goal is to create a software library that enables easy deployment of the ALL's Seldonian algorithm framework for safe and aligned AI.


Political Economy of Reinforcement Learning schedule

The date for the <@PERLS workshop@>(@Political Economy of Reinforcement Learning (PERLS) Workshop@) at NeurIPS has been set for December 14, and the schedule and speaker list are now available on the website.


Survey: classifying AI systems used in response to the COVID-19 pandemic

Samuel Curtis
Adriana Bora
Nicolas Miailhe
Rui Wang

A team at The Future Society aims to build a living database of AI systems used to respond to COVID, classified using the [OECD framework](https://www.oecd.ai/classification). I think this is an interesting example of building capacity for effective AI governance. If you were involved in developing an AI system used in the COVID response, they ask that you take [this survey](https://survey.oecd.org/index.php?r=survey/index&sid=145657&lang=en) by August 2nd.


AI Safety Career Bottlenecks Survey

[AI Safety Support](https://www.aisafetysupport.org/home) have released a career bottlenecks survey that they will use to guide their work. You can take the survey [here](https://www.guidedtrack.com/programs/n8cydtu/run).


Formal Methods for the Informal Engineer

Gopal Sarma
Jimmy Koppel
Ramana Kumar
Eric Drexler
Patrick Schultz
Gregory Malecha

This online workshop will teach engineers how to use verification tools like Z3 and Coq, and then discuss how formal verification can be applied in many different areas of software engineering (including robust machine learning). The organizers tell me they plan to produce a white-paper with high-level recommendations following the workshop. You can register [here](https://docs.google.com/forms/d/e/1FAIpQLSf2SVrwHFz-5oDj3LT9ks5moUjF6VFzNrUfnTGYshDw2XYnfg/viewform).


S-Risk Intro Seminar

The first intro seminar to s-risks will take place on the weekend of February 20 & 21, 2021. It is targeted at people who are at least seriously thinking about addressing s-risks as part of their career, and who have not yet spent a lot of time interacting with the Center on Long-Term Risk.


Action: Help expand funding for AI Safety by coordinating on NSF response

The National Science Foundation (NSF) has put out a Request for Information relating to topics they will be funding in 2023 as part of their NSF Convergence Accelerator program. The author and others are coordinating responses to increase funding to AI safety, and ask that you fill out this [short form](https://airtable.com/shrk0bAxm0EeJbyPC) if you are willing to help out with a few small, simple actions.


Researcher / Writer job

This full-time researcher / writer position would involve half the time working with [Convergence](https://www.convergenceanalysis.org/) on x-risk strategy research and the other half with [Normative](https://normative.io/) on environmental and climate change analysis documents.


Self-driving cars are here

Drive.ai will offer a self-driving car service for public use in Frisco, Texas starting in July, 2018. The post goes into details of how the cars will be rolled out, and some plans for how to make them easier for humans to interact with.


Facebook Open Sources ELF OpenGo

Facebook has created an open-source AI bot that has beaten world champion professional Go players in matches where the professional player was allowed unlimited time to think.


Introducing Stanford's Human-Centered AI Initiative

Stanford will house the Human-centered AI Initiative (HAI), which will take a multidisciplinary approach to understand how to develop and deploy AI so that it is robustly beneficial to humanity.

It's always hard to tell from these announcements what exactly the initiative will do, but it seems to be focused on making sure that AI does not make humans obsolete. Instead, AI should allow us to focus more on the creative, emotional work that we are better at. Given this, it's probably not going to focus on AI alignment, unlike the similarly named Center for Human-Compatible AI (CHAI) at Berkeley. My main question for the author would be what she would do if we could develop AI systems that could replace all human labor (including creative and emotional work). Should we not develop such AI systems? Is it never going to happen?


Human-Aligned AI Summer School: A Summary

A summary of the talks at the summer school that just happened, from one of the attendees, that covers value learning, agent foundations, bounded rationality, and side effects. Most of the cited papers have been covered in this newsletter, with the notable exceptions of Bayesian IRL and information-theoretic bounded rationality.


Conference on Fairness, Accountability, and Transparency (FAT*)

... will be held early 2019 in Atlanta, Georgia. Abstract pre-registration deadline is August 16.


RAISE is hiring

... for full-time content developers, to work at the [EA Hotel](http://effective-altruism.com/ea/1pc/ea_hotel_with_free_accommodation_and_board_for/) in Blackpool.


Announcing the second AI Safety Camp

I forgot to mention last week that the second AI safety camp will be held Oct 4-14 in Prague.


The first AI Safety Camp and onwards

The first AI safety camp was held in April, in which people interested in AI safety gathered to work on research within groups. Everyone prepared for the camp over the six weeks leading up to it, and then spent 10 days focusing on a particular research question. There were five teams of around four people, and each team wrote up some notes on the results of their project at the end of the camp.


BERI Project Grants Program

BERI is offering grants of up to $300,000 per year for work relating to their mission, with the application deadline of June 30. In their words, "We are open to any ideas you have, as long as you can explain how the project will contribute to improving human civilization’s long-term prospects for survival and flourishing."


Our essay competitions for young people

There is an essay competition for people between 16 and 25 years old, where one of the topics is "Do the benefits of artificial intelligence outweigh the risks?" Winning essays will be published on The Economist’s Open Future website and the author will be invited to attend one of the three Open Future Festival events. The deadline is July 15th.


Reframing Impact - Part 2

In <@part 1@>(@Reframing Impact - Part 1@) of this sequence, we saw that an event is _impactful_ if it _changes our ability to get what we want_. This part takes this understanding and applies it to AI alignment.

In the real world, there are many events that cause _objective_ negative impacts: they reduce your ability to pursue nearly any goal. An asteroid impact that destroys the Earth is going to be pretty bad for you, whether you want to promote human flourishing or to make paperclips. Conversely, there are many plans that produce objective positive impacts: for many potential goals, it's probably a good idea to earn a bunch of money, or to learn a lot about the world, or to command a perfectly loyal army. This is particularly exacerbated when the environment contains multiple agents: for goals that benefit from having more resources, it is objectively bad for you if a different agent seizes your resources, and objectively good for you if you seize other agents' resources.

Based on this intuitive (but certainly not ironclad) argument, we get the **Catastrophic Convergence Conjecture (CCC)**: "Unaligned goals tend to have catastrophe-inducing optimal policies because of power-seeking incentives".

Let's now consider a _conceptual_ version of <@Attainable Utility Preservation (AUP)@>(@Towards a New Impact Measure@): the agent optimizes a primary (possibly unaligned) goal, but is penalized for changing its "power" (in the intuitive sense). Intuitively, such an agent no longer has power-seeking incentives, and so (by the [contrapositive](https://en.wikipedia.org/wiki/Contraposition) of the CCC) it will not have a catastrophe-inducing optimal policy -- exactly what we want! This conceptual version of AUP also avoids thorny problems such as ontology identification and butterfly effects, because the agent need only reason about its own beliefs, rather than having to reason directly about the external world.

This was my favorite part of the sequence, as it explains the conceptual case for AUP clearly and concisely. I especially liked the CCC: I believe that we should be primarily aiming to prevent an AI system "intentionally" causing catastrophe, while not attempting to guarantee an absence of "accidental" mistakes (<@1@>(@Clarifying "AI Alignment"@), <@2@>(@Techniques for optimizing worst-case performance@)), and the CCC is one way of cashing out this intuition. It's a more crisp version of the idea that [convergent instrumental subgoals](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf) are in some sense the "source" of AI accident risk, and if we can avoid instrumental subgoals we will probably have solved AI safety.


Reframing Impact - Part 3

The final section of the sequence turns to an actual implementation of AUP, and deals with problems in how the implementation deviates from the conceptual version of AUP. We measure power by considering a set of auxiliary rewards, and measuring the change in attainable utilities of this auxiliary set as impact, and penalizing the agent for that. The first post presents some empirical results, many of which <@we've covered before@>(@Penalizing Impact via Attainable Utility Preservation@), but I wanted to note the new results on [SafeLife](https://www.partnershiponai.org/safelife/) (summarized below). On the high-dimensional world of SafeLife, the authors train a VAE to find a good latent representation, and choose a single linear reward function on the latent representation as their auxiliary reward function: it turns out this is enough to avoid side effects in at least some cases of SafeLife.

We then look at some improvements that can be made to the original AUP implementation. First, according to CCC, we only need to penalize _power_, not _impact_: as a result we can just penalize _increases_ in attainable utilities, rather than both increases and decreases as in the original version. Second, the auxiliary set of rewards only provides a _proxy_ for impact / power, which an optimal agent could game (for example, by [creating subagents](https://www.alignmentforum.org/posts/mdQEraEZQLg7jtozn/subagents-and-impact-measures-full-and-fully-illustrated), summarized below). So instead, we can penalize increases in attainable utility for the _primary_ goal, rather than using auxiliary rewards. There are some other improvements that I won't go into here.

I think the plan "ensure that the AI systems we build don't seek power" is pretty reasonable and plausibly will be an important part of AI alignment. However, the implementation of AUP is trying to do this under the threat model of optimal agents with potentially unaligned primary goals. I think this is probably going to do something quite different from the conceptual version of AUP, because impact (as defined in this sequence) occurs only when the agent's beliefs _change_, which doesn't happen for optimal agents in deterministic environments. The current implementation of AUP tries to get around this using proxies for power (but these can be gamed) or by defining "dumber" beliefs against which power is measured (but this fails to leverage the AI system's understanding of the world). See [this comment](https://www.alignmentforum.org/posts/wAAvP8RG6EwzCvHJy/reasons-for-excitement-about-impact-of-impact-measure?commentId=s48grPhMbuBEXNtyc) for more details.

Note that the author himself is more [excited](https://www.alignmentforum.org/s/7CdoznhJaLEKHwvJW/p/wAAvP8RG6EwzCvHJy) about AUP as deconfusion, rather than as a solution to AI alignment, though he is more optimistic about the implementation of AUP than I am.


Introducing SafeLife: Safety Benchmarks for Reinforcement Learning

So far, techniques to avoid negative side effects have only been tested on <@simple@>(@Measuring and avoiding side effects using relative reachability@) <@gridworlds@>(@Penalizing Impact via Attainable Utility Preservation@) <@or@>(@Learning Preferences by Looking at the World@) <@hypotheticals@>(@Test Cases for Impact Regularisation Methods@). SafeLife aims to provide a high-dimensional environment in which negative side effects are likely. It is based on Conway's Game of Life, which allows for complex effects arising out of relatively simple rules. An agent is given the ability to move, create life in an adjacent cell, or destroy life in an adjacent cell. With the specified reward function, the agent must build desired patterns, remove undesired patterns, and navigate to the exit.

The challenge comes when there are additional "neutral" patterns in the environment. In this case, we want the agent to leave those patterns alone, and not disrupt them, even if doing so would allow it to complete the main task faster. The post shows several examples of agents attempting these levels. Vanilla RL agents don't avoid side effects at all, and so unsurprisingly they do quite badly. An agent with a naive impact measure that simply says to preserve the initial state can correctly solve levels where all of the "neutral" patterns are static, but has much more trouble when the existing patterns are dynamic (i.e. they oscillate over time).

2019 NeurIPS Safety and Robustness in Decision Making Workshop

I am a big fan of benchmarks; they seem to be a prerequisite to making a lot of quantitative progress (as opposed to more conceptual progress, which seems more possible to do without benchmarks). This benchmark seems particularly nice to me because the "side effects" which need to be avoided haven't been handcoded into the benchmark, but instead arise from some simple rules that produce complex effects.

Paper: SafeLife 1.0: Exploring Side Effects in Complex Environments


Safety Gym

Safety gym contains a set of tasks with varying difficulty and complexity focused on safe exploration. In the tasks, one of three simulated robots has to move to a series of goals, push buttons or move a box to a target location, while avoiding costs incurred by hitting randomized obstacles. This is formalized as a **constrained reinforcement learning** problem: in addition to maximizing the received reward, agents also have to respect constraints on a **safety cost function**. For example, we would like self-driving cars to learn how to navigate from A to B as quickly as possible while respecting traffic regulations and safety standards. While this could in principle be solved by adding the safety cost as a penalty to the reward, constrained RL gets around the need to correctly quantify tradeoffs between safety and performance.

Measures of safety are expected to become important criteria for evaluating algorithms' performance and the paper provides first benchmarks. Constrained policy optimization, a trust-region algorithm that tries to prevent updates from breaking the constraint on the cost is compared to new lagrangian versions of TRPO/PPO that try to maximize the reward, minus an adaptive factor times the cost above the threshold. Interestingly, the lagrangian methods incur a lot less safety cost during training than CPO and satisfy constraints more reliably at evaluation. This comes at the cost of reduced reward. For some of the tasks, none of the tested algorithms is able to gain nontrivial rewards while also satisfying the constraints.

Lastly, the authors propose to use safety gym for investigating methods for learning cost functions from human inputs, which is important since misspecified costs could fail to prevent unsafe behaviour, and for transfer learning of constrained behaviour, which could help to deal with distributional shifts more safely.

I am quite excited about safety gym. I expect that the crisp formalization, as well as the availability of benchmarks and ready-made environments, combined with OpenAI's prestige, will facilitate broader engagement of the ML community with this branch of safe exploration.  As pointed out in the paper, switching from standard to constrained RL could merely shift the burden of correct specification from the reward to the cost and it is not obvious whether that helps with alignment. Still, I am somewhat optimistic because it seems like humans often think in terms of constrained and fuzzy optimization problems rather than specific tradeoffs and constrained RL might capture our intuitions better than pure reward maximization. Lastly, I am curious whether an increased focus on constrained RL will provide us with more concrete examples of "nearest unblocked strategy" failures, as the rising popularity of RL arguably did with more general examples of specification gaming.

**Rohin's opinion:** Note that at initialization, the policy doesn't "know" about the constraints, and so it must violate constraints during exploration in order to figure out what the constraints even are. As a result, in this framework we could never get down to zero violations. A zero-violations guarantee would require some other source of information, typically some sort of overseer (see <@delegative RL@>(@Delegative Reinforcement Learning@), [avoiding catastrophes via human intervention](https://arxiv.org/abs/1707.05173), and [shielding](https://arxiv.org/abs/1708.08611)).

It's unclear to me how much this matters for long-term safety, though: usually I'm worried about an AI system that is plotting against us (because it has different goals than we do), as opposed to one that doesn't know what we don't want it to do.


Designing agent incentives to avoid reward tampering

Tom Everitt
Ramana Kumar
and Marcus Hutter

Reward tampering occurs when a reinforcement learning agent actively changes its reward function. The post uses <@Causal Influence Diagrams@>(@Modeling AGI Safety Frameworks with Causal Influence Diagrams@) to analyze the problem in a simple grid world where an agent can easily change the definition of its reward. The proposed solution is **current-RF optimization**: Instead of maximizing the sum of rewards that would be given after each action (where the reward signal can dynamically change over time), the agent searches for and executes a plan of actions that would maximize the current, unchanged reward signal. The agent would then not be incentivized to tamper with the reward function since the current reward is not maximized by such tampering. There are two different flavours to this: time-inconsistency-aware agents account for future changes in their own behaviour due to modified reward signals, while TI-unaware agents ignore this in their planning. TI-aware agents have an incentive to preserve their reward signal and are therefore potentially incorrigible.

I enjoyed this application of causal diagrams and think that similar detailed analyses of the interactions between failure modes like wireheading, instrumental goals like reward preservation and the specific implementation of an agent would be quite valuable. That said, I am less excited about the feasibility of the proposed solution since it seems to require detailed knowledge of the agent about counterfactual rewards. Also, I expect the distinction between changes in the reward signal and changes in the state that happen to also affect the reward to be very fuzzy in real problems and current-RF optimization seems to require a very sharp boundary.

**Rohin's opinion:** I agree with Flo's opinion above, and I think the example in the blog post shows how the concept "what affects the reward" is fuzzy: in their gridworld inspired by the game Baba Is You, they say that moving the word "Reward" down to make rocks rewarding is "tampering", whereas I would have called that a perfectly legitimate way to play given my knowledge of Baba Is You.

Paper: Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective


Reframing Impact - Part 1

_This sequence has exercises that **will be spoiled by this summary**, so take a moment to consider whether you want to read the sequence directly._

This first part of the sequence focuses on identifying what we mean by impact, presumably to help design an impact measure in the future. The punch line: an event is **impactful to an agent** if it changes the agent's **ability to get what it wants**. This is _Attainable Utility (AU) theory_. To quote the sequence: "How could something possibly be a big deal to us if it _doesn't_ change our ability to get what we want? How could something _not_ matter to us if it _does_ change our ability to get what we want?"

Some implications and other ideas:
- Impact is _relative to an agent_: a new church is more impactful if you are a Christian than if not.
- Some impact is _objective_: getting money is impactful to almost any agent that knows what money is.
- Impact is _relative to expectations_: A burglar robbing your home is impactful to you (you weren't expecting it) but not very impactful to the burglar (who had planned it out). However, if the burglar was unsure if the burglary would be successful, than success/failure would be impactful to them.

While this may seem obvious, <@past work@>(@Measuring and avoiding side effects using relative reachability@) has talked about impact as being caused by changes in state. While of course any impact does involve a change in state, this is the wrong level of abstraction to reason about impact: fundamentally, impact is related to what we care about.

To quote myself from a discussion with Alex, "you're looking at the optimal Q-function for the optimal utility function and saying 'this is a good measure of what we care about' and of course I agree with that". (Although this is a bit inaccurate -- it's not the optimal Q-function, but the Q-function relative to what we expect and know.)

This may be somewhat of a surprise, given that I've been [pessimistic](https://www.alignmentforum.org/posts/kCY9dYGLoThC3aG7w/best-reasons-for-pessimism-about-impact-of-impact-measures#qAy66Wza8csAqWxiB) about impact measures in the past. However, my position is that it's difficult to simultaneously get three desiderata: value-agnosticism, avoidance of catastrophes, and usefulness. This characterization of impact is very explicitly dependent on values, and so doesn't run afoul of that. (Also, it just makes intuitive sense.)

This part of the sequence did change some of my thinking on impact measures as well. In particular, the sequence makes a distinction between _objective_ impact, which applies to all (or most) agents, and _value_ impact. This is similar to the idea of [convergent instrumental subgoals](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf), and the idea that <@large-scale multiagent training@>(@Emergent Tool Use from Multi-Agent Interaction@) can lead to generally useful behaviors that can be applied to novel tasks. It seems plausible to me that we could make value-agnostic impact measures that primarily penalize this objective impact, and this might be enough to avoid catastrophes. This would prevent us from using AI for big, impactful tasks, but could allow for AI systems that pursue small, limited tasks. I suspect we'll see thoughts along these lines in the next parts of this sequence.


Asymptotically Benign AGI

I'm a bit out of my depth with this summary, but let's give it a shot anyway. The setting: we are not worried about how much compute we use (except that it should be finite), and we would like to build a powerful AI system that can help us with tasks but does not try to influence the world. We'll assume that we can construct a box from which no signals can pass through, except by pressing a specific button that opens a door.

First, the simple version of BoMAI (Boxed Myopic AI). We'll put the AI system **and the operator** in the box, and the operator and the AI system can talk via text message, and the operator can enter rewards. Each episode has a maximum length (hence myopic), and if the operator ends the episode early, all future rewards are set to zero. BoMAI maximizes episodic reward in a manner similar to [AIXI](http://www.hutter1.net/ai/). It has a distribution (initially a speed prior) over all possible time-bounded Turing Machines as possible models that predict observations and rewards. BoMAI uses the maximum a posteriori (MAP) Turing Machine to predict future observations and rewards given actions, searches over all possible sequences of actions for the best one, and executes the first action of that sequence. (See [this comment](https://www.alignmentforum.org/posts/pZhDWxDmwzuSwLjou/asymptotically-benign-agi#ToDYj27wQZWHqWnH6) and its parents for the reason to use the MAP model.)

Intuitively, BoMAI has no incentive to affect anything outside the box: information can only leave the box if the episode ends, but if that happens, reward is guaranteed to be exactly zero. It might deceive the operator into giving it high reward, but any such deception will only _incidentally_ affect the outside world, it hasn't been _optimized_ to cause any such effect. However, **this is all assuming that the MAP world model is accurate**, which may not be true.

Obviously at initialization our world model will not be accurate: we need to explore. Rather than using something like epsilon-exploration, BoMAI delegates to a human explorer for safety, and reduces exploration over time.

More insidiously, a world model could simulate the world for a year, and then report the value in the computer's memory for "the reward from a year ago". This will usually be a perfect prediction of the reward the operator will give, but if BoMAI tries to _optimize_ for it, it has an incentive to try to e.g. cause the operator to build an unboxed AI whose goal is to change BoMAI's memory cell to the maximum value.

These sorts of bad plans can only happen finitely often. If BoMAI executes such a plan, it will then get the actual reward from the operator, see that it differs from the predicted reward, and update against the world model for making a bad prediction. Eventually the correct model will become more likely. But ideally this sort of thing wouldn't happen at all.

Intuitively, such world models would have to simulate the world past the end of the episode, incurring a speed penalty relative to the correct model. However, they could use that simulation to save on the computation needed for future predictions. To make sure that they do incur a speed penalty, we need to make it so that sometimes their simulations are wrong and can't be reused. So, at the end of each episode, we sample from a source of true randomness, and use that to determine whether or not the operator is given a drug that causes retrograde amnesia. In that case, world models that simulate the future will sometimes be wrong about how that decision plays out, and any simulation they did in the wrong branch is useless and leads to a speed penalty.

With a few non-trivial assumptions, we can prove that asymptotically, BoMAI will do at least as well as the human explorer at accumulating reward, and the MAP world model's rewards do not incentivize BoMAI to affect the outside world.

I think the idea of putting the operator in the box with the AI system is very interesting: with previous attempts at boxing, the human operator talking to the AI system was an obvious glaring hole in the box. In this setting, the only information escaping from the box is the fact that the operator has not yet chosen to end the episode.

I am generally skeptical of intuitive reasoning about what can or can't be done by Turing Machines using extreme amounts of computation. There are _lots_ of comments on the post that debate specifics of this. This usually cashes out as a debate about the assumptions in the proof. But it's also worth noting that the theorem is asymptotic, and allows for arbitrarily bad behavior early on. We might still expect good behavior early on for the reasons laid out in the proof, but it's not implied by the theorem, even if the assumptions hold.

Paper: Asymptotically Unambitious Artificial General Intelligence


Safety-first AI for autonomous data centre cooling and industrial control

Two years ago, DeepMind built an AI recommendation system that provided suggestions on how best to cool Google's data centers, leading to efficiency gains. Nine months ago, the AI was given autonomous control to take actions directly, rather than going through human operators, and it has been improving ever since, going from 12% savings at deployment to 30% now.

Of course, such a system must be made extremely reliable, since a failure could result in Google's data centers going down. They implemented several safety measures. They throw out any actions that the AI is not confident about. All actions are verified against a set of hand-coded safety rules, both when the actions are generated in the cloud, and at each local data center, for reliability through redundancy. There are human operators monitoring the AI to make sure nothing goes wrong, who can take over control whenever they want to. There is also an automated system that will fall back to the original system of heuristics and rules if the safety conditions are ever violated.

This is a remarkable number of safety precautions, though in hindsight it makes total sense given how bad a failure could be. None of the precautions would stop a superintelligent agent in the classical sense (that is, the sort of superintelligent agent in paperclip maximizer stories), but they seem like a really good set of precautions for anything task-based. I am curious how they chose the threshold for when to discard actions that the AI is not confident enough in (especially since AI uncertainty estimates are typically not calibrated), and how they developed the safety rules for verification (since that is a form of specification, which is often easy to get wrong).


Designing agent incentives to avoid side effects

Victoria Krakovna
Ramana Kumar
Laurent Orseau
Alexander Turner

This blog post provides details about the recent update to the [relative reachability paper](https://arxiv.org/abs/1806.01186) ([AN #10](https://mailchi.mp/d1a19c140226/alignment-newsletter-10)), which is now more a paper about the design choices available with impact measures. There are three main axes that they identify:

First, what baseline is impact measured relative to? A natural choice is to compare against the starting state, but this will penalize the agent for environment effects, such as apples growing on trees. We can instead compare against an inaction baseline, i.e. measuring impact relative to what would have happened if the agent did nothing. Unfortunately, this leads to offsetting behavior: the agent first makes a change to get reward, and then undoes the change in order to not be penalized for impact. This motivates the stepwise inaction baseline, which compares each action against what would have happened if the agent did nothing _from that step onwards_.

Second, we need a measure by which to compare states. The unreachability measure measures how hard it is to reach the baseline from the current state. However, this "maxes out" as soon as the baseline is unreachability, and so there is no incentive to avoid further irreversible actions. This motivates relative reachability, which computes the set of states reachable from the baseline, and measures what proportion of those states are reachable from the state created by the agent. [Attainable utility](https://www.alignmentforum.org/posts/yEa7kwoMpsBgaBCgb/towards-a-new-impact-measure) ([AN #25](https://mailchi.mp/0c5eeec28f75/alignment-newsletter-25)) generalizes this to talk about the _utility_ that could be achieved from the baseline for a wide range of utility functions. (This is equivalent to relative reachability when the utility functions are of the form "1 if state s is ever encountered, else 0".)

Finally, we need to figure how to penalize changes in our chosen measure. Penalizing decreases in the measure allows us to penalize actions that make it harder to do things (what the AUP post calls "opportunity cost"), while penalizing increases in the measure allows us to penalize convergent instrumental subgoals (which almost by definition increase the ability to satisfy many different goals or reach many different states).

Since the AUP post was published about half a year ago, I've been watching this unification of AUP and relative reachability slowly take form, since they were phrased very differently initially. I'm glad to see this finally explained clearly and concisely, with experiments showing the effect of each choice. I do want to put special emphasis on the insight of AUP that the pursuit of convergent instrumental subgoals leads to large _increases_ in "ability to do things", and thus that penalizing increases can help avoid such subgoals. This point doesn't typically make it into the academic writings on the subject but seems quite important.

On the topic of impact measures, I'll repeat what I've said before: I think that it's hard to satisfy the conjunction of three desiderata -- objectivity (no dependence on human values), safety (preventing any catastrophic outcomes) and usefulness (the AI system is still able to do useful things). Impact measures are very clearly aiming for the first two criteria, but usually don't have much to say about the third one. My expectation is that there is a strong tradeoff between the first two criteria and the third one, and impact measures have not dealt with this fact yet, but will have to at some point.


Shielding Atari Games with Bounded Prescience

Mirco Giacobbe
Mohammadhosein Hasanbeig
Daniel Kroening
Hjalmar Wijk

In order to study agents trained for Atari, the authors write down several safety properties using the internals of the ALE simulator that agents should satisfy. They then test several agents trained with deep RL algorithms to see how well they perform on these safety properties. They find that the agents only successfully satisfy 4 out of their 43 properties all the time, whereas for 24 of the properties, all agents fail at least some of the time (and frequently they fail on every single rollout tested).

This even happens for some properties that should be easy to satisfy. For example, in the game Assault, the agent loses a life if its gun ever overheats, but avoiding this is trivial: just don’t use the gun when the display shows that the gun is about to overheat.

The authors implement a “bounded shielding” approach, which basically simulates actions up to N timesteps in the future, and then only takes actions from the ones that don’t lead to an unsafe state (if that is possible). With N = 1 this is enough to avoid the failure described above with Assault.

I liked the analysis of what safety properties agents failed to satisfy, and the fact that agents sometimes fail the “obvious” or “easy” safety properties suggests that the bounded shielding approach can actually be useful in practice. Nonetheless, I still prefer the approach of finding an <@inductive safety invariant@>(@Neurosymbolic Reinforcement Learning with Formally Verified Exploration@), as it provides a guarantee of safety throughout the episode, rather than only for the next N timesteps.


Learning to be Safe: Deep RL with a Safety Critic

Krishnan Srinivasan
Benjamin Eysenbach
Sehoon Ha
Jie Tan
Chelsea Finn

While there has been a lot of work on verifying logical specifications and avoiding constraint violations, I’ve said before that the major challenge is in figuring out what specifications or constraints to use in the first place. This paper takes a stab at this problem by _learning_ safety constraints and transferring them to new situations.

In particular, we assume that we have some way of telling whether a given trajectory violates a constraint (e.g. a human looks at it and says whether or not a violation has happened). We also assume access to a safe environment in which constraint violations are acceptable. For example, for robots, our constraint could be that the robot never crashes, and in the safe environment the robot could be constrained to only move slowly, so that if they do crash there is no permanent damage. We then want to train the agent to perform some task in the true training environment (e.g. with no restrictions on speed), such that we avoid constraint violations with high probability _even during training_.

The key idea is to pretrain a _safety Q-function_ in the safe environment, that is, a function **Qsafe(s, a)** that specifies the probability of eventually violating a constraint if we take action **a** in state **s**. We have the agent choose actions that are estimated to be on the verge of being too risky, in order to optimize for getting more information about the constraints.

Once we have this safety Q-function, we can use it as a shield (<@1@>(@Neurosymbolic Reinforcement Learning with Formally Verified Exploration@), <@2@>(@Shielded Decision-Making in MDPs@)). Specifically, any actions whose risk is above some threshold ε have their probabilities set to zero. Using this shield, we can then train for the true task in the (unsafe) training environment using RL, while only behaving safely. Of course, this depends on the safety Q-function successfully generalizing to the new environment. We also add the safety Q-function as part of the RL objective to disincentivize constraint violations.

Their experiments show that this approach significantly reduces the number of constraint violations during training, though in absolute numbers there are often still hundreds of constraint violations (or about 1% of the number of training steps).

I’m glad to see more work on this: robustness techniques seem particularly important to get working with learned specifications, and this paper (like the next one) takes a real shot at this goal. In some sense it isn’t that clear what we gain from an approach like this -- now, instead of requiring robustness from the agent, we require robustness from the safety Q-function (since we transfer it from the safe environment to the training environment). Nonetheless, we might hope the safety Q-function is easier to learn and more likely to transfer between the two environments, since it could be simpler than a full policy.


Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones

Brijen Thananjeyan*
Ashwin Balakrishna*
Suraj Nair
Michael Luo
Krishnan Srinivasan
Minho Hwang
Joseph E. Gonzalez
Julian Ibarz
Chelsea Finn
Ken Goldberg

This paper introduces Recovery RL, which tackles the same problem as the previous paper: given a dataset of constraint violations (presumably collected from some safe environment), train an agent to perform some task while exploring safely. Like the previous paper, it starts by learning a safety Q-function from the dataset of constraint violations.

The difference is in how this safety Q-function is used. The previous paper uses it as a shield on a policy, and also uses it in the RL objective to get a policy that is performant and safe. Recovery RL instead splits these into two separate policies: there is a task policy that only optimizes for performance, and a recovery policy that only optimizes for safety. During training, the safety Q-function is used to monitor the likelihood of a constraint violation, and if a violation is sufficiently likely, control is handed over to the recovery policy which then tries to make the probability of constraint violation as low as possible.

Experiments show that the method performs significantly better than other baselines (including SQRL, the method from the previous paper). Note however that the environments on which the methods were tested were not the same as in the previous paper.


Attainable utility has a subagent problem

This post argues that regularizing an agent's impact by <@attainable utility@>(@Towards a New Impact Measure@) can fail when the agent is able to construct subagents. Attainable utility regularization uses auxiliary rewards and penalizes the agent for changing its ability to get high expected rewards for these to restrict the agent's power-seeking. More specifically, the penalty for an action is the absolute difference in expected cumulative auxiliary reward between the agent either doing the action or nothing for one time step and then optimizing for the auxiliary reward.

This can be circumvented in some cases: If the auxiliary reward does not benefit from two agents instead of one optimizing it, the agent can just build a copy of itself that does not have the penalty, as doing this does not change the agent's ability to get a high auxiliary reward. For more general auxiliary rewards, an agent could build another more powerful agent, as long as the powerful agent commits to balancing out the ensuing changes in the original agent's attainable auxiliary rewards.

I am confused about how much the commitment to balance out the original agent's attainable utility would constrain the powerful subagent. Also, in the presence of subagents, it seems plausible that attainable utility mostly depends on the agent's ability to produce subagents of different generality with different goals: If a subagent that optimizes for a single auxiliary reward was easier to build than a more general one, building a general powerful agent could considerably decrease attainable utility for all auxiliary rewards, such that the high penalty rules out this action.


Vehicle Automation Report

Last week, the NTSB released a report on the Uber automated driving system (ADS) that hit and killed Elaine Herzberg. The pedestrian was walking across a two-lane street with a bicycle. However, the car didn't slow down before impact. Moreover, even though the environment was dark, the car was equipped with LIDAR sensors which means that the car was able to fully observe the potential for collision. The report takes a closer look at how Uber had set up their ADS and notes that in addition to not considering the possibility of jay-walkers, "...if the perception system changes the classification of a detected object, the tracking history of that object is no longer considered when generating new trajectories". Additionally, in the final few seconds leading up to the crash the vehicle engaged in *action suppression*, which is described as "a one-second period during which the ADS suppresses planned braking while the (1) system verifies the nature of the detected hazard and calculates an alternative path, or (2) vehicle operator takes control of the vehicle". The reason cited for implementing this was concerns of false alarms which could cause the vehicle to engage in unnecessary extreme maneuvers. Following the crash, Uber suspended its ADS operations and made several changes. They now use onboard safety features of the Volvo system that were previously turned off, action suppression is no longer implemented, and path predictions are held across object classification changes.

**While there is a fair amount of nuance regarding the specifics of how Uber's ADS was operating it does seem as though there was a fair amount of incompetence in how the ADS was deployed.** Turning off Volvo system fail-safes, not accounting for jaywalking, and trajectory reseting seem like unequivocal *mistakes*. A lot of people also seem upset that Uber was engaging in action suppression. However, given that randomly engaging in extreme maneuvering in the presence of other vehicles can *indirectly cause* accidents I have a small amount of sympathy for why such a feature existed in the first place. Of course, the feature was removed and it's worth noting that "there have been no unintended consequences—increased number of false alarms".

Jeff Kaufman writes a [post](https://www.lesswrong.com/posts/tTg4bn5rxHYqQJXhD) summarizing both the original incident and the report. Wikipedia is also rather thorough in their reporting on the factual information. Finally, *[Planning and Decision-Making for Autonomous Vehicles](https://www.annualreviews.org/doi/pdf/10.1146/annurev-control-060117-105157)* gives an overview of recent trends in the field and provides good references for people interested in safety concerns.


Bridging Hamilton-Jacobi Safety Analysis and Reinforcement Learning

Jaime F. Fisac*
Neil F. Lugovoy*
Vicenc Rubies-Royo
Shromona Ghosh
Claire J. Tomlin

Reinforcement learning is not great at enforcing constraints that hold at all times, because the agent would violate a constraint now if it would lead to higher reward later. In robust optimal control theory, we maximize the **minimum** of the constraint reward over time to avoid this. We can do this in the Bellman equation by taking a minimum between the current reward and estimated future value (instead of summing), but this does not uniquely define a fixed point. Just as in regular RL, we can use discounting to avoid the problem: in particular, if we interpret the discount as the probability that the episode continues, we can derive a Safety Bellman equation for which Q-learning is guaranteed to converge. They demonstrate their method in classic control environments as well as half-cheetah, with a range of RL algorithms including soft actor-critic (SAC).

I really like how simple the change is here -- it should be a one-line change for many deep RL algorithms. Previously, we had to choose between unconstrained agents for high dimensional problems, or constrained agents for low dimensional problems -- I like that this work is making progress on constrained agents for high dimensional problems, similarly to [Constrained Policy Optimization](https://arxiv.org/abs/1705.10528). While this work doesn't involve a performance reward, you could use the resulting safe policy in order to guide a process of safe exploration to learn a policy that safely optimizes a performance metric. Of course, this is all assuming a specification for the constraint to satisfy.


Shielded Decision-Making in MDPs

Nils Jansen
Bettina Könighofer
Sebastian Junges
Roderick Bloem

Given a model of an MDP, we can compute a _shield_, which restricts the actions available to an RL agent to only the ones that can achieve at least some fraction of the optimal value. This results in safe exploration (since catastrophes would fall under the level that the shield guarantees), and also improves sample efficiency, since you no longer have tons of episodes in which the agent gets a large negative reward which only serve to teach it what not to do. They evaluate their approach on Pacman.

They require quite a lot of modeling in order to do this -- I think that it's specific to a particular kind of MDP, where there is an agent, and adversaries (the ghosts in Pacman), that are traversing a graph (the maze), which can have tokens (the food pellets). In theory, you should just solve the MDP and not use RL at all. Also in theory, shielding would actually require you to do this (in order to calculate the optimal values of actions), in which case it seems pointless (just use the optimal policy instead). In practice, the shield is only computed over a few timesteps. So you can think of this as a way of combining explicit, computationally-expensive forward reasoning (as in value iteration, for example) with RL, which learns from experience and can scale to much longer time horizons.

From the perspective of safety, I would be a lot more interested in approaches based on formal verification if they could work with learned features, rather than requiring that the human accurately formally model the world. This seems doable using a framework similar to [Trial without Error: Towards Safe Reinforcement Learning via Human Intervention](https://arxiv.org/abs/1707.05173), except by getting a formal safety specification iteratively instead of learning to mimic the human shield with neural nets.


Overcoming Clinginess in Impact Measures

In their [previous post](https://www.lesswrong.com/posts/H7KB44oKoSjSCkpzL/worrying-about-the-vase-whitelisting), TurnTrout proposed a whitelisting approach, that required the AI not to cause side effects not on the whitelist. One criticism was that it made the AI _clingy_, that is, the AI would also prevent any other agents in the world from causing non-whitelisted effects. In this post, they present a solution to the clinginess problem. As long as the AI knows all of the other agents in the environment, and their policies, the AI can be penalized for the _difference_ of effects between its behavior, and what the human(s) would have done. There's analysis in a few different circumstances, where it's tricky to get the counterfactuals exactly right. However, this sort of impact measure means that while the AI is punished for causing side effects itself, it _can_ manipulate humans to perform those side effects on its behalf with no penalty. This appears to be a tradeoff in the impact measure framework -- either the AI will be clingy, where it prevents humans from causing prohibited side effects, or it could cause the side effects through manipulation of humans.

With any impact measure approach, I'm worried that there is no learning of what humans care about. As a result I expect that there will be issues that won't be handled properly (similarly to how we don't expect to be able to write down a human utility function). In the previous post, this manifested as a concern for generalization ability, which I'm still worried about. I think the tradeoff identified in this post is actually a manifestation of this worry -- clinginess happens when your AI overestimates what sorts of side effects humans don't want to happen in general, while manipulation of humans happens when your AI underestimates what side effects humans don't want to happen (though with the restriction that only humans can perform these side effects).


Worrying about the Vase: Whitelisting

It's really hard to avoid negative side effects because explicitly listing out all possible side effects the agent should avoid would be far too expensive. The issue is that we're trying to build a blacklist of things that can't be done, and that list will never be complete, and so some bad things will still happen. Instead, we should use whitelists, because if we forget to add something to the whitelist, that only limits the agent, it doesn't lead to catastrophe. In this proposal, we assume that we have access to the agent's ontology (in current systems, this might be the output of an object detection system), and we operationalize an "effect" as the transformation of one object into another (i.e. previously the AI believed an object was most likely an A, and now it believes it is most likely a B). We then whitelist allowed transformations -- for example, it is allowed to transform a carrot into carrot slices. If the agent causes any transformations not on the whitelist (such as "transforming" a vase into a broken vase), it incurs a negative reward. We also don't have to explicitly write down the whitelist -- we can provide demonstrations of acceptable behavior, and any transitions in these demonstrations can be added to the whitelist. The post and paper have a long list of considerations on how this would play out in a superintelligent AI system.

Whitelisting seems like a good thing to do, since it is safe by default. (Computer security has a similar principle of preferring to whitelist instead of blacklist.) I was initially worried that we'd have the problems of symbolic approaches to AI, where we'd have to enumerate far too many transitions for the whitelist in order to be able to do anything realistic, but since whitelisting could work on learned embedding spaces, and the whitelist itself can be learned from demonstrations, this could be a scalable method. I'm worried that it presents generalization challenges -- if you are distinguishing between different colors of tiles, to encode "you can paint any tile" you'd have to whitelist transitions (redTile -> blueTile), (blueTile -> redTile), (redTile -> yellowTile) etc. Those won't all be in the demonstrations. If you are going to generalize there, how do you _not_ generalize (redLight -> greenLight) to (greenLight -> redLight) for an AI that controls traffic lights? On another note, I personally don't want to assume that we can point to a part of the architecture as the AI's ontology. I hope to see future work address these challenges!