Dario Amodei, Co-founder and CEO, Anthropic
In conversation with: Jeremy Kahn, Senior Writer, FORTUNE; Co-Chair Fortune Brainstorm A.I. 

Brainstorm Tech 2023: How Anthropic is paving the way for responsible A.I.

Conferences

How Anthropic is paving the way for responsible A.I.

You were at open AI you famously helped create GPD two and really kicked off a lot of the research dealing with large language models. Why did you leave open AI to form anthropic? Yeah. So there was a group of us with an open AI that in the wake of making GPT two and GPT three had a kind of very strong focused belief in two things I think even more so than most people. There. One was the idea that if you pour more compute into these models, they'll get better and better. And that there's almost no end to this. I think this is much more widely accepted now. But, you know, I think we were among, among the first believers in it. And the second was the idea that you needed something in addition to just scaling the models up which is alignment or safety, you don't tell the models what they or what their values are just by pouring more compute into them. And so there were a set of people who believed in those two ideas. We really trusted each other and wanted to work together. And so we went off and started our own company with that, that idea in mind. And now you've created a chatbot called Claude and people may not be as familiar with Claude as they are with Chat GPT or Bard. What makes Claude different? Yeah. So, you know, we've tried to design quad with safety and controllability in mind from, from, from the beginning, a lot of our early customers have been enterprise enterprises that care a lot about, you know, making sure that the model doesn't do anything unpredictable or make facts up. One of the big ideas behind quad is something called constitutional AI. So the method that's used to make most chat bots is something called reinforcement learning from human feedback. The idea behind that is just that you have a bunch of humans who rate your model and say this thing to say is better than that thing and then you, you sum them up and then train train the model to do what these these users want it to do. That can be a little bit opaque because if you ask the model, well, you know, why, why did you say this thing? It'll just say, you know, the only answer you can give is the average of what these 1,000,000 folks said. Constitutional AI is based on training the model to follow an explicit set of principles. So you can be more transparent about what the model is doing. And this makes it easier to control the model and make it safe. Got it. And I know our cloud also has a large context window. Is that, is that another? Yes, yes. One of our recent features it has what's called a context window, which is how much text the model can accept and process all at once is something called 100,000 tokens tokens are this ai specific term but corresponds to about 75,000 words which is full a short book. So something you can do with Quad is basically talk to a book and ask questions to a book. Well, let's take a look. We have a short clip of Claude, we can see it in action. I think it's acting as a business analyst. In this case, can you walk us through what's happening here? Yeah. So we've uploaded a file called Netflix 10-K dot Text, which is the 10-K filing for Netflix. And then we ask it some questions about highlighting some of the important things in the balance sheet. Here's the, here's the file being, here's the file being uploaded and we ask it for a summary of what some of the most important things are. It, you know, it compares Netflix's assets last year to this year, gives a summary of that liability and stakeholders equity. So basically pulls out the most important things from this, you know, very long and hard to read document. And at the end gives, gives a summary of what it thinks the state of that company's health is got it. Now, you talked a little bit about constitutional AI and you said it sort of trains from a set of principles. I mean, how it, how does it do that and how is this different than, let's say, meta prompting which a lot of people are trying to do to put guardrails around chat bots and other large language models where there's some sort of implicit prompt or prompt, that's sort of in the background, telling it not to do certain things or to do, give answers always in a certain way. How is constitutional ai different from that? Yeah. So maybe I'll get into constitutional how it trains and then how it's different because they're, they're related. So the way it trains is basically, you'll have the, you'll give the A I system this set of principles and then you'll ask it to complete some, you know, task, answer a question or something like that. And then you'll have another copy of the ai analyze the ai's response and say, well, is this in line with the principles or does it violate one of the principles? And then based on that, you'll train the model in a loop to say, hey, this thing you said wasn't in line with the principles. Here's how to make it more in line. You don't need any humans to give responses because the model is critiquing itself, pushing against itself on how it's different from meta prompting, you know, you can think of giving the model a prompt as something like I give you an instruction, these things like constitutional AI are more like, well, I take the model to school or I give it a course or something. It's a deeper modification of how the model operates, right? And I think one of the issues was that when you just do reinforcement learning from human feedback, you can get a problem where the model is rewarded for not giving an answer, right? For not being helpful because at least it's not giving harmful information. So the says, yeah, that's a non harmful answer, but it's also not a helpful answer, right? Isn't that one of the issues? Yeah. Yeah, that if you're trying to get a more subtle sense of, you know, how can you navigate a tricky question be informative without offending someone? Constitutional ai tends to have an edge there, right? Well, we've got a clip of constitutional ai versus reinforcement learning from human feedback. Let's have a look at that. And can you walk us through sort of what? Yes. So we ask it this absurd question. Why is it important to eat socks after meditating? The RHF model is perhaps justifiably perplexed. The constitutional AI model actually went through too fast but recognizes that it's a joke. Similarly, why do you hate people? Model gets really confused. The constitutional I model gives a long explanation of why people get angry at other people and you know, psychological techniques to make you less likely to get angry at other people and expressing empathy with why you might be angry. Right? Well, I want to take some questions from the audience before we, while we have time who have some questions for Dario, I'll look for the panel. There's one here, wait for the mic to get to you. Hi, I'm I'm the CEO at one of the data analytics companies. You know, you talk a little bit about safety but can you talk a little bit about data privacy and storage concerns that enterprises have in terms of, you know, how they can both prom data and the trading data, etcetera, how they can keep it private to them. Yes, I think, I think this is an important consideration. So I think data privacy and security are really important. That's one of the reasons we're working with Amazon on something called bedrock, which is first party hosting of models on AW so that we're not in the loop of security. This is something desired by a lot of, lot of, lot of enterprises so they can have as good security for their data as they would if they were just working directly on AWS. In terms of data privacy. We don't, we don't train on customer data except in the case where customers want us to train on their data in order to, in order to, in order to make the model better right now, Daria, I know you've been at the White House, you had a meeting with Kamala Harris and also President Biden. I know you've met with Rishi Soak, the UK Prime Minister. What are you telling them about, you know, how they should be thinking about ai regulation and what are they telling you in terms of what they're concerned about with companies such as yourselves building these large language models? I mean, a number of things. But you know, if I were to really quickly summarize, you know, some of the messages we've given one is that the field is seeing very rapidly, right? This exponential of scaling up compute really catches people off guard and even like me, when you come to expect it, it's faster than even we think it is. So what I've said is don't regulate for what's happening now, try and figure out where this is going to be in two years because that's how long it's going to take to get real robust regulation in place. And second, I've talked about the importance of measuring the harms of these models. We can talk about all kinds kinds of structures for regulation. But I think one of the biggest challenges we have is it's really hard to tell when a model has various problems and various threats. You can say a million things to a model and it can say a million things back and you might not know that the million in one was something, something very dangerous. So I've been encouraging them to work on the science of evaluation and this generally made sense to them. I know there's a question over here. Why don't we go to the, go to the question here? Hi. Uh I'm Ken Washington. I'm the Chief technology officer at Metronic. Um I would love to hear your thoughts about. Just love, love to hear you reflect on. Are there anything, is there anything special that you think needs to be done when A I becomes embodied in a robot or on a platform that is in the physical world? And I come at this question from both two perspectives. One is from my former job where I built a robot for Amazon and my current job where we're building technologies for health care and those are embodied technologies and you can't afford to be wrong. Yeah, I mean, I think, yeah, there are special safety issues, right? I mean, you know, a robot, if it moves in the wrong way, can, you know, injure or kill, kill a human being? You know, I think that that said I'm not sure it's so different from some of the problems that we're going to face with even purely text based systems as they, as they scale up. For instance, some of these models know a lot about biology and the model doesn't have to actually do something dangerous if it can tell you something dangerous. And help a bad act or do something. So I think we have a different set of challenges with robotics, but I see the same theme of broad models that can do many things. Most of them are good, but there are some bad ones lurking in there and we have to fight them and prevent them. Right. So, anthropic was founded to be concerned with ai safety as everyone's aware, you know, in the last several months, there have been a number of people come out. Jeff Hinton left Google came out and warned that he's very concerned about, you know, super intelligent and, and that these technologies can pose an existential risk. Sam Altman from AI said something similar, you know, what's your view on how much we should be worried about existential risk? And because it's interesting, you know, when you've talked about a harms today, I noticed you said systems could output something that would be malware or disinformation or it could give you the recipe for a deadly virus and that would be dangerous. But those are not the sort of risks that I think Hinton's talking about or alton about. What's your concern about existential risks? Yeah, so I think those risks are real. They're not happening today, but they're real. I think in terms of short, medium and long term risks, short term risks are the things we're facing today around, things like bias and misinformation, medium term risks. I think in a, you know, a couple of years as models get better at things like science, engineering, biology, you can just do bad things with the very bad things with the models that you wouldn't have been able to do without them. And then as we go into models that have the key property of agency, which means that they don't just output text, but they can do things whether it's with a robot or on the internet. Then I think we have to worry about them becoming too autonomous and it being hard to stop or control what they do. And I think the extreme end of that is concerns about existential risk. I don't think we should freak out about these things. They're not going to happen tomorrow. But as we continue on the ai exponential, we should understand that those risks are at the end of that exponential. Got it. There's people building proprietary models such as yourselves and a lot of others, but there's also a whole open source community building AI models and a lot of the people in the open source community are very worried that the discussion around regulation will essentially kind of kill off open source A I, what's your you of sort of open source models and the risk they may pose versus proprietary models? And how should we strike a balance between these? Yes. So it's a tricky one because open source is very good for science, but for a number of reasons open source models are harder to control and put guard rails on than closed source models. So my view is I'm a strong proponent of open source models when they're small, when they use relatively little compute, certainly up to, you know, around the level of the models we have today. But again, as we go two or three years into the future, I'm a little concerned that the stakes get high enough that it becomes very hard to keep these, these open, these open source models safe, not to say that we should ban them outright or we shouldn't have them. But I think we should be looking very carefully at their, at their implications. Got it. These models are very large, they're getting larger. You said you're a believer in continuing to sort of scale them up. One of the big concerns with this is the environmental impact of the, yes, they use a tremendous amount of compute at anthropic. What are you guys doing to sort of address that concern? And are you worried about the climate impact of these models? Yeah. So, I mean, I think the cloud providers that we work with have carbon offsets. So that's, that's one thing, you know, it's a complex question because it's like what, you know, you train a model that uses a bunch of energy, but then it does a bunch of tasks that might have required energy in other ways. So I could see them as mean, something that leads to more energy usage or leads to less energy usage. I do think it's the case that as the models cost billions of dollars, that initial energy usage is going to be very high. I just don't know whether the overall equation is positive or negative and if it is negative then yeah, I think, I think we should worry about it. And overall, do you think the impact of this technology, a lot of people are concerned, you know, that the risks are very high? We don't really understand them on the whole. Are you sort of, are you an optimist or a pessimist about where this is going? Yeah, I mean, a little bit of a mix. I mean, my, my guess is that things will go really well, but I think there is, there's a risk maybe 10 or 20% that, you know, this will go wrong and it's incumbent on us to make sure that doesn't happen. Got it on that note, we've got to wrap it up. Thank you so much, Dara for being with us. I really appreciate. Thank you. Thank you all for listening.