Sal Khan, Founder and CEO, Khan Academy Moderator: Jeremy Kahn, FORTUNE


Brainstorm AI 2023: How AI Is Transforming Education

Conferences

According to a sales force study, more than 70% of Gen Z report having used generative A I with such a rapid adoption rate. It is no surprise that the education system has had to address concerns around the use of gen A I for cheating and plagiarism. Our next speaker believes that instead of fearing the misuse of A I in education, we should embrace its potential. And with the right guard rails, an approach, artificial intelligence can become a personal A I tutor for every student and an A I teaching assistant for every teacher here to talk about his latest project. Khan Migo. Please welcome Khan Academy founder Sal Khan. He'll be interviewed by Fortune senior writer and brainstorm A I CO chair, Jeremy Kah. Thank you. So, thanks for joining us. Thanks for having me. Um Why don't you start by explaining a little bit about what Conmigo is for those who aren't familiar with it. Yeah, I was just introduced at a very high level, think of a tutor for every student, a teaching assistant for every student. You know, at first we just, if you think about what Khan Academy is a lot of folks know it started with me tutoring cousins almost 20 years ago and I start with one cousin, words, friends of my family, free tutoring is going on before I know it. I'm working with 1015 cousins. I started writing software for them to work at their own time and pace. I start making videos for them and, and if you think and you fast forward to even pre generative A I it's really been about how can we scale what I was originally able to do with Nadia. And so when we saw what was possible over a year ago, you open A I showed us GP T four summer of 2022 we said, wow, this could take us that much further. And so our original view was, wow, this, if a student is struck stuck on something on Khan Academy, it can help them with that if they're watching a video and they're like, why do I need to learn this? It could help them with that. We started to realize it's valuable for a teacher to do things like lesson planning, grading papers, writing progress reports. And the more we've worked on it, we realize it can do things that would even transcend a traditional tutor. It could emulate. We have simulations where you could talk to you or the donkey or Jay Gatsby um simulations you, you could talk to Alexander Hamilton or, or Thomas Jefferson. Uh We just launched last week where it won't write your essay for you. But uh you can get feedback on your essay and it's like working on a Google Doc with, uh with your editor. Uh when, when you're trying to make a paper better, uh So, so big picture, we're realizing almost every day this can go do more and more of the things that one, we were always trying to scale that a good tutor, teaching assistant would do. But we're starting to realize it can go pretty far beyond that. And how do you make sure the students aren't using it to cheat? Because I think that was one of the big fears and, and a lot of people were using chat EPT to cheat when it first came out. Yeah. And, and this is a, you know, that first weekend we had access, we're like, this could be used to cheat. Um, but it has all this other power. And so from the beginning, we had all the debates inside the organization. We're like, ok, well, first of all, we can set it up so that Conmigo itself does not cheat. So if you go to Conmigo and you're a student, there's two modes. If you're a teacher, you can put it in teacher mode and then it will help you learn how to explain it and things. But if a student says, tell me the answer, it will say I'm your tutor. I'm not going to tell you the answer how do you think you should approach the problem? On top of that, there's fears around just students getting into unproductive conversations. You know, we, we've already deployed this with tens of thousands of kids in real schools and I mean, just a stereotype 1/7 grade boy, uh will immediately. Right. Let's call it Red Hat test it in ways that very few of us would imagine. Uh So we have guardrails and the A I itself, we have a second A I that's moderating it. All of the conversations are transparent to parents and teachers. If the, if the moderating A I thinks that the conversation is shady, it'll shut it down and it'll actively notify the teachers and the administrators. So there's a lot of guard rails there. And then for the, you know, like this essay tool where we already launched the essay tool where we're going is roughly by back to school. Probably sooner a teacher is going to be able to make an assignment through Conmigo, probably develop it with Conmigo as well. Conmigo will then tell the student, hey, let's work on this paper together. I'm not going to do it for you, but I'm gonna be your coach. I'm gonna give you feedback and then when the students and then the student can submit it and, and maybe even the teacher can get some preliminary grading and things like that and what Conmigo is going to do and is already doing to some degree. It won't just give the teacher here's the essay that Jeremy wrote. It'll say here's the essay we worked on it for 4.5 hours. Jeremy had a little trouble coming up with the thesis statement. We eventually got there. And by the way, in your class of 30 about 15 of the kids had trouble coming up with the thesis statement. But icon Migo a confident it is Jeremy's work because we did it together. The writing that was produced is consistent with the other writing that I've seen of Jeremy, especially the writing that he's done in the classroom. And if you were to go to chat GP T and even before generative A I, there's plenty of websites on the internet that you could pay $5 a page and they'll hire someone in Kenya to write your paper for you. This is real. Um If you did that and you copied and pasted it in Conmigo, tell the teacher, like, I don't know where this paper came from. It just showed up, it looks shady and it's completely inconsistent with what Jeremy's been doing in the classroom. So instead of being the end of the college essay, this can actually preserve it potentially and, and extend that process and make it interactive and even go, potentially, even go beyond the, I think the essay will always have real value. Uh But often it could be a simulation. Now, it could be a conversation, an interview that you have with the A I instead of just a static. How do you make sure the information that Kono is giving the student is accurate? Because again, if you're using an LM, this has been one of the big concerns is that they'll invent information, they'll hallucinate I think for step one, anyone making an app, especially for students. But really for anyone, you have to be completely up front and say like this is not an infallible source of information. So even on Conmigo, there's things like Conmigo can sometimes make mistakes now with that said, especially who we are and what we're trying to do, we are trying to minimize that as much as possible. A big way we've been able to do that and we have a whole bunch of evaluations to measure the hallucination rate and the math error rate, anchoring it on Khan Academy content. When you anchor it on content that already exists, your hallucination rates almost go to zero. We have a couple of use cases where it isn't quite as anchored. But even there a lot of work to make sure it gets better and better on the math side of things. And this is stuff we worked on October of last year. Prompt chaining, being able to do math in the background. We, we worked a lot with a lot of the open A I researchers on ways to do multiple calls to the A I to significantly bring down the math error rate. So already we feel pretty confident and we're seeing every successive generation of models are getting better and better and you know, some of the what we're, what you're now starting to see with models, you know, with, with Google's Gemini, we'll start seeing what Gemini Ultra can start doing in, in about a week's time. Um You're going to have, you know, there's rumored models we all know about open A I. So I think that's, you know, by the end of this year it'll probably be even, it will be a lot better. Now, you guys were built on open A I GP T four when they had a sort of fiasco as Altman was fired, you know, no one knew what was going to happen. All the employees were going to leave. Were you worried? I mean, what, what was going on at Khan Academy at the time? Yeah, I mean, I was intrigued by what was going on. Um I uh very quickly we got reassured. We got, I'm sure they reassured many folks, but very quickly we got reassurances that there would be no interruption of service, etcetera, etcetera. At the same time, we did reach out there are GP T four end points you could get directly from Microsoft. So we did reach out to, you know, the highest levels of contacts we had at Microsoft to make sure that we wouldn't have an interruption of service. So I, I didn't lose a ton of sleep over it and thank God it's all kind of come back to we think normal now. But um yeah, it was, it was an interesting, I mean, for everyone here, it was an interesting couple of days. I've got some more questions. I wanted to get some questions from the audience. If there are any, anyone have a question for yourself from the audience, please raise your hand and we'll try to get a mic to you. There's one right here, this woman here, if you could please stand up and identify yourself when the mic gets to you. Hi, I'm Shanaz. D. I work at the post venture. I used to work with Sal too. Um Quick question with Con me go. Does this mean you're gonna have less videos generated on the con side or is it gonna be complimentary? How do you view that? Yeah, I think it's a very fascinating question. Long term, you know, in the, in the short term, I think short term, medium term, I think they're extremely complimentary. I think as we mentioned, like the hallucination question, you know, what can a Khan Academy video do? Well, it's, it's able to, you know, talk about it the way I did it with my cousins, etc, etc, what's its limitations that people have always known about? Well, what if a student says, why do I have to learn this or they have a follow up question, etc, etc well, now the generative A I has the context. It could either refer you to other content or it'll in that moment answer the question. We've actually now done it. So it has memory on you and, but that memory is completely transparent to the user, to teachers so they can reset it, they can delete it. They're like that inference isn't right. But if you go on to Conmigo and you say, why should I have to learn this? It'll say, well, what do you care about? And you say, oh, you know, I want to be a doctor one day and so it'll be able to connect it and then it'll use that information in the future to hopefully try to keep you engaged. Now, a big conversation we've been having internally at Khan Academy and I'm someone who's personally and I made videos yesterday. I personally made 7000 videos. So I don't want them to be all irrelevant in five years. But um I've come to terms with they, they might be a transition step uh in three years, four years, five years, it's very plausible that Conmigo is going to essentially create a video on the fly for you that it's not really a video anymore. It's an interaction and you're going to be able to talk to it. It's just gonna feel like you're blackboard together like a Khan Academy video. So we'll, we'll see where that goes. We're we're definitely trying to, I tell the team at Khan Academy every day, like we should, we're canceling projects and adding projects on a daily basis. We're canceling the ones that we think are going to be irrelevant in three years because of this dynamic. And we want to work on the ones that will only become more and more relevant as the models get better. Great other questions. Do you have any other questions? I've got a question for. So which is, do you think this technology ultimately will help inequality? Because you're going now really be able to put a personal tutor in every student's hand or is this going to be something where you know, the better off kids, they're going to know how to use this technology, they're going to have, you know, a mobile phone that they can run it on and, and essentially they're just going to pull ahead and it will actually exacerbate inequality. So yes and yes, I think you're absolutely going to see all of our Children adopting this first. Now with that said we are, you know, on the drive up here, I was talking to some states who are talking, who are talking about adopting Conmigo for the state. So we're talking to a lot of districts, I think by this time next year, we're going to be looking at many hundreds of thousands of kids in mainstream public schools, if not millions using this as part of their day to day. So that will help drive. But with that said, I think everyone in this room, you and your Children are going to be on the bleeding edge of that. But what I always tell folks is and look, some of this is true of even classic Con Academy. There's much more awareness in this room of it than if I were to go into more underserved parts of the country. And we've obviously been trying to work on that. But I always tell people, you don't look at whether it's helping or driving inequality by looking at the proportion who's using it. You think about it by saying what happens if it disappears. So I could probably, you know, we're right next to the Museum of Modern Art right here. If I just observed who's going in and out, it's probably 80% people who are educated, people who are affluent who are using it. Does that mean that it's driving inequality? Well, I would say what happens if that museum disappears. If that museum disappears, everyone with resources will fly to another museum or look at art books or find some other way to get exposure while that 20% is going to have nothing. And so we say the same thing about something like Conmigo, if Conmigo didn't exist, people are going to find another tool, you're going to get a paid tutor, you're going to do whatever. But there's that 40 50% of kids. If it disappears, they're no longer gonna have something that can advise them on college, something that can help with their essay, uh, something that can tutor them. Great. I want to get one more question and, uh, this gentleman here, thank you. First of all, thank you, have a parent and uh, we thank you for all that you do for our kids. And uh, thank you very much for all the math. He help that you've given me over the over the years. Um Quick question on Yeah, you good. Your brand is built on trust. We send our Children to Khan Academy, whether it's the app or the website, um teachers, et cetera. So how do you, what advice do you have for brands who are in the information business and the trust business? How do you think about the risk reward? Why did you jump in so early um in an environment where there's a lot of trust for you to lose if you don't get it? Right. Yeah. And it's fascinating because I, I even before a I, I used to point out to our team and even, you know, our philanthropist, I was like, look edu education. It tells you something about the space that there aren't a lot of brands in the space. No one not to pick on anyone but like no one says, oh, that's a Pearson textbook. It's going to be awesome, right? Um Some of them might be but I, I'm just saying, you know, there's not, it's not, it's known, but it's not a brand in that way. And, and I, I hopefully we have built that people say, oh, it's approachable. It's quirky, it's connected, but it's high quality. It's conceptual. We had those debates inside our, our own organization this time last year. Um You know, and what I just told the team is, you, you know, we can't just run from this. Our brand is also about innovation. And so let's put the guardrails in place every one of our fears about brand or otherwise, let's write it down and let's turn them all into features. And, you know, I talked a lot about them and I remember when chat GP T came out that day, I immediately slacked Greg Brockman and I'm like, what's going on? You guys have us under an ND A. We're not supposed to release until March of 2023. You just launch something. And Greg's like, oh, this isn't a new product. It's just a chat interface on GP T 3.5. It's been out for months and the world has just exploded. And I was like, oh, people are going to throw out the baby with the bathwater. We're working on all this stuff. All these school districts are starting to ban it because of cheating. In hindsight, it was a blessing because everyone allowed them to process a very imperfect tool and frankly, by about March. They start saying if only someone we could trust puts the right guard rails around it, avoids cheating, makes it transparent, helps solve the bias the data. And then we're like, here you go. And honestly, we have, that's great. I'm going to have to stop you there because you're right out of time. But I thought fears to features is a, is a great takeaway. Thank you so much Saul for joining us and thank you all for listening. Thank you.