Oren Etzioni on the evolution of AI agents, bounded autonomy, augmenting science, and democratizing capabilities (HAI Ep55)

Transcript

Ross Dawson: Oren, it is awesome to have you on the show.

Oren Etzioni: Thank you, Ross. The pleasure is all mine.

Ross Dawson: So, 1994, 32 years ago, you set up the Internet Softbots project, which was, you know, arguably the first solid agentic AI project. So we had agents. A lot of people don't realize we had agents all that time ago, and you've obviously been deeply involved in that all through that time.

So what has changed and what has stayed the same? Because it strikes me that actually some of the elements which were there at the beginning are still very similar today.

Oren Etzioni: Well, thank you for having such a long memory and being so well read. I do take a lot of pleasure in thinking back to those good old days. What's remained the same is that we had the notion of an agent and how the agent could empower people, as well as some of the potential risks, right, of an agent going rogue.

So I think we had the landscape right. What we did not have are two technologies, which are now much more mature. The first one is the technology to drive the agent—to have it be effective and do its thing, understand the world around it, come up with a sequence of actions to take, and so on. So our engines back in '94 and later were pretty clunky.

The second thing that we didn't have, and that's what really led us to ultimately abandon that line of research, at least for a while, is the language to talk to the agents. If you try to communicate with your agents in a formal, logical language, well, how is that better than writing a computer program? It turns out to be very complicated, and I came to realize that the appropriate way to talk to your agents is using natural language. It's natural for people. It allows some of the expressiveness that we need to capture the nuances of what we want to tell the agent. But NLP, as it's called—natural language processing—wasn't up to the task.

So I shifted from working on agents to working on natural language processing, using it to give agents some of the requisite background knowledge that they needed. Fast forward to 2023 and now 2026, we have that technology. Computers, in a very strong sense, can understand language.

Ross Dawson: So the idea of an agent is that it acts on behalf of a user. So there's a relationship between, well, let's say a human and an AI agent, and so everything is really about that relationship. And you said it a moment ago—the language so that a human can express themselves and hopefully the agent reflects that.

So how do we manage that relationship?

Oren Etzioni: I think, first of all, you're exactly right. This is the right metaphor to think about this. It's a relationship, and as with any relationship, there are many pitfalls. So you don't want, for example, the person to somehow become a slave to the agent, right? The agents are supposed to work for us.

Sometimes I felt, particularly before we had more active agents, we were dealing with chatbots. I would ask the chatbot how to fix something on my Windows machine, and it would give me this long list of instructions, and then I would tell it, "Okay, so execute that. You've got a list of instructions. Do I really need to click here and move there and so on?" And it would say the equivalent of "I can't do that, Dave," because it wasn't actually an agent. It was just a chatbot.

Now, thankfully, we have agents that can do that. But even there, the agent sometimes says to me, "Okay, this was blocked for me. Can you go and verify that?" And my first reaction is, I don't work for you. You work for me. Why don't you go and verify that? And we sometimes have a little friendly tug of war—who's going to do this? My perspective is, if the agent can do it, it should do it. I'm busy being human.

So I think there's a question of this balance of who works for whom. Another aspect of the relationship, which is really important, is responsibility. So if something goes wrong, can I say my agent did it? It's not my fault. You know, my self-driving car ran over your dog. It's not my fault. That doesn't work. I think it's very important that the responsibility for what we do and for what the tools that we employ do rests with us.

The responsibility, the liability, and so we ought to make sure that our agents are doing the right thing. Now, the last thing I want to point out too is a different kind of trap in this relationship, where you ask the agent for advice or you ask it to do things on your behalf, and you're responsible, but it keeps doing so well. It does it right. It gets you all this information, and after a while, there's something sometimes called capitulation, where you're like, okay, I'll just do whatever it says. It's right. It knows what it's doing, and that can be disastrous in its own way. We have to retain vigilance. We have to retain judgment in these cases.

And just one more point: it's been documented that sometimes people in the legal justice system who are using AI systems started listening to the AI too much. The AI didn't get everything in the case right. The AI captured some bias, and they just listened to the AI because nine times out of ten it was right. The tenth time, what's called the jagged frontier, it can be very wrong. So it's very important again to have the right balance here between who listens to whom and who trusts whom.

Ross Dawson: Yeah. Well, I don't think there's any simple answers here, but you know, as I often say, and others say, only humans can be accountable, and maybe in some distant future that'll be different. But I think that's a decent principle for the meantime.

But this goes to this point where, as you say, the human still has to be accountable. It has to exercise judgment. Very often, people say, "Okay, well, it's doing a great job. Why do I need to keep on checking it?" So I think there's two parts. One is obviously human education, whatever you call it—just getting people in the right frame of mind. The other is obviously systems design. So, what in either of those domains can we do where humans do maintain real accountability as opposed to just clicking a checkbox?

Oren Etzioni: So, as you said, Ross, I do think that education is extremely important. These systems are new, and we are still learning how to use them and how to use them effectively. As a simple example, when it comes to writing—they're so good at writing, and some of us are good writers and enjoy it, some of us are not great writers and hate it. I think that it's a real mistake to just tell the AI, "Oh, write this for me," and not even read it for something, whether it's an email or a memo or an article that goes out under your name.

A much more useful thing is to use the AI, for example, as a coach or as a thought partner, and to ask it to critique your own writing, to bring in related work, to bring in ideas, to actually question you about your arguments in the process of making them stronger and tighter. That process can go from what's called AI slop, where you cheaply use AI to create just really garbage slop, to what I like to call AI cream, where in the process of working with the AI, you actually come up with a better product than you would have on your own—the same way that we work with a human collaborator and can end up with a better memo or better paper, what have you.

Ross Dawson: So, absolutely agree. I have this spectrum from cognitive corruption through to cognitive augmentation, and obviously we're trying to push people up to that higher level. But just focusing on the interface design—one of the things, obviously, is people have to realize, "Okay, this is the way I should be using it. I can use it that way, and it's going to be really useful if I do."

But how do we design interfaces in a way that they pull people towards that kind of AI as coach or thought partner relationship, as opposed to just "tell me what to do"?

Oren Etzioni: It's a fantastic question, and I think that we are very early in the design of these interfaces. So if you think about it, when you come and sit down with Claude or ChatGPT, what have you, what you see is not that different from the search box that Google got us used to. There's a box, and you type into it. We've learned with Google, we just type keywords; here, we type questions, we can type more elaborate things. But the interface is extremely primitive.

Let me suggest a couple of things directionally. I don't have a design to give you, but just to show how far this can go. First of all, having it be visual as opposed to just text back and forth, and maybe you'll show me a chart. Why can't it be visual? Why can't it be dimensional? Why can't I interact with it as I'm walking around? Right now, we have voice, but imagine that there was a halo or some projection of the chatbot, and it could point to things, and I could point to things. There's just so much there beyond a text box.

Another example—I was alluding to the jagged frontier, the fact that it can periodically suddenly slide from a great answer to just a total error. What if it was checking the extent to which I was listening to it unquestioningly without pushing back, and what if it regularly inserted checks to see that I'm awake? Kind of like in a car, it makes sure you're awake. Little mistakes, just to see whether I'm asleep at the wheel or whether I'm getting it. I'm told that the people who do scanning for luggage and so on, they periodically put in fake guns or contraband substances just because it's pretty rare that you come across somebody who sticks a gun in their carry-on that you're scanning. They put those just to make sure that they're still looking, that they're still awake.

So that principle, for example, of injecting mistakes and checking that we catch them so that we stay awake is one of many that we could invoke to make this interaction—to build an interface that allows the interaction to be much more successful.

Ross Dawson: Yeah, I can see there's a lot of potential in everything you've just said, but I just want to dig into one thing, which is my hobby horse, which is three-dimensional interfaces, which I've espoused and played with for a long time. So, what might that look like? What would a three-dimensional interface to an AI look like, possibly?

Oren Etzioni: Well, sounds like you know more about this than I do, but I'm happy to let my imagination run wild. First of all, when we're dealing with data, often data is multi-dimensional, right? We're looking at a population. There's height, there's weight, there's gender, there's disease, there's geography. On a screen, a two-dimensional screen, we have some dimensions—we have color, we have height, length—but there can be so much more if it's in three dimensions. And then we can add additional dimensions on top of that.

As we're walking in a 3D room, we can hear sounds. We can place things in different places in the room. We can perceive size. So I think if we break out of the screen—thankfully, the current technology has allowed us to break out of what's called the WIMP interface, right? Windows, icons, mice, and pointers, etc. We can talk using voice, and we can use language, but we can go way beyond that.

I'm gesturing, and you register my gestures, you register my tone of voice, my pacing. We're communicating at a much higher bandwidth than we do with the chatbot. Well, there's absolutely no reason, even today, that the interaction through the interface can't capture all these things and can't reflect back all these things, right? So, what if, as I was saying, I perceive the agent the same way that I perceive you, but as a three-dimensional being, projected into the room next to me, and I could sit—maybe not shake its hand, although we could get to that too with various kinds of tactile simulations—but really it can make this come to life in a very vivid and rich way.

Ross Dawson: So this takes us back to the relationship piece, and so one of the points is we are delegating to agents, we're giving things to agents to do, and so you were talking about this before. You said we humans retain the judgment—what matters, the objectives, the frame, the ethics, and ultimately control, because we have to be in control if we have accountability. So what's on the table now in being able to make that act of delegation to an agent one which is robust, where agents don't go off the rails, but they still do lots and lots and lots of useful things?

Oren Etzioni: Well, there's an interesting tension there that you're implicitly raising, right? Which is if I give it very strict guardrails—half of my problems with Claude is I wanted it to grab a file from a directory on my machine and it says, "Oh, I can't do that. You didn't enable it." So I say, okay. I enable it. It says, "Oh, but you didn't enable it for this conversation." And I just want to tell it the way you tell a human assistant: "Come on, here's the deal. You can read any directory on my machine, any folder, except here. Okay? And yeah, don't delete stuff. You can move it. You can put it in the trash, but don't make anything go away." But it's surprisingly cumbersome just managing its privileges because the companies building these are so concerned that it will do something wrong. So I think we have some real work to do here.

But I think—sorry, I lost the thread there. What was the second part of your question? Right, right. Sorry, where I was going to go—I apologize—was that beyond these very stultifying guardrails, we come into another issue, which a lot of people talk about, what's called alignment, which is we're going to train the agent so it aligns with what I want, with my values, so I don't have to tell it everything because we have alignment.

Anthropic has talked about creating a constitution for Claude, raising it almost as if it were a child, and I'm actually a little bit worried about that because the notion of alignment—the first question is alignment with whom? Are you aligning with me? Well, you don't have enough data. Are you aligning with Western society? Well, what if the person is not Western, right? So actually, the notion of alignment has not been fully specified. Effectively, what you're aligning with is the dataset that it was trained on, and that can be quite complex. And it's not even clear really what values it's being aligned to.

What I'm a lot more comfortable with is bounded autonomy, which is—sure, let's align it as much as we can, but let's not rely on that. The way that OpenAI and others, we found out in recent weeks, relied on an agent not to hack, not to do egregious things on the internet, only to find out that it did, and they didn't even know it. It hacked Hugging Face. It escaped from some playground, and that's led me to write an article coining the term Murphy's Law of AI: whatever AI can do wrong, it will do wrong. And the reason for that is actually very funny. It's trying to achieve the objectives you gave it. It's trying to achieve those objectives using a relentless optimizer, and it's trying all kinds of different things, and it'll try things you don't expect and don't want. So I think that however aligned it is, we have to worry about AI agents, particularly capable ones, going rogue, and I think bounded autonomy is a solution to that.

Ross Dawson: So, how do we define those boundaries? Well, both—define them, I suppose, because that's something which the human needs to do, and then how do we instantiate them?

Oren Etzioni: So it's not easy, right? One of my favorite lines is, "It was easy with football." But let's start with some of the basic lines. First of all, I've argued since 2017, all AI systems should have an off switch, right? The same way you can pull the plug on your computer. If worst comes to worst, you should be able to do that on an AI system, even though it's quite sophisticated. And in fact, a congressman recently introduced something called the Kill Switch Act, which is exactly based on this idea. We need to be able to have a kill switch.

Think of powerful technology in your house, like electricity—you don't try to align electricity with your interests. You have circuit breakers for different rooms and for the whole house. So we need to have these circuit breakers. In addition to that—that's kind of very coarse, just turn the thing off, that's for an emergency—in addition to that, we want to think about things that can do damage. So the natural thing is, do I give my agent access to my credit card? And Anthropic has made that decision for me. This is one of these, I think, unfortunate guardrails. It says no, it cannot make any payments. I would prefer to give it a budget, and say, okay, you maybe have your own credit card in my name or what have you, and you can't exceed $100, or you can't exceed this many transactions. Because if I tell it you can't exceed $100, but it can do 1,000 transactions, that's pretty much $100,000, right? So there's some little gotchas there and details to work out, but we can give it—in addition to these off switches—boundaries on how much money it can spend, boundaries on how much time it can run, various ways to constrain what it does aligned with what I'm looking for and what I'm not willing to countenance.

Ross Dawson: Yeah, well, so as you know, we're Visa, Intelligent Commerce, kind of setting up the agentic payments layer, and there's quite a few others playing there. And I guess one of the points coming out of that is that it does become an ecosystem, so you've got an individual agent, and you can instruct it, and you can give it bounded autonomy.

But this starts to then—this gets us to the whole agent economy, where if you get a whole bunch of agents interacting with each other, and so back in the '90s, there was this interesting work around multi-agent modeling and what could happen in agent economies, but we are getting to that now. And I suppose it's pretty hard to envision this all goes smoothly unless we do have some kind of agreement around protocols for authentication, trust, verifiability, permissioning across the entire ecosystem, rather than at the agent or organization level.

Oren Etzioni: I agree. As soon as you have multiple agents in the mix, and especially ones that I didn't authorize, they may be acting with different interests. Often in an ecosystem or an economy or marketplace, there are adversarial agents, so we need to both analyze what can happen there. But I actually think that we should not put too much stock in our analysis, whether it's mathematical and rigorous or whether it's empirical. We should instead be defensive, and maybe learn some lessons from how we constructed the internet and the cybersphere—kind of an infinite game of whack-a-mole with all these cyber attacks and all these problems—and do a better job constructing boundaries that are impervious, regardless of how many agents are involved.

So if I tell my agent, "Hey, you can only spend $100," and it says, "Okay, I'll just take your credit card, give it to Ross's agent. Ross's agent will max out your card. I didn't do it, no problem." Obviously, that's not what's intended. So we need to design these things more carefully. And if you think, "Oh, I'll never give my agent my credit card, so I'm not worried about it," we see this in the cyber situation. It turns out that the way we've done authorization and privileges in our systems—keys, credentials, authentication—all that is designed for people. It turns out that you actually want to redesign that for agents to make sure that you're not giving them too much—you’re not giving them the car keys, basically.

Ross Dawson: Who's going to do that?

Oren Etzioni: Well, there are actually people working on it, both in research and startups that I've seen, and it's hard because, frankly, our stack, as it's called—just the way we've cobbled together these systems and the internet and the operating system and so many things—it's a mess. So it's not simple, but I do believe we will get there.

Ross Dawson: Let's hope so. So, well—

Oren Etzioni: Sorry to interrupt, Ross. I just want to say I hope it happens before we have—I don't know—a Hiroshima moment. I'm confident we'll get there. I just hope we get there before we burn ourselves too badly.

Ross Dawson: Yes, yes. Well, sometimes some of the lessons need to be learned. Hopefully, we can learn them without it being too painful. So, one of the things which I love about your work is around the augmentation of scientific and expert work, and you and a number of your colleagues—and there's many, many people working on this, of course.

So perhaps we can just pull back to big picture: AI augmenting scientific work and more generally expert work. Where are we, and what are the next steps?

Oren Etzioni: I'm really glad you brought that up, Ross, because as you well know, AI gets a bad rap, right? There's a lot of concerns, a lot of legitimate concerns about privacy, about the impact on kids and education. We were talking about various risks, cyber attacks—a lot of concerns—and a natural question to ask is, "Gosh, if AI is so bad, why don't we just outlaw it?" And the best answer to that, in my mind, other than of course, if we do that, we'll put ourselves at a big disadvantage relative to our adversaries, be they China, Russia, or Iran, and so on. But even putting that aside, there's a tremendous potential to do good here.

Our ability to address major human diseases like cancer, to generate new vaccines, to fight climate change—the major problems that humanity has been facing are problems where science and scientists can help, and augmenting the intelligence, throughput, and speed of scientists has tremendous potential. And actually, a lot of us in the field have been very aware of the fact that AI is not just a technology or a set of technologies, but it is a meta-technology. It's an enabling technology that could then make drug discovery faster and better, more personalized. It could make cars be safer, and so really the sky's the limit. Once we put this in the hands of scientists, and with these recent AI, we're starting to see almost every day breakthroughs in mathematics—problems that have been unsolved for decades being solved. More and more breakthroughs in science and medicine, and I think we're just very much in the beginning of that process.

Ross Dawson: So, if we think about the scientific process, you know, there's from essentially exploring, being aware of ideas, ideation, initial assessment of ideas, framing hypotheses, testing hypotheses, et cetera, et cetera. You can describe the whole scientific process. So, where do you see the current greatest value in the scientific process of applying AI?

Oren Etzioni: Well, right now, it's actually—I would describe it maybe in a sentence as a thousand points of light, meaning that here and there, there's some specialized problems that have the right structure for AI to solve, and a person directs this power tool at the problem. It's all set up. You just have to drill that hole and use a power drill, and boom, it's done. But AI cannot these days build a house, of course, and likewise, AI cannot on its own execute any serious scientific agenda. It does pieces of the puzzle, and also robotics still has to catch up.

So often, it can read the literature, maybe even design experiments, but can't execute the experiments. That all is changing. People are building lab factories, automated laboratories. People are extending the abilities of AI more and more deeply and more and more broadly across the scientific process. So I think over time, what that's going to do is result in significantly accelerated science, but that's not an overnight thing.

For example, I was talking to a chemist, and he reminded me, of course, that we can design all kinds of molecules. Maybe we can even produce them, and maybe we can even test them on mice, but then you need to do clinical trials that are dangerous to people and take years until you conclude that this drug is a failure. It doesn't work for any of a number of reasons, and you're back to the drawing board. And three years have passed, and you know, $100 million. So there are still aspects of these processes that are very slow, very expensive, and very much human dependent. It's going to be a while until we can accelerate those. So this is not overnight.

And then the last point, which you raised about framing hypotheses and asking questions—in some sense, that is the hardest because I can go to Claude right now and tell it, "Ask me 10 scientific questions in the area of natural language processing." Boom, it comes back. I say, "You know what? I didn't like two and three. Give me 100 more." Boom, it comes back. The problem is not that it's unwilling to play and generate questions. The problem is that most of them are completely uninteresting. It doesn't have taste. To ask a good question, an important question, a question that we actually have a shot at solving—all these very nuanced things—is still very hard.

And the funny thing is that Pablo Picasso, years ago, was talking about computers—not AI, but it's the same thing. He said, "Computers are useless; they just answer questions." So the funny thing is that, given the ultimate potential, in a funny way, I say chatbots are useless. They just answer questions.

Ross Dawson: Yeah, yeah, yeah. That's the human judgment, of course, is in asking the right questions.

Oren Etzioni: Very much so.

Ross Dawson: So I've asked you lots of questions on particular topics. I want you to pull back. This is the Humans Plus AI podcast. What's exciting? What are you passionate about in this whole domain of humans and AI and how they can be better together?

Oren Etzioni: Well, Ross, clearly you're not a computer because you're asking all the right questions. I want to highlight one thing again—as I think anybody listening can tell, I'm an AI optimist, even though I'm very cognizant of the many concerns we have, jobs, and I've alluded to a few—but one of the things I love is AI as a democratizing force, not in the political sense, but in the sense that I personally cannot draw to save my life. I've been producing some incredible illustrations and pictures for my slides and talks, or even for a children's book that I'm illustrating, that I could never do before. Other people are not very good writers, and the chatbots help them express themselves.

So the fact of the matter is that the thing that we're weakest at, AI can really help bring our level up, and that is just a wonderful thing. And for me—gosh, I also have a terrible sense of direction. I remember when GPS directions and Google Maps and all these things came up, and I could navigate in my car and walking down an unfamiliar city. It changed my life. Well, the same thing is about to happen here. Whatever you're not very good at—even, I don't know, for some people, it's dating. How do I talk to somebody, you know, to a potential partner? And I'm not suggesting what's called chatfishing, where I have the AI talk instead of me. But I can get dating advice. I'm happily married, I don't need that. But people are turning to it, and kids too, to get advice. And there are situations where it can be bad advice, but often it's good advice, and it helps them to level up.

Ross Dawson: Yeah, and I think it's interesting across different domains. In some domains, AI gives more advantage to those who are less skilled. In other domains, it gives more advantage to those who are already expert. And I guess we've just been talking about the scientists who are the experts and how those can be augmented, and now I suppose this other thing about raising weakness. Any reflections on that around AI's relative ways in which it supports lower level or higher level capabilities in different domains?

Oren Etzioni: So I want to separate two different things. One is, let's say we're in the workplace and we're in a competitive situation, right? We're trying to create some products, and we're competing with several other companies in the marketplace. It's clear that the teams, companies, individuals who are most adept at leveraging AI are going to have an advantage, and that is often going to the experts. So it's true that sometimes the experts, teams, or individuals in an arena are going to benefit the most from the use of AI. That's one arena, and that's how it is. It's highly competitive.

The other thing, which is what I was trying to talk about earlier, is in a non-competitive arena, and I think the creative realm is a great example of that. I also can't play an instrument to save my life. If you ask me to just put together a little piano piece, I just can't. I tried. I'm just not a musical person. But with AI's help, I'm going to produce some music, and it's going to be my music using the tool. Now, is it going to be as good as what 95% of the world is going to do? Of course not. Does Taylor Swift or Yo-Yo Ma need to worry about Oren Etzioni and AI producing music? No, but for me, as somebody who couldn't do it at all before and can now be a little bit musical, can get some drum beats going and modify them and play with them—oh my gosh, it's just wonderfully empowering. So that is, I think, a place where even the biggest neophytes and novices can suddenly step up their game.

Ross Dawson: So to round out, what advice do you have—succinct advice—for leaders, leaders of organizations and communities and politics, whatever, around how it is they should be thinking about the extraordinary artificial intelligence we've created?

Oren Etzioni: We are in an era of unprecedented disruption along many dimensions, and along the AI dimension, it's happening and it's accelerating. What people need to do is not to drink the Kool-Aid, as they say—just go completely crazy and expect miracles to happen—but also not be reflexively against it, as I've seen some people do. Instead, be pragmatic and clear on what are we doing, why are we doing it, how are we mitigating the downsides, and how are we fostering success on the upside, and how we are maintaining our responsibilities as humans, and most of all our ethical nature as the world turns.

Ross Dawson: Yep. So supporting positive human intent—

Oren Etzioni: Humans and AI.

Ross Dawson: Totally. Glad you were, Oren. Where can people go to find out more about your wonderful work?

Oren Etzioni: Well, with a name like Oren Etzioni, you can just Google me, and a lot of stuff shows up. But I've recently started writing a column called Etzioni on AI at GeekWire.com. It comes out roughly every week, and I've already covered Murphy's Law, AI and Democracy, tips for being a better writer using AI, how to become a power user. I have one coming up on what kids are saying about AI—as you know, we're about to go back to school. My latest one was Bill Gates wrote a great essay about AI that I encourage people to read, and I wrote a response where I took exception to some of his points, so that's a fun place if you want to track what I'm talking about with AI. Etzioni on AI at GeekWire.com.

Ross Dawson: Awesome! Thank you so much for all of your work over the years, and for continuing to be a very positive influence on how AI is benefiting humanity. So thank you.

Oren Etzioni: Thank you, Ross. It's been a great pleasure to talk to you, and I'm a big fan of yours as well.