Cleotilde Gonzalez on human-AI complementary intelligence, theory of mind in teams, dynamic decision making, and metrics of complementarity (HAI Ep61)

Transcript

Ross: Coty, it is a delight to have you on the show.

Cleotilde Gonzalez: Thank you so much for the invitation. Happy to be here.

Ross: I love your work. It is just so aligned with the humans plus AI mission, and all of my interests, so I'm very excited to learn from you in this conversation. To start, you've written that the goal of human-AI interaction is complementarity. Could you explain why you believe that is the case?

Cleotilde: Certainly. What we have been experiencing recently is not complementarity, but that should be our goal. I think the goal is not to replace humans, but really to create this complementary intelligence where humans and AI collaborate in useful ways, and when this combination of humans and AI can do better than what AI alone can do. So that's the general idea.

It's not about which one is more intelligent, or whether we can replace the humans, but really, how can the different capabilities be combined?

Ross: In fact, you wrote that in response to somebody who said that AI should be used for augmenting humans, and you disagreed.

Cleotilde: Well, it's not that I disagree with augmenting humans. Certainly, that is a good goal to have. But we need to make a distinction between augmentation and complementarity.

Augmentation is when you add AI to what humans do, and the combination of humans and AI is better than humans only. That's augmentation, and that's of course a great thing to have. But there is no complementarity because then it is always possible that the combination of humans and AI doesn't do better than what AI can do alone, right? And if that is the case, then you would wonder why do we have human and AI together? If AI can do better alone, let's just use AI and replace the human, which is exactly the issue that we are having now.

Ross: Yes, absolutely. So, just at a very high level, to your point, there are many experiments showing that human-AI combinations are not performing as well as AI alone, or sometimes humans alone. My complaint about many of those studies is that they are in a single context, often with poor design, and they generalize from a single instance of human-AI combinations, suggesting that AI is superior or humans are superior to the combination, but just because it's a poor design.

So, how can we move towards understanding better the principles of what truly is effective design or combination for human-AI teams?

Cleotilde: Exactly, that is where the crux of the problem is, and I think where the opportunities for research are. Putting humans and AI together does not necessarily produce a better team. That's the first point. The best AI teammate is not always going to be the one that performs best.

You can have an excellent AI, but then the combination of humans and AI are not the appropriate ones. And the question is why, right? You can think of this in terms of human teams also. If you think of two people who are excellent at doing a particular task and making decisions in a particular environment, and they try to do the same thing effectively, they are not collaborating, and they are not complementing each other. They are actually on top of each other.

The key is in figuring out what AI is good at, what humans are good at, and how we can make the best of the two types of skills, because there is no question that AI is better than humans at many things. But there is also the case that humans are more aware of context, more aware of ethics in situations that AI might not know how to interpret. So it's a combination of the best of both types of agents that's going to result in a better team, and that doesn't happen just by having a better AI performer.

Ross: I guess one way of framing this, and I'm very interested in both at a team level and at an organizational level, is that we can look at human-only teams or human-only organizations and see what are the best structures, and we can look at AI agent-only teams, or by extension AI agent-only organizations, and see what are optimal organizational structures, communication patterns, handoffs, and so on.

If we are looking for complementarity, then we're looking at team or organizational structures where humans and AI are both present, and looking at the balance, and again what structures are appropriate. Where are we? Because I think we're relatively early in the research in understanding what are the best human-AI team and organizational structures. How far have we got in understanding that?

Cleotilde: I think we are very early indeed, because we haven't had this type of capability in AI before. We are trying to learn from human teams—what's the best, for example, from shared situational awareness, from collective intelligence, which are concepts that we've known for a long time in human teams, but that change now that we have a different type of teammate. Indeed, we are very early on that. One of the things that we are doing and trying to study is how to allocate different roles based on these capabilities. The other important thing is the dynamics.

When you are putting a team together, you might not know your teammate as well, and you might start to work with that teammate and start to develop this mutual understanding and theory of mind together. As you move forward, you start to understand each other, and you become a good team. But you were not initially a good team. These are the types of questions that we have: what roles should humans and AI take at what time, when should humans delegate to AI, when should AI take initiative and do things alone, and how do humans and AI influence each other together to become better teams.

Ross: On the theory of mind question, as you know, on a basic level, human teams perform better when they get to know each other better. Of course, it's a different dynamic when some of those team members are AI agents, and that goes two ways: the agents can get a better understanding of the cognition of the individual humans on the team and how they think and how they can be best interacted with, but also for the humans to better understand the nature of the agents and how they work and function.

At best, people who interact the most with AI tend to have a somewhat effective theory of mind of the agents, and other people don't. How can we build that better mutual theory of mind in human-AI teams?

Cleotilde: I think that is a point in which a lot of the current AI is lacking. They—and I say "they" because I don't consider myself a developer of AI per se—have focused on efficiency and accuracy of tasks. But in reality, they haven't focused on understanding the human, which is essential to be able to work with humans.

They haven't done a lot of human cognition, and that's in part why our work in cognitive science can have an impact on machine learning. For years, we have developed cognitive models, and these are computational representations of how humans make decisions. We have developed some theories that are actually pretty accurate at predicting and describing how humans make choices. What is puzzling is why machine learning models are not learning from those cognitive models.

They should, because those are going to have a better representation of human behavior—how humans adapt to different partners, how they collaborate, how they can be more understandable. In many ways, these are the mental models of humans that machine learning can learn from, and that's what we've been working on: figuring out ways in which cognitive models can inform machine learning models so that they can have a better representation of humans.

Ross: That sounds wonderful, and I guess you would hope the frontier labs or the LLM makers would be building this into their systems, but they probably aren't to the degree which they should. So again, where are we, and how do we get better at putting this into practice?

Cleotilde: I do think—and I don't know many of their secrets, I would say—but I do think that many companies are really investing a lot in this issue, in really representing human behavior, and not only considering AI that is optimal and effective, and precisely is to try to achieve this goal of complementarity. But from what I've seen up to this point, they are not taking the right approach. We have been stuck with generative models and with increasing the capabilities of generative models to learn more about how humans do it, and in modifying these models to make them more human-like in some ways. But in my opinion, it really needs a completely different frame of mind, because the principles of these models are based on data and patterns, and humans are just not like that.

Humans are a different type of machine. The whole concept of representing human behavior, I think, needs to start from psychology and from cognitive science—who are the ones that have been studying human behavior for decades, or even a century or more. I think it's the approach that they are taking that I'm concerned about, because I think they are going to end up—which we are observing now—mimicking behavior but not really explaining or knowing how or why humans behave in certain ways. They are just trying to mimic at the surface, at the outcome level.

Ross: Yes, and this goes a little bit to something I'm thinking about a lot, which is cognitive interfaces. Human cognition is, of course, limited, and we can take in information in text, in voice, in images, and so on. Along the way, I think that essentially, the LLMs are not very good at presenting an optimal cognitive interface to humans to best be a complement or best tap the distinctive capabilities of humans. So I think we've got a long way to go in cognitive interface design.

Cleotilde: Yeah, I agree. Some things might look like human behavior, but that doesn't mean that the machine has the capabilities that humans have. Some people argue, does that matter? That's the question. As long as it behaves like a human, why would it matter? I've heard that argument quite often. I think it matters because if we understand human error, in which ways humans can behave wrongly, we can prevent those errors. If we can explain the successes and failures of humans' decisions, we can accommodate the environment in better ways. So I think it does matter.

It is not about the outcome only; it's about the explanation and the whole process by which we make those choices or come up with an answer.

Ross: I'm very focused on learning loops, and so of course we are building recursive self-improvement or moving towards that and learning loops in AI. I think we can very much build learning loops for humans alone, and of course, building those learning loops for the humans plus AI systems. Some of your work has been around essentially feedback to teams—what you call collaborative process metrics to improve team performance. Where are we in being able to gather those collaborative process metrics, gather the data that actually will help us to improve the performance of human-AI teams?

Cleotilde: We're very behind, I would say. My work from very early on has focused on learning, but in particular, what I call learning from experience or dynamic decision making. This is because a large majority of decision sciences was focused on one-shot decisions. That has been changing over the years, and we have now more people interested in decisions from experience and the big contrast that exists between experiential choice and one-shot decisions.

We know that we learn from reinforcement; there's no question. Humans learn from reinforcement, but it's not as simple as a reinforcement learning algorithm presents, because humans also have cognitive constraints. We can remember things that happen more often, that happen more recently, that are similar to other things we have observed in the world. Our cognitive machinery is different from a simple reinforcement and just going for the reward. In many ways, it's what Simon referred to as bounded rationality. We are boundedly rational because we do want to be good and we do pursue a good goal, but we are constrained by our cognition, by the environment, by our individual experience, etc.

The learning loop itself, just with the human and the environment, is actually quite complicated, and it still needs quite a lot of work. But add to that the AI, where you are learning not just with an environment, but with another being that can be as intelligent in many ways as you are, or as agentic as you are. Now we are in a cycle in which we influence each other. We can influence each other for good, and we can influence each other for bad. One can be better than another, and it's a lot of very interesting dynamics that can occur when you put two intelligent agents together versus one intelligent agent in an environment. Even when the environment can be dynamic, it is not the same as learning with another intelligent agent. All this being said, I think in terms of research, we are very early in understanding the dynamics, in particular in how humans and AI can influence each other for the better.

Ross: But coming back to the metrics point, are there any specific metrics? Let's say we have a humans plus AI team. Are there one or two or three metrics you might say, "Okay, measure this, and this will help you improve"?

Cleotilde: One of our metrics is, of course, a metric of complementarity, which we didn't invent—we borrowed from other people—but that is at the team level, where you basically compare the human-AI team over AI alone and human alone. But if you just look at the human-AI team and how they share their roles and their workload, what we have seen—and I think this is interesting, it's in a paper in preparation—is that naturally, we have seen how AI can take a role and human can take another role without even communicating. They can exercise the subtasks that they are best at—what the human is best at and what the AI is best at. The goal is to achieve some balance, and this balance doesn't mean that you do half of this and I do the other half.

It really means how we distribute the various subtasks in a way that we have a reasonably equal workload. That's something we have observed, and this is measurable. We can look at the workload of the AI and that of the human in the different subtasks that compose a more complex decision task, and we have seen that when there is more balance, it is closer to achieving this complementarity.

Ross: Right, and I presume observing how that changes over time relative to team outcomes.

Cleotilde: That's right.

Ross: One of your wonderful recent papers is "Toward a Science of Human-AI Teaming for Decision Making," and I suppose that implies that we don't yet have a science because we're moving towards that science. The field of decision making—it's very clear that the world is full of decisions: personal decisions, organizational decisions, governmental decisions, and all of those decisions will or should include both humans and AI at some level, at least in terms of oversight. There are many different types of decisions, and what will work will play out very differently.

How should we begin to start thinking about the different structures or different ways of design for bringing together humans and AI in better decisions?

Cleotilde: Partly in that paper, we try to start by identifying what AI can be good at at different levels—in perception, in cognition, in organization—and what humans are better at. We have a table in that paper in which we separated what humans are good at, what AI is good at, and how complementarity can be achieved. But it is difficult because it depends on the task a lot, in particular for decision making. If the task is relatively simple, of course AI can do it on its own, and we actually want accuracy in those cases, and we want an AI that is fast and accurate. Let's say, for example, a calculator or GPS providing an accurate route—maybe speed matters the most for me, and so I want recommendations that are accurate and good, etc.

For simpler tasks, maybe you don't need so much this idea of complementarity. I think it depends on the task. For very high-stakes decision making, like in the case of disaster management, you might send a robot with AI into a non-reachable place and try to figure out whether to save a victim, but the robot does not have the sense of fairness, the sense of urgency, the sense of ethics. In those kinds of high-level decision making, I don't think anybody would be comfortable with having the AI make the choice alone. That's when we want the intervention. So I think this idea of cognitive alignment and getting the human and AI to work together becomes more important the more significant and high-stakes the decisions we have to make.

Ross: One broader point here is where many people talk about "human in the loop," and almost always they mean putting a human at the end for approval, which I think is deeply flawed for many reasons. For some decisions that's appropriate, but in many cases, it means that the human essentially pays less attention. It erodes their judgment because they're not involved in the entire process along the way. It's accountability in theory, but not necessarily having a useful human role.

So, if we're going beyond human in the loop to really bring humans into the decision process in a way which draws on, as you say, the context and the ethics and the judgment and perspective, what are ways in which we can design that?

Cleotilde: I think the human should be in the loop. But what does this mean? If we think of this as a team, the AI and the human in principle would have similar capabilities in terms of autonomy. The AI could be autonomously deciding on something, and the human could autonomously decide on other things. The key is to know what things the human should be deciding on and what the AI should be deciding on.

In an example of cybersecurity—it might be very complicated, but I'm going to try to explain it—for cyber defenders, we put together a team of humans and an AI working together to defend a network against a very knowledgeable attacker. Both the human and the AI were able to essentially do the same tasks, which involve monitoring, stopping a server, or even deploying honeypots and things like that. But the question was: Should AI be deploying honeypots and shutting down systems? There were some relevant points in which the human needed to be responsible for that. If the AI is monitoring or identifying, go ahead and do it without any human in the loop. That's great; that's what AI should be doing.

But what we did is say, okay, every time you are going to deploy a honeypot or shut down a system, the human would need to review that. The human could say, "Oh, that's a great idea. Go ahead," or the human could say, "No, I don't want you to do that." So the human would need to have some say in what the AI needs to do for decisions that matter. In that case, we compared a cognitive model with a strategic partner—playing together as a team—and we found actually the cognitive model not only produced a better team, but also the humans liked it more. They did the task with less effort and more efficiently with a human-like agent compared to a strategic and optimal agent. That's an example of how the human needs to be in the loop, meaning really collaborating with the AI, but also where the guardrails should be for high-level decision making.

Ross: That's great to have that grounded example. But more generally, let's say in an organization, we have a set of important decisions, and we can see that these are important decisions. Coming back to the ones you were talking about before—the strategic, very high-impact ones, which are human first. In this case, these are framed by the humans, understanding what is the question or the issue or the decision, and obviously making final decisions along the way.

What are ways to bring in AI, and how should we be thinking around optimizing the most complex strategic decisions using human-AI complementarity? Where is the complementarity? What are the roles? How can we design these very complex strategic decisions?

Cleotilde: Remember that here we are talking about optimizing in terms of complementarity again. That is important because it's not about optimizing the task itself, but optimizing the way we work together. I think the decisions need to be clear in terms of what decisions need to be made. The agency needs to be clear—who should be responsible for making that kind of decision? Those depend on the impact and the stakes that you have. If it is dealing with human lives, for example, is that what we want the AI to be making? Each organization should have a clear understanding of all the decisions that are going to be made and who's going to be responsible.

We talked about the roles, right? We cannot give the role of, let's say, summarizing all the information in an organization to the human because we know that the AI is best, but once the information is summarized, we cannot let the AI fire an employee because we know that humans can make that better judgment. I think it depends on clarifying the roles for decision making, and also when those roles need to be deployed, when those decisions need to be made. The context will matter a lot, so it would depend on exactly the situation we are talking about. But in general, I think for high-level decisions, that is what we should be thinking about.

Ross: One of the things which I'm most interested in at the moment, and I think many other people are, is this idea of humans plus AI organizations. As I mentioned before, all organizations will comprise humans and AI agents, and so we can't assume that we have the human organization forms and then just throw in some AI agents in various places. How can we best design—of course, it's context specific for lots of reasons in industry and culture and so on—but how can we start to build the new organizational forms that are optimized for human-AI complementarity?

Cleotilde: I'm afraid I don't have any specific advice beyond what we've already discussed in terms of roles, etc. I imagine that the tendency in many companies would be to bring in AI and force people to use AI or assume that their work is going to be better if they use AI and request the use of AI. But as we have discussed, adding AI is not going to make a better team necessarily. So, really understanding the capabilities of people and the capabilities of the AI for a particular task—there is no question that it would be good for some subtasks, but not for everything. I'm afraid that there is no clear recipe here, or at least not yet.

Ross: Yes, well, I think as with many other things we've discussed, we are very early. We need to do experiments, we need to learn, we need to design, and those organizations that are doing the experiments and learning from those earliest will be the best at discovering what works for them. So, to round it—go on, sorry.

Cleotilde: I think organizations also should pay attention to research and what research is telling us.

Ross: The very pragmatic research of seeing what works for them. So, to round out, I think we can call it a science of human-AI complementarity. Where do you see this field over the next few years? What needs to be done? What will we discover? What are the directions that the field of human-AI complementarity needs to go?

Cleotilde: With the speed at which AI is increasing its capabilities, I think it's becoming even more relevant that we better understand what makes us human, and that we understand the aspects that humans are unique at. To achieve complementarity, we need to figure out all the ways in which humans can contribute that AI cannot contribute, and certainly there are many ways. We live in a physical world, and we are talking about an information world. Humans are going to have many advantages in the physical world over the information world, but I think it is indeed taking over very rapidly. So it becomes more important to reflect on what makes humans human.

What are the unique capabilities of humans that AI cannot have, and will never have, and will not have. I don't believe that we would ever have AI overtaking the world and finishing humanity, but it becomes more important for us to reflect on that. It is also extremely important to figure out metrics that demonstrate this complementarity. Demonstrations of this complementarity, I think, are also very relevant for me now.

Ross: Yes, there's a lot of very valuable work to be done.

Cleotilde: Yes.

Ross: So, where can people go to find out more about your wonderful work and research?

Cleotilde: Certainly. I am the co-director of the AI Institute for Societal Decision Making at Carnegie Mellon. AISDM is a national institute funded by the National Science Foundation. We have podcasts, we have talks that happen every week, we have a lot of resources there. It's a big organization that involves not only universities but also companies and nonprofit organizations like the American Red Cross, for example, and many others. So you can learn from there. If you are interested in dynamic decision making in particular, I'm the founding director of the DDM Lab since 2001. There is also a lot of resources in dynamic decision making. We have a set of demonstrations of dynamic decision making tasks. We use many video games for studying dynamic decision making. So, there is quite a lot of resources.

Ross: That is fantastic. I really, really deeply appreciate your focus and your work and everything you are doing in such an important domain. Thank you for all of your work and your time and insights on the podcast.

Cleotilde: Oh, thank you so much. This was fun, and I really appreciate the opportunity to talk about all this with you, Ross.