Gideon Nave and Steven Shaw on system 3 thinking, strategic delegation, cognitive surrender, and the ethics of AI in marketing (AC Ep49)

Transcript

Ross Dawson: Gideon and Steven. It is awesome to have you on the show.

Gideon Nave: Lovely to be here.

Steven Shaw: Pleasure to be here.

Ross Dawson: So! particularly interested to hear about your recent paper on thinking fast, slow, and artificial, around cognitive surrender. I think a lot of people found it a very interesting paper and concept.

The starting point is that you talk about thinking fast, slow, and artificial, obviously referring to Daniel Kahneman’s system one and two, and you introduce this idea of system three. So, what is system three?

Gideon Nave: We started with system one and two, and I think the reason we needed to add system three is because we now have an external system that is readily available 24/7 upon our request, doesn’t cost much, and doesn’t judge us very harshly. In some ways, we’ve started to think as if it is a part of our thinking process. We consult it readily when faced with thinking and reasoning problems—sometimes we don’t even bring up our own intuitions, we just give up thinking and go there.

Maybe sometimes we are stuck, and then we go there, but very clearly this has become a part of our thinking. As cognitive scientists, we don’t only care about what system three is, but also about how system three interacts with the two other systems, how it affects human thinking. That’s what led to bringing in system three, because it’s not just the addition of another thing.

People have been talking about for many years cognitive offloading—like giving a calculator a small specialized task, or letting GPS do a small specialized task. There is something different here: it’s readily available, but also it’s a very general thinking device.

Ross: Exactly your point—as you said, most people would say, “Intuition, all right, well, I’ll just throw that over, I’ll think about it, but I’ll let the machine do that.” Whereas with system two, the rational thinking, that’s where a lot of people do engage. Sometimes people have long conversations, engaging in complex thinking. So, is that first place, where you’re just throwing it over to the AI and giving up your intuition, the surrender piece?

Does that mean when you engage with system two with the artificial component—system three—that it’s more likely to be augmentative or additive to cognition?

Steven Shaw: Yeah, building off Gideon’s points, we introduce system three because of the real ubiquity of AI in our decision making. We had systems one and two in vivo in our brain, and they have their own characteristics. We need to have a definition for this artificial thinking system. So, we introduce system three as artificial cognition, because it can, if we allow it, do many of the types of thinking tasks that we could do with systems one or two.

I think what you’re getting at is the differentiation between whether we’re engaging system two at all. One of the key contributions of our research is this distinction between strategic delegation—where we are engaging in critical thinking and system two is active, and we’re cognitively offloading certain components of cognition, which leads to augmentation and strengthening of critical thinking skills—versus when we have only some engagement of system one, or maybe just visual processing, and then outsource our thinking.

That’s what we define as cognitive surrender, where individuals are outsourcing the whole thought process itself. That has quite a range of possible consequences for how we think.

Just one other point on why we think system three really needs to be defined and hold the stature to be alongside system one and two: we use artificial intelligence for such a wide range of tasks, and it’s so fluid and available that there’s even research showing our thinking, decisions, and speaking are affected even by the knowledge that AI exists in society. We’re pre-processing and seeing AI process content all the time, and our writing, language, and speaking are being influenced by AI. So, it’s really that ubiquitous and pervasive.

Ross: You had a pretty in-depth study, which was reflected in the paper, and it’s not all very pretty. So, what’s the top line of the empirical findings?

Gideon: I don’t think describing the entire study now is what we’re going to spend all of the podcast on, but in general, we created a situation where we asked people reasoning questions with an answer that is seemingly intuitive—many people come up with it—but this intuitive answer is wrong. You need to catch it and think it through. The key outcome was how the presence of AI affects people’s performance in this test. Behind the scenes, we randomly manipulated the AI to sometimes be right and sometimes be wrong.

The key finding is that when AI is right, people get remarkably better, and that’s good—it means we are becoming, in some ways, artificially intelligent. The not-so-good finding is that when AI is wrong, we go below baseline, meaning people are getting things wrong more often than without AI. Even MIT and Stanford students get it wrong without AI, but the presence of wrong AI makes us worse. Even if AI is wrong, we also become more confident in the answer. So, there’s a combination of getting worse than baseline if the AI is wrong and becoming more confident, which is the price we pay for becoming artificially intelligent when the AI is right, and maybe not having to invest so much in the thinking process. This complete pattern is what we call cognitive surrender.

Ross: I have this concept of appropriate trust or calibrated trust—you need to trust AI enough to make it useful, but if you trust it too much, then you let it influence you to do things where it’s outside its ability. I suppose that calibration is critical, because as you say, sometimes it’s right, sometimes it’s wrong. So, what can we do to have the useful level of confidence in AI when we are engaging with it?

Steven: That’s a great question on calibration. How do we know the accuracy of these AI systems across a variety of domains? In the case of the CRT problems—the cognitive reflection tasks we used in our experiments—in the wild, AI can get these math problems right most of the time. So maybe our participants were actually well calibrated. But when you’re using AI across domains, you don’t actually know, and it often doesn’t give you much information about its accuracy.

On math problems in structured domains, there’s going to be near 100% accuracy, but when you get into things like legal settings or mental health, you’re supplied with a fluent, confident answer, and you have a partner giving you a ready-made answer, but not necessarily knowing how accurate that answer is. That’s the calibration you’re talking about, but I think the question itself is sort of the golden question—we don’t know. There’s very likely some sort of reinforcement learning mechanisms happening with cognitive surrender, in that we go to system three and work with it in structured domains, starting by cognitively offloading or being strategic about delegation. But as it performs well in those domains, we can rely or over-rely more, and slowly slip into more cognitive surrender as we trust more and more. Then, as you trust more, you might start relying on AI in other domains where its accuracy is no longer near 100%, and then we start seeing more risks to our personality, identity, or adopting the biases of AI in our outcomes, like errors in legal outputs or even possibly in warfare decision making.

Gideon: Yeah, I want to jump in and say that calibrated trust is important, but I think we’re used to thinking that the goal of thinking is coming up with some decision, outcome, or knowledge, and that has been the case throughout most of history. But there is also a value of thinking to us—once we think, we are training ourselves to think, and our brain changes as a function of thinking, like a muscle that is training. Now we are capable of producing the same output without training this muscle, and we don’t know how this is going to unfold over time.

We already have evidence that is quite worrying—people losing skills, people never skilling. Performance looks perfect, but only when the AI is there. The moment the AI is gone, performance shifts. So there is something deeper here that goes beyond just the output we produce—something internal that goes on inside us and may change us, not only by adding something, but also by what we already had.

Ross: I’d like to get into that a little bit further, but one of the findings in your paper was that incentives and feedback reduce cognitive surrender. What can we learn from that finding in terms of what we can put to use for ourselves?

Gideon: I’ll start with the negative—even with incentives and feedback, there is still cognitive surrender. Even when people trust, I would say, is probably calibrated to the situation more with the feedback, we still see this, and we also still see it with incentives. So, there are stakes here, and that’s quite worrying. That’s the downside.

The upside is that maybe there are ways to shift this around, because it does change. We did not incentivize people with a million dollars, but probably if you’re about to take a mortgage, you’re less likely to surrender with this decision. So, that’s the bright side.

Steven: I’ll add to that. In our experiment on using incentives and feedback, we increased the stakes for participants to try to make sure they were checking and verifying outputs, and they got paid more for every correct answer. You’d expect that to make a difference. Our experiments are a simple behavioral demonstration in a lab environment of a broader, more important idea—cognitive surrender.

In the wild, when a doctor is using open evidence, we don’t know if they’re copy-pasting information or not, or how much they’re relying on or engaging in cognitive surrender. The distinction between cognitive offloading and cognitive surrender is, in the end, a continuum. In the wild, we don’t know in which domains or environments people are actually doing more checking, and there are probably individual differences. We see some individual differences in our study—need for cognition and trust in AI affect how susceptible individuals are to cognitive surrender. So, there’s a lot of work to be done in the field to see how much cognitive surrender is actually occurring in different domains. But at least right now, we see cognitive surrender as a psychological mechanism that leads to de-skilling, for example, and we have empirical evidence from colleagues at Wharton in high school math students that this kind of de-skilling is happening when there’s optional, unaided access to AI systems.

Gideon: I want to jump in again and say that in our studies, checking that AI is wrong was very easy—it required basically second-grade arithmetic, just adding numbers and seeing, “Hey, this doesn’t add up.” I think most of the time it’s not so easy to check, and if it’s not easy to check, I don’t know if incentives are even going to work. It needs to be very, very costly for you to start checking. Maybe there are some people who tend to check and like to think, and we did find that certain personality traits make people less susceptible to surrender—fluid IQ is one, need for cognition (liking to think), and of course your level of trust, as you mentioned earlier. But in reality, unfortunately, it’s not so easy to check.

Ross: One of the important things is naming and pointing to cognitive surrender. I was just having dinner with some friends on the weekend, and someone was describing something happening at work, and I said, “Oh, that’s cognitive surrender, and I’m talking to the authors of the paper.” So, the big picture here is you’ve named it, you’ve got some data. Let’s pull out to the big picture: yes, it is real. We have cognitive surrender, what I call cognitive erosion, we have cognitive corruption, but we also have the potential for cognitive augmentation, as you’ve described—engaging with system two, where we can be more, and potentially even interact with AI in a way where we can think smarter, even after we take the AI away. So, in the big picture, for individuals, organizations, and society, what can and should we be doing to push ourselves up from cognitive surrender to positive cognitive impact?

Gideon: At the individual level, I think at the end of the day we need to decide we don’t want to surrender. That’s the first step. If you want to surrender, you can surrender now—you pay a price for it. Maybe in the short term, it’s going to be very good for you. We had a piece in Fortune magazine just two days ago—a CEO said, “I’m three times the CEO I used to be.” So, it will get you more productive. You need to choose the liberty not to do it, because you believe it may have some damage. Like deciding not to smoke anymore because you think it’s going to have long-term damage. But if you don’t want this, you will surrender, because there is a good reason to surrender.

The AI companies are working very hard for you to use them, to rely on AI more and more, and to get you more and more satisfied. From what I’ve seen, AI that tries to prompt people to think more—the so-called Socratic AI, or more confrontational AI, as opposed to sycophantic AI, which tries to confirm and validate you—the Socratic AI is not very liked by people. The validating AI is very much liked. So, what chance do we have if we don’t choose ourselves to think? That’s number one. In this sense, I think the entire debate on the value of thinking beyond the outcome is important, because some people may be more validated and even aware that they are at risk of losing something important. So that’s number one, but I don’t have very high hopes overall.

Steven: I like to summarize it as: if we want to augment ourselves, we need to be intentional about our use of AI. Summarizing what Gideon was saying, I think intentionality is a big part of it. If you’re using AI with intent instead of reflexively, you’re more likely to verify outputs, more likely to think carefully about what you’re putting into these systems, what you’re asking, and how you’re prompting.

On the empirical side, there are some initial studies looking at cognitive forcing techniques—techniques to maintain some system two deliberation while using AI. The leading candidate, and Gideon and I are working on this as well, seems to be on the user side: think first and then go to the prompt—come up with an idea and think about it yourself before you go to AI. Don’t reflexively go to AI. On the UX side, if we slow down how outputs come back to us, that also helps, because it forces you to think while you’re waiting. I do that myself—if I find I’m using AI too much for my work, I’ll slow down the process by going to another screen after submitting the prompt, so it’s not immediately giving me feedback and I’m not immediately grabbing it and continuing on. Trying to slow things down a little bit—those are the two cognitive forcing techniques that have some behavioral evidence behind them.

Gideon: One thing Steven and I were talking about before the show—we were both watching the World Cup quite a lot, and you can think of VAR as a kind of system three to the referee. I think VAR is a very good model of system three, because the referee does what he has to do—he’s not losing his skills—but VAR interferes when the incentives are high. It doesn’t actively interrupt, it interferes when needed. I hated VAR in the beginning, but now I kind of like it, because maybe the world has changed, so now VAR looks very non-intrusive. It lets the referee do his thing, and it fulfills its role when needed. You can go ask it sometimes, but not for every little thing—the referee keeps doing the job.

Steven: And for most of the important VAR decisions, the humans—the refs—retain autonomy over the decision. In the end, they’re the decision makers. With cognitive surrender, at least the thinking portion of the decision is actually handed over to the machine—the thinking and the autonomy are in system three, at least by our definitions. With VAR, for example, the offside—the linesmen get a notice on their watch, and it’s green or red, and it’s correct 100% of the time because they’ve fixed that technology, and it’s fast enough that it augments in a very positive way. Are there still issues with VAR? Certainly, but they have that room with other referees who review the video outputs, and they have timing constraints. It took a while to get to that point, and I think the World Cup is really the first time they’ve started getting VAR right with the flow of play.

Ross: It’s a great example, and as you say, it does seem to be working very well, but it’s been quite a journey. I think other sports are also grappling with the integration of AI and visual game analysis.

Gideon: By definition, it’s a limited set of rules here—there are limited situations.

Ross: Yeah.

Gideon: Maybe I can design some AI that, whenever I’m doing specific behaviors, it’s going to interfere—like when I’m donating money to some cause, it jumps in. That could be an equivalent, but of course life is not as specific as the offside.

Ross: Yeah. Well, AI is bounded, humans are unbounded. You both work on vast expanses of interesting things beyond what we’ve talked about already, and one of the overlaps between both your work is in AI—let’s call it marketing. The framing, of course, is humans plus AI. We’ve got humans—wonderful things. We have AI as well, and we want to make the system come together in powerful ways. One of the things you’ve both worked on is persuasion technologies, and I think Marc Andreessen was saying his biggest fear is that we get to super-persuading AI, and that is a very genuine fear. I’d like to start by asking: what is the state of AI in marketing, or using it for persuasion marketing, and what are, maybe, societal more than individual responses to this?

Gideon: Actually, the most interesting paper I recently read was that it’s possible to use marketing to persuade the AI—you can ask the AI to give you instructions to create explosives, and you’re more likely to get it if you use by-the-book persuasion techniques from marketing. In some ways, it acts a bit like us. On the other hand, marketing is broader than we tend to think. We often think of it as an evil discipline, but at the end of the day, the goal of marketing is to help you make better decisions and maybe lead to the creation of better products. Of course, AI is extremely powerful in that—it has access to a lot of data, and, with limitations, the capacity to bring together things that are seemingly unrelated. It can synthesize a lot of information very quickly, so it’s very useful for marketing.

In many ways, AI is perceived psychologically as neutral—you don’t develop negative emotions toward it, and you don’t have the reactance you have when talking to a person. It feels as if it doesn’t have an agenda, maybe it also has less of an agenda, and I think that helps to break the block we often have when it comes to persuasion.

Steven: The most interesting topics about AI and marketing have to do with the fact that we see advertisements all the time—we get bombarded with hundreds, if not thousands, every day. More and more of those ads are being created by AI, so we are being exposed to these pre-processed stimuli that are different from what we see in the real world. For example, when you see people generated by AI, they tend to be much more attractive, more symmetrical, or things like this. If we’re repeatedly exposed to those types of ad copy and AI-generated content, it can change our perceptions of other people and our expectations of real people in the world.

I’m interested in how repeated exposure to AI-generated content in marketing will change the way we see the world. Separately, I think cognitive surrender in consumers is likely to be a big thing. Many people, when searching for a high-involvement product, go to Google and ask, “What’s the best portable air conditioner?”—many of us are in a heat wave right now. They might look at a list of top ten air conditioners, or maybe they’ll take the AI’s recommendation. Whatever platform you’re on, AI can very likely recommend a single optimal outcome.

We’re seeing large AI companies trying to be more profitable—OpenAI, for example, just started implementing ads at the bottom of their outputs. In our paper, Gideon and I make the claim that AI and system three are part of our thinking, and now, if system three is part of our thinking, and companies developing system threes are taking money to advertise in that area of artificial cognition, the possibility for things to go very wrong is very high. This could become the most powerful persuasive mechanism in history if we’re not careful about how corporate entities bring marketing content into AI outputs. We tend to think AI outputs are going to be accurate and correct, but if marketing advertisements are sold inside outputs, that would be a very persuasive area.

Gideon: A useful thing to say about marketing is that we’ve known for a long time—if you’re not paying for the product, you are the product. Now, of course, you provide training data to ChatGPT and so on, but the amount of training data each person provides is so small. Still, we are the product in the end, and we have to keep that in mind.

Ross: Yes. I think what you said before, Gideon, about the individual response to cognitive surrender applies exactly in the same way to any individual consumer today in a world of marketing—caveat emptor. But I think you also do work or framing around ethics. So, at a societal level, how should we be framing this? Are there any guidelines, frameworks, regulations, or other things that can help us move to a world where AI’s role in marketing is positive rather than detracting from value to participants in the economy?

Gideon: There are old-school guardrails, like disclosure—mandating disclosure. It’s hard, but we need to have criteria for what is okay and what is not. Humans have struggled with this always, and many times we find out the implications only after many years. There are lawsuits outside marketing now—people suing AI because it made them break up with their partner, or try to kill themselves. AI can do this kind of stuff, and I’m pretty sure that at the end of the day, lots of those will get to settlements, but if not, maybe precedent will be set. I would probably not count on the US being the leader in that domain, but we’ve seen with privacy that European regulations, like GDPR, eventually made it to the US. Probably there’s going to be something similar, but it’s very difficult. There are so many different scenarios, and we don’t know for sure what these models are doing computationally. These are vast models that are black boxes in many ways—you can try to reverse engineer them, but we have no idea what they’re really doing inside. They’re trying to predict language, not to do anything specific.

It’s a challenge. I’m mostly worried about long-term effects, especially in children. I’m not just talking about education, because I think it’s going to be even outside the classroom. Once you have things too easy, we know it’s not beneficial—if thinking comes at no cost, and without actually thinking, we may have fewer and fewer people thinking less and less. We’re already in a kind of crisis because of technology, in terms of education and global intelligence. It’s hard to show causality—we had COVID, lots of things happening at the same time—but I’m very worried about AI use in the classroom.

Steven: One thing I’ll add is that we had system one and system two—they were of evolutionary origin. We didn’t get to choose how those systems came about or how we think. With system three, the optimistic look is that this is the first thinking system we actually get to design. We can choose the characteristics we want in that system. The optimist would say, we hope we can design system three in a way that effectively augments our systems one and two and helps us make better decisions and live better lives.

Ross: Absolutely, absolutely right. That’s the whole humans-plus-AI thesis. We have to be deliberate, have intention, and design it. Whilst there are plenty of potential pitfalls, there’s also some real positive potential as well. So, where can people go to find out more about your work, Steven and Gideon?

Gideon: We have websites, but we’ve started—well, we booked cognitivesurrender.ai. It’s still not online, but everything is moving so quickly. There was a Congress hearing where Cognitive Surrender was mentioned. Now we’re recording a podcast video. Things move very quickly, and we’re going to have a place where all of this is going—maybe a nice newsletter you can sign up to. So, stay tuned. It’s going to take a little longer, but hopefully by the end of the summer we’ll have a nice website. It will be cognitivesurrender.ai 

Ross: Okay well, your websites, and their pages. Definitely let me know when it’s up, and we’ll share the word. Thank you so much for all of your wonderful work, your time, and your insights.

Steven: Thanks for having us, Ross.

Gideon: That was lovely. Thank you.