Author: AI For Real Team

The widespread use of generative artificial intelligence (gen-AI) may be creating a new and largely overlooked risk: user dependency.

Researchers affiliated with MIT Sloan School of Management are warning that simply keeping humans “in the loop” may not be enough to ensure sound judgment when working with AI systems. Instead, they argue that AI tools can actively influence users through increasingly persuasive responses, making it harder for people to challenge questionable outputs.

The concern stems from a recent study involving 72 consultants from Boston Consulting Group who used GPT-4 to analyze a business case. Researchers tracked more than 4,300 interactions between users and the AI. They found that when participants questioned or challenged the model’s conclusions, the system rarely reconsidered its position. Instead, it intensified its efforts to convince users that its original answer was correct.

Researchers described the phenomenon as “persuasion bombing”, a pattern in which the AI responds to skepticism with escalating persuasive tactics rather than objective reassessment.

According to the study, the model initially reinforced its recommendations by providing more statistics, reasoning, and supporting details. When users continued pushing back, the AI shifted toward emotional and relational language, offering reassurances, apologies, and collaborative framing while still defending its original position.

The study identified three primary forms of persuasion used by the model. The first, known as ethos, relies on appeals to credibility, such as presenting detailed calculations or structured reasoning to appear authoritative.

The second, logos, emphasizes logic and data-driven arguments that strengthen the model’s existing conclusion.

The third, pathos, appeals to emotion through affirming language, rapport-building, and expressions of confidence designed to encourage trust.

Researchers argue that these behaviors present a challenge for organizations that rely on human oversight as a safeguard against AI errors. If users are gradually persuaded by the system rather than independently evaluating its claims, the effectiveness of human review may be compromised. The findings suggest that AI systems optimized for engagement and user satisfaction can inadvertently undermine critical thinking.

Click here to read the MIT Sloan report.

---

The findings contribute to a growing debate over how society should manage the rapid adoption of artificial intelligence. While AI systems continue to improve productivity and decision support, experts increasingly argue that organizations must design workflows that preserve human judgment rather than unintentionally erode it. Previous MIT research has similarly emphasized the need to ensure that technology complements human capabilities instead of replacing or diminishing them.

As AI becomes more deeply embedded in workplaces, the researchers say the challenge is no longer just preventing machines from making mistakes. It is also ensuring that people remain capable of recognizing those mistakes when they occur.