Mistakes are part of being human; pardoning them is a divine trait. However, when considering autonomous AI “agents” that are now performing duties once done by people, what level of imperfection is acceptable?
TL;DR
- Businesses are struggling with AI agent implementation due to security and governance concerns.
- Companies are in early phases of AI agent adoption, with few reaching full autonomy.
- Trust in AI agents is hindered by the need for duration and the risk of errors.
- Human oversight remains critical, especially in high-stakes fields like healthcare.
During Coins2Day's recent Brainstorm AI gathering in San Francisco, a panel of specialists debated this very topic, with industry leaders explaining their organizations' strategies for security and oversight—a concern that is now surpassing more immediate hurdles like data availability and processing capabilities. Businesses are engaged in a competitive struggle to deploy AI agents into their operations that can handle assignments independently and with minimal human oversight. However, numerous entities are encountering a core dilemma that is significantly hindering implementation: Rapid progress necessitates trust, while simultaneously establishing confidence demands considerable duration.
Dev Rishi, general manager for AI at Rubrik, joined the security company last summer following its acquisition of his deep learning AI startup Predibase. Afterward, he spent the next four months meeting with executives from 180 companies. He used those insights to divide agentic AI adoption into four phases, he told the Brainstorm AI audience. (To level set, agentic adoption refers to businesses implementing AI systems that work autonomously, rather than responding to prompts.)
According to Rishi’s learnings, the four phases he unearthed include the early experimentation phase where companies are hard at work on prototyping their agents and mapping goals they think could be integrated into their workflows. The second phase, said Rishi, is the trickiest. That’s when companies shift their agents from prototypes and into formal work production. The third phase involves scaling those autonomous agents across the entire company. The fourth and final stage—which no one Rishi spoke with had achieved—is autonomous AI.
Rishi's findings indicated that approximately 50% of the 180 surveyed businesses were engaged in experimentation and prototyping. Additionally, 25% were actively working to solidify their prototypes, with another 13% in the process of scaling up. The remaining 12% had not yet initiated any AI endeavors. Nevertheless, Rishi anticipates a significant shift in the near future: based on their strategic plans, those currently in the 50% category expect to advance to the second stage within the coming two years.
“I think we’re going to see a lot of adoption very quickly,” Rishi told the audience.
However, there’s a major risk holding companies back from going “fast and hard,” when it comes to speeding up the implementation of AI agents in the workforce, he noted. That risk—and the No.1 blocker to broader deployment of agents— is security and governance, he said. And because of that, companies are struggling to shift from agents being used for knowledge retrieval to being action oriented.
“Our focus actually is to accelerate the AI transformation,” said Rishi. “I think the number one risk factor, the number one bottleneck to that, is risk [itself].”
Incorporating artificial intelligence into the labor force
Kathleen Peters, chief innovation officer at Experian overseeing product strategy, stated that the slowdown stems from an incomplete grasp of the dangers when AI agents exceed the established limitations by companies and the necessary safety measures for such occurrences.
“If something goes wrong, if there’s a hallucination, if there’s a power outage, what can we fall back to,” she questioned. “It’s one of those things where some executives, depending on the industry, are wanting to understand ‘How do we feel safe?’”
Determining that aspect will vary for each business and will probably be especially challenging for firms in heavily regulated sectors, she pointed out. Chandhu Nair, senior vice president for data, AI, and innovation at the home improvement store Lowe’s, remarked that it's “fairly easy” to create agents, yet individuals don't grasp their nature: Are they a virtual worker? Is it a team? How will it be integrated into the company's structure?
Nair stated that “It’s almost like hiring a whole bunch of people without an HR function,”. “So we have a lot of agents, with no kind of ways to properly map them, and that’s been the focus.”
The company has been working through some of these questions, including who might be responsible if something goes wrong. “It’s hard to trace that back,” said Nair.
Peters from Experian anticipated that the upcoming years will witness many of those exact inquiries being debated publicly, occurring at the same time as discussions unfold privately within boardrooms and among high-level compliance and strategy groups.
“I actually think something bad is going to happen,” Peters said. “There are going to be breaches. There are going to be agents that go rogue in unexpected ways. And those are going to make for a very interesting headlines in the news.”
Significant disputes will draw considerable notice, Peters elaborated, and the standing of the company will be at stake. This will compel discussions about who bears responsibility for software and representatives, and it will probably result in stricter oversight, she stated.
“I think that’s going to be part of our societal overall change management in thinking about these new ways of working,” Peters said.
Nevertheless, tangible instances demonstrate how AI can offer advantages to businesses when it's integrated in manners that connect with both staff and clientele.
Nair said Lowe’s has seen strong adoption and “tangible” return on investment from the AI it has embedded into the company’s operations thus far. For instance, among its 250,000 store associates, each has an agent companion with extensive product knowledge across its 100,000 square foot stores that sell anything from electrical equipment, to paints, to plumbing supplies. A lot of the newer entrants to the Lowe’s workforce aren’t tradespeople, said Nair, and the agent companions have become the “fastest-adopted technology” so far.
“It was important to get the use cases right that really resonate back with the customer,” he said. In terms of driving change management in stores, “if the product is good and can add value, the adoption just goes through the roof.”
Who’s watching the agent?
He further stated that for individuals employed at the main office, the strategies for managing change must differ, thereby increasing the intricacy of the situation.
Numerous organizations find themselves at an initial dilemma: should they develop their own agents or depend on the artificial intelligence functionalities created by prominent software providers.
Rakesh Jain, who serves as the executive director for cloud and AI engineering at the healthcare system Mass General Brigham, stated that his company is adopting a cautious stance. Given that prominent platforms such as Salesforce, Workday, and ServiceNow are developing their proprietary agents, it might lead to duplication of effort if his organization were to develop its own agents concurrently.
“If there are gaps, then we want to build our own agents,” said Jain. “Otherwise, we would rely on buying the agents that the product vendors are building.”
In healthcare, Jain said there’s a critical need for human oversight given the high stakes.
“The patient complexity cannot be determined through algorithms,” he said. “There has to be a human involved in it.” In his experience, agents can accelerate decision making, but humans have to make the final judgment, with doctors validating everything before any action is taken.
Still, Jain also sees enormous potential upside as the technology matures. In radiology, for example, an agent trained on the expertise of multiple doctors could catch tumors in dense tissue that a single radiologist might miss. But even with agents trained on multiple doctors, “you still have to have a human judgment in there,” said Jain.
And the danger of an agent overstepping its bounds, even one intended to be reliable, is always a possibility. He likened a wayward agent to an autoimmune disorder, a condition notoriously challenging for medical professionals to identify and manage due to the internal nature of the threat. If an agent within a system “becomes corrupt,”, he stated, “it’s going to cause massive damages which people have not been able to really quantify.”
Rishi stated that a way ahead exists, notwithstanding the unresolved issues and anticipated difficulties. He outlined two prerequisites for fostering confidence in agents. Primarily, organizations require frameworks that assure agents are functioning within established guidelines. Secondly, they must establish unambiguous protocols and steps for situations where errors are bound to happen—a policy with real enforcement. Nair further contributed three elements for cultivating trust and progressing prudently: establishing identity and responsibility, understanding the agent's role; assessing the uniformity of the quality in each agent's results; and, examining the retrospective record that clarifies the reasons and timing of errors.
Nair stated that “Systems can make mistakes, just like humans can as well,”. “ But to be able to explain and recover is equally important.”
