AT&T Transforms AI Orchestration to Enhance Productivity and Cost Efficiency

AT&T has recently restructured its approach to artificial intelligence (AI) orchestration in response to its staggering daily token usage, which reached 8 billion. Chief Data Officer Andy Markus and his team recognized the need for a more efficient system that would not only manage this scale but also reduce costs. They developed a multi-agent stack utilizing LangChain, where large language model “super agents” oversee smaller, task-specific “worker” agents. This new orchestration layer has resulted in a significant increase in processing capacity, allowing AT&T to handle 27 billion tokens daily, a threefold increase in just a few months. Markus emphasized the effectiveness of smaller language models, stating, “I believe the future of agentic AI is many, many, many small language models (SLMs).”

VentureBeat

The Ask AT&T Workflows platform, built on this re-architected stack, has been deployed to over 100,000 employees, with more than half using it daily. Active users report productivity gains of up to 90%. The platform offers two user journeys: a pro-code option for those familiar with programming and a no-code drag-and-drop interface designed for ease of use. Interestingly, even technically proficient employees have shown a preference for the no-code option during recent hackathons. Employees utilize these agents across various functions, such as network management, where agents can autonomously identify issues, generate trouble tickets, and propose solutions, all while being supervised by human engineers.

VentureBeat

Markus highlighted a transformative approach to coding, termed "AI-fueled coding," which streamlines the software development cycle. This method reduces the iterative back-and-forth typically seen in traditional coding practices, enabling faster production of high-quality code. Non-technical teams can also leverage this technique, allowing them to create software prototypes in a fraction of the time—20 minutes instead of six weeks for a curated data product. “We develop software with it, modify software with it, do data science with it, do data analytics with it, do data engineering with it,” Markus stated, calling it a “game changer.”

VentureBeat

Markus noted that AT&T's strategy involves rigorous evaluations of both homegrown tools and off-the-shelf options, adapting to the rapidly evolving AI landscape. He cautioned against overcomplicating solutions, urging developers to consider whether simpler, single-turn generative solutions could suffice. “Sometimes we over complicate things,” he remarked, emphasizing the importance of accuracy, cost, and responsiveness in tool selection.

VentureBeat

While AT&T's advancements in AI orchestration have garnered praise, some critics may argue that reliance on multiple smaller models could lead to inconsistencies in performance across different tasks. Additionally, the shift towards no-code solutions might raise concerns about the depth of technical understanding among employees, potentially impacting long-term innovation.

VentureBeat

As AT&T continues to refine its AI orchestration strategies, the company plans to phase out homegrown tools in favor of more efficient, off-the-shelf solutions. This ongoing evolution reflects the fast-paced nature of AI technology, with Markus indicating that the company must remain adaptable to stay ahead in the industry.

VentureBeat