The compute rethink: Scaling AI where data lives, at the edge

Presented by ArmAI is no longer confined to the cloud or data centers. Increasingly, it’s running directly where data is created — in devices, sensors, and networks at the edge. This shift toward on-device intelligence is being driven by latency, privacy, and cost concerns that companies are confronting as they continue their investments in AI. For leadership teams, the opportunity is clear, says Chris Bergey, SVP and GM, of Arm’s Client Business: Invest in AI-first platforms that complement cloud usage, deliver real-time responsiveness, and protect sensitive data. “With the explosion of connected devices and the rise of IoT, edge AI provides a significant opportunity for organizations to gain a competitive edge through faster, more efficient AI,” Bergey explains. “Those who move first aren’t just improving efficiency, they’re redefining what customers expect. AI is becoming a differentiator in trust, responsiveness, and innovation. The sooner a business makes AI central to its workflows, the faster it compounds that advantage.” Use cases: Deploying AI where data livesEnterprises are discovering that edge AI isn’t just a performance boost — it’s a new operational model. Processing locally means less dependency on the cloud and faster, safer decision-making in real time. For instance, a factory floor can analyze equipment data instantly to prevent downtime, while a hospital can run diagnostic models securely on-site. Retailers are deploying in-store analytics using vision systems while logistic companies are using on-device AI to optimize fleet operations. Instead of sending vast data volumes to the cloud, organizations can analyze and act on insights where they emerge. The result is a more responsive, privacy-preserving, and cost-effective AI architecture.The consumer expectation: Immediacy and trustWorking with Alibaba’s Taobao team, the largest Chinese ecommerce platform, Arm (Nasdaq:Arm) enabled on-device product recommendations that update instantly without depending on the cloud. This helped online shoppers find what they need faster while keeping browsing data private.Another example comes from consumer tech: Meta’s Ray-Ban smart glasses, which blend cloud and on-device AI. The glasses handle quick commands locally for faster responses, while heavier tasks like translation and visual recognition are processed in the cloud.”Every major technology shift has created new ways to engage and monetize,” Bergey says. “As AI capabilities and user expectations grow, more intelligence will need to move closer to the edge to deliver this kind of immediacy and trust that people now expect.” This shift is also taking place with the tools people use every day. Assistants like Microsoft Copilot and Google Gemini are blending cloud and on-device intelligence to bring generative AI closer to the user, delivering faster, more secure, and more context-aware experiences. That same principle applies across industries: the more intelligence you move safely and efficiently to the edge, the more responsive, private, and valuable your operations become. Building smarter for scaleThe explosion of AI at the edge demands not only smarter chips but smarter infrastructure. By aligning compute power with workload demands, enterprises can reduce energy consumption while maintaining high performance. This balance of sustainability and scale is fast becoming a competitive differentiator.”Compute needs, whether in the cloud or on-premises, will continue to rise sharply. The question becomes, how do you maximize value from that compute?” he said. “You can only do this by investing in compute platforms and software that scale with your AI ambitions. The real measure of progress is enterprise value creation, not raw efficiency metrics.”The intelligent foundationThe rapid evolution of AI models, especially those powering edge inferencing, multimodal applications, and low-latency responses, demands not just smarter algorithms, but a foundation of highly performant, energy-efficient hardware. As workloads grow more diverse and distributed, legacy architectures designed for traditional workloads are no longer adequate. The role of CPUs is evolving, and they now sit at the center of increasingly heterogenous systems that deliver advanced on-device AI experiences. Thanks to their flexibility, efficiency, and mature software support, modern CPUs can run everything from classic machine learning to complex generative AI workloads. When paired with accelerators such as NPUs or GPUs, they intelligently coordinate compute across the system — ensuring the right workload runs on the right engine for maximum performance and efficiency. The CPU continues to be the foundation that enables scalable, efficient AI everywhere.Technologies like Arm’s Scalable Matrix Extension 2 (SME2) bring advanced matrix acceleration to Armv9 CPUs. Meanwhile, Arm KleidiAI, its intelligent software layer, is extensively integrated across leading frameworks to automatically boost performance for a wide range of AI workloads, from language models to speech recognition to computer vision, running on Arm-based edge devices — without needing developers to rewrite their code.”These technologies ensure that AI frameworks can tap into the full performance of Arm-based systems without extra developer effort,” he says. “It’s how we make AI both scalable and sustainable: by embedding intelligence into the foundation of modern compute, so innovation happens at the speed of software, not hardware cycles.”That democratization of compute power is also what will facilitate the next wave of intelligent, real-time experiences across the enterprise, not just in flagship products, but across entire device portfolios. The evolution of edge AI As AI moves from isolated pilots to full-scale deployment, the enterprises that succeed will be those that connect intelligence across every layer of infrastructure. Agentic AI systems will depend on this seamless integration — enabling autonomous processes that can reason, coordinate, and deliver value instantly.”The pattern is familiar as in every disruptive wave, incumbents that move slowly risk being overtaken by new entrants,” he says. “The companies that thrive will be the ones that wake up every morning asking how to make their organization AI-first. As with the rise of the internet and cloud computing, those who lean in and truly become AI-enabled will shape the next decade.”Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.