AI inference chips see surge in new architectures and European entrants

The autumn of 2026 is set to see significant changes in AI inference. As inference becomes the dominant AI workload, success depends not only on powerful chips but also on system-level innovation from new architectures and memory designs. Several new chips are coming to the AI datacentre, from OpenAI’s Jalapeneo and Intel’s Crescent Island GPU to start up unicorn Fractile. Other start ups such as Taalus and Tensordyne have been developing dedicated chips for AI inference in the datacentre rather than programmable graphics processors (GPUs). The edge AI market is also changing in the next few months, with Hailo being bought by Microchip and Nvidia going head-to-head with Apple with desktop machines. At the same time AMD is to combine its fifth-generation AMD GPU, the MI455X, with the wafer-scale inference engine developed by Cerebras. Then there is also a chiplet technology called Titania being developed by Axelera in the Netherlands and a new chip by Graphcore in the UK, again for datacentre applications later this year. Other established companies such as SambaNova and Tenstorrent are also pitching their chips for AI inference, with cloud providers starting to adopt these technologies. However, with datacentres hoovering up the supply of high-bandwidth memory (HBM) stacks and even DRAM, the memory architecture and support is increasingly important. Jalapeño OpenAI’s Jalapeño inference chip is built from the ground up for current and future LLMs from the company and was developed from design to production in nine months, with Broadcom using OpenAI’s models to speed up the chip design. This will be deployed at gigawatt scale with datacentre partners over multiple generations, says Broadcom, and starts deploying by the end of 2026. Engineering samples of the Jalapeño chip are running ML workloads in the lab at production target frequency and power, including GPT-5.3-Codex-Spark. OpenAI designed the chip from scratch around its deep understanding of LLM fundamentals, informed by its roadmap of models, kernels, serving systems, and product needs, with partners Broadcom and Celestica, helping industrialize the platform through chip implementation, board, rack system integration, high-performance networking, and scalable production systems. Jalapeño is designed with flexibility to work with all LLMs. The architecture reduces data movement and balances compute, memory, and networking resources to achieve realized utilization much closer to theoretical peak performance. Broadcom’s silicon implementation and networking technologies, including Tomahawk networking silicon, help bring the platform to large-scale production. While a detailed technical report is still to be published, OpenAI says early testing shows that Jalapeño will deliver performance per watt substantially better than current state-of-the-art chips. A detailed technical report on performance will be presented in the coming months. “The world is moving to a compute-powered economy,” said Greg Brockman, President and Co-Founder, OpenAI. “Jalapeño is part of our long-term full-stack infrastructure strategy to make compute more abundant, resulting in AI which is faster, more reliable, more affordable for people and businesses, and can be used to solve more important problems. By designing more of the stack ourselves, we can serve more intelligence with greater efficiency and keep pushing advanced AI toward broader access.” “Jalapeño was designed from the ground up for LLM inference using detailed insights from our close collaboration with OpenAI researchers,” said Richard Ho, who leads OpenAI’s hardware program. “We optimized the architecture around the kernels, memory movement, networking, and serving patterns that matter most for frontier AI models. Based on early testing, Jalapeño will efficiently execute our most important workloads close to the hardware’s theoretical limits.” “Our collaboration with OpenAI represents a fundamental commitment to scaling the physical infrastructure required for the next decade of AI,” said Hock Tan, President and CEO, Broadcom. “This is just the beginning of a multi-generation roadmap. By co-developing our industry-leading silicon directly with OpenAI, we are enabling the deployment of gigawatt scale data centers with Microsoft and other partners beginning in 2026.” Crescent Island Intel is returning to the GPU market with a datacentre device code-named Crescent Island for AI inference workloads with high memory capacity and energy-efficient performance.    “AI is shifting from static training to real-time, everywhere inference—driven by agentic AI,” said Sachin Katti, CTO of Intel. “Scaling these complex workloads requires heterogeneous systems that match the right silicon to the right task, powered by an open software stack. Intel’s Xe architecture datacentre GPU will provide the efficient headroom customers need—and more value—as token volumes surge.” Intel worked with datacentre standards group the Open Compute Project (OCP) on the design, which is optimised for air-cooled enterprise servers and incorporates large amounts of memory capacity and bandwidth for inference workflows. The chip is based on the Xe3P microarchitecture that is used in the coming Nova Lake-S and -H CPUs with optimised performance per watt and support for 160GB of low-power LPDDR5X memory. There is also support for a broad range of data types for “tokens-as-a-service” providers and inference. Intel’s software stack for heterogeneous AI systems is currently being developed and tested on Arc Pro B-Series GPUs to enable early optimisation, and customer samples are due in the second half of 2026. Maia 200 Meanwhile, hyperscalers are developing their own chips. Microsoft has developed the Maia 200, launched in January, with a redesigned architecture that focuses on memory throughput. The 750W chip is built on TSMC’s 3nm process technology and uses a hierarchical micro-architecture with a series of tiles. Each tile integrates two complementary execution engines: a Tile Tensor Unit (TTU) for high-throughput matrix multiply and convolution, and a Tile Vector Processor (TVP) as a highly programmable SIMD engine. These engines are fed by multi-banked Tile SRAM (TSRAM) and a tile-level DMA subsystem for moving data into and out of that SRAM without stalling the compute pipeline. A lightweight Tile Control Processor (TCP) runs the low-level code emitted by the software stack and orchestrates TTU and DMA work issuance, while hardware semaphores provide fine-grained synchronization between data movement and compute. All of this means the chip has 272 GB of SRAM on the die and links to 216 GB of HBM3e memory that can scale up to 6144 devices in a cluster. It is already deployed in datacentres in Iowa and Arizona, with plans to expand to Italy, Australia, and South Korea later this year. AI inference sovereignty UK-based startup Fractile is highlighting the need for a European source for inference chips. Backed by the NATO Investment Fund and a recent $212m funding round, the company is valued at over $1bn for its RISC-V-based compute-in-memory architecture. It plans to release its first chip in the second half of 2026. CEO interview: Walter Goodwin, Fractile Another UK chip designer, although owned by SoftBank of Japan, is Graphcore. Back in 2020, it developed the second generation of its Colossus chip, the GC200, that integrated 1472 cores and 900Mbits of SRAM memory. This ran software developed by the company before the advent of LLMs, and the company also had to develop its own boards and racks. The company has key expertise in memory integration and compute-in-memory and is expanding its design teams in Bristol and Cambridge in the UK as well as a 500-strong team in Bengaluru in India. These teams are developing the Izanagi chip which is also planned for later this year. This is likely to take advantage of the wafer-to-wafer bonding technology that Graphcore has already demonstrated to link the processor to very large amounts of SRAM memory. This would boost performance while avoiding the current problems of sourcing HBM memory stacks and even LPDDR memories. Other inference AI chips There are several inference AI chips under development. Taalus is taking specific AI models and creating custom hardware for them to keep power consumption down while boosting performance. The Taalas HC1 Technology Demonstrator is built in a 6nm process at TSMC and runs the Llama 3.1 8B AI model in a 2.5kW server. The challenge is choosing the right model to turn into hardware. Another startup is Tensordyne, which started out in 2017 as Recogni with a focus on automotive vision AI. The shift to datacentre inference uses a logarithmic number system (LNS) instead of conventional floating-point maths. Its 3nm Napier inference AI processor converts complex matrix multiplications into simple additions, shrinking compute logic area to optimise the performance per watt. The processor has 48 compute nodes optimised for transformer AI models and links to 144 GB of HBM memory. It is packaged into an air-cooled pod with a 72-chip chassis delivering 76.7 petaflops of FP16 equivalent compute at 30 kW, a third of other leading-edge systems. Edge AI inference Nvidia fired the gun on edge AI devices with its Project Digits design, now shipping as the DGX Spark. This is based on the standalone GB10 Grace Blackwell superchip developed with Mediatek and integrates 20 ARM cores (10 Cortex-X925 and 10 Cortex-A725) with 6144 CUDA cores in the Nvidia Blackwell architecture. With 128 GB of LPDDR5x coherent unified system memory, this delivers up to 1 PetaFLOP at FP4 4bit precision or 1,000 AI TOPS. Power consumption is one of the other key criteria for inference chips, whether in a rack or in a box, and the GB10 has a thermal envelope of 140W and the DGX Spark box needs a 240W power supply. This can run leading-edge AI models such as Kimi-K2 Thinking, DeepSeek-V3.2, Mistral Large 3, Meta Llama 4 Maverick, Qwen3 and OpenAI gpt-oss-120b. “The Nvidia GB300 is typically deployed as a rack-scale system,” said Kaichao You, core maintainer of vLLM. “This makes it difficult for projects like vLLM to test and develop directly on the powerful GB300 superchip. DGX Station changes this dynamic. By delivering GB300 in a compact, single-system form factor deskside, DGX Station enables vLLM to test and develop GB300-specific features at a significantly lower cost. This accelerates development cycles and makes it easy for vLLM to continuously validate and optimize against GB300.” “DGX Station brings data-centre-class GPU capability directly into my room,” said Jerry Zhou, community contributor to SGLang. “It is powerful enough to serve very large models like Qwen3-235B, test training frameworks with large model configurations and develop CUDA kernels with extremely large matrix sizes, all locally without relying on cloud racks. This dramatically shortens the iteration loop for systems and framework development.” A range of manufacturers have developed their own versions of the DGX Station, with systems starting to ship from Asus, Boxx, Dell Technologies, Gigabyte, HP Inc, MSI and Supermicro. Mediatek has also helped Nvidia with a similar version of the GB10 for PCs running Windows 11 to use AI agent frameworks such as OpenClaw. The RTX Spark chip has the same 128 GB of unified memory as the GB10 and is aimed at AI laptops shipping later this year, and has a power envelope of 80 to 110W. “We are strong supporters of deploying agents like OpenClaw securely into the Windows ecosystem,” said Vincent Koc, chief architect at the OpenClaw Foundation. “Running solutions like OpenShell and the Microsoft security primitives on RTX Spark will enable users to leverage a fully integrated stack for private, personal agents running on device.” Metis Axelera points to a developer running millions of tokens a day on two mini PCs, ditching cloud APIs entirely to cut costs. This is a key shift. As AI becomes more capable and more embedded in daily operations, the case for running it locally, on in-house hardware, grows stronger. Open-weight models have matured and the mini AI PC has evolved into a serious computing platform capable of replacing traditional desktops across a wide range of workloads. So the Axelera Mini PC is powered by its first generation Metis chip on a PCI M.2 card for edge AI. Axelera plans datacentre inference AI push For AI vision, that shift is particularly significant. A factory line running continuous defect inspection generates too much data to route to the cloud efficiently. Over 40% of manufacturing plants globally are projected to use edge-enabled computer vision systems by 2027, according to analysts Market Mind Partners. The Axelera AI mini PC based on the Metis M.2 card Bringing AI vision in-house means processing data where it’s generated, scaling deployments for actual operational needs rather than being constrained by cloud architecture, says Axelera. This also allows systems to remain fully functional regardless of network conditions or API pricing changes. Apple M5 Ultra While Nvidia has been benchmarking its DGX Station against the two-year-old Apple M4 Max processor, a new chip is set to provide more performance for AI inference. The M5 Ultra is set for launch later this year and is expected to combine two M5 dies. These dies have up to 10 custom ARM cores with up to 10 GPU cores that are optimised for AI. The M5 Max combines two of these dies, so the specification of the M5 Ultra is expected to include more memory to support larger AI models. Hailo Microchip Technology is to acquire Israeli chip developer Hailo in the next two months for an undisclosed sum. This will provide a significant boost for Microchip’s edge AI, adding the Hailo-8, Hailo-10 and Hailo-15 chips for computer vision, while adding advanced camera, ISP, DSP, video encoding and AI video stream processing capabilities for intelligent edge systems. This follows the acquisition of Neuronix AI Labs, which added neural network optimization technology for AI/ML workloads on FPGAs and system-on-chip devices. “The acquisition of Hailo accelerates Microchip’s expansion into high-performance edge AI processing,” said Mark Reiten, Senior Corporate Vice President. “Hailo’s AI acceleration, advanced vision processing and software ecosystem directly complement Microchip’s embedded processing, FPGA, connectivity, security, power and analog portfolio. Together, we can help customers build more capable intelligent edge systems with the right balance of performance, power efficiency, reliability and system cost.” “Joining Microchip would give Hailo the opportunity to scale our accelerated edge AI technology through a global embedded systems leader,” said Orr Danon, CEO of Hailo. “Microchip’s customer reach, channel scale and broad technology portfolio would create a strong platform for bringing advanced vision processing and AI acceleration to a broader range of intelligent edge applications.” The boom in AI- is driving new chip architectures and memory designs, with the technology this year driving down from the datacentre to the desktop. A wide range of chip technologies are emerging over the next few months which are set to boost productivity in edge AI deployments but also change the economics of AI inference. The post AI inference at the crossroads appeared first on eeNews Europe.