Beyond TOPS: The Efficiency Contest for On-Device Large-Model Chips | WAIC UP! Highlights

Yu Hao proposes that evaluating on-device large-model chips should shift from TOPS to comprehensive metrics such as effective memory bandwidth, energy efficiency, and tokens/s. A 3D-DRAM stacked cubic systolic TPU architecture measured about 80 tokens/s for INT4-quantized Decode of a 3B model at an average power consumption of 5W, with token energy efficiency of 15 token/W and bandwidth utilization of about 80%. The related 3D-TPU solution is at the sample stage, and Maitexin's AI agent MetaClaw has been adapted to Qwen, MiniCPM, and Gemma and is undergoing pilot applications.