LoRAX (LoRA eXchange) is a framework that allows users to serve thousands of fine-tuned models on a single GPU, dramatically reducing the cost of serving without compromising on throughput or latency.

Abstract: Off-the-shelf x86 servers are widely deployed as middleboxes in edge and public clouds, such as cloud gateways and load balancers. They follow the “run-to-completion” model and achieve ...