omlx

oMLX is a macOS application designed for optimized inference of large language models (LLMs) specifically targeting Apple Silicon hardware. It features continuous batching and tiered key-value caching, allowing users to manage model performance directly from the menu bar, persist context across dynamic conversations, and utilize custom kernel support for enhanced efficiency. Its primary use case is to facilitate practical local LLM operations for coding tasks, improving both speed and control compared to traditional LLM servers.