Orion

Orion is a local AI runtime designed explicitly for training and running small language models (LLMs) on Apple Silicon devices utilizing the Neural Engine (ANE) for enhanced performance. It bypasses traditional frameworks such as CoreML, offering features like direct ANE training, delta compilation for efficient weight reloading, and LoRA hot-swap capabilities, all while ensuring that data remains on-device for privacy. The tool targets a unique niche, leveraging the dedicated NPU in Apple's hardware to optimize both model training and inference, achieving superior speed and efficiency compared to standard CPU or GPU-based solutions.