grok-1

Grok-1 is an open-weights model implemented in JAX, featuring a mixture of experts architecture with 314 billion parameters primarily designed for advanced natural language processing tasks. It utilizes a maximum context length of 8,192 tokens, incorporates rotary embeddings, and supports activation sharding and 8-bit quantization, making it capable of handling large-scale inputs efficiently. Users must run the provided code on a GPU-equipped machine with adequate memory to sample from the model using the pre-trained checkpoints.