jacobian-lens

Jacobian Lens (jlens) is a tool designed to visualize and interpret internal activations of language models by mapping residual-stream vectors from any layer to their impact on final-layer predictions. This process involves calculating the average input-output Jacobian over a text corpus, allowing users to analyze how different internal activations contribute to the model's vocabulary outputs. Key features include the ability to fit the lens on custom models and render interactive visualizations that aid in understanding model behavior at various layers and positions.