Identifying Interactions at Scale for LLMs

Berkeley AI Research introduces SPEX and ProxySPEX, two algorithms that use structural assumptions such as sparsity, low degree, and hierarchy to turn interaction discovery in ablation attribution into a sparse recovery problem, identifying the key interactions that drive LLM outputs.