Detecting Phase Transitions

Can we detect phase transitions purely from weights? ... No. Not yet. 

Summary

• Our aim was to develop tools that could detect phase transitions (parts of training in which the model quickly learns a particular subtask) purely from weights.
• We ended up blocked by finding suitable datasets in which to study phase transitions. 
  • We attempted several techniques to control and induce transitions: "graduating the data" and studying bounded polynomials of varying difficulty, but these all ran into problems. 
  • We also looked at well-known tasks with transitions (grokking) and learning without transitions (MNIST & CIFAR-10).
• We hereby lay the seeds for (future) phase detectors.
  • You can find a GitHub repo with the (ongoing) work here.
  • Notebooks: (graduated) MNIST, bounded polynomials, CIFAR-10.