A downloadable project

Can we detect phase transitions purely from weights? ... No. Not yet. 

Summary

  • Our aim was to develop tools that could detect phase transitions (parts of training in which the model quickly learns a particular subtask) purely from weights.
  • We ended up blocked by finding suitable datasets in which to study phase transitions. 
    • We attempted several techniques to control and induce transitions: "graduating the data" and studying bounded polynomials of varying difficulty, but these all ran into problems. 
    • We also looked at well-known tasks with transitions (grokking) and learning without transitions (MNIST & CIFAR-10).
  • We hereby lay the seeds for (future) phase detectors.

Download

Download
Write-up
External
Download
GitHub repo
External

Leave a comment

Log in with itch.io to leave a comment.