I was apparently too excited by a bit of convergence in thinking from Noah Smith and me in this old post (I recently used it as a reference, hence the re-read) that I forgot to call Noah out for not understanding information theory.