Garuav Sood writes:
You had expressed slight frustration with some ML/CS papers that read more like advertisements than anything else. The attached paper by Zachary Lipton and Jacob Steinhardt flags four reasonable concerns in modern ML papers:
Recent progress in machine learning comes despite frequent departures from these ideals. In this paper, we focus on the following four patterns that appear to us to be trending in ML scholarship:
1. Failure to distinguish between explanation and speculation.
2. Failure to identify the sources of empirical gains, e.g. emphasizing unnecessary modifications to neural architectures when gains actually stem from hyper-parameter tuning.
3. Mathiness: the use of mathematics that obfuscates or impresses rather than clarifies, e.g. by confusing technical and non-technical concepts.
4. Misuse of language, e.g. by choosing terms of art with colloquial connotations or by overloading established technical terms.
I don’t know that machine learning is worse than any other academic field, but I have noticed a sort of arms race by which it becomes almost necessary to use certain buzzwords, just because other papers are using those buzzwords.
One of my pet peeves is the word “provably.” Everyone wants to say that their method is “provably” wonderful in some way. OK, fine, but “proof” is just another word for “assumption.” Indeed, all Bayesian inferences are provably optimal, if you average over the assumed prior and data models.
I’m not saying that proofs are bad—a proof can give insight, especially into the conditions required for a desired assertion to be true. Sometimes a proof is helpful, other times there’s no real reason for it. My problem is that, once there’s the expectation that a CS paper will have “provably” in its abstract, then that can create a push for everyone to have that “provably” there, just to compete.
Again, this is not a problem unique to CS. The problem is, once there’s a culture of hype, it can sustain itself through the competitive review and publicity process.