well, you don't actually have to have your models fight their models. there's lots of things to try that aren't the SOTA rat race.
like, if you want to show relative improvement of some new variation of an RL algorithm, this could be a good way to do it. or if you have a new environment that you want to solve for yourself. right now if you try to train anything in a moderately interesting environment on a PC, it takes just a little too long to get results -- makes the whole research process pretty painful.
I'm afraid that computing resources are not and have never been the limiting factor for innovative work in machine learning in general and in deep learning in particular. I have quoted the following interview with Geoff Hinton a number of times on HN - apologies if this is becoming repetitious:
GH: One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas.
Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad.
What we should be going for, particularly in the basic science conferences, is radically new ideas. Because we know a radically new idea in the long run is going to be much more influential than a tiny improvement. That's I think the main downside of the fact that we've got this inversion now, where you've got a few senior guys and a gazillion young guys.
In other words, yes, unfortunately, everything is the SOTA rat race. At least anything that is meant for publication, which is the majority of research output.
at the same time, if you go to this year's ICML papers and ctrl-F "policy", there are several RL papers that come up with a new variant on policy gradient and validate it using only relatively small computing resources on simpler environments without any claim of being state of the art. probably many would directly benefit from this well-optimized policy gradient code.
Well, that's encouraging. "Pourvu que ça dure !" (as Letizia Bonaparte said).
It's funny, but older machine learning papers (most of what was published throughout the '70s, '80s and '90s) was a lot less focused on beating the leaderboard and much more on the discovery and understanding of general machine learning principles. As an example that I just happened to be reading recently, Pedro Domingos and others wrote a series of papers discussing Occam's Razor and why it is basically inappropriate in the form where it is often used in machine learning (or rather, data mining and knowledge discovery, since that was back in the '90s). It seems there was a lively discussion about that, back then.
like, if you want to show relative improvement of some new variation of an RL algorithm, this could be a good way to do it. or if you have a new environment that you want to solve for yourself. right now if you try to train anything in a moderately interesting environment on a PC, it takes just a little too long to get results -- makes the whole research process pretty painful.