This is one of the first papers in the neuromorphic vein that I think may hold up. It would be amazing if it did too due to the following properties:
-Linear (transformer) complexity at training time
-Linear scaling with number of tokens
-Online learning(!!!)
The main point that made me cautiously optimistic:
-Empirical results on par with GPT-2
I think this is one of those ideas that needs to be tested with scaled up experiments sooner rather than later, but someone with budget needs to commit. Would love to see HuggingFace do a collab and throw a bit of $$$ at it with a hardware sponsor like Nvidia.
I guarantee if there's even a 0.1% chance of this architecture eventually outperforming traditional ones, then Zuckerberg et al are already eating the cost and have teams spinning up experiments doing just that.
That's not true. The AI industry appears to play a game of follow the leader copying other companies and major researchers. There's all kinds of good ideas we never see applied by big companies. So, it's not safe to assume they tried them all and they didn't work.
In fact, we've sometimes seen new companies show up with models based on research big companies didn't use, the new models are useful or better in some way, and people use them or big companies acquire them. I'd say that's proof big companies miss a lot of good ideas internally.
Not every company is investigating every direction. Like, it's clear that Google is investing a lot in embodiment and multimodal understanding, but Anthropic barely cares about either. Across the field though?
I think it's fairly safe to say that every remotely promising thing that showed up in the papers was tried at some big lab at least once. If it showed good results, they'd pick it up.
Absolutely agreed, but we may not even hear about it as Meta has made it clear they're not necessarily committed to the open source first policy at this point.
-Linear (transformer) complexity at training time
-Linear scaling with number of tokens
-Online learning(!!!)
The main point that made me cautiously optimistic:
-Empirical results on par with GPT-2
I think this is one of those ideas that needs to be tested with scaled up experiments sooner rather than later, but someone with budget needs to commit. Would love to see HuggingFace do a collab and throw a bit of $$$ at it with a hardware sponsor like Nvidia.