here's a summary for non-technical people
imagine you have a really smart friend who's great at understanding language, but they're terrible at math. you also have another friend who's a math whiz, but they can't understand a word you say. this paper is about combining these two friends to make a super-powered friend who can do both.
the "smart friend" is a transformer, a type of AI that's really good at understanding natural language. the "math whiz" is a neural algorithmic reasoner (NAR), a type of AI that's really good at solving math problems.
the researchers combined these two AIs into a new system called TransNAR. they trained TransNAR on a bunch of math problems that were written in natural language. they found that TransNAR was much better at solving these problems than the transformer alone.
this is because TransNAR can use the NAR to help it understand the math problems better. it's like having a math tutor who can explain the problems in a way that the transformer can understand.
the researchers also found that TransNAR was better at solving problems that were different from the ones it was trained on. this is because the NAR is really good at generalizing to new problems.
so, TransNAR is a new way to combine the strengths of two different types of AI. it's a promising step towards building AIs that can solve complex problems in the real world.
for technical people here's a good vid on progress by one of the authors of the paper: [https://www.youtube.com/watch?v=btF19HOrWC4](https://www.youtube.com/watch?v=btF19HOrWC4)
The paper cites a similarity to multi modal models where the Transformers representation vectors are sort of 'glued' to a vision transformer's (or CNN) representation vectors. This hopefully allows the models to learn / utilize each others representations.
It also makes the conversation about "LLM's and OpenAI setting AGI progress back by 5-10 years" a little silly.
Clearly DeepMind is proof that not every dollar and researcher is invested into just making the newest and flashiest LLM without any thought to long term changes and breakthroughs.
There is always work going on behind the scenes, it just seems that ChatGPT in the public's eyes is getting all the plaudits.
Yeah the naysayers will always be here, lack of imagination on their part I suspect. Hot take: there is no wall that we'll hit in our lifetime, instead progress will continue steadily then exponentially and end in super intelligence. Language models are understated in their capability and imo already smarter than the median human - most people are really dumb.
>The article explores a novel approach to combining Transformers, a neural network architecture adept at understanding natural language, with neural algorithmic reasoners (NARs), which are proficient at algorithmic tasks. The authors propose a hybrid architecture, TransNAR, that leverages the strengths of both Transformers and NARs to achieve robust algorithmic reasoning. Notably, TransNAR outperforms a baseline Transformer model on the CLRS-Text benchmark, a collection of text-based algorithmic problems.
>Future AI is expected to benefit from the marriage of Transformers and NARs in several ways. First, this approach can enhance the ability of AI systems to solve algorithmic problems expressed in natural language. Second, it alleviates the issue of Transformers struggling with out-of-distribution generalization, where the test data deviates from the training data. Third, TransNAR paves the way for AI systems to leverage external tools and knowledge sources to inform their reasoning.
>However, there are limitations to consider. TransNAR necessitates both textual and graph-based inputs, which may not always be available. Future research will need to address this limitation to make this approach more applicable in real-world scenarios. Overall, the findings presented here are a promising step towards more robust and versatile AI systems.
After all the recent talk about in and out of distribution reasoning, especially after Francois's podcast appearance, this seems almost intentionally timed.
To me it's obvious people are working on modifications to improve reasoning in transformer models, but it's nice to actually see what's happening behind closed doors
This is probably one of the most popular research topics in AI. I don't think it's timed to any one event or discussion, it's just one of the hottest topics of the last 3-4 years.
It always amazes me that all of the research seems to be coming out of DeepMind, but OpenAI still seems to be at the cutting edge in terms of available products. What is it that’s holding DeepMind back exactly?
Everything "seems" this way probably because you spend too much time on this sub which is up OpenAI's ass. See how much gpt-5 is hyped compared to Gemini 2? And how quickly Google caught up from a sub-par PaLM 2 model to Gemini 1.5 pro in one year (which is now #2 in the arena)? And how quickly Gemini models are improving (1.5 pro was released like 3 months after 1.0 pro and a few weeks after Ultra)? And that they have 2M token context window (nothing else even comes close, AND it's multimodal), combined with the fact that it is also way cheaper to serve than any model of comparable quality?
Nothing is holding DeepMind back - they're in the lead if you ask me, and their models are improving more quickly than anyone elses. Gemini 2 is going to be a big step forward for them - there's a reason 1.5 was "only" called 1.5, Google knows that their next generation of models will be truly on another level.
Exactly, DeepMind is the clear favorite for the AGI race. And now that Google is *actually trying* to make good AI products they are making astonishing progress.
OAI just always leaps frog the competition, GPT-2, GPT-3, GPT-4 their models are always impressive and SOTA on release.
This exact situation we are in now, is very reminiscent of the state of pre-gpt4. Where it seems like OAI is falling very behind and players like NVIDIA, Meta, Google etc. are all releasing similarly performant models to what GPT-3 was and certainly above GPT-3s performance, everyone thought wtf was OAI doing, only to be later destroyed by GPT-4 and it took months to catch up.
Circumstances are a bit different but if feels extremely similar to that time. History repeats itself, am I right?
But I don't expect that to be fully true in this case. I do think OpenAI has the lead, but Anthropic and Deepmind are closing in. Gemini 2 should be an impressive model and I have high expectations, same with GPT-5 and Anthropic's equivalent of GPT-5.
Christ what the hell are you guys saying. OpenAI just doesn’t publish because they don’t want to share information. This is just a company stance OpenAI follows, while deep mind chooses to share SOME THINGS. Even deepmind is hiding a lot its research. I can’t believe you guys said what you said.
Publishing your research publicly has nothing to do with how ahead you are of other competitors. You just choose to make it public.
this. remember how everyone lost their mind when Google came out with Bard demo (old Gemini) and it got some facts incorrect? Google lost $100 billion in market value. This was a little over a year ago.
The latest Gemini Pro is basically at GPT-4o level, maybe a tiny bit worse. OTOH, Gemini Flash is far better than anything OpenAI has in the same price range.
I wouldn't say Google is behind in terms of product quality. They are slightly behind in mindshare, because OpenAI released their models first, but that's about it.
ChatGPT is a cool product but I still don't think it's as cool as Waymo, ChatGPT is great as Google Assistant on steroids, but it still is difficult to trust it for any serious tasks. I would trust a Waymo taxi with my life on the other hand.
The reason why Deepmind has more research publications is simply because they actually publish whereas OpenAI keeps their research to themselves. Why does Deepmind continue to publish then? I assume it’s the same reason Facebook open sourced their model weights, to destroy OpenAI’s moat.
This, at scale, across many domains (math problems, reasoning problems, programming, etc), and fully grokked should basically solve the reasoning issue.
The next AI research comes out of Google again. The 1st product that uses this research will come out of some other company because Google's executives are so bad at their jobs. They won't be able to turn this into a good product.
A lot of different types of biomedical data are efficiently encoded in graphs. Especially sequence data like RNA, DNA and protein and interaction networks at all scales (molecular, cellular, tissue, etc). This could be a killer model for integrating the scientific literature with the raw data the literature is based on. For biomed that might look like mining pubmed for articles and NCBI for sequence data.
here's a summary for non-technical people imagine you have a really smart friend who's great at understanding language, but they're terrible at math. you also have another friend who's a math whiz, but they can't understand a word you say. this paper is about combining these two friends to make a super-powered friend who can do both. the "smart friend" is a transformer, a type of AI that's really good at understanding natural language. the "math whiz" is a neural algorithmic reasoner (NAR), a type of AI that's really good at solving math problems. the researchers combined these two AIs into a new system called TransNAR. they trained TransNAR on a bunch of math problems that were written in natural language. they found that TransNAR was much better at solving these problems than the transformer alone. this is because TransNAR can use the NAR to help it understand the math problems better. it's like having a math tutor who can explain the problems in a way that the transformer can understand. the researchers also found that TransNAR was better at solving problems that were different from the ones it was trained on. this is because the NAR is really good at generalizing to new problems. so, TransNAR is a new way to combine the strengths of two different types of AI. it's a promising step towards building AIs that can solve complex problems in the real world. for technical people here's a good vid on progress by one of the authors of the paper: [https://www.youtube.com/watch?v=btF19HOrWC4](https://www.youtube.com/watch?v=btF19HOrWC4)
gemini assisted?
So another approach to MOE, which with SLMs will go far in making bots ubiquitous
The paper cites a similarity to multi modal models where the Transformers representation vectors are sort of 'glued' to a vision transformer's (or CNN) representation vectors. This hopefully allows the models to learn / utilize each others representations.
How does this have so many upvotes? It's oversimplifies to the point of not communicating anything useful.
There is so much to research and experiment with, but caveman brains can't see beyond whether GPT5 is going to be better to judge if we reach a wall.
It also makes the conversation about "LLM's and OpenAI setting AGI progress back by 5-10 years" a little silly. Clearly DeepMind is proof that not every dollar and researcher is invested into just making the newest and flashiest LLM without any thought to long term changes and breakthroughs. There is always work going on behind the scenes, it just seems that ChatGPT in the public's eyes is getting all the plaudits.
Right? So many idiots chomping at the bit to say we’ve hit a wall when we’ve barely dipped our toe into all this, relatively speaking.
Yeah the naysayers will always be here, lack of imagination on their part I suspect. Hot take: there is no wall that we'll hit in our lifetime, instead progress will continue steadily then exponentially and end in super intelligence. Language models are understated in their capability and imo already smarter than the median human - most people are really dumb.
Agree, i'm dumb and i'm glad i'm not alone.
huh? what does this mean to people who also agree as you?
champing
I looked it up before I even commented and “chomping” is more common in American English, even if the original saying was “champing”
I've also never heard it referred to as "champing" either. Chomping sounds right. MassiveWasabi has my support.
Don't trust them! They're an ASI trying to turn the whole world into wasabi! We have to stop them!
Just look at the 10 year timescale, we haven't hit a wall at all.
Amen, Praise Sweet Digital Messiah!
It beats dying.
If chatgpt5 sounds hot its proof deepmind is behind on research
That seems like quite a bit of a performance jump in their tested tasks. The concept is really cool.
>The article explores a novel approach to combining Transformers, a neural network architecture adept at understanding natural language, with neural algorithmic reasoners (NARs), which are proficient at algorithmic tasks. The authors propose a hybrid architecture, TransNAR, that leverages the strengths of both Transformers and NARs to achieve robust algorithmic reasoning. Notably, TransNAR outperforms a baseline Transformer model on the CLRS-Text benchmark, a collection of text-based algorithmic problems. >Future AI is expected to benefit from the marriage of Transformers and NARs in several ways. First, this approach can enhance the ability of AI systems to solve algorithmic problems expressed in natural language. Second, it alleviates the issue of Transformers struggling with out-of-distribution generalization, where the test data deviates from the training data. Third, TransNAR paves the way for AI systems to leverage external tools and knowledge sources to inform their reasoning. >However, there are limitations to consider. TransNAR necessitates both textual and graph-based inputs, which may not always be available. Future research will need to address this limitation to make this approach more applicable in real-world scenarios. Overall, the findings presented here are a promising step towards more robust and versatile AI systems.
After all the recent talk about in and out of distribution reasoning, especially after Francois's podcast appearance, this seems almost intentionally timed. To me it's obvious people are working on modifications to improve reasoning in transformer models, but it's nice to actually see what's happening behind closed doors
This is probably one of the most popular research topics in AI. I don't think it's timed to any one event or discussion, it's just one of the hottest topics of the last 3-4 years.
Finally, LMMs with GNNs! This does have huge potential in theory.
It always amazes me that all of the research seems to be coming out of DeepMind, but OpenAI still seems to be at the cutting edge in terms of available products. What is it that’s holding DeepMind back exactly?
Everything "seems" this way probably because you spend too much time on this sub which is up OpenAI's ass. See how much gpt-5 is hyped compared to Gemini 2? And how quickly Google caught up from a sub-par PaLM 2 model to Gemini 1.5 pro in one year (which is now #2 in the arena)? And how quickly Gemini models are improving (1.5 pro was released like 3 months after 1.0 pro and a few weeks after Ultra)? And that they have 2M token context window (nothing else even comes close, AND it's multimodal), combined with the fact that it is also way cheaper to serve than any model of comparable quality? Nothing is holding DeepMind back - they're in the lead if you ask me, and their models are improving more quickly than anyone elses. Gemini 2 is going to be a big step forward for them - there's a reason 1.5 was "only" called 1.5, Google knows that their next generation of models will be truly on another level.
Exactly, DeepMind is the clear favorite for the AGI race. And now that Google is *actually trying* to make good AI products they are making astonishing progress.
OAI just always leaps frog the competition, GPT-2, GPT-3, GPT-4 their models are always impressive and SOTA on release. This exact situation we are in now, is very reminiscent of the state of pre-gpt4. Where it seems like OAI is falling very behind and players like NVIDIA, Meta, Google etc. are all releasing similarly performant models to what GPT-3 was and certainly above GPT-3s performance, everyone thought wtf was OAI doing, only to be later destroyed by GPT-4 and it took months to catch up. Circumstances are a bit different but if feels extremely similar to that time. History repeats itself, am I right? But I don't expect that to be fully true in this case. I do think OpenAI has the lead, but Anthropic and Deepmind are closing in. Gemini 2 should be an impressive model and I have high expectations, same with GPT-5 and Anthropic's equivalent of GPT-5.
Christ what the hell are you guys saying. OpenAI just doesn’t publish because they don’t want to share information. This is just a company stance OpenAI follows, while deep mind chooses to share SOME THINGS. Even deepmind is hiding a lot its research. I can’t believe you guys said what you said. Publishing your research publicly has nothing to do with how ahead you are of other competitors. You just choose to make it public.
Extreme bureaucracy: While Google DeepMind can innovate, smaller companies can implement changes more quickly due to their ability to take risks.
this. remember how everyone lost their mind when Google came out with Bard demo (old Gemini) and it got some facts incorrect? Google lost $100 billion in market value. This was a little over a year ago.
A week later they were valued higher than the day before the announcement. It had no overall impact on the stock price, it was just headlines.
The latest Gemini Pro is basically at GPT-4o level, maybe a tiny bit worse. OTOH, Gemini Flash is far better than anything OpenAI has in the same price range. I wouldn't say Google is behind in terms of product quality. They are slightly behind in mindshare, because OpenAI released their models first, but that's about it.
Gemini exists, it’s a very good product. And was multimodal before ChatGPT, and has a larger content window.
ChatGPT is a cool product but I still don't think it's as cool as Waymo, ChatGPT is great as Google Assistant on steroids, but it still is difficult to trust it for any serious tasks. I would trust a Waymo taxi with my life on the other hand.
It only seems that way because this subreddit is full of OAI shills.
The reason why Deepmind has more research publications is simply because they actually publish whereas OpenAI keeps their research to themselves. Why does Deepmind continue to publish then? I assume it’s the same reason Facebook open sourced their model weights, to destroy OpenAI’s moat.
Deepminds published research is what led to GPT….
Technically Google Brain, a part of current Google DeepMind but at the time it was separate.
Google has always published research. It's not something they started doing due to OpenAI.
This, at scale, across many domains (math problems, reasoning problems, programming, etc), and fully grokked should basically solve the reasoning issue.
LLMs, Neurosymbolic AI, LLMs + GNNs, LMMs (even directions like neuromorphic computing as well) etc. can all get to AGI and beyond imo.
The next AI research comes out of Google again. The 1st product that uses this research will come out of some other company because Google's executives are so bad at their jobs. They won't be able to turn this into a good product.
We used the transformers to defeat the transformers
A lot of different types of biomedical data are efficiently encoded in graphs. Especially sequence data like RNA, DNA and protein and interaction networks at all scales (molecular, cellular, tissue, etc). This could be a killer model for integrating the scientific literature with the raw data the literature is based on. For biomed that might look like mining pubmed for articles and NCBI for sequence data.
ELI5. Implications?
So a Modular NN