Not sure if ditching infinity cache/L3 for a bigger L2 would make sense. It seems at least plausible to me that growing the L2 cache that much would make its latency worse, thus losing most of what has been gained. In the image you linked, Nvidia's L2 latency is like twice as high as AMD's!
Keep in mind that the 'big' desktop chips have bigger caches too. The L2 on a 5080 and 'L3' on a 9070 XT are both 64 MB. Additionally, AMD has already been growing the L2, going to 8 MB on the 9070 XT vs. 4 MB on the 7800 XT and 6800 XT.
To clarify, the idea that AMD is ditching IC is not mine, I find it puzzling too.
Their marketing materials to OEMs have been leaked, and they no longer mention IC at all, instead proudly displaying (much higher) L2 amounts per GPU. This of course doesn't necessarily mean that they are definitely removing it, but it certainly hints at that.
Keep in mind that the 'big' desktop chips have bigger caches too. The L2 on a 5080 and 'L3' on a 9070 XT are both 64 MB. Additionally, AMD has already been growing the L2, going to 8 MB on the 9070 XT vs. 4 MB on the 7800 XT and 6800 XT.