I think it's hard to argue that there exists a vast space of untapped design potential for games that can't be realized only because of the limitations of off the shelf game engines. Most people who use custom engines use them for mainstream common game concepts, because they disagree with architectural decisions about the engine itself (most likely the language being used) and they would rather start from scratch than work with the engine.
Handmade Hero can be made in Unity or Godot. So could Braid. I'm actually struggling to think of a game built in a custom engine that's so radically out of pocket design-wise that it needs a custom engine. I'm not arguing that the use case doesn't exist, I'm arguing that most of the time using a custom engine is a matter of convenience and comfort rather than creative expression, and isn't strictly necessary.
A custom engine is never _needed_ technically speaking since COTS engines are often customizable to the point where you can do anything you want with them. That doesn't mean that they don't influence the design of games, though.
There's a talk that Casey gives where he explains how he implemented the movement system for The Witness, in which he shows examples of Unity-based "walking simulator"-type games dealing with limitations of the engine in ways that The Witness was able to totally avoid. This allowed the game's artists to be creative with set design without worrying as much about performance issues or collision bugs, thus potentially opening up more design space.
Here's some praise of The Witness by Fabien Giesen:
> That’s where I am right now. I have never seen another game as cohesive as this. Not even close. Without getting into specifics or spoiler territory, I have never seen a game with such manifest intention behind every single detail, nor one where all the details cohere this well into a whole. This goes for the design and gameplay itself, but also across traditional hard boundaries. Game design and art play off each other, and both of these are tightly coupled with some of the engine tech in the game. It’s amazing. There is no single detail in the game that would be hard to accomplish by itself, but the whole is much more than the sum of its parts, with no signs of the endless compromise that are normally a reality in every project.
> A custom engine is never _needed_ technically speaking since COTS engines are often customizable to the point where you can do anything you want with them
Yeah, no. You can make a lot of things with an engine, and if you don't already have the necessary talent to make a custom engine you probably should just not.
But there are certainly games which would not exist as "Just use COTS" games. The first which comes to mind is Outer Wilds.
Outer Wilds is running a tiny solar system model. Such a model isn't stable for very long even if you had a lot of compute power which a video game console does not, but in Outer Wilds there's an excuse for that [spoiler] the sun is about to explode, it's a time loop game. Still, this is a heavily specialised engine because normal games centre on the camera or player, Outer Wilds can't do that, the model would explode almost immediately if you do that, so the centre is the sun at all times.
I didn't know that. Although it seems as though they're sufficiently far outside what its makers thinks Unity is for that their attempt to go to Unity 5 had big obstacles. Thanks for correcting me!
[Also, another way I was wrong: to make Unity work the centre of the world is the player, and so the universe is implemented in reverse, your orbit around the sun is calculated with you at the middle, this works correctly in our actual Einsteinian universe - no position is privileged, there is no "center", but it would be crazy to do the maths, for the short time Outer WIlds needs it works well enough with their simple Newtonian physics model]
Handmade Hero can be made in Unity or Godot. So could Braid. I'm actually struggling to think of a game built in a custom engine that's so radically out of pocket design-wise that it needs a custom engine. I'm not arguing that the use case doesn't exist, I'm arguing that most of the time using a custom engine is a matter of convenience and comfort rather than creative expression, and isn't strictly necessary.