It seems to very heavily depend on your exact project and how well it's represented in the training set.
For instance, AI is great at react native bullshit that I can't be bothered with. It absolutely cannot handle embedded development. Particularly if you're not using Arduino framework on an Atmel 328. I'm presently doing bare metal AVR on a new chip and none of the AI agents have a single clue what they're doing. Even when fed with the datasheet and an entire codebase of manually written code for this thing, AI just produces hot wet garbage.
If you're on the 1% happy path AI is great. If you diverge even slightly from the top 10 most common languages and frameworks it's basically useless.
The weird thing is if you go in reverse it works great. I can feed bits of AVR assembly in and the AI can parse it perfectly. Not sure how that works, I suspect it's a fundamentally different type of transformation that these models are really good at
For instance, AI is great at react native bullshit that I can't be bothered with. It absolutely cannot handle embedded development. Particularly if you're not using Arduino framework on an Atmel 328. I'm presently doing bare metal AVR on a new chip and none of the AI agents have a single clue what they're doing. Even when fed with the datasheet and an entire codebase of manually written code for this thing, AI just produces hot wet garbage.
If you're on the 1% happy path AI is great. If you diverge even slightly from the top 10 most common languages and frameworks it's basically useless.
The weird thing is if you go in reverse it works great. I can feed bits of AVR assembly in and the AI can parse it perfectly. Not sure how that works, I suspect it's a fundamentally different type of transformation that these models are really good at