When every operation needs to be approved (every button click, every form entry, etc.) does it even make sense to use an agent?
And it’s not like you can easily “always allow” let’s say, certain actions on certain websites, because the issue is less with the action, and more with the data passed to it.
Sure, just look at the examples in TFA like finding emails that demand a response or doing custom queries on Zillow.
You probably are just going to grant it read access.
That said, having thought about it, the most successful or scarier injections probably aren't going to involve things like crafting noisy destructive actions but rather silently changing what the LLM does during trusted/casual flows like reading your emails.
So I can imagine a dichotomy between pretty low risk things (Zillow/Airbnb queries) and things that demand scrutiny like doing anything in your email inbox where the LLM needs to read emails, and I can imagine the latter requiring such vigilance that you might be right.
It'll be very interesting and probably quite humbling to see this whole new genre of attacks pop up in the wild.
And it’s not like you can easily “always allow” let’s say, certain actions on certain websites, because the issue is less with the action, and more with the data passed to it.