I.e. annex is really in a different problem space than "big files in git", despite the obvious overlap.

A good way to think about it is that git-annex is sort of a git-native and distributed solution to the storage problem at the "other side" ("server side") of something like LFS, and to reason about it from there.