This is literally what private detectives do when they tail someone. So yes, this is legal as long as it's not harassing or there's a restraining order or something. Did you think it was not?
Not OP, but yes, I think it is not. At least, not legal in the same expansive way that you are implying. AFAIK private detective work is very much regulated, most likely because it is otherwise known as stalking.
stalking implies harassment, threats, and real or perceived potential for danger
it is illegal because it means the stalker will attack / rape / otherwise damage or harass the victim.
however watching or tracking someone in public is plenty legal, and actual PIs have ethical and legal obligation to weed out stalkers and dubious behavior
So I know you're saying that's too much, but honestly even if the bar was set at that level it would fix mass surveillance. If it takes one hour of labor to track one person for an hour, things work out fine.