Everything in JavaScript is passed the same way, whether primitive values or objects. Trying to graft a distinction between pass-by-value and pass-by-reference on top of this single passing method is unlikely to do anything other than confuse people learning JavaScript.
You're talking about the call site; I'm talking about the semantics. There is undeniably a crucial semantic distinction, whatever terminology you want to use to talk about it.