Ok, but then we might agree that there is no sun, that frogs are born from rocks and mice from leaves, that the stars are holes in the sphere of the heavens and so on - as many people did for millenia. They had consensus - and they were utterly wrong. The actual world actually exists. The world isn't any less round for members of the Flat Earth Society.
Niels Bohr supposedly kept a horseshoe in his office. A visitor asked "what's that for?" "Good luck." "Surely you don't really believe in that." "No, but they say it works even if you don't believe in it."
The joke of course is quantum mechanics also works even if you don't believe in it (or so they say). If someone rejects quantum physics and embraces magic horseshoes, will anything punish them for being objectively wrong? Is there anything you can do to force them to believe the objective truth? If the answer is no to both, then what makes quantum physics the objective truth and magic horseshoes utterly wrong? Maybe there is an objective truth, but all we can have is belief about which things are objective truths. I can be contrarian and say flat earthers are correct, and there's nothing you can do to force me to agree otherwise. So if nothing eventually forces agreement on that matter, what makes one side objectively true?
For the record, I do actually believe in reality. I'm just interested to see what happens when we turn the problem of consciousness on its head. Instead of assuming reality and understanding consciousness within it, assume consciousness and try to paint a picture of reality within it. This is all academic exercise.