I wonder if the model could be refined on the fly by rendering small test patches using traditional methods and using that as the feedback for a LoRA tuning layer or some such.