My answer is different from what you might want to hear:
Forget the belt and learn to do the exercises strict without it AND increase your strength in the lower back. Nobody needs a belt while not attempting new maximum lifts, and even on that, some world class lifters don't use belts.
In my opinion, the belt gives you a false feeling of safety and might tempt you to use higher weights than are good for you. Better to build your muscles to hold yourself while lifting. A belt can never replace that.
But if you use it, yes, it should be as tight as possible.
I concur with you. I was about to type almost word for word what you said.