I there any way to disable it with some custom installer setting and thereby bypass these popups?
Yes, disable it in installer.
Isn’t it a bug then that this isn’t explained to the user in the popup windows?
No. We don’t know what is detected and shown by Windows. In your case, it was FF. In another’s case, it might happen to be LibreOffice itself, that user forgot to close, or hung in the background. In one case, it might be OK to ignore, in others not.
I disagree. It is nowadays unusual for installers in Windows to require closing other applications and/or rebooting. If other installers can manage that then why not LO?
Which specific installer you refer to? This statement is nonsense. There are hundreds of programs that don’t have shell extensions; and even when you update an application with such components, you not always have such warnings. So your personal feeling that other deal that might not reflect the reality.