This should be a font problem. LibreOffice has set (at least on Win) some variants of ‘Liberation’ fonts by default for anything. As these fonts lack many glyphs, some of them needed for ‘Math’, this component depends on a well eworking font replacement.
You may get your OS (which one?) to manage this better for you, you may explicitly override the system’s font replacement for LibreOffice in general, or you may select fonts containing all the needed symbols via the font settings of Math.
I am using Windows 7. The symbols I have most problems with are the braces in math formulas. Do you know which fonts I should use for this? Thanks in advance for any assistance.
Perhaps you’re missing the OpenSymbol font?
You can reinstall it by repairing the LibreOffice installation or by downloading the .ttf file and installing it manually.