How to improve font rendering in Writer?

As you can see above, Cambria Math render is decent, but any Latex related font - like Latin Modern - is ugly and pixelized on screen until I print it or export it as a PDF.

I enabled OpenGL, which slighty improved the font rendering, but its still meh.

Running LibreOffice Writer 6.2.8 on Windows 10 / Lenovo Thinkpad x240 / 13" LCD screen - 1366*768 / 8 gb ram / i5-4300U @ 1.90 GHz

Unfortunately there is not much you can do.
This issue comes up over-and-over again.
Another recent example: very poor font display quality

Two issues are the problem.

First, a big problem is the lack of precision in calculating the character placement.
It needs to be changed to floating-point calculations for finer character positions.
That is why you see the odd gaps between characters and why they jump around as you zoom-in.
This bug has been in the tracker for many years.

Second, the anti-aliasing appears to be like the old GDI, and not the current DirectWrite.
Your own image shows this.
I zoomed into your image and made screenshots.
Top is Writer - old GDI style horizontal-only, grayscale-only anti-aliasing (like Word 2010).
Bottom is Chrome - DirectWrite style ClearType, full color, horizontal and vertical anti-aliasing (like Word 2016).
ClearType is clearly better looking, especially at smaller font sizes.

This needs to be fixed in LibreOffice.
But since it only affects we lowly Windows users, there does not seem to be much interest in fixing it.

.

What you wrote is not two, but one problem. The bottom enlarged image shows subpixel rendering - hence the colouring of b&w text. And the subpixel positioning implies using subpixel precision.

And no, that not only affects Windows users. Any platform suffers from that; the relevant bug and its see-also have messages from macOS and Linux users - but of course, the problem disappears on any platform with using hi-dpi displays (hence some messages like “no problem om platform X for me”). It’s simply lack of those who dare to change this. Volunteers welcome.

By the way, see also this essay on topic.

for LINUX install GTK -
libreoffice-gtk3

@Getafix: this module package will affect rendering of the icon set in the UI and must be consistent with the active desktop manager: libreoffice-gtk3 for GNOME and kin, libreoffice-kf5 for KDE Plasma. The default libreoffice-x11 is effectively a poor fallback.

Font rendering is not impacted by the UI renderer.