I’d like to outright prohibit the entry of non-numerical characters into cells (except the . for the decimal).
A conditional format that changes that cell with a non numerical character, to red, would suffice. I tried but was unable to find a conditional format argument that would do this.
Conditional formatting changes how cell content and results are displayed, conditional and number formatting is not input masks. To restrict input use Validity, menu Data → Validity…, in your case Allow Decimal and set Data to not equal some (impossible) value, or some other better fitting condition.
If you only want to display bad input / cell content differently then conditional formatting is fine, assuming you invoked the dialog on cell A1 then as condition use Formula is with NOT(ISNUMBER(A1)) and Apply Style Error, for example.