I am using indows 7.32 bit. downloaded LO 5.1. there is a problem while writing in bangla, some special character like signs appearing. also, while I tried to user bangla spell checker it is showing boxes as suggestion.
I looked at a similar problem 66047 with regard to Sanskrit, have a look at it. I cannot reproduce your problem on my Linux-Mint-Mate with 5.1.1.3 EN-GB using the Lohit Bengali provided with LibO. You could try that font to see whether it is a Font problem, Windows or the version of LibO. With Lohit I was able to type using Insert > Special Characters > জ অ আ ই ঈ উ ঊ ঋ ঌ এ ঐ ও ঔ ক খ চ চ ব ঢ় ৩ ৩ ৪ ৰ.
If you are seeing a small rectangle a little like this square  but with a code this is usually a sign that the font directly chosen or substituted by LibO using the FONT REPLACEMENT TABLE does not contain your character. � is a character error.
Also, make certain that there is not a graphics problem caused by OpenGL or Anti Aliasing.
Sorry not to give a complete answer, but this may point you in a few areas to try…Peter