I am trying to use .jpg file as a backgroud of the custom-style page. Both page and image are in a5 ratio. Some images fits perfectly, when some others (created on the same soft, with identical proportions) gets randomly stretched and deformed. I tried fifferent options available through the style editor but nothing works. Any ideas what may be wrong or how to fix it?
Once you have inserted your image, select it (click on it making sure you select the image, not the frame it is in) and go to Format → Picture..., Type tab. In the Size section, check the Keep ratio box and eventually reset the size to what you need if it changed (click Original size then scale using either Width or Height; both are in sync because of Keep ratio).
If this answer helped you, please accept it by clicking the check mark 
to the left and, karma permitting, upvote it. If this resolves your problem, close the question, that will help other people with the same question.
created on the same soft, with identical proportions
You could try to import your image file on page > properties/area-background. See attached screenshot… Be aware that the proportions have to orientate to the real text area, i.e. type area plus headers and/or footers.
If you use Page style’s Area tab to assign a bitmap as the background, then there are options how the image is inserted: you may use “Original” style (not tiled or stretched), and adjust width and height (in percents). You should keep in mind that the page background only takes the area inside margins (i.e., page body and header/footer).
It could be that some images have different horizontal/vertical density, and that could affect the image’s aspect ratio. You should post here a link to a sample document with a problematic background image (uploaded to a public share), instead of drying to express something with words. That would allow others to see exact problem, and come with actually useful answer.
You are right, here is the link to the file.
On the first page I used a custom style with a background and it works fine. In the last page I used another custom style with another background image and it is weirdly stretched. Both images used have exact same proportions and pixel dimensions. One work, the other one - not so.
https://drive.google.com/file/d/0B25jSZQ8STakMmhVbXJhel9QbVE/view?usp=sharing
Works for me with 5.4.0.3