Simple
=A1
if I drag this down I get =A2 which is expected.
How can I get the same result while dragging horizontally?
Furthermore how can I apply formulas to columns or rows and control how such formulae are to change on each hop?
Your question is a misconception. Dragged formulas don’t change: their references’ parts are either absolute (like in $E$2), or relative (like in A5). When they are absolute, they never change and point to the absolute address. When they are relative they never change and point to “two columns to the left, one row above”.
When you think that you see a changed formula, you actually see the name of the cell that is pointed by that unchanged relative reference. So when you (drag-)copied a formula below, the relative reference naturally points below. You can’t change that.
You can, of course, use smart formulas like involving INDIRECT or OFFSET and current cell’s ROW and COLUMN to construct a reference using other rules, but that’s not “How to control how dragdown formulas change”.
When they are relative they never change and point to “two columns to the left, one row above”.
That sounds very interesting. Does this mean, that the formula I see (is shown to the user), is also just some kind of a representation of what is stored in the cell (similiar to formatting)?
In a sense that internal representation is not a shown string, yes. Additionally: try to change formula settings like “use English function names” (if using non-English UI) or argument separators or reference style (A1/R1C1) … And see how already existing formulas change their display.
Additionally: try …
Did that immediately after having read your answer… and got rid of some own misconception-based questions on “How the hell this is done”