I personally don’t see it as an annoyance though, like you, there are a plethora of built-in styles I’ve never used. Moreover many of them are in a “prototypal” form and must be customised before being useful (think of all List n or Numbering n which are not fit for list out of the box because they are not associated by default with “list” styles).
However, those built-in styles suggest some usage (Body Text for main flow, Heading n for level-n headings, etc.) and are clues for “structured writing”. Built-in styles are not sacred. If the name appeals to you for a usage, don’t hesitate to customise them to fit your needs. But built-in styles have a special property (contrary to custom styles designed by user): their names get translated when the UI language is changed. I know this is of little interest for a particular user, but it is of great importance in this site to explain notions and good practices around a common vocabulary.
After this long introduction, I now give you two suggestions to reduce clutter in the style sidepane:
Hierarchical in the bottom drop-down menu
Hierarchy (dependency) between styles will be shown with indentation. You’ll see the various families at a glance and you can expand/shrink a full sub-tree with a click. You’ll also have a suggestion on which style to use in contexts (heading, header, footer, lists, cover page, etc.).
This may not actually reduce the number of displayed styles but display is structured.
Applied styles in the bottom drop-down menu
This restricts the displayed styles to those actively used in the document. This mode is effective (useful and uncluttered) only after the document is long enough to cover all contexts of your text (headings, text, header, …). It needs some “warm up” before the list settles to something quite definitive.
In long documents, the same styles tend to be used over and over with rare additions if your template or collection is well designed.
You can alternate between these two modes to sporadically reference a “new” style.
To show the community your question has been answered, click the ✓ next to the correct answer, and “upvote” by clicking on the ^ arrow of any helpful answers. These are the mechanisms for communicating the quality of the Q&A on this site. Thanks!
EDIT 2019-04-19: minor adjustments to add missing words and correct typos; no change in significance