I understood your purpose as such. IMHO, there is a bug in handling direct formatting on note anchor. I already detected a similar discrepancy on list number/bullet (see tdf#158835) and I commented tdf#136554 to draw developers’ attention on the similarity and possible same cause.
Usually, the styling precedence is paragraph < character < direct formatting. Here this precedence is followed except when colours are involved (text colour or background highlighting colour). This twist is really upsetting.
I could not find any valid workaround. The only thing you can do is to apply bold (but this gives you only two categories and this is not very noticeable due to size reduction of the anchor), italics (same comment), underline (idem). Changing font face, though allowing for more categories, again is not visually sufficient due to the size of the anchor.