How to restrict number of recent font colors

In font-color, recent are 12 of the most recent colors. I would like to reduce this quantity to 3 or 4 of the colors I use often. I can not remove unused colors except by adding new colors which I also don’t want. Is there a way to restrict the number of recent colors?

Thanks,