If this was to potentially be a thing, each of the gems could resemble a different ability.
For example, Emerald Would give the attack size buff when made into an accessory (If I remember correctly the icon for it is green)
You would have to go to a forge/weapons smith and pay a toll to craft the amulet. The more gems you use in it, the better chance for it to be high quality.