Might be bugged. I just used that earlier and it seems you cant remove the gems, and there was one instance where it consumed my gem yet didnt appear on the display. (Also, the prompt to insert a gem can also be bugged where if you select another non-gem item, it says you can insert it there)
Im thinking of doing a bug report later when I have time