Try soaking a cotton wool ball with pure alcohol (Isopropyl, bought from hardware stores). Then soak the back of the rhinestone. This may be difficult on caps because of the thickness and layers behind the rhinestone..worth a try. 
Not sure there is a way to remove rhinestones without leaving behind glue spots on the fabric.We have pressed rhinestones on some hats and need to remove them. Is this possible? If how?