Yes, the edges will probably frey if they aren't sewn down. But can you explain to us what is on the inside of your shirt that is making it scratchy. Are you talking about the stiching is making it scratchy. If its just stitching that you're talking about, then I'm not sure what to do about that.
But if you are using any
other sort of fabric on the
inside of your shirt to avoid shifting, that's what is giving you the scratchy feeling. That's what I'm guessing the problem is, but you never answered that part of the question
. That's to what I was referring to using a backing or stabilizer.
What kind of machine are you using to apply your pieces to your shirts?
I'd go to the nearest sewing/embroidery supply store and ask them for the softest stabilizer that they have. Also, take them a sample of the blank shirt that you are using so that they make sure they sell you something that won't weigh down the shirt or be visable from the outside of the garment (especially if you're using light colored garments).
You might want a soft mesh cut away.