I think it depends.
My labels have come in at about a grand. I have something very specific in mind, though.
So this is preventing me from launching the brand. First 1000 shirts would probably cost $250 to sew the tags in, so we're looking at an additional $1250 to begin with, on top of blank purchase and actual printing.
$1250 is not something to be sniffed at. However, my shirts are being released in our own dedicated flagship store and in another independent store that stocks majorly-huge industry brands, so we HAVE to release a final polished product.
If I was selling online only, I'd already have put it out there without any concern. As Tim said, it depends on what your customers expect.
Bear in mind that most people don't even look at the label, let alone the brand of shirt you use. If you are really creating a true niche brand, this is the only time you should be concerned. Most people on here are not and they do really well. Many many brands don't have accent tags and just print their logo and care instructions on the neck.
As to whether you are creating a true brand or not - if you are, you will already have market awareness, brand strategy in place and have an absolute understanding of what will and won't harm your target market.
You'd be amazed at how many brands have the original labels still attached to shirts. I would go round some shops that sell to your target market and start to turn the shirts inside out to hunt for tags and see what they do!