As opposed to the nice round number of $2.39 for shipping?I think a buck or two makes sense. More doesn't. Medium amounts ($0.75, $1.50) look odd. Same rules as any other retail pricing basically.
If you think you need to charge it, charge it. It's pretty common and not many people will be put off.
We charge $2 extra for XXL on online sales but when we sell at an event we don't charge anything extra. It makes our lives easier when doing the math. We usually emphasize that we don't charge extra when talking to bigger customers and they often end up buying two shirt instead of just one just because of that. Sweeet!
If I had bigguyshirts.com or something like that, I would tack on a surcharge for smaller sizes. You're not going to lose anything as your primary customers will be buying big shirts. They will probably appreciate it that they aren't the ones paying extra for just being who they are.