you are both. you sell to retailers, which make you a wholesale business, and you sell directly to consumers, making you a retailer.
I have worked for big clothing companys, which were selling to retailers. So they were wholesale operations. But in the end they started selling to consumers in their own flagshipstores or, more recently, through the internets. But in order to do this, they would start a new retaildivision thus separating the two activities. They were basicly selling the clothes to their own daughtercompanies.
You are probably to small to do that

. If you
have to choose, I'd say you're a wholesale operation, if only since the biggest volume and turnover will be in the wholesale. (unless you start a multibrand store)