T-Shirt Forums banner
1 - 7 of 7 Posts

·
Registered
Joined
·
55 Posts
Discussion Starter · #1 ·
Do any of you use these? Do the spiders rather your site have one even if it only has the following?

User-agent: *
Disallow:

I went to one of those free URL submission sites and it suggested I put one up. Any thoughts on this?
 

·
Registered
Joined
·
1,029 Posts
That is useless in my opinion. Dont use free search engine submission sites either. The way you can get your site crawled in 1-2 days is to get a link on a well established site. Do this with link exchanges to start out with. This wont get you listed in the search engine results immediately though. It will just get you crawled. To get listed in results will take alot of links from quality pages, and well optimized content. There is no quick fix with SEO no matter what anyone tells you.
 

·
Registered
Joined
·
1,906 Posts
There is no reason not to have a robots.txt. Go ahead and put the basic disallow nothing, takes all of half a minute to make and put up. I am not sure if it actually makes search engines like you more, but it definitely doesn't make them like you less.

If the html compressor took that much off, you probably had some definate bloat (not just all the empty spaces which it took out too) that could've been addressed directly. I don't really like having it optimzed automatically like that personally (I'd rather trim it myself), but smaller is better considering how many people are still on dialup so I suppose if the page still works fine there isn't really any harm.
 
1 - 7 of 7 Posts
Top