for example, the following example we want to shield /abcd/ this directory page, some people might write Disallow: /abcd. This is a problem? Write that there is still a problem, but also a big problem. It can shield all the pages below /abcd/ in this directory. But it will also transfer to the information search engine like, that is at the beginning of all pages for /abcd shielding. The effect of equivalent /abcd ‘> Disallow:
error two: shielding a page, the page without former slash "/"
I think this is a lot of stationmaster is also very easy to ignore, for example, we want to abcd.html search engine shield in the root directory of the web page, some people might do in robots: Disallow: abcd.html, on the surface may not have what problem, but I want to ask you to inform this is a page search engine shield in what directory? If we do not take it, the search engine spiders cannot identify where a page. The correct method is: Disallow: /abcd.html, so as to the real screen located in the root directory of the abcd.html below this page.
Robots.txt file appears to be only a few lines of letters, but in fact there are many details that need our attention. Because if you don’t pay attention to these details, some statements will not take effect, or even effect. The robots.txtt file is the first file search engine into our site after the visit, the write good relationship to the site of the Shanghai dragon can proceed smoothly. The author will analyze three examples in the robots.txt file we are prone to the details wrong.
error three: behind the shield did not keep up with a slash "/"
this statement at first we can get is to tell the search engine in the ABCD directory page cannot crawl, but all other pages can be. But how the actual results of the segment statement? Effect is to draw further apart, why? We know the search engine spiders to the rules in the robots file is executed from top to bottom. This will cause the Disallow statement fails, the correct approach is to put the Disallow: /abcd/ in Allow: / in order to play the effect we want.
error: Allow and disallow order
First we see a