http://www.robotstxt.org/orig.html says:
Disallow: /help disallows both /help.html and /help/index.html
Now, google.com/robots.txt lists:
Disallow: /search
Allow: /search/about
Upon running robotparser.py
, it returns false
for both the above cases in Googles robots.txt
.
Would somebody please explain me, what's the use of Allow
in Allow: /search/about
as it would return a false
based on the Disallow
entry above it?
See Question&Answers more detail:
os 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…