Skip to main content
Help Center
The GoDaddy Community will undergo maintenance starting on Tuesday, August 3rd at 3pm PST / 6pm EST. Learn more
  • GoDaddy Community
  • How to Market Your Business
  • How to Market Your Business

    Showing results for 
    Show  only  | Search instead for 
    Did you mean: 
    Getting Started

    Search engine optimization ignore rbots.txt & sitemap


    I'm use search engine optimization, but I can't understand why this is tools analysis pages which close for index. For example tools ask add meta descriptions for

    but I was close this is URL in file robots.txt 





    Also I'm add "no index" for this is pages. But not add "no follow", I was read "no follow" not good crawls. 

    2 REPLIES 2
    Helper VI

    @stas does the tool have access to your files? I mean, can it open them?

    You should also know that the robots.txt may be not be followed by some crawlers. It is a directive, but some may do not listen to it, like the DNT http header.


    Hope it helps!

    If you find my replies helpful, give me a like or kudo 🙂
    And if I solved your issue, mark it as solution 😄
    Getting Started

    also what is mean error?



    If im click on alexa (check rating my site (from goddady panel) I will see error((



    But really site indexed in ypu can check 



    Why tools show wrong information?