cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
New

Domain forwarding not working for robots.txt

In my DNS management for IGBOTIQUE.COM, I have domain forwarding set-up to forward requests https://www.igbotique.com.

 

This works well, for all requests except http://igbotique.com/robots.txt.

 

I have output my curl requests below. For every path it redirects it as expected. But as soon as the path is 'robots.txt', it times out. Please how do I resolve this issue?

 

 

$ curl http://igbotique.com/j -i
HTTP/1.1 301 Moved Permanently
Server: nginx/1.12.2
Date: Mon, 05 Aug 2019 12:55:36 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Location: https://www.igbotique.com/j

 

$ curl http://igbotique.com/robots -i
HTTP/1.1 301 Moved Permanently
Server: nginx/1.12.2
Date: Mon, 05 Aug 2019 12:55:42 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Location: https://www.igbotique.com/robots
$ curl http://igbotique.com/robots.txt -i
curl: (56) Recv failure: Operation timed outP

Please help me. I want it to forward my robots.txt request along with all other requests.

 

 

 

 

 

 

9 REPLIES 9
Super User IV

@AdaI 

 

I just tested this on a domain where I am using forwarding with the same result. I believe that this is similar to the fact that you can't forward HTTPS traffic through GoDaddy.

 

I follow what you are looking to do - I just do not believe it is possible - it's one of those "reserved" file names that doesn't working with forwarding.

I am a GoDaddy End User - Just Like You
Check out my site! | I currently manage over 300 WordPress Websites
* Please note that I offer free advice on this forum. Thank You Info If you would like personalized help, please contact me. Otherwise, please ask your question in the proper forum so the answer can assist EVERYONE in the community and not just you. Thanks! *

Once your issue is resolved,
please be sure to come back and click accept for the solution

Get Better Support on the Community Boards!
Etiquette When Asking for Help from the Community

Who marked this as solved? This is anything but solved. We need an answer from GoDaddy support on if in fact this is a reserved filename for domain forwarding, or if this is just an error or problem that can be resolved. 

 

In our case, we're trying to setup an SEO campaign with Moz, and they require the site's robots.txt file be accessible on the non-www version of the domain. But for other reasons, we want the site to always load with www in front. This should all be manageable from the DNS settings and it's not. 

 

See my example: 

 

 

curl -I http://carloscarpets.com/robots123.txt
HTTP/1.1 301 Moved Permanently
Server: nginx/1.12.2
Date: Tue, 27 Aug 2019 14:57:11 GMT
Content-Type: text/html; charset=utf-8
Connection: close
Location: http://www.carloscarpets.com/robots123.txt

 

That URL string forwards perfectly. 

 

However, this one doesn't: 

curl -I http://carloscarpets.com/robots.txt
curl: (56) Recv failure: Connection reset by peer

GoDaddy support claims that the domain is showing the www version of the domain in their browsers that they're testing with, but that doesn't mean the domain forwarding feature is setup correctly.  The browser could just be guessing to put the www in front, when the non-www version doesn't immediately establish a connection. 

 

Then they tried blaming it on my hosting, but domain forwarding happens at a higher level than hosting. Hosting has nothing to do with this. So frustrating that no one understands this issue, or can explain it.

Yeah, I don't know who marked it as solved because it wasn't me. 

For me, the "solution" was to transfer to another DNS provider.

 

If GoDaddy support is claiming that it's working in the browsers that they are testing with, then they are being silly. Robots.txt is rarely accessed with a browser. If it's not working in curl, it's not working

Good for you! We have hundreds of domains pointed this way and transferring all would be a burden though not something we've ruled out.

 

The most frustrating part is that we can't even have an email conversation with GoDaddy support. They force customers into phone or chat with Tier 1 tech support who can't take the time to do the proper research, presumably because they're encouraged to keep support convos as short as possible. Finding the right person who understands this technical nuance is infuriatingly difficult, and so far has been impossible for me.

 

Does anyone know how we are supposed to escalate support problems when our questions aren't answered?

@AdaI  & @jbmm 

 

As the GoDaddy community is not a "support" channel but rather end users helping each other out, so you wouldn't see a GoDaddy support reply.....

My "understanding" of how the GoDaddy forwarding feature works (vs a regular DNS entry)  based on my own experience... I note this is probably an oversimplification, and I don't have direct knowledge of the actual setup..... is that GoDaddy has a server with a MASSIVE .htaccess file and when you setup forwarding of a domain it is doing a URL re-write / redirect - however due to the nature of the forwarding server, there re certain files which the server itself will not forward.

 

This is a similar reason to why you can't forward HTTPS to HTTPS via the forwarding feature as otherwise this same server/platform would need an SSL for every domain it was forwarding from.

 

I believe the reason this was marked as resolved was there was no further response and based on my original response the moderators felt it answered the question.

 

I am going to ask one of the GoDaddy Community Admins (GoDaddy Employee) if they can provide any additional insight to this.

I am a GoDaddy End User - Just Like You
Check out my site! | I currently manage over 300 WordPress Websites
* Please note that I offer free advice on this forum. Thank You Info If you would like personalized help, please contact me. Otherwise, please ask your question in the proper forum so the answer can assist EVERYONE in the community and not just you. Thanks! *

Once your issue is resolved,
please be sure to come back and click accept for the solution

Get Better Support on the Community Boards!
Etiquette When Asking for Help from the Community

Thank you, @PL281 .

Community Manager

Hi @jbmm and @AdaI. Thanks for posting. I'm not sure why you were seeing an issue with your cURL requests. However, I was able to check with the dev team for the forwarding service and they confirmed that there isn't any type of block on /robots.txt. I was able to pull this information from a test domain that was forwarded to a test site and it worked for me as well. Further, it looks like @jbmm's site is producing a redirect when I test it using cURL from a cPanel plan.

 

[removed]@p3plcpnl0119 [~]$ curl http://carloscarpets.com/robots.txt -i
HTTP/1.1 301 Moved Permanently
Server: nginx/1.12.2
Date: Wed, 28 Aug 2019 20:00:11 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Location: http://www.carloscarpets.com/robots.txt

[removed]@p3plcpnl0119 [~]$ curl http://carloscarpets.com/robots.txt -I
HTTP/1.1 301 Moved Permanently
Server: nginx/1.12.2
Date: Wed, 28 Aug 2019 20:01:53 GMT
Content-Type: text/html; charset=utf-8
Connection: close
Location: http://www.carloscarpets.com/robots.txt

If you're still having this issue, then perhaps it is some combination of where you're sending the request from and where the site is hosted. Happy to take a closer look if you can provide further examples. 

 

JesseW - GoDaddy | Community Manager | 24/7 support available at x.co/247support | Remember to choose a solution and give kudos.

@JesseW 

 

I tired via Terminal on my VPS and got the same thing you did, however when I tested via a browser I got the same result as the @AdaI I also tried via redirect-checker.org and it  just spins

 
Could this be an inside the GoDaddy network vs outside the network issue???
I am a GoDaddy End User - Just Like You
Check out my site! | I currently manage over 300 WordPress Websites
* Please note that I offer free advice on this forum. Thank You Info If you would like personalized help, please contact me. Otherwise, please ask your question in the proper forum so the answer can assist EVERYONE in the community and not just you. Thanks! *

Once your issue is resolved,
please be sure to come back and click accept for the solution

Get Better Support on the Community Boards!
Etiquette When Asking for Help from the Community

Community Manager

That's a good point @PL281. I tested outside of our network and I do see the same issue as well. I did some further digging and it appears this is related to some security filtering we have in place on the forwarding platform. It is frequently the target of attacks and is fairly locked down. I'm going to continue working with our security team to see if there's some way to allow this type of request to go through. I'll update here if I find anything else out. Thanks Everyone for your patience. 

 

JesseW - GoDaddy | Community Manager | 24/7 support available at x.co/247support | Remember to choose a solution and give kudos.