Here’s a common issue when you’re running the famous link checker script: the script reports response code 403 for a number of URLs.
The site loads fine, and checking the same URLs with your browser, FireBug, Screaming Frog or any other tool does not return any error. Yet still, the link checker script reports these damned 403 error(s).
A 403 Forbidden response code typically indicates that the server understands the request but refuses to authorize it; it doesn’t give access to the requested resource.
There can be loads of reasons for this annoying issue.
Here’s the top 4:
1. User-Agent Header:
Some servers check the User-Agent header to identify the client making the request. If the User-Agent header sent by UrlFetchApp is different from a standard browser’s, the server might block it. You could try setting the User-Agent header to mimic a common browser in your UrlFetchApp request.
Fix: If the server is configured to block requests from non-browser User-Agent strings, you can modify the User-Agent string in your UrlFetchApp request to mimic a common browser. (More details in the link below.)
2. IP Whitelisting:
Some servers allow requests only from specific IP addresses or ranges. If the server’s configuration is set to allow requests only from certain IPs (like your own), requests from other sources (like Google’s servers) might be blocked. The web host is blocking Google in their firewall.
Fix: make sure to whitelist all IP addresses used by the UrlFetchApp service. Here’s the list of IP addresses: https://www.gstatic.com/ipranges/goog.txt
3. Authentication:
If your website requires some form of authentication (like cookies or tokens), the request from UrlFetchApp may lack these credentials, leading to a 403 error.
Fix: Make sure that any required authentication headers or cookies are included in the UrlFetchApp request.
4. Rate Limiting:
If the server has rate limiting in place and UrlFetchApp is making requests too frequently, it might trigger a 403 response. This might not be an issue with your browser if you’re not hitting those rate limits (your browsing behavior typically will not hit these limits, whereas an automated process will).
Fix: decrease the frequency and volume of the requests to your server, or change the server settings to allow for higher limits.
Because of the frequency of this 403 error, I’ve added an answer with some more causes and details to my Google Ads Scripts FAQ.
So, next time any of your team members gets this error message, simply send them this link:
https://nilsrooijmans.com/google-ads-scripts-faq/error-403-response-code/
Haven’t installed any link checker solutions yet? C’mon. It’s a no-brainer and only takes 5 minutes! Here’s the link: https://nilsrooijmans.com/the-best-adwords-script-i-have-ever-used/
– Nils