Here’s a common issue when you’re running the link checker script:
the script reports response code 403 for a number of URLs.
However, the site loads fine, and checking the same URL’s with your browser, FireBug, Screaming Frog or any other tool does not return any error.
Here’s a list of the most frequent causes:
A 403 Forbidden response code typically indicates that the server understands the request but refuses to authorize it; it doesn’t give access.
If your website is returning a 403 response to the UrlFetchApp service while returning a 200 OK response to your browser, there could be several reasons for this discrepancy.
1. User-Agent Header:
Some servers check the User-Agent header to identify the client making the request.
If the User-Agent header sent by UrlFetchApp is different from a standard browser’s, the server might block it. You could try setting the User-Agent header to mimic a common browser in your UrlFetchApp request.
Fix: If the server is configured to block requests from non-browser User-Agent strings, you can modify the User-Agent string in your UrlFetchApp request to mimic a common browser.
Here’s an example of how you might set the User-Agent header to mimic Google Chrome:
var url = "https://yourserver.com";
var options = {
"headers": {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36"
}
};
var response = UrlFetchApp.fetch(url, options);
var content = response.getContentText();
Logger.log(content);
Please note that the User-Agent string provided above is just an example.
User-Agent strings vary across browsers and versions, so you might want to use the exact User-Agent string that matches the browser you’re using.
You can find your browser’s User-Agent string by searching “What’s my User-Agent” in your browser or checking the browser’s developer tools.
It’s also worth noting that while mimicking a browser’s User-Agent can solve this particular issue, it might not be the best long-term solution, especially if you don’t control the server.
If the server’s rules or the browsers’ User-Agent strings change, your code might stop working.
If you control the server, it would be better to adjust its rules to allow the UrlFetchApp requests explicitly.
2. IP Whitelisting:
Some servers allow requests only from specific IP addresses or ranges.
If the server’s configuration is set to allow requests only from certain IPs (like your own), requests from other sources (like Google’s servers) might be blocked. The web host is blocking Google in their firewall.
Fix: make sure to whitelist all IP addresses used by UrlFetchApp service. Here’s the list of IP addresses https://www.gstatic.com/ipranges/goog.txt
3. Authentication:
If your website requires some form of authentication (like cookies or tokens), the request from UrlFetchApp may lack these credentials, leading to a 403 error.
Fix: Make sure that any required authentication headers or cookies are included in the UrlFetchApp request.
4. Rate Limiting:
If the server has rate limiting in place and UrlFetchApp is making requests too frequently, it might trigger a 403 response.
This might not be an issue with your browser if you’re not hitting those rate limits (your browinsg behavior typically will not hit these limits, whereas an automated process will)
Fix: decrease the frequency and volume of the requests to your server, or change the server settings to allow for higher limits.
Other potential causes that are less frequent:
CORS (Cross-Origin Resource Sharing) Policy:
If your server has a strict CORS policy and UrlFetchApp’s request doesn’t satisfy it, this might lead to a 403 response.
Although this is more common with browsers making cross-origin requests, it’s worth checking if CORS settings might affect your scenario.
Referrer Policy:
Some servers might expect a specific referrer header that is present when you make the request from a browser but absent or different when you make the request via UrlFetchApp.
Custom Security Measures:
The server might have custom security rules that are specifically blocking the requests from UrlFetchApp.
This could include rules based on headers, request rate, origin, or other factors. Make sure to check with your IT department and security experts to see if this is the case.
Join thousands of PPC geeks who already have access:
If the button above isn’t working for you, you can sign up here to get access.