Repository navigation
Expand file tree
/
Copy pathstatus.json
More file actions
222 lines (222 loc) · 18.5 KB
/
Copy pathstatus.json
File metadata and controls
222 lines (222 loc) · 18.5 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
[
{
"code": 200,
"label": "OK",
"type": "active",
"description": "The request succeeded and the page loaded normally.",
"scangovStatus": "Scan successful",
"statusDescription": "We were able to fully scan your site.",
"problem": "None.",
"recommendation": "None.",
"recommendationTechnical": "None."
},
{
"code": 301,
"label": "Moved Permanently",
"type": "redirect",
"description": "The URL has permanently moved to a new address.",
"scangovStatus": "Scan failed",
"statusDescription": "We weren't able to fully scan your site.",
"problem": "Links and search engines pointing to this URL land on a redirect instead of the final page, which can slow load times, dilute SEO signals, and cause automated scans that don't follow redirects to miss the real content.",
"recommendation": "Point your links at the new address instead of the old one, and make sure the redirect is one you meant to set up.",
"recommendationTechnical": "Update internal links, sitemaps, and external references to point directly at the destination URL. Confirm the redirect is intentional and permanent, not a leftover from an old site migration."
},
{
"code": 302,
"label": "Found",
"type": "redirect",
"description": "The URL has temporarily moved to a new address.",
"scangovStatus": "Scan failed",
"statusDescription": "We weren't able to fully scan your site.",
"problem": "Search engines treat this as a temporary move, so the original URL keeps its indexing weight even though visitors are sent elsewhere. It's easy to leave a 302 in place long after the move became permanent.",
"recommendation": "Decide whether this move is permanent. If it is, ask your web team to make it a permanent redirect. If it's temporary, remove it once you no longer need it.",
"recommendationTechnical": "If the move is permanent, switch to a 301 redirect. If it's genuinely temporary, confirm the destination is correct and remove the redirect once it's no longer needed."
},
{
"code": 400,
"label": "Bad Request",
"type": "inaccessible",
"description": "The server could not understand the request due to invalid syntax or formatting.",
"scangovStatus": "Scan failed",
"statusDescription": "We weren't able to fully scan your site.",
"problem": "The server rejected the request outright, so the page never loads for visitors, search engines, or AI crawlers hitting this URL.",
"recommendation": "Open the address in a browser and see if it works. If it doesn't, ask your web team to look at what's wrong with it.",
"recommendationTechnical": "Check the URL for malformed query parameters or encoding issues, and review server logs for the exact validation error being triggered."
},
{
"code": 401,
"label": "Unauthorized",
"type": "inaccessible",
"description": "The requested resource requires authentication. The client must log in to access it.",
"scangovStatus": "Scan failed",
"statusDescription": "We weren't able to fully scan your site.",
"problem": "This page is locked behind a login, so it's invisible to the public, search engines, and AI tools — even if it's meant to be publicly available.",
"recommendation": "Decide whether this page is meant to be public. If it is, ask your web team to remove the login requirement. If it isn't, take it out of your sitemap and off your public pages.",
"recommendationTechnical": "If the page should be public, remove the authentication requirement. If it's intentionally private, remove it from your sitemap and any public-facing links."
},
{
"code": 403,
"label": "Forbidden",
"type": "inaccessible",
"description": "The server understood the request but refused to fulfill it. The client does not have permission to access this resource.",
"scangovStatus": "Scan failed",
"statusDescription": "We weren't able to fully scan your site.",
"problem": "The server actively blocked access, which may indicate misconfigured permissions, a firewall rule, or a bot-blocking tool that's also blocking legitimate crawlers (like <a href=\"https://scangov.com/bot\">ScanGovBot</a>).",
"recommendation": "Something on your site is turning visitors away, and it's catching good bots too. Ask whoever manages your security settings to let good bots through. <a href=\"https://scangov.com/bot\">Learn more</a> about what to allow.",
"recommendationTechnical": "Review server/firewall rules and any bot-protection service (e.g. a WAF or Cloudflare) to confirm search engines and monitoring tools aren't being blocked unintentionally. Allow the ScanGovBot user-agent, or allowlist 44.236.34.143."
},
{
"code": 404,
"label": "Not Found",
"type": "inaccessible",
"description": "The server could not find the requested page or resource.",
"scangovStatus": "Scan failed",
"statusDescription": "We weren't able to fully scan your site.",
"problem": "The page no longer exists at this address, creating a broken link for anyone who visits it, including from search results, bookmarks, or other sites linking here.",
"recommendation": "Put the page back, or send this address to wherever the content lives now. Then update any menus or links still pointing here.",
"recommendationTechnical": "Restore the page at this URL, or set up a redirect to its new location or the closest relevant page. Update internal links and the sitemap to remove the dead URL."
},
{
"code": 408,
"label": "Request Timeout",
"type": "inaccessible",
"description": "The server closed the connection because the request took too long to complete.",
"scangovStatus": "Scan failed",
"statusDescription": "We weren't able to fully scan your site.",
"problem": "The server took too long to respond, so the page failed to load — this can point to performance problems or an overloaded server.",
"recommendation": "Your site is taking too long to answer. Ask your web team to look at why pages are slow.",
"recommendationTechnical": "Investigate server response times and resource usage. If timeouts are frequent, consider caching, scaling server resources, or optimizing slow backend queries."
},
{
"code": 410,
"label": "Gone",
"type": "inaccessible",
"description": "The requested resource has been permanently removed and will not be available again.",
"scangovStatus": "Scan failed",
"statusDescription": "We weren't able to fully scan your site.",
"problem": "The page was intentionally and permanently removed, so anyone linking to it — including search engines — hits a dead end with no path forward.",
"recommendation": "If this content moved somewhere else, send this address there instead. If it's really gone, remove links to it from your menus and sitemap.",
"recommendationTechnical": "If the content moved rather than disappeared, use a redirect instead of a 410. Otherwise, remove references to this URL from menus, sitemaps, and external links."
},
{
"code": 429,
"label": "Too Many Requests",
"type": "inaccessible",
"description": "The client sent too many requests in a short period and the server is rate-limiting access.",
"scangovStatus": "Scan failed",
"statusDescription": "We weren't able to fully scan your site.",
"problem": "The server is rate-limiting requests to this URL, which can prevent monitoring tools, search engines, or real visitors from reliably reaching the page during busy periods.",
"recommendation": "Your site is turning away visitors for asking too often. Ask your web team to give good bots more room. Good bots already crawl slowly. <a href=\"https://scangov.com/bot\">Learn more</a> about what to allow.",
"recommendationTechnical": "Review and relax rate-limiting thresholds for legitimate traffic, including known crawlers, or allowlist the scanning/monitoring service's IP (44.236.34.143 for ScanGovBot)."
},
{
"code": 500,
"label": "Internal Server Error",
"type": "inaccessible",
"description": "The server encountered an unexpected error and could not complete the request.",
"scangovStatus": "Scan failed",
"statusDescription": "We weren't able to fully scan your site.",
"problem": "An unhandled error on the server prevented the page from loading at all, affecting every visitor who requests this URL.",
"recommendation": "Something broke on your site. Ask your web team to check the error logs and fix it.",
"recommendationTechnical": "Check server error logs for the underlying exception and fix the code or configuration issue causing the crash."
},
{
"code": 502,
"label": "Bad Gateway",
"type": "inaccessible",
"description": "The server received an invalid response from another server while processing the request.",
"scangovStatus": "Scan failed",
"statusDescription": "We weren't able to fully scan your site.",
"problem": "A server acting as a proxy or gateway got an invalid response from an upstream server, so the page failed to load — often a sign of backend or infrastructure trouble.",
"recommendation": "One of the systems behind your site isn't responding. Ask your web team or hosting provider to check it.",
"recommendationTechnical": "Check the health of upstream/backend servers and the configuration of the proxy or load balancer sitting in front of them."
},
{
"code": 503,
"label": "Service Unavailable",
"type": "inaccessible",
"description": "The server is temporarily unable to handle the request, often due to maintenance or overload.",
"scangovStatus": "Scan failed",
"statusDescription": "We weren't able to fully scan your site.",
"problem": "The server can't currently handle the request, often due to maintenance or overload, so the page is temporarily unreachable.",
"recommendation": "Your site is down or too busy to answer. If this is planned maintenance, it should clear up on its own. If not, ask your hosting provider to look into it.",
"recommendationTechnical": "If this is planned maintenance, confirm it's temporary and communicated to users. If it's overload, investigate server capacity and traffic spikes."
},
{
"code": 993,
"label": "Certificate error",
"type": "inaccessible",
"description": "A non-standard code ScanGov uses when a site's security certificate fails validation, so no page can be loaded over HTTPS.",
"scangovStatus": "Scan failed",
"statusDescription": "We couldn't scan your site because its security certificate didn't pass validation.",
"problem": "The site's HTTPS certificate is expired, issued for a different host name, or signed by an authority browsers don't trust. Browsers show a full-page security warning instead of the site, and automated tools, including <a href=\"https://scangov.com/bot\">ScanGovBot</a>, refuse the connection. Until the certificate is fixed, most users can't reach the site at all.",
"recommendation": "Your site's security certificate isn't valid. Ask whoever manages your hosting to renew or reissue it, and check that it covers every host name people use to reach your site, with and without www.",
"recommendationTechnical": "Check the certificate chain with a tool such as the SSL Labs server test. Common causes are an expired certificate, a missing intermediate certificate, or a certificate whose subject alternative names don't include the requested host. Renew or reissue it, confirm the full chain is served, and redirect every host name to the one the certificate covers."
},
{
"code": 994,
"label": "Page failed to load",
"type": "inaccessible",
"description": "A non-standard code ScanGov uses when the server accepted the request but the page never finished loading or never displayed any content.",
"scangovStatus": "Scan failed",
"statusDescription": "We couldn't scan your site because the page never finished loading in our browser.",
"problem": "The server accepted the connection, but the page didn't display any content within the time allowed, or it stopped responding partway through. That can be a page that depends on a script or resource that never arrives, a server that stalls under load, or protection that quietly withholds the page from automated tools, including <a href=\"https://scangov.com/bot\">ScanGovBot</a>. Users on slow connections or older devices are likely to hit the same wall.",
"recommendation": "Your home page didn't load in our browser. Ask your web team to open it in a fresh browser session and watch for a blank screen, a spinner that never stops, or errors in the browser console.",
"recommendationTechnical": "Load the page in a clean browser profile and in a headless browser from outside your network. Check for render-blocking scripts or stylesheets that time out, oversized images that delay first paint, and firewall or CDN rules that stall non-browser clients. ScanGov scans from 44.236.34.143 and announces itself as ScanGovBot."
},
{
"code": 995,
"label": "No page content",
"type": "inaccessible",
"description": "The server answered the request, but the page it returned had no content for the scanner to read.",
"scangovStatus": "Scan blocked",
"statusDescription": "We couldn't fully scan your site because it responded without any page content.",
"problem": "The server returned a success status, but the page had no readable content. That usually means a bot-protection layer (a CDN, firewall, or hosting feature) served an empty or interstitial page to a visitor it didn't recognize, including <a href=\"https://scangov.com/bot\">ScanGovBot</a>. Search engines and AI tools that hit the same wall see nothing either, so the site can't be evaluated or indexed by them.",
"recommendation": "Something on your site is serving an empty page to good bots. Ask whoever manages your security settings to let good bots through. <a href=\"https://scangov.com/bot\">Learn more</a> about what to allow.",
"recommendationTechnical": "Check your bot-protection or CDN rules for responses that return 200 with a challenge, interstitial, or empty body. Allow the ScanGovBot user-agent, or allowlist 44.236.34.143, and confirm the homepage HTML is returned to non-browser clients."
},
{
"code": 996,
"label": "Challenge page",
"type": "inaccessible",
"description": "A non-standard code ScanGov uses when a site answers with a captcha or \"checking your browser\" page instead of the page that was asked for. A person with a browser usually gets through; an identified bot does not.",
"scangovStatus": "Scan blocked",
"statusDescription": "We couldn't scan your site because it showed our bot a captcha or browser check instead of the page.",
"problem": "A bot-protection service in front of your site (a CDN, firewall or hosting feature) is challenging visitors it doesn't recognize. ScanGovBot identifies itself and follows robots.txt but cannot solve challenges, so the site's content can't be verified or shown here, and other good bots are probably being turned away too.",
"recommendation": "Ask whoever manages your hosting, CDN or firewall to let good bots through by name instead of challenging every unknown visitor. <a href=\"https://scangov.com/bot\">Learn more</a> about what to allow.",
"recommendationTechnical": "Allow-list the ScanGovBot user-agent (or 44.236.34.143) in your bot-management rules so it is not served a challenge (Cloudflare: a WAF custom rule or Bot Fight Mode exception; SiteGround: the anti-bot AI setting; Imperva: a bot policy exception). Prefer rate limiting to challenging for automated traffic that identifies itself."
},
{
"code": 997,
"label": "Crawl-delay too long",
"type": "inaccessible",
"description": "A non-standard code ScanGov uses when a site's robots.txt asks crawlers to wait longer between requests than ScanGov will (30 seconds). The site itself loaded normally.",
"scangovStatus": "Scan skipped",
"statusDescription": "We didn't scan your site because its robots.txt asks crawlers to wait longer than we can.",
"problem": "Your robots.txt sets a Crawl-delay above 30 seconds. ScanGov honours crawl-delay, and at that pace a scan can't finish, so the site is skipped and its content can't be verified here.",
"recommendation": "Ask whoever manages your robots.txt to lower the crawl-delay for good bots, or to leave it out for ScanGovBot. <a href=\"https://scangov.com/bot\">Learn more</a> about what to allow.",
"recommendationTechnical": "Set \"Crawl-delay: 30\" or lower in the robots.txt group that applies to ScanGovBot, or add a \"User-agent: ScanGovBot\" group with no Crawl-delay directive. Rules in a specific ScanGovBot group take precedence over \"User-agent: *\"."
},
{
"code": 998,
"label": "Blocked by robots.txt",
"type": "inaccessible",
"description": "A non-standard code ScanGov uses when a site's robots.txt asks ScanGovBot not to crawl it. The site itself loaded normally.",
"scangovStatus": "Scan skipped",
"statusDescription": "We didn't scan your site because its robots.txt asked us not to.",
"problem": "Your robots.txt disallows ScanGovBot (or all crawlers). ScanGov honours that request and does not scan the site, so its content can't be verified or shown here until the file allows the bot.",
"recommendation": "If you want ScanGov to scan your site, ask whoever manages your robots.txt to allow good bots. <a href=\"https://scangov.com/bot\">Learn more</a> about what to allow.",
"recommendationTechnical": "Add a group to /robots.txt that allows ScanGovBot: \"User-agent: ScanGovBot\" followed by \"Allow: /\". A blanket \"User-agent: *\" / \"Disallow: /\" rule blocks ScanGovBot unless a more specific ScanGovBot group allows it."
},
{
"code": 999,
"label": "Request Denied",
"type": "inaccessible",
"description": "A non-standard code used by some sites to actively block automated or bot traffic.",
"scangovStatus": "Scan failed",
"statusDescription": "We weren't able to fully scan your site.",
"problem": "The site is actively blocking automated tools, including this scanner, so its content can't be verified or indexed by anyone using similar tools.",
"recommendation": "Your site is blocking automated visitors on purpose, and that includes good bots. Ask whoever set that up to let good bots through. <a href=\"https://scangov.com/bot\">Learn more</a> about what to allow.",
"recommendationTechnical": "Review bot-blocking rules to allow legitimate crawlers and monitoring services, or confirm this restriction on this URL is intentional. Allow the ScanGovBot user-agent, or allowlist 44.236.34.143."
}
]