itsulu/FediFetcher
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

shorten http timeout for robots.txt fetch

Browse source
This commit is contained in:
nanos 2024-06-25 16:32:47 +01:00
parent dec718db76
commit 01a2719918
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
find_posts.py
View file

@ -1017,7 +1017,7 @@ def can_fetch(user_agent, url):
else: else:
try: try:
# We are getting the robots.txt manually from here, because otherwise we can't change the User Agent # We are getting the robots.txt manually from here, because otherwise we can't change the User Agent
robotsTxt = get(robots, ignore_robots_txt=True) robotsTxt = get(robots, timeout = 2, ignore_robots_txt=True)
if robotsTxt.status_code in (401, 403): if robotsTxt.status_code in (401, 403):
ROBOTS_TXT[robots] = False ROBOTS_TXT[robots] = False
return False return False