Yep, just make sure to add them to your robots.txt file so that only the bad rob... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		RobKohr on March 20, 2025 \| parent \| context \| favorite \| on: FOSS infrastructure is under attack by AI companie... Yep, just make sure to add them to your robots.txt file so that only the bad robots are harmed.

ethan_smith on March 20, 2025 [–]

Temporary solution and only works if only some of us are doing it. What if these bots have a "manager" LLM agent that takes the decision on what pages to scrape?

Consider applying for YC's Summer 2026 batch! Applications are open till May 4
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact