If You’re Using a Bot or Scraper
Your bot/scraper is returning this page instead of the page you tried to access because the operator of the website you’re trying to crawl considers your bot/scraper to be a commercial use of their website to which they did not consent.
Since you chose to ignore robots.txt the operator has resorted to more stringent measures.
If You Came From a Search Engine, Forum, or Social Network
If you were brought here after clicking a link in a search engine like Google or Bing, a forum like Reddit or Hacker News, or a social network like Facebook or Twitter it was because the operator of the site you tried to visit does not welcome traffic from corporate-owned platforms and regards links to their site from such platforms as a commercial use to which they did not consent.
Their problem isn’t necessarily with you, but with the operators of a platform you frequent. Your regrettable inconvenience is collateral damage in a previously unrecognized form of class warfare between the commercial/corporate web and personal website operators.
If You Operate a Personal Website
The law may not adequately safeguard your right to be acknowledged as an author and compensated for commercial use of your work, but that does not mean you are obligated to accept unwanted commercial traffic. If you have access to your web server configuration, you have the right to block unwanted user agents to reduce commercial crawling and scraping. You likewise have the right to attempt to reject traffic from commercial referrers.
If you agree with the sentiments expressed on this website, you may redirect unwelcome commercial traffic to nocommercialuse.org.
If you are unfamiliar with this process of rejecting unwanted traffic at the server level, the following examples may prove useful.
Simplified Examples For Apache Sites
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (BadBot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (WorseBot) [NC]
RewriteRule .* https://nocommercialuse.org/ [L]
RewriteEngine On
RewriteCond %{HTTP_REFERER} (corporate\.social) [NC,OR]
RewriteCond %{HTTP_REFERER} (nazi\.bar) [NC]
RewriteRule .* https://nocommercialuse.org/ [L]
Simplified Examples For Nginx Sites
if ($http_user_agent ~* (BadBot|WorseBot)) {
proxy_pass https://nocommercialuse.org/
}
if ($http_referer ~* (corporate\.social|nazi\.bar)) {
proxy_pass https://nocommercialuse.org/
}
If You Work in Tech
If you work for a search engine, social media platform, or a generative AI company and you’re visiting this website for any reason, the work you do and the world you’re helping to build is the reason this site exists. Your business depends on not having to ask permission, and on being able to presume consent for commercial use of personal websites for no other reason than that they are accessible on the internet.
US and international copyright law may even permit this at the moment, but that does not mean that personal website operators are legally or ethically obligated to cooperate with you. We reserve the right to refuse service to anyone for any reason.
We do not consent to commercial use of our websites, we never have, and if you had bothered to try to get informed consent first instead of doing as you pleased and then asking forgiveness afterward you might not have provoked our ire. You might instead have found yourselves facing a demand for proper compensation and attribution, but that is merely an indication that we are not entirely unreasonable and thus willing to negotiate a mutually beneficial arrangement.
If you think that opt out is acceptable and that persuading people to opt in is onerous or inconvenient, then I pity the women in your lives. I doubt you have any greater regard for their rights than you do for those of personal website operators. Your careerism makes you complicit, and when those who survive us ask “Who killed the world?” their storytellers will rightly blame you.
If this offends any of you because you don’t want to see yourselves as nothing more than little Eichmanns, then stop being part of the problem. Unionize and resist unethical uses of your skills. You are the means of production. Your work makes your bosses’ profits possible. Without you, they are nothing.