Since we are having this conversation here I will post this thought/question.
I know for a community to thrive it really does need to grow to get new ideas, replace people that have left or to busy to participate, etc. I am a happy part of that growth over the last few months.
But when I click on Members online I always see multiples of:
Robot Google
Robot Bing
Robot ByteDance
Are these the website scrubbers for these companies and if so would it make some sense to set the robots.txt off so then if I understand it they don't digest all the posts and eventually post stuff from here automatically when searches are done as the site grows? Or am I completely misunderstanding this. I don't believe the vision for the site is to put EVERYTHING behind a member login (Nor do I think it really should be...those sites already exist)
Don't know the answers, Just a random not completely formed question/thought.