Ever had content crawled by Google which you didn’t intend to expose? Of course you have, you just don’t want to admit it
I recently had a client getting their entire staging environment indexed by Google all because someone wondered what the robots.txt file was and instead of asking they decided to delete it. So now hundreds of pages from the live environment had a duplicate from the staging environment showing up in the search result and exposing their staging site to the entire world. Now I’m all against exposing the stage server to the public web but since it’s what the client wanted, it’s what they got.
So, what to do in this case?