What is robots.txt?

A text file at the root of a site instructing crawlers which paths they may or may not request. Crucially it does NOT stop indexing — to prevent indexing, use a <meta name="robots" content="noindex">.