Definition
A file at a site’s root that gives crawler access instructions. It is a crawling control, not a way to secure private content.
In practice
Check robots.txt when an important page is not crawled, and avoid using it to hide private data.
What to check next
A technical check can identify a possible issue on a page. Verify the finding against the page’s intended behavior before changing it.