Robot.txt files are simply a list of URLs for Googlebot to crawl and to index. You can tell the crawlers for example to crawl or not to crawl a page. This allows the search engines to more efficiently use their crawl budget, so only the pages / posts you want crawled do get indexed and regularly “crawled and indexed” by Google.