How to implement and verify robots.txt

Create a `robots.txt` file in the `/public` folder specifying which crawlers can access which parts of your site. Include `Sitemap: https://yourdomain.com/sitemap.xml`. For AI crawlers, allow them explicitly if you want indexing by Perplexity, ChatGPT, etc. Test via the Robots Tester in Google Search Console.