Robots.txt
Please can you add a Robots.txt to the sites.
I signed up for the Google Search Console and they expect to get this.
Robots. txt file is a set of instructions for bots (in particular the Googlebot crawler) . This file is included in the source files of most websites.
Log in to comment and vote
Comments3
Brendon McCarthy
Aug 15, 2024
robots.txt is most often used to “Disallow” as “Allow” is the default.
https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt
To set robots.txt on a site that does not have a setting and are using Cloudflare as your DNS host, you could potentially use a Cloudflare Worker:
https://github.com/simonw/til/blob/main/cloudflare/robots-txt-cloudflare-workers.md
https://developers.cloudflare.com/workers/configuration/routing/routes/
Kyle Davidson
Jun 25, 2024
Go to Design Settings and scroll down to SEO
Grab the link for the sitemap xml file and upload that to Google search console:
https://app.jobboardly.com/design_settings/edit
I also recorded a video on how to do it here: https://www.youtube.com/watch?v=9YibyOpOsxE
Meleva
Jul 5, 2024
Hi Kyle,
Sitemap and robot.txt are not the same thing