technical seo audit robots.txt check

robots.txt checking