Questions about robots.txt
Read more about robots.txt
python (65.1k questions)
javascript (44.2k questions)
reactjs (22.7k questions)
java (20.8k questions)
c# (17.4k questions)
html (16.3k questions)
r (13.7k questions)
android (12.9k questions)
How to parse a sitemap index that has compressed links
I've made a program that reads the /robots.txt and the /sitemap.xml of a page and substracts the available sitemaps and stores them on the siteMapsUnsorted list.
Once there I use crawler-commons libra...
Santiago Luca
Votes: 0
Answers: 1
Can domains be used in robots.txt?
We have a development server at dev.example.com that is being indexed by Google. We are using AWS Lightsail to duplicate the development server to our production environment in totality — the same rob...
franklylately
Votes: 0
Answers: 3
Could we add ajax requests disallow in robots.txt in codeigniter
My question is that could we add ajax requests disallow in robots.txt in codeigniter
And is that compulsory to add robots.txt file in home directory or we place it in views with defined routes
Developer Ali Usman
Votes: 0
Answers: 0
How To Disallow URLs in Robots.txt ending with numeric values
I need help removing or DisAllowing some malicious URLs that Google has configured with my main domain. I didn't really pay attention until my site broke down and I found hundreds of pages like websit...
Hireme1st
Votes: 0
Answers: 1