After all, Google has been saying this for years back in 2015, John Mueller said “you probably shouldn't use the noindex in the robots.txt file”. In fact, Google released its robots.txt parser as an open source project along with this announcement the other day. Removed references to the deprecated Ajax Crawling Scheme.Updated formal syntax to be valid Augmented Backus-Naur Form (ABNF) per RFC5234 and to cover for UTF-8 characters in the robots.txt.Google currently enforces a size limit of 500 kibibytes (KiB), and ignores content after that limit.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |