14-09-2014, 12:36 PM
He encontrado esta lista de bots que rastrean el contenido en la red:
Si me he dejado algún Bot ponerlo!! Indicar la URL donde sale el nombre que emplea ese bot en el User-Agent
Código:
https://ahrefs.com/robot/
bot => AhrefsBot
http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
bot => MJ12bot
http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
bot => Majestic-12
http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
bot => Majestic-SEO
http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
bot => DSearch
http://moz.com/help/pro/rogerbot-crawler
bot => Rogerbot
http://www.semrush.com/bot.html
bot => SemrushBot
http://archive.org/about/exclude.php
bot => ia_archiver
http://scoutjet.com
bot => ScoutJet
http://crawler.sistrix.net
bot => sistrix
http://www.searchmetrics.com/en/searchmetrics-bot/
bot => SearchmetricsBot
http://www.seokicks.de/robot.html
bot => SEOkicks-Robot
http://www.lipperhey.com/en/website-spider/
bot => Lipperhey Spider
http://www.exalead.com/search/webmasterguide
bot => Exabot
https://twitter.com/NetComber/status/334476871691550721
bot => NCBot
http://www.backlinktest.com/crawler.html
bot => BacklinkCrawler
http://archive.org/details/archive.org_bot
bot => archive.org_bot
https://meanpath.com/meanpathbot.html
bot => meanpathbot
http://www.botsvsbrowsers.com/details/1002332/index.html
bot => PagesInventory
http://www.aboundex.com/crawler/
bot => Aboundexbot
http://www.seoprofiler.com/bot/
bot => spbot
http://www.linkdex.com/about/bots/
bot => linkdexbot
http://www.useragentstring.com/pages/Gigabot/
bot => Gigabot
http://en.wikipedia.org/wiki/DotBot
bot => dotbot
http://nutch.apache.org/bot.html
bot => Nutch
http://webmeup-crawler.com/
bot => BLEXBot
http://graphicline.co.za/blogs/what-is-ezooms-bot
bot => Ezooms
http://blekko.com/about/blekkobot
bot => BlekkoBot
http://help.yandex.com/search/?id=1112030
bot => Yandex
https://support.google.com/webmasters/answer/182072?hl=en
bot => googlebot
https://support.google.com/webmasters/answer/178852
bot => Feedfetcher-Google
http://en.wikipedia.org/wiki/Bingbot
bot => BingBotSi me he dejado algún Bot ponerlo!! Indicar la URL donde sale el nombre que emplea ese bot en el User-Agent
Grupo Telegram
