1. Halo Guest, pastikan Anda selalu menaati peraturan forum sebelum mengirimkan post atau thread baru.

[SHARE] Settingan Robots.txt Untuk PBN

Discussion in 'SEO' started by algozali, Aug 31, 2016.

Tags:
Page 1 of 2
  1. algozali

    algozali Dollar Hunter

    Joined:
    May 26, 2013
    Messages:
    877
    Likes Received:
    57
    Location:
    Pejwan Gugel [all TLD]
    Pagi semua warga ads.id yg tercinta :x
    langsung saja karena ngetik pake virtual keyboard capek euy :p
    untuk yg ingin bangun pbn tapi gak tau settingan robots.txt gimana?
    copas aja langsung
    Code:
    User-agent: duggmirror
User-agent: libwww-perl
User-agent: libwwwperl
User-agent: attach
User-agent: ASPSeek
User-agent: appie
User-agent: AbachoBOT
User-agent: autoemailspider
User-agent: anarchie
User-agent: antibot
User-agent: asterias
User-agent: B2w
User-agent: BackWeb
User-agent: BackDoorBot
User-agent: Bandit
User-agent: BatchFTP
User-agent: Black\ Hole
User-agent: Baidu
User-agent: BlowFish
User-agent: BuiltBotTough
User-agent: Bot\ mailto
User-agent: BotALot
User-agent: Buddy
User-agent: Bullseye
User-agent: bumblebee
User-agent: BunnySlippers
User-agent: ClariaBot
User-agent: curl
User-agent: clsHTTP
User-agent: ChinaClaw
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: Crescent
User-agent: CherryPickerSE
User-agent: CherryPickerElite
User-agent: Collector
User-agent: COAST\ WebMaster
User-agent: cosmos
User-agent: CopyRightCheck
User-agent: ColdFusion
User-agent: Copier
User-agent: Crescent
User-agent: DA
User-agent: DTS\ Agent
User-agent: DISCo\ Pump
User-agent: DittoSpyder
User-agent: Diamond
User-agent: Download\ Demon
User-agent: Download\ Wonder
User-agent: Downloader
User-agent: dloader
User-agent: Drip
User-agent: eCatch
User-agent: EirGrabber
User-agent: Express\ WebPictures
User-agent: Extreme\ Picture\ Finder
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EasyDL
User-agent: EirGrabber
User-agent: EroCrawler
User-agent: ExtractorPro
User-agent: EyeNetIE
User-agent: FAST\ WebCrawler
User-agent: FileHound
User-agent: Fetch\ API\ Request
User-agent: FlashGet
User-agent: FlickBot
User-agent: FrontPage
User-agent: FreeFind.com
User-agent: GetRight
User-agent: GetSmart
User-agent: Generic
User-agent: Go!Zilla
User-agent: Go-Ahead-Got-It
User-agent: gotit
User-agent: Grabber
User-agent: GrabNet
User-agent: Grafula
User-agent: Gulliver
User-agent: Harvest
User-agent: HMView
User-agent: Heretrix
User-agent: HitboxDoctor
User-agent: HTTPapp
User-agent: HTTrack
User-agent: HTTPTrack
User-agent: HTTPviewer
User-agent: httplib
User-agent: httpfetcher
User-agent: httpscraper
User-agent: hloader
User-agent: humanlinks
User-agent: ia_archiver
User-agent: InterGET
User-agent: Internet\ Ninja
User-agent: InfoNaviRobot
User-agent: InternetSeer.com
User-agent: Iria
User-agent: IRLbot
User-agent: JetCar
User-agent: JOC
User-agent: JOC\ Web\ Spider
User-agent: JoBo
User-agent: Java
User-agent: JustView
User-agent: Jonzilla
User-agent: JennyBot
User-agent: Kenjin\ Spider
User-agent: Keyword\ Density
User-agent: larbin
User-agent: LeechFTP
User-agent: Lachesis
User-agent: LexiBot
User-agent: libWeb
User-agent: Libby_
User-agent: LinkScan
User-agent: LinkWalker
User-agent: LinkextractorPro
User-agent: lftp
User-agent: likse
User-agent: Link
User-agent: lwp-trivial
User-agent: lwp\ request
User-agent: Magnet
User-agent: Mag-Net
User-agent: Mass\ Downloader
User-agent: MIIxpc
User-agent: Microsoft\ URL\ Control
User-agent: MSFrontPage
User-agent: MSIECrawler
User-agent: MicrosoftURL
User-agent: Missigua
User-agent: Mewsoft\ Search\ Engine
User-agent: moget
User-agent: Mata\ Hari
User-agent: Memo
User-agent: Metacarta
User-agent: Mercator
User-agent: MIDown\ tool
User-agent: MFC_Tear_Sample
User-agent: Mirror
User-agent: MIIxpc
User-agent: Mister\ PiX
User-agent: NationalDirectory\ WebSpider
User-agent: NICErsPRO
User-agent: Nikto
User-agent: Navroad
User-agent: NearSite
User-agent: NetAnts
User-agent: NetSpider
User-agent: NICErsPRO
User-agent: NetResearchServer
User-agent: NetMechanic
User-agent: Net\ Vampire
User-agent: Net\ Probe
User-agent: NetZip
User-agent: nexuscache
User-agent: Ninja
User-agent: NPBot
User-agent: our\ agent
User-agent: onestop
User-agent: oBot
User-agent: Octopus
User-agent: Offline\ Explorer
User-agent: Openfind
User-agent: Openfind\ data\ gatherer
User-agent: OrangeBot
User-agent: PageGrabber
User-agent: Papa\ Foto
User-agent: PHP\ version
User-agent: PHP
User-agent: PHPot
User-agent: Perl
User-agent: pcBrowser
User-agent: pavuk
User-agent: Pockey
User-agent: Ping
User-agent: PingALink\ Monitoring\ Services
User-agent: ProWebWalker
User-agent: ProPowerBot
User-agent: Pump
User-agent: Pompos
User-agent: psbot
User-agent: Python\ urllib
User-agent: Python-urllib
User-agent: QueryN
User-agent: RealDownload
User-agent: Reaper
User-agent: Recorder
User-agent: RepoMonkey
User-agent: psycheclone
User-agent: RMA
User-agent: Rico
User-agent: Robozilla
User-agent: ReGet
User-agent: Siphon
User-agent: SiteSnagger
User-agent: sitecheck.internetseer.com
User-agent: SmartDownload
User-agent: Snake
User-agent: spanner
User-agent: Stealer
User-agent: SpaceBison
User-agent: SpankBot
User-agent: Spinne
User-agent: Stripper
User-agent: slysearch
User-agent: Sucker
User-agent: Snoopy
User-agent: ScoutAbout
User-agent: Scooter
User-agent: SuperBot
User-agent: SuperHTTP
User-agent: Snapbot
User-agent: Surfbot
User-agent: suzuran
User-agent: Szukacz
User-agent: Sqworm
User-agent: tAkeOut
User-agent: Teleport\ Pro
User-agent: Telesoft
User-agent: TurnitinBot
User-agent: turingos
User-agent: toCrawl
User-agent: TightTwatBot
User-agent: True_Robot
User-agent: The\ Intraformant
User-agent: TheNomad
User-agent: Titan
User-agent: UrlDispatcher
User-agent: URLy\ Warning
User-agent: Vayala
User-agent: Vagabondo
User-agent: Vintage
User-agent: Vacuum
User-agent: VCI
User-agent: VoidEYE
User-agent: W3C_Validator
User-agent: Webdownloader
User-agent: Web\ Downloader
User-agent: Webhook
User-agent: Webmole
User-agent: Webminer
User-agent: Webmirror
User-agent: Websucker
User-agent: Websites
User-agent: Web\ Image\ Collector
User-agent: Web\ Sucker
User-agent: WebAuto
User-agent: WebCopier
User-agent: WebFetch
User-agent: WebReaper
User-agent: WebSauger
User-agent: Website
User-agent: Webster
User-agent: WebStripper
User-agent: WebCopier
User-agent: WebViewer
User-agent: WebWhacker
User-agent: WebEnhancer
User-agent: Wells
User-agent: WebZIP
User-agent: Wget
User-agent: Whacker
User-agent: Widow
User-agent: Xaldon
User-agent: Wildsoft\ Surfer
User-agent: WinHttpRequest
User-agent: WinHttp
User-agent: Webster\ Pro
User-agent: Web\ Image\ Collector
User-agent: WebZip
User-agent: WebAuto
User-agent: Website\ Quester
User-agent: WWWOFFLE
User-agent: WWW-Collector-E
User-agent: Xaldon\ WebSpider
User-agent: Xenu
User-agent: Xara
User-agent: Y!TunnelPro
User-agent: YahooYSMcm
User-agent: Zade
User-agent: ZBot
User-agent: Zeus
User-agent: Rogerbot
User-agent: Exabot
User-agent: MJ12bot
User-agent: Dotbot
User-agent: Gigabot
User-agent: AhrefsBot
User-agent: BlackWidow
User-agent: Bot\ mailto:[email protected]
User-agent: ChinaClaw
User-agent: Custo
User-agent: DISCo
User-agent: Download\ Demon
User-agent: eCatch
User-agent: EirGrabber
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: Express\ WebPictures
User-agent: ExtractorPro
User-agent: EyeNetIE
User-agent: FlashGet
User-agent: GetRight
User-agent: GetWeb!
User-agent: Go!Zilla
User-agent: Go-Ahead-Got-It
User-agent: GrabNet
User-agent: Grafula
User-agent: HMView
User-agent: HTTrack
User-agent: Image\ Stripper
User-agent: Image\ Sucker
User-agent: Indy\ Library
User-agent: InterGET
User-agent: Internet\ Ninja
User-agent: JetCar
User-agent: JOC\ Web\ Spider
User-agent: larbin
User-agent: LeechFTP
User-agent: Mass\ Downloader
User-agent: MIDown\ tool
User-agent: Mister\ PiX
User-agent: Navroad
User-agent: NearSite
User-agent: NetAnts
User-agent: NetSpider
User-agent: Net\ Vampire
User-agent: NetZIP
User-agent: Octopus
User-agent: Offline\ Explorer
User-agent: Offline\ Navigator
User-agent: PageGrabber
User-agent: Papa\ Foto
User-agent: pavuk
User-agent: pcBrowser
User-agent: RealDownload
User-agent: ReGet
User-agent: SiteSnagger
User-agent: SmartDownload
User-agent: SuperBot
User-agent: SuperHTTP
User-agent: Surfbot
User-agent: tAkeOut
User-agent: Teleport\ Pro
User-agent: VoidEYE
User-agent: Web\ Image\ Collector
User-agent: Web\ Sucker
User-agent: WebAuto
User-agent: WebCopier
User-agent: WebFetch
User-agent: WebGo\ IS
User-agent: WebLeacher
User-agent: WebReaper
User-agent: WebSauger
User-agent: Website\ eXtractor
User-agent: Website\ Quester
User-agent: WebStripper
User-agent: WebWhacker
User-agent: WebZIP
User-agent: Wget
User-agent: Widow
User-agent: WWWOFFLE
User-agent: Xaldon\ WebSpider
Disallow: /
    Apa fungsi robots.txt ini?

    untuk ngeblock bot biar gak meng-crawl pbn agan, seperti AhrefsBot jadi blog agan gak terdeteksi jika ada backlink dari pbn tersebut. tujuannya? biar kompetitor gak tau link building kita.
    contoh lain ia_archiver bot dari wayback machine untuk cek history web

    udah gitu ajah
    untuk yang mau nambahin settingan .htacces monggo
    saya gak nolak like loh :malu:
     
    algozali, Aug 31, 2016
    #1
    Tapuy, pebriscreamo, angger07 and 6 others like this.
  2. Ahmad Sarnuji

    Ahmad Sarnuji Super Hero

    Joined:
    May 30, 2015
    Messages:
    1,918
    Likes Received:
    201
    Location:
    Kab.Tangerang, Desa Sumur Bandung
    menyimak :kaget2::kaget2::kaget2::kaget2::kaget2:
     
    Ahmad Sarnuji, Aug 31, 2016
    #2
  3. Ahmad Sarnuji

    Ahmad Sarnuji Super Hero

    Joined:
    May 30, 2015
    Messages:
    1,918
    Likes Received:
    201
    Location:
    Kab.Tangerang, Desa Sumur Bandung
    menyimak :kaget2::kaget2::kaget2::kaget2::kaget2:
     
    Ahmad Sarnuji, Aug 31, 2016
    #3
  4. yanzbatara

    yanzbatara Super Hero

    Joined:
    Feb 23, 2011
    Messages:
    936
    Likes Received:
    40
    Location:
    Bandung
    thanks om.. :ok::ok:
     
    yanzbatara, Aug 31, 2016
    #4
  5. iidbae

    iidbae Super Hero

    Joined:
    Nov 12, 2012
    Messages:
    2,314
    Likes Received:
    168
    Location:
    Pusdai-Bandung
    terima kasih
     
    iidbae, Aug 31, 2016
    #5
  6. algozali

    algozali Dollar Hunter

    Joined:
    May 26, 2013
    Messages:
    877
    Likes Received:
    57
    Location:
    Pejwan Gugel [all TLD]
    Ok monggo disedot gan [emoji11]
     
    algozali, Aug 31, 2016
    #6
  7. Fay xmuts

    Fay xmuts Newbie

    Joined:
    May 26, 2015
    Messages:
    13
    Likes Received:
    0
    Ini yg ane cari hee..mksh gan
     
    Fay xmuts, Aug 31, 2016
    #7
  8. algozali

    algozali Dollar Hunter

    Joined:
    May 26, 2013
    Messages:
    877
    Likes Received:
    57
    Location:
    Pejwan Gugel [all TLD]
    Sama" gan
     
    algozali, Aug 31, 2016
    #8
  9. hamdi1387

    hamdi1387 Banned

    Joined:
    Dec 6, 2014
    Messages:
    377
    Likes Received:
    7
    Location:
    Jakarta
    gelar tiker dulu :D
     
    hamdi1387, Aug 31, 2016
    #9
  10. robetjanuar

    robetjanuar Newbie

    Joined:
    Aug 31, 2016
    Messages:
    15
    Likes Received:
    0
    Mantab lanjut disedot
     
    robetjanuar, Sep 1, 2016
    #10
  11. gorian

    gorian Ads.id Pro

    Joined:
    Dec 27, 2011
    Messages:
    301
    Likes Received:
    44
    Sangar-e rek...... kompetitor bisa mati kutu :cilukba::cilukba::cilukba::cilukba::cilukba:
     
    gorian, Sep 1, 2016
    #11
  12. indra666

    indra666 Ads.id Fan

    Joined:
    Feb 1, 2013
    Messages:
    187
    Likes Received:
    4
    Yg buat web arcive wayback machine pilih yg mana gan codenya ?

    dikirim dari dalam kamar
     
    indra666, Sep 1, 2016
    #12
  13. algozali

    algozali Dollar Hunter

    Joined:
    May 26, 2013
    Messages:
    877
    Likes Received:
    57
    Location:
    Pejwan Gugel [all TLD]
    Monggo gan

    Hahaha iya gan dikirain gak pake link building padahal backlinknya dari pbn semua [emoji41]
    Itu emang settingan standart buat PBN gan

    User-agent : ia_archiver
    Disallow : /
     
    algozali, Sep 1, 2016
    #13
  14. budispdst

    budispdst Hero

    Joined:
    Jan 8, 2015
    Messages:
    610
    Likes Received:
    10
    semua user agent di atas, saya ngk tau fungsi, peran dari masing2 user agent :bosan:
     
    budispdst, Sep 1, 2016
    #14
    robetjanuar likes this.
  15. daffyshaci

    daffyshaci S.E.O Freak

    Joined:
    Aug 26, 2011
    Messages:
    905
    Likes Received:
    1,433
    menggunakan settingan yg sama di semua PBN bisa gampang terdetect oleh google. hxxp://domainpbn.com/robot.txt
    kompetitor ga ngedetect, malah google yg ngedetect,.. hehe
     
    daffyshaci, Sep 2, 2016
    #15
  16. cowobreng

    cowobreng Ads.id Fan

    Joined:
    Feb 3, 2014
    Messages:
    192
    Likes Received:
    25
    bro orderan sisa 4 pbn lg gmn? ente bener2 mau kabur gitu aja? di whatsapp sama ym gak pernah bales? udah lewat 3 bulan nih. Parah!
     
    cowobreng, Sep 2, 2016
    #16
  17. mjrasta

    mjrasta Ads.id Fan

    Joined:
    May 26, 2011
    Messages:
    215
    Likes Received:
    6
    Jejak dulu,
     
    mjrasta, Sep 2, 2016
    #17
  18. algozali

    algozali Dollar Hunter

    Joined:
    May 26, 2013
    Messages:
    877
    Likes Received:
    57
    Location:
    Pejwan Gugel [all TLD]
    Saya juga bingung gan :komunis:

    Iya om, tp yang terpenting ngeblock bot backlink checker seperti ahrefsbot. Jadi bisa di akali dengan mengurangi beberapa User-agent atau tidak mengatur robots.txt pada beberapa PBN.

    Waduh ada apa inih.
    Semoga cepet kelar masalahnya gan

    Monggo
     
    algozali, Sep 2, 2016
    #18
  19. siijo

    siijo Super Hero

    Joined:
    Apr 8, 2013
    Messages:
    1,141
    Likes Received:
    162
    Location:
    Jogja - Purworejo
    Like pencet gann.. ini yang ane tunggu2 ada yang share robot txt, maklum kalo cari di web bule rada ng fly mata ane wkakaka
     
    siijo, Sep 3, 2016
    #19
  20. angger07

    angger07 Hero

    Joined:
    Jun 26, 2011
    Messages:
    652
    Likes Received:
    45
    Location:
    sumatra selatan
    Ajib.thanks
     
    angger07, Sep 3, 2016
    #20
Page 1 of 2

Share This Page