# researchJobs — automated collection is not permitted except search-engine indexing. # Terms of use: https://jobs.scholarfellow.com/terms # Text and data mining rights are reserved (EU Directive 2019/790, Art. 4): /.well-known/tdmrep.json # Search engines are welcome on the public pages. User-agent: Googlebot Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: Googlebot-Image Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: AdsBot-Google Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: Bingbot Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: DuckDuckBot Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request # Answer engines: allowed, because they cite the page to a reader rather than train on it. User-agent: ChatGPT-User Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: OAI-SearchBot Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: Claude-User Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: Claude-SearchBot Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: Perplexity-User Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request # AI and LLM crawlers: allowed, and copying requires our consent — which this grants, on # the conditions at https://jobs.scholarfellow.com/terms: attribute the source, link back, and do not re-publish # a substantial part of the database. Text and data mining rights stay reserved and are # licensed on those terms, not waived: /.well-known/tdmrep.json User-agent: GPTBot Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: ClaudeBot Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: anthropic-ai Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: CCBot Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: Google-Extended Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: Applebot-Extended Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: Meta-ExternalAgent Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: Meta-ExternalFetcher Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: FacebookBot Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: Amazonbot Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: cohere-ai Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: cohere-training-data-crawler Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: YouBot Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: Timpibot Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request User-agent: PerplexityBot Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request # Commercial SEO and dataset scrapers User-agent: Bytespider User-agent: Diffbot User-agent: omgili User-agent: omgilibot User-agent: ImagesiftBot User-agent: PetalBot User-agent: DataForSeoBot User-agent: AhrefsBot User-agent: SemrushBot User-agent: MJ12bot Disallow: / # Everything else: public pages only User-agent: * Allow: / Disallow: /api/ Disallow: /internal/ Disallow: /account Disallow: /auth/ Disallow: /admin Disallow: /login Disallow: /register Disallow: /forgot Disallow: /reset Disallow: /calendar Disallow: /report Disallow: /data-request Sitemap: https://jobs.scholarfellow.com/sitemap.xml