This is an example of our most recent optimized robots.txt file:
User-Agent: * Disallow: /cgi-bin Disallow: /wp- Disallow: /?s= Disallow: *&s= Disallow: /search Disallow: /author/ Disallow: *?attachment_id= Disallow: */rss Disallow: */embed Disallow: /wp-admin/ Disallow: /*?*wordfence_lh Allow: /wp-admin/admin-ajax.php Allow: /wp-content/uploads/ Allow: /wp-content/themes/ Allow: /*/*.js Allow: /*/*.css Allow: /wp-*.png Allow: /wp-*.jpg Allow: /wp-*.jpeg Allow: /wp-*.gif Allow: /wp-*.svg Allow: /wp-*.pdf
This is what each means: