Peter Wilson 122cb2864b #43590: Use robots meta tag to better discourage search engines. ...

This changes the "discourage search engines" option to output a `noindex, nofollow` robots meta tag. `Disallow: /` is removed from the `robots.txt` to allow search engines to discover they are requested not to index the site.

Disallowing search engines from accessing a site in the `robots.txt` file can result in search engines listing a site with a fragment (a listing without content).

Props donmhico, jonoaldersonwp.
Fixes #43590.



git-svn-id: https://develop.svn.wordpress.org/trunk@45928 602fd350-edb4-49c9-b593-d223f7449a82

2019-09-02 02:26:55 +00:00

..

e2e

Build Tools: Miscellaneous local-env improvements.

2019-08-12 08:28:33 +00:00

phpunit

#43590: Use robots meta tag to better discourage search engines.

2019-09-02 02:26:55 +00:00

qunit

Regenerate the REST API fixtures after [45443].

2019-06-07 20:23:46 +00:00