Skip to main content

Documentation: Domain per Language plugin

A robots.txt per domain (PRO)

A multilingual site on several domains has one robots.txt file, and that file can only name one sitemap and one set of rules. The PRO edition serves a separate robots.txt on every domain, straight from the plugin.

Filling it in

Every domain row has a robots.txt field. It starts with the content of Joomla's own robots.txt.dist, so you can add to it instead of writing it from scratch. Add the sitemap of that domain yourself:

User-agent: *
Disallow: /administrator/
Disallow: /tmp/

Sitemap: https://www.example.nl/sitemap.xml

Leave the field empty to let Joomla handle robots.txt for that domain as it normally would.

How it is served

A request for /robots.txt on a configured domain is answered by the plugin as text/plain, with exactly what you typed. Nothing is added and no placeholder is replaced, so what you see in the field is what a crawler gets.

Two conditions

  • No robots.txt file in the root of your site. The web server serves an existing file itself and Joomla never sees the request. A fresh Joomla ships robots.txt.dist, not robots.txt, so this is usually fine; if a file is there, the plugin warns you on its settings screen. Rename or remove it.
  • URL rewriting must send missing files to Joomla. The .htaccess of Joomla does this out of the box, and a normal nginx configuration with try_files $uri $uri/ /index.php?$args does too.

Checking the result

Open https://www.example.nl/robots.txt and https://www.example.com/robots.txt in a browser: each should show its own content. The domain health check does the same automatically and tells you when a domain serves something else than what you filled in.