Documentation: Domain per Language plugin
A robots.txt per domain (PRO)
A multilingual site on several domains has one robots.txt file, and that file can only name one sitemap and one set of rules. The PRO edition serves a separate robots.txt on every domain, straight from the plugin.
Filling it in
Every domain row has a robots.txt field. It starts with the content of Joomla's own robots.txt.dist, so you can add to it instead of writing it from scratch. Add the sitemap of that domain yourself:
User-agent: *
Disallow: /administrator/
Disallow: /tmp/
Sitemap: https://www.example.nl/sitemap.xml
Leave the field empty to let Joomla handle robots.txt for that domain as it normally would.
How it is served
A request for /robots.txt on a configured domain is answered by the plugin as text/plain, with exactly what you typed. Nothing is added and no placeholder is replaced, so what you see in the field is what a crawler gets.
Two conditions
- No robots.txt file in the root of your site. The web server serves an existing file itself and Joomla never sees the request. A fresh Joomla ships
robots.txt.dist, notrobots.txt, so this is usually fine; if a file is there, the plugin warns you on its settings screen. Rename or remove it. - URL rewriting must send missing files to Joomla. The
.htaccessof Joomla does this out of the box, and a normal nginx configuration withtry_files $uri $uri/ /index.php?$argsdoes too.
Checking the result
Open https://www.example.nl/robots.txt and https://www.example.com/robots.txt in a browser: each should show its own content. The domain health check does the same automatically and tells you when a domain serves something else than what you filled in.