How important is a robots.txt file for cybersecurity?

A robots.txt file has very limited direct cybersecurity significance, but it does have indirect security implications, it is a signal not an enforcement how to behave.

The safest way to explain it is:

robots.txt is meant for crawler behavior, not hide content. If security matters, robots.txt should be irrelevant.

A modern robots.txt should not list content that is meant to be hidden, because that hidden content should not be publicly accessible in the first place.

robots.txt assumes:

• public access,
• boluntary compliance,
• good-faith crawlers.

Security assumes:

• hostile actors (I will not follow instructions, and thanks for telling me where /admin directory is...),
• intentional abuse,
• zero trust.

And these two assumptions are incompatible.