User-agent: * # Disallow index, but allow individual files Disallow: /files/$ Disallow: /files/internal/