Removed x-robots-header helpers file

[vincenzodnp]

The X-Robots-Tag "noindex, nofollow"; HTTP header is added twice for development environment because it's already added by build-deploy-tool for autogenerated and/or custom routes in a development environment.
No more readons to keep it in the nginx image unless useful for local development.

See https://github.com/uselagoon/build-deploy-tool/blob/7ad87f8c3248effcbd30f34997f2f64724141cee/internal/templating/ingress/templates_ingress.go#L136-L138

[shreddedbacon]

This may not be required. The duplicate headers are the same, its more that its just presented twice.

Since the build-deploy-tool adds the ingress-nginx annotation, this means it will only work if the cluster running the workload has ingress-nginx, but it means every development or autogenerated route will receive the header, even if it isn't an nginx image.

This being in the nginx image only, means that for those clusters not running ingress-nginx, they will still get the header too, but only in containers built from this.

If duplicate headers are a problem for someone, if their hosted Lagoon uses ingress-nginx then they can simply remove this file in their nginx image builds to prevent their nginx service from adding the duplicate header.

Edit: Maybe it is worth just describing this behaviour somewhere in the Lagoon documentation