I've installed kube-prometheus-stack through helm, and works well. I added a additional scrape job, following https://github.com/prometheus-operator/prometheus-operator/issues/3608 this and it was successful. Job looks like this:

scrape_configs:
  - job_name: 'my_job'
    static_configs:
      - targets: ['x.x.x.x:9100']

Now, I want to replace that IP address(it's not a pod's IP, it's an instance located completely different from cluster or Node) to a host name. I think a proper options is using hostAliases, but can not get it to work. So far, I've tried:

prometheus:

  enabled: true

  annotations: {}
  
  hostAliases:
  - ip: "127.0.0.1"
    hostnames:
    - "foo.local"
    - "bar.local"
  - ip: "10.1.2.3"
    hostnames:
    - "foo.remote"
    - "bar.remote"



prometheusSpec:
    ## If true, pass --storage.tsdb.max-block-duration=2h to prometheus. This is already done if using Thanos
    ##
    hostAliases:
    - ip: "127.0.0.1"
      hostnames:
      - "foo.local"
      - "bar.local"
    - ip: "10.1.2.3"
      hostnames:
      - "foo.remote"
      - "bar.remote"

adding hostAliases(both under prometheus and prometheusSpec) in values.yaml didn't work. So I tried to modify templates/prometheus/prometheus.yaml ,but adding hostAliases right under spec generates error:

Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(Prometheus.spec): unknown field "hostAliases" in com.coreos.monitoring.v1.Prometheus.spec

referencing How to add extra hosts entries in helm charts ,I've tried this to templates/prometheus/prometheus.yaml:

spec:
  spec:
    {{- with .Values.hostAliases }}
    hostAliases:
{{ toYaml . | indent 8 }}
    {{- end }}

and didn't work. (template under spec caused error, so I deleted template.)

So desperate, I even tried to modify /etc/hosts file by getting into prometheus pods(which I think is not a good solution but does work),but I can't even find a way to modify /etc/hosts because I don't have root access. Any solutions?

To check if hosts are properly added, this:

kubectl exec prometheus-prometheus-kube-prometheus-prometheus-0 -- cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
172.0.99.34 prometheus-prometheus-kube-prometheus-prometheus-0.prometheus-operated.prometheus.svc.cluster.local prometheus-prometheus-kube-prometheus-prometheus-0