I have installed the "monitoring" app (via rancher UI) which is provided by kubernetes. On top of it, I have deployed thanos-sidecar as 4th container in the prometheus pod running in cattle-monitoring-system with 3 containers. And this thanos-sidecar is able to push metrics to AWS s3 bucket (object-storage-config provides as input).
prometheus-rancher-monitoring-prometheus-0 4/4 Running 0 28mAfter this I have deployed following thanos components using bitnami/thanos chart
- Thanos Query
- Thanos Bucket Web
- Thanos Compactor
- Thanos Store Gateway
Thanos chart was deployed enabling the following components:- Thanos Query- Thanos Bucket Web- Thanos Compactor- Thanos Store GatewayThanos Query can be accessed through following DNS name from within your cluster: thanos-query.cattle-monitoring-system.svc.cluster.local (port 9090)To access Thanos Query from outside the cluster execute the following commands:1. Get the Thanos Query URL by running these commands: export SERVICE_PORT=$(kubectl get --namespace cattle-monitoring-system -o jsonpath="{.spec.ports[0].port}" services thanos-query) kubectl port-forward --namespace cattle-monitoring-system svc/thanos-query ${SERVICE_PORT}:${SERVICE_PORT} & echo "http://127.0.0.1:${SERVICE_PORT}"2. Open a browser and access Thanos Query using the obtained URL.I am not getting any kind of errors in these deployed thanos components pod's logs. And if I configure this thanos-query as a new data source in grafana dashboard, I am able to query and list the metrics without any issues.
But I am facing the below error when I try to access thanos-query UI page
Warning: Unexpected response status when fetching server time:Warning: Error fetching metrics list: Unexpected response status when fetching metrics namesWarning: Error fetching stores list: Unexpected response status when fetching storesWarning: Error fetching flags list: Unexpected response status when fetching flags
It would be really helpful if someone helps me to resolve this issue. I am not sure how to debug this issue as I am not seeing any error logs as such in the thanos components.