FEATURE: improve new time series registration speed on systems with high number of CPU cores. Thanks to @misutoth for the initial idea and implementation.
FEATURE: make background merge more responsive and scalable. This should help the following issues: 5190, 3425, 648.
FEATURE: graphite: add support for negative index in groupByNode and aliasByNode functions. Thanks to @rbizos for the pull request.
FEATURE: vmagent: reduce initial memory usage when scraping targets, which return huge responses (for example, kube-state-metrics in large Kubernetes cluster may return 100MB+ responses), without the need to explicitly enable stream parsing mode. See this issue.
FEATURE: vmagent: expose ability to set OAuth2 endpoint parameters per each -remoteWrite.url via the command-line flag -remoteWrite.oauth2.endpointParams. See these docs. Thanks to @mhill-holoplot for the pull request.
FEATURE: vmagent: do not send unfinished aggregation state on shutdown or hot config reload by default, as it tend to produce unexpected anomalies with lower values. The old behavior can be restored by specifying flush_on_shutdown: true setting in streaming aggregation config. See more details here.
FEATURE: streaming aggregation: expand %{ENV_VAR} placeholders in config files with the corresponding environment variable values.
FEATURE: vmalert: expose ability to set OAuth2 endpoint parameters via the following command-line flags:
-datasource.oauth2.endpointParams for -datasource.url
-notifier.oauth2.endpointParams for -notifier.url
-remoteRead.oauth2.endpointParams for -remoteRead.url
-remoteWrite.oauth2.endpointParams for -remoteWrite.url
FEATURE: vmauth: add ability to proxy incoming requests to different backends based on the requested host via src_hosts option at url_map. See these docs.
FEATURE: vmauth: expose vmauth_user_request_backend_errors_total and vmauth_unauthorized_user_request_backend_errors_totalmetrics, which track the number of failed requests because of backend errors. See this feature request.
FEATURE: all VictoriaMetrics components: break HTTP client connection if an error occurs after the server at -httpListenAddr already sent response status code. Previously such an error couldn't be detected at client side. Now the client will get an error about invalid chunked response. The error message is simultaneously written to the server log and in the last line of the response. This should help detecting errors when migrating data between VictoriaMetrics instances by vmctl. See this issue.
FEATURE: all VictoriaMetrics components: add ability to specify arbitrary HTTP headers to send with every request to -pushmetrics.url. See push metrics docs.
FEATURE: all VictoriaMetrics components: add -metrics.exposeMetadata command-line flag, which allows displaying TYPE and HELP metadata at /metrics page exposed at -httpListenAddr. This may be needed when the /metrics page is scraped by collector, which requires the TYPE and HELP metadata such as Google Cloud Managed Prometheus.
FEATURE: all VictoriaMetrics components: add ability to dynamically re-read auth keys and passwords from files and urls when using file:///path/to/file or http://host/path syntax for the following command-line flags: -configAuthKey, -deleteAuthKey, -flagsAuthKey, -forceMergeAuthKey, -forceFlushAuthKey, -httpAuth.password, -metricsAuthKey, -pprofAuthKey, -reloadAuthKey, -search.resetCacheAuthKey, -snapshotAuthKey. For example, -httpAuth.password=file:///path/to/password. See these docs for details.
FEATURE: dashboards/cluster: add panels for detailed visualization of traffic usage between vmstorage, vminsert, vmselect components and their clients. New panels are available in the rows dedicated to specific components.
FEATURE: dashboards/cluster: update "Slow Queries" panel to show percentage of the slow queries to the total number of read queries served by vmselect. The percentage value should make it more clear for users whether there is a service degradation.
FEATURE: dashboards/single: change dashboard title from VictoriaMetrics to VictoriaMetrics - single-node. The new title should provide better understanding of this dashboard purpose.
FEATURE: vmctl: rename cmd-line flag vm-native-disable-retries to vm-native-disable-per-metric-migration to better reflect its meaning.
FEATURE: vmctl: add -vm-native-src-insecure-skip-verify and -vm-native-dst-insecure-skip-verify command-line flags for native protocol. It can be used for skipping TLS certificate verification when connecting to the source or destination addresses.
FEATURE: Alerting rules for VictoriaMetrics: add job label to DiskRunsOutOfSpace alerting rule, so it is easier to understand to which installation the triggered instance belongs.
FEATURE: vmstorage: add tenant identifier for log messages regarding dropping excessive labels due to limits defined by -maxLabelsPerTimeseries or -maxLabelValueLen command-line flags. Previously, it was hard to understand to which tenant the dropped labels belong.
FEATURE: vmui: add the ability to export and import query reports:
add a Query Analyzer page that allows you to build graphs from JSON data containing the results of executing a query request.
add an Export query button to the graph that saves the result of executing the query in JSON. See this pull request.
FEATURE: vmui: add -vmui.defaultTimezone flag to set a default timezone. See this issue and these docs.
FEATURE: vmui: include UTC in the timezone selection dropdown for standardized time referencing. See this issue.
FEATURE: vmui: add the ability to expand/collapse all tracing entries and download tracing data in .json format. See this issue.
BUGFIX: properly return the list of matching label names and label values from /api/v1/labels and /api/v1/label/.../values when the database contains more than -search.maxUniqueTimeseries unique time series on the selected time range. Previously VictoriaMetrics could return the number of matching timeseries exceeds ... error in this case. See this issue.
BUGFIX: properly return errors from export APIs. Previously these errors were silently suppressed. See this pull request.
BUGFIX: VictoriaMetrics cluster: properly return full results when -search.skipSlowReplicas command-line flag is passed to vmselect and when vmstorage groups are in use. Previously partial results could be returned in this case.
BUGFIX: vmstorage: properly expire storage/prefetchedMetricIDs cache. Previously this cache was never expired, so it could grow big under high churn rate. This could result in increasing CPU load over time.
BUGFIX: vmalert: check -external.url schema when starting vmalert, must be http or https. Before, alertmanager could reject alert notifications if -external.url contained no or wrong schema.
BUGFIX: vmalert: automatically add exported_ prefix for original evaluation result label if it's conflicted with external or reserved one, previously it was overridden. See this issue.
BUGFIX: vmalert: autogenerate ALERTS_FOR_STATE time series for alerting rules with for: 0. Previously, ALERTS_FOR_STATE was generated only for alerts with for > 0. The change aligns with Prometheus behavior. See more details in this issue.
BUGFIX: vmalert: fix data race during hot-config reload. The result of the race could cause all rules from updated group to fail with context cancelled error.
BUGFIX: MetricsQL: consistently sort results for q1 or q2 query, so they do not change colors with each refresh in Grafana. See this issue.
BUGFIX: MetricsQL: properly handle queries, which wrap rollup functions with multiple arguments without explicitly specified lookbehind window in square brackets into aggregate functions. For example, sum(quantile_over_time(0.5, process_resident_memory_bytes)) was resulting to expecting at least 2 args to ...; got 1 args error. Thanks to @atykhyy for the pull request.
BUGFIX: vmctl: retry on import errors in vm-native mode. Before, retries happened only on writes into a network connection between source and destination. But errors returned by server after all the data was transmitted were logged, but not retried.
BUGFIX: vmagent: properly discover targets for role: endpoints and role: endpointslice in kubernetes_sd_configs. Previously some endpoints and endpointslice targets could be left undiscovered or some targets could have missing __meta_* labels when performing service discovery in busy Kubernetes clusters with large number of pods. See this pull request.
BUGFIX: vmagent: respect explicitly set series_limit: 0 in scrape_config. This allows removing series_limit restriction on a per-scrape_config basis when global limit is set via -promscrape.seriesLimitPerTarget. Previously, 0 value was ignored in favor of -promscrape.seriesLimitPerTarget.
BUGFIX: vmagent: do not discover scrape targets for already terminated pods and containers in kubernetes_sd_configs. Such pods and containers cannot be scraped and cannot resurrect, so there is no sense in generating scrape targets for them.
BUGFIX: MetricsQL: properly process queries with too big lookbehind window such as foo[100y]. Previously, such queries could return empty responses even if foo is present in database. See this issue.
BUGFIX: MetricsQL: properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See this issue.
BUGFIX: vmselect: vmsingle/vmselect returns http status 429 (TooManyRequests) instead of 503 (ServiceUnavailable) when max concurrent requests limit is reached.
BUGFIX: vmselect: set proper timestamps in instant query results if the query contains rollup functions with lookbehind windows in square brackets exceeding 3 hours. For example, sum(avg_over_time(up[24h])). This bug has been introduced in v1.95.0. See this and this issues.
How to run VictoriaMetrics
Unpack the victoria-metrics-*.tar.gz archive and read these docs.
vmutils-*.tag.gz archive contains the following tools:
This PR contains the following updates:
| Package | Update | Change |
|---|---|---|
| [victoriametrics/vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics) | minor | `v1.96.0` -> `v1.97.0` |
---
### Release Notes
<details>
<summary>VictoriaMetrics/VictoriaMetrics (victoriametrics/vmagent)</summary>
### [`v1.97.0`](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.0)
[Compare Source](https://github.com/VictoriaMetrics/VictoriaMetrics/compare/v1.96.0...v1.97.0)
Released at 2024-01-30
- SECURITY: upgrade base docker image (Alpine) from 3.19.0 to 3.19.1. See [alpine 3.19.1 release notes](https://www.alpinelinux.org/posts/Alpine-3.19.1-released.html).
- SECURITY: upgrade Go builder from Go1.21.5 to Go1.21.6. See [the list of issues addressed in Go1.21.6](https://github.com/golang/go/issues?q=milestone%3AGo1.21.6+label%3ACherryPickApproved).
- FEATURE: improve new [time series](https://docs.victoriametrics.com/keyConcepts.html#time-series) registration speed on systems with high number of CPU cores. Thanks to [@​misutoth](https://github.com/misutoth) for the initial idea and [implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5212).
- FEATURE: make [background merge](https://docs.victoriametrics.com/#storage) more responsive and scalable. This should help the following issues: [5190](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5190), [3425](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425), [648](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648).
- FEATURE: [graphite](https://docs.victoriametrics.com/#graphite-render-api-usage): add support for negative index in `groupByNode` and `aliasByNode` functions. Thanks to [@​rbizos](https://github.com/rbizos) for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5581).
- FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add support for discovering [Hetzner Cloud](https://www.hetzner.com/cloud) and [Hetzner Robot](https://docs.hetzner.com/robot) scrape targets. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3154) and [these docs](https://docs.victoriametrics.com/sd_configs.html#hetzner_sd_configs).
- FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add support for [DataDog v2 data ingestion protocol](https://docs.datadoghq.com/api/latest/metrics/#submit-metrics). See [these docs](https://docs.victoriametrics.com/#how-to-send-data-from-datadog-agent) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4451).
- FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): reduce initial memory usage when scraping targets, which return huge responses (for example, [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) in large Kubernetes cluster may return 100MB+ responses), without the need to explicitly enable [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5567).
- FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): expose ability to set OAuth2 endpoint parameters per each `-remoteWrite.url` via the command-line flag `-remoteWrite.oauth2.endpointParams`. See [these docs](https://docs.victoriametrics.com/vmagent.html#advanced-usage). Thanks to [@​mhill-holoplot](https://github.com/mhill-holoplot) for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5427).
- FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add ability to set `attach_metadata.node=true` option for all the [`kubernetes_sd_configs`](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs) defined at [`-promscrape.config`](https://docs.victoriametrics.com/vmagent.html#quick-start) via `-promscrape.kubernetes.attachNodeMetadataAll` command-line flag. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4640). Thanks to [@​wasim-nihal](https://github.com/wasim-nihal) for [the initial implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5593).
- FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): do not send unfinished [aggregation state](https://docs.victoriametrics.com/stream-aggregation.html) on shutdown or [hot config reload](https://docs.victoriametrics.com/stream-aggregation.html#configuration-update) by default, as it tend to produce unexpected anomalies with lower values. The old behavior can be restored by specifying `flush_on_shutdown: true` setting in streaming aggregation config. See more details [here](https://docs.victoriametrics.com/stream-aggregation.html#aggregation-outputs).
- FEATURE: [streaming aggregation](https://docs.victoriametrics.com/stream-aggregation.html): expand `%{ENV_VAR}` placeholders in config files with the corresponding environment variable values.
- FEATURE: [vmalert](https://docs.victoriametrics.com/vmagent.html): expose ability to set OAuth2 endpoint parameters via the following command-line flags:
- `-datasource.oauth2.endpointParams` for `-datasource.url`
- `-notifier.oauth2.endpointParams` for `-notifier.url`
- `-remoteRead.oauth2.endpointParams` for `-remoteRead.url`
- `-remoteWrite.oauth2.endpointParams` for `-remoteWrite.url`
- FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth.html): add ability to proxy incoming requests to different backends based on the requested host via `src_hosts` option at `url_map`. See [these docs](https://docs.victoriametrics.com/vmauth.html#generic-http-proxy-for-different-backends).
- FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth.html): expose `vmauth_user_request_backend_errors_total` and `vmauth_unauthorized_user_request_backend_errors_total` [metrics](https://docs.victoriametrics.com/vmauth.html#monitoring), which track the number of failed requests because of backend errors. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5565).
- FEATURE: [vmauth](https://docs.victoriametrics.com/vmauth.html): add an ability to specify additional labels for [per-user metrics](https://docs.victoriametrics.com/vmauth.html#monitoring) via `metric_labels` section. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5565).
- FEATURE: all VictoriaMetrics components: break HTTP client connection if an error occurs after the server at `-httpListenAddr` already sent response status code. Previously such an error couldn't be detected at client side. Now the client will get an error about invalid chunked response. The error message is simultaneously written to the server log and in the last line of the response. This should help detecting errors when migrating data between VictoriaMetrics instances by [vmctl](https://docs.victoriametrics.com/vmctl.html). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5645).
- FEATURE: all VictoriaMetrics components: add ability to specify arbitrary HTTP headers to send with every request to `-pushmetrics.url`. See [`push metrics` docs](https://docs.victoriametrics.com/#push-metrics).
- FEATURE: all VictoriaMetrics components: add `-metrics.exposeMetadata` command-line flag, which allows displaying `TYPE` and `HELP` metadata at `/metrics` page exposed at `-httpListenAddr`. This may be needed when the `/metrics` page is scraped by collector, which requires the `TYPE` and `HELP` metadata such as [Google Cloud Managed Prometheus](https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type).
- FEATURE: all VictoriaMetrics components: add ability to dynamically re-read auth keys and passwords from files and urls when using `file:///path/to/file` or `http://host/path` syntax for the following command-line flags: `-configAuthKey`, `-deleteAuthKey`, `-flagsAuthKey`, `-forceMergeAuthKey`, `-forceFlushAuthKey`, `-httpAuth.password`, `-metricsAuthKey`, `-pprofAuthKey`, `-reloadAuthKey`, `-search.resetCacheAuthKey`, `-snapshotAuthKey`. For example, `-httpAuth.password=file:///path/to/password`. See [these docs](https://docs.victoriametrics.com/#security) for details.
- FEATURE: dashboards/cluster: add panels for detailed visualization of traffic usage between vmstorage, vminsert, vmselect components and their clients. New panels are available in the rows dedicated to specific components.
- FEATURE: dashboards/cluster: update "Slow Queries" panel to show percentage of the slow queries to the total number of read queries served by vmselect. The percentage value should make it more clear for users whether there is a service degradation.
- FEATURE: dashboards/single: change dashboard title from `VictoriaMetrics` to `VictoriaMetrics - single-node`. The new title should provide better understanding of this dashboard purpose.
- FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): rename cmd-line flag `vm-native-disable-retries` to `vm-native-disable-per-metric-migration` to better reflect its meaning.
- FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): add `-vm-native-src-insecure-skip-verify` and `-vm-native-dst-insecure-skip-verify` command-line flags for native protocol. It can be used for skipping TLS certificate verification when connecting to the source or destination addresses.
- FEATURE: [Alerting rules for VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts): add `job` label to `DiskRunsOutOfSpace` alerting rule, so it is easier to understand to which installation the triggered instance belongs.
- FEATURE: `vmstorage`: add tenant identifier for log messages regarding dropping excessive labels due to limits defined by `-maxLabelsPerTimeseries` or `-maxLabelValueLen` command-line flags. Previously, it was hard to understand to which tenant the dropped labels belong.
- FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add the ability to export and import query reports:
- add a `Query Analyzer` page that allows you to build graphs from `JSON` data containing the results of executing a query request.
- add an `Export query` button to the graph that saves the result of executing the query in `JSON`. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5497).
- FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add `-vmui.defaultTimezone` flag to set a default timezone. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5375) and [these docs](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmui#timezone-configuration).
- FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): include UTC in the timezone selection dropdown for standardized time referencing. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5375).
- FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add the ability to expand/collapse all tracing entries and download tracing data in .json format. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5677).
- FEATURE: add [VictoriaMetrics datasource](https://github.com/VictoriaMetrics/grafana-datasource) to docker compose environment. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5363).
- BUGFIX: properly return the list of matching label names and label values from [`/api/v1/labels`](https://docs.victoriametrics.com/url-examples.html#apiv1labels) and [`/api/v1/label/.../values`](https://docs.victoriametrics.com/url-examples.html#apiv1labelvalues) when the database contains more than `-search.maxUniqueTimeseries` unique [time series](https://docs.victoriametrics.com/keyConcepts.html#time-series) on the selected time range. Previously VictoriaMetrics could return `the number of matching timeseries exceeds ...` error in this case. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5055).
- BUGFIX: properly return errors from [export APIs](https://docs.victoriametrics.com/#how-to-export-time-series). Previously these errors were silently suppressed. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5649).
- BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html): properly return full results when `-search.skipSlowReplicas` command-line flag is passed to `vmselect` and when [vmstorage groups](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#vmstorage-groups-at-vmselect) are in use. Previously partial results could be returned in this case.
- BUGFIX: `vminsert`: properly accept samples via [OpenTelemetry data ingestion protocol](https://docs.victoriametrics.com/#sending-data-via-opentelemetry) when these samples have no [resource attributes](https://opentelemetry.io/docs/instrumentation/go/resources/). Previously such samples were silently skipped.
- BUGFIX: `vmstorage`: added missing `-inmemoryDataFlushInterval` command-line flag, which was missing in [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html) after implementing [this feature](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337) in [v1.85.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.0).
- BUGFIX: `vmstorage`: properly expire `storage/prefetchedMetricIDs` cache. Previously this cache was never expired, so it could grow big under [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate). This could result in increasing CPU load over time.
- BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): check `-external.url` schema when starting vmalert, must be `http` or `https`. Before, alertmanager could reject alert notifications if `-external.url` contained no or wrong schema.
- BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): automatically add `exported_` prefix for original evaluation result label if it's conflicted with external or reserved one, previously it was overridden. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5161).
- BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): autogenerate `ALERTS_FOR_STATE` time series for alerting rules with `for: 0`. Previously, `ALERTS_FOR_STATE` was generated only for alerts with `for > 0`. The change aligns with Prometheus behavior. See more details in [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5648).
- BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): fix data race during hot-config reload. The result of the race could cause all rules from updated group to fail with `context cancelled` error.
- BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): consistently sort results for `q1 or q2` query, so they do not change colors with each refresh in Grafana. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5393).
- BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): properly return results from [bottomk](https://docs.victoriametrics.com/MetricsQL.html#bottomk) and `bottomk_*()` functions when some of these results contain NaN values. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5506). Thanks to [@​xiaozongyang](https://github.com/xiaozongyang) for [the fix](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5509).
- BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): properly handle queries, which wrap [rollup functions](https://docs.victoriametrics.com/MetricsQL.html#rollup-functions) with multiple arguments without explicitly specified lookbehind window in square brackets into [aggregate functions](https://docs.victoriametrics.com/MetricsQL.html#aggregate-functions). For example, `sum(quantile_over_time(0.5, process_resident_memory_bytes))` was resulting to `expecting at least 2 args to ...; got 1 args` error. Thanks to [@​atykhyy](https://github.com/atykhyy) for [the pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5414).
- BUGFIX: [vmctl](https://docs.victoriametrics.com/vmctl.html): retry on import errors in `vm-native` mode. Before, retries happened only on writes into a network connection between source and destination. But errors returned by server after all the data was transmitted were logged, but not retried.
- BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): properly assume role with [AWS IRSA authorization](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). Previously role chaining was not supported. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3822) for details.
- BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): exit if there is config syntax error in [`scrape_config_files`](https://docs.victoriametrics.com/vmagent.html#loading-scrape-configs-from-multiple-files) when `-promscrape.config.strictParse=true`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5508).
- BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): properly discover targets for `role: endpoints` and `role: endpointslice` in [kubernetes_sd_configs](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs). Previously some `endpoints` and `endpointslice` targets could be left undiscovered or some targets could have missing `__meta_*` labels when performing service discovery in busy Kubernetes clusters with large number of pods. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5557).
- BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): respect explicitly set `series_limit: 0` in [scrape_config](https://docs.victoriametrics.com/sd_configs.html#scrape_configs). This allows removing [`series_limit` restriction](https://docs.victoriametrics.com/vmagent.html#cardinality-limiter) on a per-`scrape_config` basis when global limit is set via `-promscrape.seriesLimitPerTarget`. Previously, `0` value was ignored in favor of `-promscrape.seriesLimitPerTarget`.
- BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): do not discover scrape targets for already terminated pods and containers in [`kubernetes_sd_configs`](https://docs.victoriametrics.com/sd_configs.html#kubernetes_sd_configs). Such pods and containers cannot be scraped and cannot resurrect, so there is no sense in generating scrape targets for them.
- BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): properly follow redirects to another hostnames when `follow_redirects: true` option is set in [`scrape_configs`](https://docs.victoriametrics.com/sd_configs/#http-api-client-options) while [stream parsing mode](https://docs.victoriametrics.com/vmagent/#stream-parsing-mode) is disabled. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945) and [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4931).
- BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix a link for the statistic inaccuracy explanation in the cardinality explorer tool. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5460).
- BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix the display of autocomplete results and cache the results. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5472) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5470).
- BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): send `step` param for instant queries. The change reverts [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3896) due to reasons explained in [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3896#issuecomment-1896704401).
- BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix the `Enter` key press issue in the `Query` field when using autocomplete. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5667).
- BUGFIX: all: fix potential panic during components shutdown when [metrics push](https://docs.victoriametrics.com/#push-metrics) is configured. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5548). Thanks to [@​zhdd99](https://github.com/zhdd99) for the [pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5549).
- BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): properly process queries with too big lookbehind window such as `foo[100y]`. Previously, such queries could return empty responses even if `foo` is present in database. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5553).
- BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5571).
- BUGFIX: `vmselect`: vmsingle/vmselect returns http status 429 (TooManyRequests) instead of 503 (ServiceUnavailable) when max concurrent requests limit is reached.
- BUGFIX: `vmselect`: set proper timestamps in [instant query](https://docs.victoriametrics.com/keyconcepts/#instant-query) results if the query contains [rollup functions](https://docs.victoriametrics.com/metricsql/#rollup-functions) with lookbehind windows in square brackets exceeding 3 hours. For example, `sum(avg_over_time(up[24h]))`. This bug has been introduced in [v1.95.0](https://docs.victoriametrics.com/changelog\_2023/#v1950). See [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5659) and [this](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5730) issues.
##### How to run VictoriaMetrics
Unpack the `victoria-metrics-*.tar.gz` archive and read [these docs](https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/Single-server-VictoriaMetrics#operation).
`vmutils-*.tag.gz` archive contains the following tools:
- [vmagent](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md)
- [vmalert](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmalert/README.md)
- [vmalert-tool](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmalert-tool/README.md)
- [vmauth](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmauth/README.md)
- [vmbackup](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmbackup/README.md)
- [vmctl](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmctl/README.md)
- [vmrestore](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmrestore/README.md)
`vmutils-*-enterprise.tar.gz` archive contains the following additional [enterprise tools](https://victoriametrics.com/enterprise.html):
- [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager.html)
- [vmgateway](https://docs.victoriametrics.com/vmgateway.html)
The corresponding docker images are available [here](https://hub.docker.com/r/victoriametrics/).
Cluster version is available [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster).
</details>
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.
♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box
---
This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4xNjIuMSIsInVwZGF0ZWRJblZlciI6IjM3LjE2Mi4xIiwidGFyZ2V0QnJhbmNoIjoibWFzdGVyIn0=-->
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
This PR contains the following updates:
v1.96.0->v1.97.0Release Notes
VictoriaMetrics/VictoriaMetrics (victoriametrics/vmagent)
v1.97.0Compare Source
Released at 2024-01-30
SECURITY: upgrade base docker image (Alpine) from 3.19.0 to 3.19.1. See alpine 3.19.1 release notes.
SECURITY: upgrade Go builder from Go1.21.5 to Go1.21.6. See the list of issues addressed in Go1.21.6.
FEATURE: improve new time series registration speed on systems with high number of CPU cores. Thanks to @misutoth for the initial idea and implementation.
FEATURE: make background merge more responsive and scalable. This should help the following issues: 5190, 3425, 648.
FEATURE: graphite: add support for negative index in
groupByNodeandaliasByNodefunctions. Thanks to @rbizos for the pull request.FEATURE: vmagent: add support for discovering Hetzner Cloud and Hetzner Robot scrape targets. See this feature request and these docs.
FEATURE: vmagent: add support for DataDog v2 data ingestion protocol. See these docs and this feature request.
FEATURE: vmagent: reduce initial memory usage when scraping targets, which return huge responses (for example, kube-state-metrics in large Kubernetes cluster may return 100MB+ responses), without the need to explicitly enable stream parsing mode. See this issue.
FEATURE: vmagent: expose ability to set OAuth2 endpoint parameters per each
-remoteWrite.urlvia the command-line flag-remoteWrite.oauth2.endpointParams. See these docs. Thanks to @mhill-holoplot for the pull request.FEATURE: vmagent: add ability to set
attach_metadata.node=trueoption for all thekubernetes_sd_configsdefined at-promscrape.configvia-promscrape.kubernetes.attachNodeMetadataAllcommand-line flag. See this feature request. Thanks to @wasim-nihal for the initial implementation.FEATURE: vmagent: do not send unfinished aggregation state on shutdown or hot config reload by default, as it tend to produce unexpected anomalies with lower values. The old behavior can be restored by specifying
flush_on_shutdown: truesetting in streaming aggregation config. See more details here.FEATURE: streaming aggregation: expand
%{ENV_VAR}placeholders in config files with the corresponding environment variable values.FEATURE: vmalert: expose ability to set OAuth2 endpoint parameters via the following command-line flags:
-datasource.oauth2.endpointParamsfor-datasource.url-notifier.oauth2.endpointParamsfor-notifier.url-remoteRead.oauth2.endpointParamsfor-remoteRead.url-remoteWrite.oauth2.endpointParamsfor-remoteWrite.urlFEATURE: vmauth: add ability to proxy incoming requests to different backends based on the requested host via
src_hostsoption aturl_map. See these docs.FEATURE: vmauth: expose
vmauth_user_request_backend_errors_totalandvmauth_unauthorized_user_request_backend_errors_totalmetrics, which track the number of failed requests because of backend errors. See this feature request.FEATURE: vmauth: add an ability to specify additional labels for per-user metrics via
metric_labelssection. See this feature request.FEATURE: all VictoriaMetrics components: break HTTP client connection if an error occurs after the server at
-httpListenAddralready sent response status code. Previously such an error couldn't be detected at client side. Now the client will get an error about invalid chunked response. The error message is simultaneously written to the server log and in the last line of the response. This should help detecting errors when migrating data between VictoriaMetrics instances by vmctl. See this issue.FEATURE: all VictoriaMetrics components: add ability to specify arbitrary HTTP headers to send with every request to
-pushmetrics.url. Seepush metricsdocs.FEATURE: all VictoriaMetrics components: add
-metrics.exposeMetadatacommand-line flag, which allows displayingTYPEandHELPmetadata at/metricspage exposed at-httpListenAddr. This may be needed when the/metricspage is scraped by collector, which requires theTYPEandHELPmetadata such as Google Cloud Managed Prometheus.FEATURE: all VictoriaMetrics components: add ability to dynamically re-read auth keys and passwords from files and urls when using
file:///path/to/fileorhttp://host/pathsyntax for the following command-line flags:-configAuthKey,-deleteAuthKey,-flagsAuthKey,-forceMergeAuthKey,-forceFlushAuthKey,-httpAuth.password,-metricsAuthKey,-pprofAuthKey,-reloadAuthKey,-search.resetCacheAuthKey,-snapshotAuthKey. For example,-httpAuth.password=file:///path/to/password. See these docs for details.FEATURE: dashboards/cluster: add panels for detailed visualization of traffic usage between vmstorage, vminsert, vmselect components and their clients. New panels are available in the rows dedicated to specific components.
FEATURE: dashboards/cluster: update "Slow Queries" panel to show percentage of the slow queries to the total number of read queries served by vmselect. The percentage value should make it more clear for users whether there is a service degradation.
FEATURE: dashboards/single: change dashboard title from
VictoriaMetricstoVictoriaMetrics - single-node. The new title should provide better understanding of this dashboard purpose.FEATURE: vmctl: rename cmd-line flag
vm-native-disable-retriestovm-native-disable-per-metric-migrationto better reflect its meaning.FEATURE: vmctl: add
-vm-native-src-insecure-skip-verifyand-vm-native-dst-insecure-skip-verifycommand-line flags for native protocol. It can be used for skipping TLS certificate verification when connecting to the source or destination addresses.FEATURE: Alerting rules for VictoriaMetrics: add
joblabel toDiskRunsOutOfSpacealerting rule, so it is easier to understand to which installation the triggered instance belongs.FEATURE:
vmstorage: add tenant identifier for log messages regarding dropping excessive labels due to limits defined by-maxLabelsPerTimeseriesor-maxLabelValueLencommand-line flags. Previously, it was hard to understand to which tenant the dropped labels belong.FEATURE: vmui: add the ability to export and import query reports:
Query Analyzerpage that allows you to build graphs fromJSONdata containing the results of executing a query request.Export querybutton to the graph that saves the result of executing the query inJSON. See this pull request.FEATURE: vmui: add
-vmui.defaultTimezoneflag to set a default timezone. See this issue and these docs.FEATURE: vmui: include UTC in the timezone selection dropdown for standardized time referencing. See this issue.
FEATURE: vmui: add the ability to expand/collapse all tracing entries and download tracing data in .json format. See this issue.
FEATURE: add VictoriaMetrics datasource to docker compose environment. See this pull request.
BUGFIX: properly return the list of matching label names and label values from
/api/v1/labelsand/api/v1/label/.../valueswhen the database contains more than-search.maxUniqueTimeseriesunique time series on the selected time range. Previously VictoriaMetrics could returnthe number of matching timeseries exceeds ...error in this case. See this issue.BUGFIX: properly return errors from export APIs. Previously these errors were silently suppressed. See this pull request.
BUGFIX: VictoriaMetrics cluster: properly return full results when
-search.skipSlowReplicascommand-line flag is passed tovmselectand when vmstorage groups are in use. Previously partial results could be returned in this case.BUGFIX:
vminsert: properly accept samples via OpenTelemetry data ingestion protocol when these samples have no resource attributes. Previously such samples were silently skipped.BUGFIX:
vmstorage: added missing-inmemoryDataFlushIntervalcommand-line flag, which was missing in VictoriaMetrics cluster after implementing this feature in v1.85.0.BUGFIX:
vmstorage: properly expirestorage/prefetchedMetricIDscache. Previously this cache was never expired, so it could grow big under high churn rate. This could result in increasing CPU load over time.BUGFIX: vmalert: check
-external.urlschema when starting vmalert, must behttporhttps. Before, alertmanager could reject alert notifications if-external.urlcontained no or wrong schema.BUGFIX: vmalert: automatically add
exported_prefix for original evaluation result label if it's conflicted with external or reserved one, previously it was overridden. See this issue.BUGFIX: vmalert: autogenerate
ALERTS_FOR_STATEtime series for alerting rules withfor: 0. Previously,ALERTS_FOR_STATEwas generated only for alerts withfor > 0. The change aligns with Prometheus behavior. See more details in this issue.BUGFIX: vmalert: fix data race during hot-config reload. The result of the race could cause all rules from updated group to fail with
context cancellederror.BUGFIX: MetricsQL: consistently sort results for
q1 or q2query, so they do not change colors with each refresh in Grafana. See this issue.BUGFIX: MetricsQL: properly return results from bottomk and
bottomk_*()functions when some of these results contain NaN values. See this issue. Thanks to @xiaozongyang for the fix.BUGFIX: MetricsQL: properly handle queries, which wrap rollup functions with multiple arguments without explicitly specified lookbehind window in square brackets into aggregate functions. For example,
sum(quantile_over_time(0.5, process_resident_memory_bytes))was resulting toexpecting at least 2 args to ...; got 1 argserror. Thanks to @atykhyy for the pull request.BUGFIX: vmctl: retry on import errors in
vm-nativemode. Before, retries happened only on writes into a network connection between source and destination. But errors returned by server after all the data was transmitted were logged, but not retried.BUGFIX: vmagent: properly assume role with AWS IRSA authorization. Previously role chaining was not supported. See this issue for details.
BUGFIX: vmagent: exit if there is config syntax error in
scrape_config_fileswhen-promscrape.config.strictParse=true. See this issue.BUGFIX: vmagent: properly discover targets for
role: endpointsandrole: endpointslicein kubernetes_sd_configs. Previously someendpointsandendpointslicetargets could be left undiscovered or some targets could have missing__meta_*labels when performing service discovery in busy Kubernetes clusters with large number of pods. See this pull request.BUGFIX: vmagent: respect explicitly set
series_limit: 0in scrape_config. This allows removingseries_limitrestriction on a per-scrape_configbasis when global limit is set via-promscrape.seriesLimitPerTarget. Previously,0value was ignored in favor of-promscrape.seriesLimitPerTarget.BUGFIX: vmagent: do not discover scrape targets for already terminated pods and containers in
kubernetes_sd_configs. Such pods and containers cannot be scraped and cannot resurrect, so there is no sense in generating scrape targets for them.BUGFIX: vmagent: properly follow redirects to another hostnames when
follow_redirects: trueoption is set inscrape_configswhile stream parsing mode is disabled. See this issue and this pull request.BUGFIX: vmui: fix a link for the statistic inaccuracy explanation in the cardinality explorer tool. See this issue.
BUGFIX: vmui: fix the display of autocomplete results and cache the results. See this issue and this issue.
BUGFIX: vmui: send
stepparam for instant queries. The change reverts this issue due to reasons explained in this comment.BUGFIX: vmui: fix the
Enterkey press issue in theQueryfield when using autocomplete. See this issue.BUGFIX: all: fix potential panic during components shutdown when metrics push is configured. See this issue. Thanks to @zhdd99 for the pull request.
BUGFIX: MetricsQL: properly process queries with too big lookbehind window such as
foo[100y]. Previously, such queries could return empty responses even iffoois present in database. See this issue.BUGFIX: MetricsQL: properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See this issue.
BUGFIX:
vmselect: vmsingle/vmselect returns http status 429 (TooManyRequests) instead of 503 (ServiceUnavailable) when max concurrent requests limit is reached.BUGFIX:
vmselect: set proper timestamps in instant query results if the query contains rollup functions with lookbehind windows in square brackets exceeding 3 hours. For example,sum(avg_over_time(up[24h])). This bug has been introduced in v1.95.0. See this and this issues.How to run VictoriaMetrics
Unpack the
victoria-metrics-*.tar.gzarchive and read these docs.vmutils-*.tag.gzarchive contains the following tools:vmutils-*-enterprise.tar.gzarchive contains the following additional enterprise tools:The corresponding docker images are available here.
Cluster version is available here.
Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR has been generated by Renovate Bot.