> For the complete documentation index, see [llms.txt](https://mariadb.com/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://mariadb.com/docs/gridgain/gridgain-8/reference/monitoring/generic-metrics.md). # Generic Metrics The GridGain generic metrics system โ€” metric registers, exporters (JMX, SQL view, log, OpenCensus, OpenTelemetry), and the full catalog of available metrics. This page describes generic GridGain metrics - a new system that is designed to replace the [legacy metrics system](/docs/gridgain/gridgain-8/reference/monitoring/jmx-metrics.md). Generic metrics provide a cleaner decoupling between metric collecting/storing and viewing/exporting. This page offers the basics of the new metrics system and explains how you can use it to monitor your cluster. There are different types of metrics in GridGain. Each metric has a name and a return value. The return value can be a simple value like `string`, `long`, or `double`. Alternatively, the value can represent a Java object. Some metrics represent [Histograms](#histograms). There are different ways to export metrics - so called [Metric Exporters](#metric-exporters). ## Metric Registers Metrics are grouped into categories called *registers*. Each register has a name. The full name of a specific metric within the register consists of the register name followed by a dot, followed by the name of the metric: `.`. For example, the register for data storage metrics is called `io.datastorage`. The metric that return the storage size is called `io.datastorage.StorageSize`. ## Metric Exporters An exporter provides a mechanism for accessing all the available metrics. GridGain includes the following exporters: * [JMX](#jmx) (default) * [SQL View](#sql-view) * [Log](#log) * [OpenCensus](#opencensus) * [OpenTelemetry](#opentelemetry) You can create a custom exporter by implementing the [MetricExporterSpi](https://www.gridgain.com/sdk/latest/javadoc/org/apache/ignite/spi/metric/MetricExporterSpi.html) interface. If you want to enable metrics, configure one or multiple metric exporters in the node configuration. This is a node-specific configuration, which means it enables metrics only on the node where it is specified. {% tabs %} {% tab title="XML" %} ```xml ``` {% endtab %} {% tab title="Java" %} ```java IgniteConfiguration cfg = new IgniteConfiguration(); cfg.setMetricExporterSpi(new JmxMetricExporterSpi()); Ignite ignite = Ignition.start(cfg); ``` {% endtab %} {% tab title="C#/.NET" %} unsupported {% endtab %} {% tab title="C++" %} unsupported {% endtab %} {% endtabs %} The following sections describe the exporters available in Ignite by default. ### JMX `org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi` exposes metrics via JMX beans. {% tabs %} {% tab title="Java" %} ```java IgniteConfiguration cfg = new IgniteConfiguration(); JmxMetricExporterSpi jmxExporter = new JmxMetricExporterSpi(); //export cache metrics only jmxExporter.setExportFilter(mreg -> mreg.name().startsWith("cache.")); cfg.setMetricExporterSpi(jmxExporter); ``` {% endtab %} {% tab title="C++" %} unsupported {% endtab %} {% endtabs %} ### SQL View `SqlViewMetricExporterSpi` is enabled by default, `SqlViewMetricExporterSpi` exposes metrics via the `SYS.METRICS` view. Each metric is displayed as a single record. You can use any supported SQL tool to view the metrics: ```shell > select name, value from SYS.METRICS where name LIKE 'cache.myCache.%'; +-----------------------------------+--------------------------------+ | NAME | VALUE | +-----------------------------------+--------------------------------+ | cache.myCache.CacheTxRollbacks | 0 | | cache.myCache.OffHeapRemovals | 0 | | cache.myCache.QueryCompleted | 0 | | cache.myCache.QueryFailed | 0 | | cache.myCache.EstimatedRebalancingKeys | 0 | | cache.myCache.CacheEvictions | 0 | | cache.myCache.CommitTime | [J@2eb66498 | .... ``` ### Log `org.apache.ignite.spi.metric.log.LogExporterSpi` prints the metrics to the log file at regular intervals (1 min by default) at INFO level. {% tabs %} {% tab title="XML" %} ```xml ``` {% endtab %} {% tab title="Java" %} If you use programmatic configuration, you can change the print frequency as follows: ```java IgniteConfiguration cfg = new IgniteConfiguration(); LogExporterSpi logExporter = new LogExporterSpi(); logExporter.setPeriod(600_000); //export cache metrics only logExporter.setExportFilter(mreg -> mreg.name().startsWith("cache.")); cfg.setMetricExporterSpi(logExporter); Ignite ignite = Ignition.start(cfg); ``` {% endtab %} {% tab title="C#/.NET" %} unsupported {% endtab %} {% tab title="C++" %} unsupported {% endtab %} {% endtabs %} ### OpenCensus `org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi` adds integration with the OpenCensus library. To use the OpenCensus exporter: 1. [Enable the 'ignite-opencensus' module](/docs/gridgain/gridgain-8/gridgain8-development/project-setup.md#enabling-modules). 2. Add `org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi` to the list of exporters in the node configuration. 3. Configure OpenCensus StatsCollector to export to a specific system. See [OpenCensusMetricsExporterExample.java](https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/opencensus/OpenCensusMetricsExporterExample.java) for an example and OpenCensus documentation for additional information. Configuration parameters: * `filter` - predicate that filters metrics. * `period` - export period. * `sendInstanceName` - if enabled, a tag with the Ignite instance name is added to each metric. * `sendNodeId` - if enabled, a tag with the Ignite node id is added to each metric. * `sendConsistentId` - if enabled, a tag with the Ignite node consistent id is added to each metric. ### OpenTelemetry `org.apache.ignite.spi.metric.otlp.OpenTelemetryMetricExporterSpi` adds integration with [OpenTelemetry Protocol (OTLP)](https://opentelemetry.io/docs/specs/otlp/). When enabled, the exporter periodically sends metrics to the configured Open Telemetry exporter. To use the OpenTelemetry exporter: * [Enable the 'ignite-opentelemetry' module](/docs/gridgain/gridgain-8/gridgain8-development/project-setup.md#enabling-modules). * Add `org.apache.ignite.spi.metric.otlp.OpenTelemetryMetricExporterSpi` to the list of exporters in the node configuration. * Configure OpenTelemetry exported to send metrics to your endpoint. The example below shows a simple configuration: {% tabs %} {% tab title="XML" %} ```xml ``` {% endtab %} {% tab title="Java" %} ```java IgniteConfiguration cfg = new IgniteConfiguration(); OpenTelemetryMetricExporterSpi otelExporter = new OpenTelemetryMetricExporterSpi(); otelExporter.setEndpoint("http://my-endpoint:4317"); otelExporter.setProtocol(Protocol.GRPC); otelExporter.setCompression(Compression.GZIP); otelExporter.setPeriod(30_000L); otelExporter.setServiceNamespace("my-org"); otelExporter.setServiceName("ignite-cluster"); //export cache metrics only otelExporter.setExportFilter(mreg -> mreg.name().startsWith("cache.")); cfg.setMetricExporterSpi(otelExporter); ``` {% endtab %} {% tab title="C#/.NET" %} unsupported {% endtab %} {% tab title="C++" %} unsupported {% endtab %} {% endtabs %} Each node identifies itself by using the following [OpenTelemetry service resource attributes](https://opentelemetry.io/docs/specs/semconv/resource/service): * `service.namespace` - set via `serviceNamespace`. * `service.name` - set via `serviceName`. Defaults to the cluster tag if not specified. * `service.instance.id` - automatically set to the node's consistent ID. The following configuration parameters are available: * `endpoint` - OTLP endpoint to connect to. Default value is `http://localhost:4317`. * `protocol` - the protocol to use. Supported values are `GRPC` (default) and `HTTP`. * `compression` - payload compression type. Supported values are `NONE` (default) and `GZIP`. * `period` - export period in milliseconds. * `filter` - predicate that filters metric registries. Can only be set programmatically via `setExportFilter()`. The filter operates at the register level โ€” an entire register is either exported or skipped. For example: ```java // Export only cache metrics: otelExporter.setExportFilter(mreg -> mreg.name().startsWith("cache.")); // Export everything except SQL metrics: otelExporter.setExportFilter(mreg -> !mreg.name().startsWith("sql.")); ``` * `serviceNamespace` - logical namespace for the service name. * `serviceName` - logical name of the service. Defaults to the cluster tag. * `connectionHeaders` - arbitrary HTTP/gRPC metadata headers (e.g. authentication tokens) added to every export request. Can only be set programmatically via `setConnectionHeaders(Map)`. * `sslEnabled` - enables SSL/TLS for the connection. Default is `false`. * `useIgniteSslContextFactory` - if `true`, reuses the `SSLContext` factory from `IgniteConfiguration`. Default is `true`. * `sslContextFactory` - a custom `Factory` for secure connections. An instance of `SslContextFactory` is recommended as it provides both the `SSLContext` and the `TrustManager`. * `trustManagerFactory` - a custom `Factory`. Required when using a custom `sslContextFactory` that is not an instance of `SslContextFactory`. ## Available Metrics ### System System metrics such as JVM or CPU metrics. Register name: `sys` | Name | Type | Description | | ----------------------------------- | ---------------- | -------------------------------------------------------------------------------------------------------------- | | CpuLoad | double | CPU load. | | CurrentThreadCpuTime | long | The total CPU time for the current thread in nanoseconds. | | CurrentThreadUserTime | long | The CPU time that the current thread has executed in user mode in nanoseconds. | | DaemonThreadCount | integer | The current number of live daemon threads. | | GcCpuLoad | double | GC CPU load. | | PeakThreadCount | integer | The peak live thread count since the Java virtual machine started or peak was reset. | | SystemLoadAverage | java.lang.Double | The system load average for the last minute. | | ThreadCount | integer | The current number of live threads including both daemon and non-daemon threads. | | TotalExecutedTasks | long | Total executed tasks. | | TotalStartedThreadCount | long | The total number of threads created and also started since the JVM. | | UpTime | long | The uptime of the Java virtual machine in milliseconds. | | memory.heap.committed | long | The amount of memory in bytes that is committed for the JVM to use. | | memory.heap.init | long | The amount of memory in bytes that the JVM initially requests from the operating system for memory management. | | memory.heap.used | long | The amount of used memory in bytes. | | memory.nonheap.committed | long | The amount of memory in bytes that is committed for the JVM to use. | | memory.nonheap.init | long | The amount of memory in bytes that the JVM initially requests from the operating system for memory management. | | memory.nonheap.max | long | The maximum amount of memory in bytes that can be used for memory management. | | memory.nonheap.used | long | The amount of used memory in bytes. | | memory.direct.buffers.max | long | The maximum amount of memory that can be reserved for all direct byte buffers. | | memory.direct.buffers.totalCapacity | long | The estimated total capacity of all buffers in the direct memory pool, in bytes. | | memory.direct.buffers.used | long | The estimated amount of memory, in bytes, currently used by the direct memory pool. | ### Caches Cache metrics. Register name: `cache.{cache_name}.{near}` {% hint style="info" %} Metrics such as `QueryCompleted`, `QueryExecuted`, `QueryFailed`, `QueryMaximumTime`, `QueryMinimalTime`, and `QuerySumTime` track operations performed via the `IgniteCache.query()` API, including scan, continuous, and text queries. However, they do not cover SQL queries executed through JDBC. {% endhint %} | Name | Type | Description | | --------------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CacheEvictions | long | The total number of evictions from the cache. | | CacheGets | long | The total number of gets to the cache. | | CacheHits | long | The number of get requests that were satisfied by the cache. | | CacheMisses | long | A miss is a get request that is not satisfied. | | CachePuts | long | The total number of puts to the cache. | | CacheRemovals | long | The total number of removals from the cache. | | CacheTxCommits | long | Total number of transaction commits. | | CacheTxRollbacks | long | Total number of transaction rollbacks. | | CacheSize | long | Local cache size. | | CommitTime | histogram | Commit time in nanoseconds. | | CommitTimeTotal | long | The total time of commit, in nanoseconds. | | EntryProcessorHits | long | The total number of invocations on keys, which exist in cache. | | EntryProcessorInvokeTimeNanos | long | The total time of cache invocations for which this node is the initiator, in nanoseconds. | | EntryProcessorMaxInvocationTime | long | So far, the maximum time to execute cache invokes for which this node is the initiator. | | EntryProcessorMinInvocationTime | long | So far, the minimum time to execute cache invokes for which this node is the initiator. | | EntryProcessorMisses | long | The total number of invocations on keys, which don't exist in cache. | | EntryProcessorPuts | long | The total number of cache invocations, caused update. | | EntryProcessorReadOnlyInvocations | long | The total number of cache invocations, caused no updates. | | EntryProcessorRemovals | long | The total number of cache invocations, caused removals. | | EstimatedRebalancingKeys | long | Number estimated to rebalance keys. | | GetAllTime | histogram | GetAll time for which this node is the initiator, in nanoseconds. | | GetTime | histogram | Get time for which this node is the initiator, in nanoseconds. | | GetTimeTotal | long | The total time of cache gets for which this node is the initiator, in nanoseconds. | | HeapEntriesCount | long | Onheap entries count. | | IndexBuildPartitionsLeftCount | int | The number of local node partitions that remain to be processed to complete indexing. | | IndexRebuildKeyProcessed | long | The number of keys with rebuilt indexes. | | IsIndexRebuildInProgress | boolean | True if index build or rebuild is in progress. | | OffHeapBackupEntriesCount | long | Offheap backup entries count. | | OffHeapEntriesCount | long | Offheap entries count. | | OffHeapEvictions | long | The total number of evictions from the off-heap memory. | | OffHeapGets | long | The total number of get requests to the off-heap memory. | | OffHeapHits | long | The number of get requests that were satisfied by the off-heap memory. | | OffHeapMisses | long | A miss is a get request that is not satisfied by off-heap memory. | | OffHeapPrimaryEntriesCount | long | Offheap primary entries count. | | OffHeapPuts | long | The total number of put requests to the off-heap memory. | | OffHeapRemovals | long | The total number of removals from the off-heap memory. | | PutAllTime | histogram | PutAll time for which this node is the initiator, in nanoseconds. | | PutTime | histogram | Put time for which this node is the initiator, in nanoseconds. | | PutTimeTotal | long | The total time of cache puts for which this node is the initiator, in nanoseconds. | | QueryCompleted | long | Count of completed queries. | | QueryExecuted | long | Count of executed queries. | | QueryFailed | long | Count of failed queries. | | QueryMaximumTime | long | Maximum query execution time. | | QueryMinimalTime | long | Minimum query execution time. | | QuerySumTime | long | Query summary time. | | RebalanceClearingPartitionsLeft | long | Number of partitions need to be cleared before actual rebalance start. | | RebalanceStartTime | long | Rebalance start time. | | RebalancedKeys | long | Number of already rebalanced keys. | | RebalancingBytesRate | long | Estimated rebalancing speed in bytes. | | RebalancingKeysRate | long | Estimated rebalancing speed in keys. | | RemoveAllTime | histogram | RemoveAll time for which this node is the initiator, in nanoseconds. | | RemoveTime | histogram | Remove time for which this node is the initiator, in nanoseconds. | | RemoveTimeTotal | long | The total time of cache removal for which this node is the initiator, in nanoseconds. | | RollbackTime | histogram | Rollback time in nanoseconds. | | RollbackTimeTotal | long | The total time of rollback, in nanoseconds. | | TotalRebalancedBytes | long | Number of already rebalanced bytes. | | getCacheTouches | long | The total number of `touch()` requests to the cache. Equal to the sum of hits and misses. | | getCacheTouchHits | long | The number of `touch()` requests that were satisfied by the cache, i.e., "hits." | | getCacheTouchMisses | long | The number of `touch()` requests that were not satisfied by the cache, i.e., "misses," either because the requested key was not found in the cache or TTL value was not changed. | | getCacheTouchHitPercentage | float | The percentage of cache `touch()` requests that were satisfied by the cache. Calculated as `getCacheTouchHits` divided by `getCacheTouches` multiplied by 100. | | getCacheTouchMissPercentage | float | The percentage of cache `touch()` requests that were not satisfied by the cache. Calculated as `getCacheTouchMisses` divided by `getCacheTouches` multiplied by 100. | ### Cache Groups Register name: `cacheGroups.{group_name}` | Name | Type | Description | | ------------------------------- | ------------------- | ---------------------------------------------------------------------------------------------------- | | AffinityPartitionsAssignmentMap | java.util.Map | Affinity partitions assignment map. | | Caches | java.util.ArrayList | List of caches | | IndexBuildCountPartitionsLeft | long | Number of partitions need processed for finished indexes create or rebuilding. | | LocalNodeMovingPartitionsCount | integer | Count of partitions with state `MOVING` for this cache group located on this node. | | LocalNodeOwningPartitionsCount | integer | Count of partitions with state `OWNING` for this cache group located on this node. | | LocalNodeRentingEntriesCount | long | Count of entries remains to evict in `RENTING` partitions located on this node for this cache group. | | LocalNodeRentingPartitionsCount | integer | Count of partitions with state `RENTING` for this cache group located on this node. | | MaximumNumberOfPartitionCopies | integer | Maximum number of partition copies for all partitions of this cache group. | | MinimumNumberOfPartitionCopies | integer | Minimum number of partition copies for all partitions of this cache group. | | MovingPartitionsAllocationMap | java.util.Map | Allocation map of partitions with state `MOVING` in the cluster. | | OwningPartitionsAllocationMap | java.util.Map | Allocation map of partitions with state `OWNING` in the cluster. | | PartitionIds | java.util.ArrayList | Local partition ids. | | SparseStorageSize | long | Storage space allocated for group adjusted for possible sparsity, in bytes. | | StorageSize | long | Storage space allocated for group, in bytes. | | TotalAllocatedPages | long | Cache group total allocated pages. | | TotalAllocatedSize | long | Total size of memory allocated for group, in bytes. | ### Transactions Transaction metrics. Register name: `tx` | Name | Type | Description | | ----------------------------- | ----------------- | ----------------------------------------------------------------------- | | AllOwnerTransactions | java.util.HashMap | Map of local node owning transactions. | | LockedKeysNumber | long | The number of keys locked on the node. | | OwnerTransactionsNumber | long | The number of active transactions for which this node is the initiator. | | TransactionsHoldingLockNumber | long | The number of active transactions holding at least one key lock. | | LastCommitTime | long | Last commit time. | | nodeSystemTimeHistogram | histogram | Transactions system times on node represented as histogram. | | nodeUserTimeHistogram | histogram | Transactions user times on node represented as histogram. | | LastRollbackTime | long | Last rollback time. | | totalNodeSystemTime | long | Total transactions system time on node. | | totalNodeUserTime | long | Total transactions user time on node. | | txCommits | integer | Number of transaction commits. | | txRollbacks | integer | Number of transaction rollbacks. | | txDeadlocks | integer | Number of transaction deadlocks. | ### Partition Map Exchange Partition map exchange metrics. Register name: `pme` | Name | Type | Description | | --------------------------------------- | --------- | -------------------------------------------------------------------- | | CacheOperationsBlockedDuration | long | Current PME cache operations blocked duration in milliseconds. | | CacheOperationsBlockedDurationHistogram | histogram | Histogram of cache operations blocked PME durations in milliseconds. | | Duration | long | Current PME duration in milliseconds. | | DurationHistogram | histogram | Histogram of PME durations in milliseconds. | ### Compute Jobs Register name: `compute.jobs` | Name | Type | Description | | -------------------------- | ---- | ------------------------------------------------------------------------- | | compute.jobs.Active | long | Number of active jobs currently executing. | | compute.jobs.Canceled | long | Number of cancelled jobs that are still running. | | compute.jobs.ExecutionTime | long | Total execution time of jobs. | | compute.jobs.Finished | long | Number of finished jobs. | | compute.jobs.Rejected | long | Number of jobs rejected after more recent collision resolution operation. | | compute.jobs.Started | long | Number of started jobs. | | compute.jobs.Waiting | long | Number of currently queued jobs waiting to be executed. | | compute.jobs.WaitingTime | long | Total time jobs spent on waiting queue. | ### Thread Pools Register name: `threadPools.{thread_pool_name}` | Name | Type | Description | | ----------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | ActiveCount | long | Approximate number of threads that are actively executing tasks. | | CompletedTaskCount | long | Approximate total number of tasks that have completed execution. | | CorePoolSize | long | The core number of threads. | | KeepAliveTime | long | Thread keep-alive time, which is the amount of time which threads in excess of the core pool size may remain idle before being terminated. | | LargestPoolSize | long | Largest number of threads that have ever simultaneously been in the pool. | | MaximumPoolSize | long | The maximum allowed number of threads. | | PoolSize | long | Current number of threads in the pool. | | QueueSize | long | Current size of the execution queue. | | RejectedExecutionHandlerClass | string | Class name of current rejection handler. | | Shutdown | boolean | True if this executor has been shut down. | | TaskCount | long | Approximate total number of tasks that have been scheduled for execution. | | Terminated | boolean | True if all tasks have completed following shut down. | | Terminating | long | True if terminating but not yet terminated. | | ThreadFactoryClass | string | Class name of thread factory used to create new threads. | ### Cache Group IO Register name: `io.statistics.cacheGroups.{group_name}` | Name | Type | Description | | --------------- | ------- | ---------------------------------------- | | LOGICAL\_READS | long | Number of logical reads | | PHYSICAL\_READS | long | Number of physical reads | | grpId | integer | Group id | | insertedBytes | long | Number of bytes inserted into the store. | | name | string | Name of the index | | removedBytes | long | Number of bytes removed from the store. | | startTime | long | Statistics collect start time | ### Sorted Indexes Register name: `io.statistics.sortedIndexes.{cache_name}.{index_name}` | Name | Type | Description | | ---------------------- | ------ | -------------------------------------------- | | LOGICAL\_READS\_INNER | long | Number of logical reads for inner tree node | | LOGICAL\_READS\_LEAF | long | Number of logical reads for leaf tree node | | PHYSICAL\_READS\_INNER | long | Number of physical reads for inner tree node | | PHYSICAL\_READS\_LEAF | long | Number of physical reads for leaf tree node | | indexName | string | Name of the index | | name | string | Name of the cache | | startTime | long | Statistics collection start time | ### Hash Indexes Register name: `io.statistics.hashIndexes.{cache_name}.{index_name}` | Name | Type | Description | | ---------------------- | ------ | -------------------------------------------- | | LOGICAL\_READS\_INNER | long | Number of logical reads for inner tree node | | LOGICAL\_READS\_LEAF | long | Number of logical reads for leaf tree node | | PHYSICAL\_READS\_INNER | long | Number of physical reads for inner tree node | | PHYSICAL\_READS\_LEAF | long | Number of physical reads for leaf tree node | | indexName | string | Name of the index | | name | string | Name of the cache | | startTime | long | Statistics collection start time | ### Index Operations Register name: `index.{schema_name}.{table_name}.{index_name}` GridGain creates one register per SQL secondary index, containing a pair of metrics for each type of B+tree page operation the index performs. In the metric names below, `{operation}` is one of `Search`, `Insert`, `Replace`, `RemoveFromLeaf`, `AskNeighbor`, `LockTail`, `LockTailExact`, `LockTailForward`, `LockBackAndTail`, or `LockBackAndRmvFromLeaf`. | Name | Type | Description | | ------------------ | ---- | ---------------------------------------------------------- | | `{operation}Count` | long | Number of `{operation}` operations | | `{operation}Time` | long | Total duration of `{operation}` operations, in nanoseconds | These metrics are collected only while statistics are enabled for the cache the index belongs to. If you disable statistics, the metrics stop accumulating and retain their current values; enabling statistics again resumes collection. See [Disabling Cache Metrics](/docs/gridgain/gridgain-8/gridgain8-management/monitoring/configuring-metrics.md#disabling-cache-metrics). The register is removed when the index or its cache is dropped. To stop collecting these metrics altogether, see [Disabling Index Operation Metrics](/docs/gridgain/gridgain-8/gridgain8-management/monitoring/configuring-metrics.md#disabling-index-operation-metrics). ### Communication IO Register name: `io.communication` | Name | Type | Description | | ----------------------------- | --------- | -------------------------------------------------------------------------------------------------------- | | ActiveSessionsCount | integer | Active TCP sessions count. | | OutboundMessagesQueueSize | integer | Outbound messages queue size. | | SentMessagesCount | integer | Sent messages count. | | SentBytesCount | long | Sent bytes count. | | ReceivedBytesCount | long | Received bytes count. | | ReceivedMessagesCount | integer | Received messages count. | | RejectedSslSessionsCount | integer | TCP sessions count that were rejected due to the SSL errors (metric is exported only if SSL is enabled). | | SslEnabled | boolean | Indicates whether SSL is enabled. | | SslHandshakeDurationHistogram | histogram | Histogram of SSL handshake duration in milliseconds (metric is exported only if SSL is enabled). | ### TCP Communication Register name: `communication.tcp` | Name | Type | Description | | ---------------------------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | maxOutboundMessagesQueueSize | long | Maximum number of messages that were waiting to be sent to a remote node over the last time interval. The default interval is 60 seconds. See [Configuring Maximum-Value Metrics](/docs/gridgain/gridgain-8/gridgain8-management/monitoring/configuring-metrics.md#configuring-maximum-value-metrics). | A growing queue can point to a failed node or a network problem. To get a warning in the log when the outbound queue for a node exceeds a threshold, set the `IGNITE_TCP_COMM_MSG_QUEUE_WARN_SIZE` system property to the maximum acceptable queue size. The default value is `0`, which disables the warning. The warning is printed at most once every 30 seconds. ### TCP Communication Connection Pool GridGain keeps a pool of TCP connections to each remote node. The pool is described by one shared register, which contains common parameters, and one register containing the values โ€‹โ€‹of local metrics of a specific pool of a cluster node. Register name: `communication.tcp.connectionPool` | Name | Type | Description | | ----------------- | ------- | ------------------------------------------------------------------------------------------------------------------ | | maxConnectionsCnt | integer | The maximum number of connections to a remote node. This is the value of `TcpCommunicationSpi.connectionsPerNode`. | | isPaired | boolean | Indicates whether paired connections are used (`TcpCommunicationSpi.usePairedConnections`). | | isAsync | boolean | Indicates whether connections put data in a queue instead of sending it immediately. | Register name: `communication.tcp.connectionPool.{node_id}`, where `{node_id}` is the ID of the remote node. The register is created when a connection to the node is first set up, and removed when the node leaves the cluster. | Name | Type | Description | | ------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | consistentId | string | The consistent ID of the remote node. | | currentConnectionsCnt | integer | The number of current connections to the remote node. | | outboundMessagesQueueSize | integer | The overall number of messages waiting to be sent to the remote node, across all connections to it. | | maxNetworkIdleTime | long | The maximum idle time of physical sending or receiving data, in milliseconds. The value is the highest idle time observed since the register was created; it does not decrease. | | avgConnectionLifetime | long | The average connection lifetime, in milliseconds. | | removedConnectionsCnt | long | The total number of removed connections. | | acquiringThreadsCnt | integer | The number of threads that are currently acquiring a connection. | ### Ignite Thin Client Connector Register name: `client.connector` | Name | Type | Description | | ------------------------------------ | --------- | -------------------------------------------------------------------------------------------------------- | | ActiveSessionsCount | integer | Active TCP sessions count. | | AcceptedSessionsCount | integer | Total number of TCP sessions accepted since server startup. | | AffinityKeyRequestsHits | long | The number of affinity-aware cache key requests that were sent to the primary node. | | AffinityKeyRequestsMisses | long | The number of affinity-aware cache key requests that missed the primary node. | | AffinityQueryRequestsHits | long | The number of affinity-aware query requests that were sent to the primary node. | | AffinityQueryRequestsMisses | long | The number of affinity-aware query requests that missed the primary node. | | ReceivedBytesCount | long | Received bytes count. | | RejectedSslSessionsCount | integer | TCP sessions count that were rejected due to the SSL errors (metric is exported only if SSL is enabled). | | RejectedSessionsTimeout | integer | TCP sessions count that were rejected due to handshake timeout. | | RejectedSessionsAuthenticationFailed | integer | TCP sessions count that were rejected due to failed authentication. | | RejectedSessionsTotal | integer | Total number of rejected TCP connections. | | `{clientType}.AcceptedSessions` | integer | Number of successfully established sessions for the client type. | | `{clientType}.ActiveSessions` | integer | Number of active sessions for the client type. | | SentBytesCount | long | Sent bytes count. | | SslEnabled | boolean | Indicates whether SSL is enabled. | | SslHandshakeDurationHistogram | histogram | Histogram of SSL handshake duration in milliseconds (metric is exported only if SSL is enabled). | ### Ignite REST Client Connector Register name: `rest.client` | Name | Type | Description | | ----------------------------- | --------- | -------------------------------------------------------------------------------------------------------- | | ActiveSessionsCount | integer | Active TCP sessions count. | | ReceivedBytesCount | long | Received bytes count. | | RejectedSslSessionsCount | integer | TCP sessions count that were rejected due to the SSL errors (metric is exported only if SSL is enabled). | | SentBytesCount | long | Sent bytes count. | | SslEnabled | boolean | Indicates whether SSL is enabled. | | SslHandshakeDurationHistogram | histogram | Histogram of SSL handshake duration in milliseconds (metric is exported only if SSL is enabled). | ### Discovery IO Register name: `io.discovery` | Name | Type | Description | | --------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CoordinatorSince | long | Timestamp since which the local node became the coordinator (metric is exported only from server nodes). | | Coordinator | UUID | Coordinator ID (metric is exported only from server nodes). | | CurrentTopologyVersion | long | Current topology version. | | ClientRouterNodeId | string | ID of the server node that the local client node uses as its router (metric is exported only from client nodes). | | JoinedNodes | integer | Joined nodes count. | | LeftNodes | integer | Left nodes count. | | MaxMsgQueueSize | long | Maximum size of the discovery message queue over the last time interval (metric is exported only from server nodes). The default interval is 60 seconds. See [Configuring Maximum-Value Metrics](/docs/gridgain/gridgain-8/gridgain8-management/monitoring/configuring-metrics.md#configuring-maximum-value-metrics). | | MessageWorkerQueueSize | integer | Current message worker queue size. | | Next | UUID | ID of the next node in the discovery ring (metric is exported only from server nodes). | | PendingMessagesRegistered | integer | Pending registered messages count. | | RejectedSslConnectionsCount | integer | TCP discovery connections count that were rejected due to the SSL errors. | | SslEnabled | boolean | Indicates whether SSL is enabled. | | TotalProcessedMessages | integer | Total processed messages count. | | TotalReceivedMessages | integer | Total received messages count. | ### Data Region IO Register name: `io.dataregion.{data_region_name}` | Name | Type | Description | | ------------------------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | AllocationRate | long | Allocation rate (pages per second) averaged across rateTimeInternal. | | CheckpointBufferSize | long | Checkpoint buffer size in bytes. | | DirtyPages | long | Number of pages in memory not yet synchronized with persistent storage. | | EmptyDataPages | long | Calculates empty data pages count for region. It counts only totally free pages that can be reused (e. g. pages that are contained in reuse bucket of free list). | | EvictionRate | long | Eviction rate (pages per second). | | EvictionsStarted | boolean | True if page eviction was triggered due to data region memory pressure. | | LargeEntriesPagesCount | long | Count of pages that fully occupied by large entries that go beyond page size | | OffHeapSize | long | Offheap size in bytes. | | OffheapUsedSize | long | Offheap used size in bytes. | | PagesFillFactor | double | The percentage of the used space. | | PagesRead | long | Number of pages read from last restart. | | PagesReadTime | long | Total time spent reading pages from persistent storage since last restart (nanoseconds). | | PagesReplaceAge | long | Average age at which pages in memory are replaced with pages from persistent storage (milliseconds). | | PagesReplaceRate | long | Rate at which pages in memory are replaced with pages from persistent storage (pages per second). | | PagesReplaceTime | long | Total time spent replacing pages in memory with pages from persistent storage since last restart (nanoseconds). | | PagesReplaced | long | Number of pages replaced from last restart. | | PagesWritten | long | Number of pages written from last restart. | | PhysicalMemoryPages | long | Number of pages residing in physical RAM. | | PhysicalMemorySize | long | Total size of pages loaded to the RAM, in bytes. | | SizeUsedByData | long | The number of bytes, occupied by data. Similar to `TotalUsedSize`, but it also takes into account the empty space in non-empty pages. | | TotalUsedSize | long | Gets an amount of bytes, occupied by non-empty pages allocated in the data region | | TotalUsedPages | long | Gets an amount of non-empty pages allocated in the data region. | | TotalAllocatedPages | long | Total number of allocated pages. | | TotalAllocatedSize | long | Gets a total size of memory allocated in the data region, in bytes. | | TotalThrottlingTime | long | Total throttling threads time in milliseconds. The Ignite throttles threads that generate dirty pages during the ongoing checkpoint. | | UsedCheckpointBufferSize | long | Gets used checkpoint buffer size in bytes | ### Data Storage Data Storage metrics. Register name: `io.datastorage` | Name | Type | Description | | --------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------- | | CheckpointBeforeLockHistogram | histogram | Histogram of checkpoint action before taken write lock duration in milliseconds. | | CheckpointFsyncHistogram | histogram | Histogram of checkpoint fsync duration in milliseconds. | | CheckpointHistogram | histogram | Histogram of checkpoint duration in milliseconds. | | CheckpointListenersExecuteHistogram | histogram | Histogram of checkpoint execution listeners under write lock duration in milliseconds. | | CheckpointLockHoldHistogram | histogram | Histogram of checkpoint lock hold duration in milliseconds. | | CheckpointLockWaitHistogram | histogram | Histogram of checkpoint lock wait duration in milliseconds. | | CheckpointMarkHistogram | histogram | Histogram of checkpoint mark duration in milliseconds. | | CheckpointPagesWriteHistogram | histogram | Histogram of checkpoint pages write duration in milliseconds. | | CheckpointSplitAndSortPagesHistogram | histogram | Histogram of splitting and sorting checkpoint pages duration in milliseconds. | | CheckpointTotalTime | long | Total duration of checkpoint | | CheckpointWalRecordFsyncHistogram | histogram | Histogram of the WAL fsync after logging CheckpointRecord on begin of checkpoint duration in milliseconds. | | CheckpointWriteEntryHistogram | histogram | Histogram of entry buffer writing to file duration in milliseconds. | | LastCheckpointBeforeLockDuration | long | Duration of the checkpoint action before taken write lock in milliseconds. | | LastCheckpointCopiedOnWritePagesNumber | long | Number of pages copied to a temporary checkpoint buffer during the last checkpoint. | | LastCheckpointDataPagesNumber | long | Total number of data pages written during the last checkpoint. | | LastCheckpointDuration | long | Duration of the last checkpoint in milliseconds. | | LastCheckpointFsyncDuration | long | Duration of the sync phase of the last checkpoint in milliseconds. | | LastCheckpointListenersExecuteDuration | long | Duration of the checkpoint execution listeners under write lock in milliseconds. | | LastCheckpointLockHoldDuration | long | Duration of the checkpoint lock hold in milliseconds. | | LastCheckpointLockWaitDuration | long | Duration of the checkpoint lock wait in milliseconds. | | LastCheckpointMarkDuration | long | Duration of the checkpoint mark in milliseconds. | | LastCheckpointPagesWriteDuration | long | Duration of the checkpoint pages write in milliseconds. | | LastCheckpointTotalPagesNumber | long | Total number of pages written during the last checkpoint. | | LastCheckpointSplitAndSortPagesDuration | long | Duration of splitting and sorting checkpoint pages of the last checkpoint in milliseconds. | | LastCheckpointStart | long | Start timestamp of the last checkpoint. | | LastCheckpointWalRecordFsyncDuration | long | Duration of the WAL fsync after logging CheckpointRecord on the start of the last checkpoint in milliseconds. | | LastCheckpointWriteEntryDuration | long | Duration of entry buffer writing to file of the last checkpoint in milliseconds. | | SparseStorageSize | long | Storage space allocated adjusted for possible sparsity, in bytes. | | StorageSize | long | Storage space allocated, in bytes. | | WalArchiveSegments | integer | Current number of WAL segments in the WAL archive. | | WalBuffPollSpinsRate | long | WAL buffer poll spins number over the last time interval. | | WalFsyncTimeDuration | long | Total duration of fsync | | WalFsyncTimeNum | long | Total count of fsync | | WalLastRollOverTime | long | Time of the last WAL segment rollover. | | WalLoggingRate | long | Average number of WAL records per second written during the last time interval. | | WalTotalSize | long | Total size in bytes for storage wal files. | | WalWritingRate | long | Average number of bytes per second written during the last time interval. | ### Defragmentation Defragmentation monitoring and management metrics. Register name: `defragmentation` | Name | Type | Description | | ------------------- | ------- | ---------------------------------------------------------------------- | | inProgress | boolean | True if the defragmentation is currently in progress, false otherwise. | | processedPartitions | integer | Number of partitions that have been processed during defragmentation. | | totalPartitions | integer | Total number of partitions to be defragmented. | | startTime | long | Timestamp when the current defragmentation process started. | ### Snapshots {% hint style="info" %} Snapshot metrics are available in GridGain Ultimate Edition only. {% endhint %} GridGain keeps one register for each type of snapshot operation. Each register describes the last operation of that type requested on the local node. The registers appear when the cluster is activated, and the metrics keep their final values after the operation finishes. Until an operation of that type runs, `SnapshotId`, `StartTime`, `EndTime`, and `TotalPages` report `-1`. {% hint style="info" %} Starting with GridGain 8.10, the snapshot metrics are available as generic metrics. `SnapshotMetricsMXBean` is deprecated; use the [JMX exporter](#jmx) to expose the registers as JMX beans. {% endhint %} Register names: `snapshot` (create), `snapshot-restore`, `snapshot-check`. **Create (`snapshot`)** | Name | Type | Description | | -------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------- | | SnapshotId | long | ID of the last created snapshot requested on this node. | | OperationId | string | ID of the last create operation requested on this node. | | StartTime | long | Start time of the last create operation, in milliseconds. | | EndTime | long | Finish time of the last create operation, in milliseconds, or `-1` if it has not started or is still in progress. | | Error | string | Error message of the last create operation. Empty if it succeeded or none has finished yet. | | TotalPages | long | Total number of pages to write. | | BytesWritten | long | Bytes written by the last create operation. | | TotalBytesWrittenOnStorage | long | Total bytes actually written to the storage device (after compression). | | WriteThrottlingTime | long | Time the snapshot writer threads spent parked because of throttling, in milliseconds. | | PartitionsWritten | long | Number of distinct partitions written. | | FreeSpace | long | Free space on the file system of the snapshot working directory, in bytes. | **Restore (`snapshot-restore`) and check (`snapshot-check`)** Both registers contain the same metrics. The values refer to the last restore or check operation on this node, respectively. | Name | Type | Description | | -------------- | ------ | ------------------------------------------------------------------------------------------------------- | | SnapshotId | long | ID of the snapshot that is being restored or checked. | | OperationId | string | ID of the last operation requested on this node. | | StartTime | long | Start time of the last operation, in milliseconds. | | EndTime | long | End time of the last operation, in milliseconds, or `-1` if it has not started or is still in progress. | | Error | string | Error message of the last operation. Empty if it succeeded. | | TotalPages | long | Total number of pages to restore or check, or `-1` if not known yet. | | ProcessedPages | long | Number of pages restored or checked so far. | ### Cluster Cluster metrics. Register name: `cluster` | Name | Type | Description | | ------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ActiveBaselineNodes | integer | Active baseline nodes count. | | Rebalanced | boolean | True if the cluster has fully achieved rebalanced state. Note that an inactive cluster always has this metric in False regardless of the real partitions state. | | TotalBaselineNodes | integer | Total baseline nodes count. | | TotalClientNodes | integer | Client nodes count. | | TotalServerNodes | integer | Server nodes count. | ## SQL SQL metrics. ### Memory Quotas Register name: `sql.memory.quotas` | Name | Type | Description | | ---------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------- | | OffloadedQueriesNumber | number | Number of queries that were offloaded to disk locally | | OffloadingRead | bytes | Number of bytes read from the disk during SQL query offloading | | OffloadingWritten | bytes | Number of bytes written to the disk during SQL query offloading | | freeMem | bytes | Amount of memory left available for the queries on this node, in bytes (negative value if SQL memory quotas are disabled) | | maxMem | bytes | Total amount of memory available for all queries on the current node (negative value if SQL memory quotas are disabled) | | requests | number | Total number of times memory quota has been requested on the current node by all the queries | ### Parser Cache Register name: `sql.parser.cache` | Name | Type | Description | | ------ | ------ | ---------------------------------- | | hits | number | Number of hits for queries cache | | misses | number | Number of misses for queries cache | ### User Queries Register name: `sql.parser.cache` | Name | Type | Description | | ----------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | | canceled | number | Number of canceled queries initiated by the current node. This number is included in the general 'failed' metric. | | failed | number | Number of failed queries (including OOME) initiated by the current node | | failedByOOM | number | Number of queries failed due to out of memory protection initiated by the current node. This number is included in the general 'failed' metric. | | success | number | Number of successfully executed queries initiated by the current node | ### Throttling Throttling metrics for the Write operation. Speed-based throttling protects the checkpoint buffer and the clean pages in the region. The checkpoint buffer needs a stronger protection because an overflow of this buffer makes a node crash. When performing a Write operation, we first check whether the checkpoint buffer is in danger. If it is, we employ the exponential backoff algorithm to protect the buffer: each subsequent "sleep" time is K times longer than the previous one. If the checkpoint buffer is not in danger, we calculate the "sleep" time using the speed-based algorithm to protect the pool of clean pages. Register name: `io.dataregion` | Name | Type | Description | | ------------------------------ | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | SpeedBasedThrottlingPercentage | double | Fraction of throttling time within average marking time (e.g., "quarter" = 0.25). | | MarkDirtySpeed | long | Speed of marking pages dirty, in pages/second. Value is averaged over the last 3 fragments, 0.25 sec each, plus the current fragment, 0-0.25 sec (0.75-1.0 sec total). | | CpWriteSpeed | long | Checkpoint write speed, in pages/second. Value is averaged over the last 3 checkpoints plus the current one. | | LastEstimatedSpeedForMarkAll | long | Last estimated speed of marking all clean pages dirty to the end of a checkpoint, in pages/second. | | CurrDirtyRatio | double | Current ratio of dirty pages (dirty vs total), expressed as a fraction. The fraction is computed for each segment in the current region, and the highest value becomes "current." | | TargetDirtyRatio | double | Ratio of dirty pages (dirty vs total), expressed as a fraction. Throttling starts when this ratio is reached. | | ThrottleParkTime | long | Park (sleep) time for the Write operation, in nanoseconds. Value is averaged over the last 3 fragments, 0.25 sec each, plus the current fragment, 0-0.25 sec (0.75-1.0 sec total). It defines park periods for either the checkpoint buffer protection or the clean page pool protection. | | CpTotalPages | int | Number of pages in the current checkpoint. | | CpEvictedPages | int | Number of evicted pages in the current checkpoint. | | CpWrittenPages | int | Number of written pages in the current checkpoint. | | CpSyncedPages | int | Number of fsynced pages in the current checkpoint. | | CheckpointBufferPagesCount | int | Number of occupied pages in the checkpoint buffer. | | CheckpointBufferPagesSize | int | Total number of pages in the checkpoint buffer. | ## Data Center Replication ### Receiver Metrics Register name: `dr.receiver.dc.` | Attribute | Type | Description | Scope | | ------------------- | ------ | ------------------------------------------------------------------------------------------------------- | ----- | | BatchesReceived | int | Total size of batches received from remote DC . | Node | | EntriesReceived | long | Total number of entries received from remote DC . | Node | | BytesReceived | long | Total number of batches received from remote DC . | Node | | BytesAcked | long | Total size of batches acked from remote DC in bytes. | Node | | EntriesAcked | long | Total number of entries acked from remote DC. | Node | | BatchesAcked | int | Total number of batches acked from remote DC. | Node | | BatchesSent | int | Number of batches waiting to be stored in cache. | Node | | EntriesSent | long | Number of entries waiting to be stored in cache. | Node | | BytesSent | long | Number of entries waiting to be stored in cache. | Node | | AverageBatchAckTime | double | Average time in milliseconds between sending batch to receiver cache nodes and successfully storing it. | Node | ### Receiver Trace Metrics Register name: `dr.receiver.trace.dc.` These metrics show how long replicated updates take to reach the receiving data center, and whether the clocks of the source and receiving data centers disagree. The sender records the time of each update when the update happens, and sends the times of the oldest and newest update with each batch. The receiver hub compares these times with its own clock when the batch arrives. The receiver hub creates the register when the first batch for the data center and cache arrives, and removes it when the receiver hub stops. | Attribute | Type | Description | Scope | | ------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- | | Batches | long | Number of batches received from remote DC for this cache. | Node | | BatchesWithTrace | long | Number of batches that carried update times and were used for the delay metrics. | Node | | DelayFloorLast | long | Receive time minus the time of the newest update in the last batch, in milliseconds. This is the lower bound of the replication delay. | Node | | DelayFloorMin | long | Smallest `DelayFloor` value observed, in milliseconds. `0` until a batch is observed. | Node | | DelayFloorMax | long | Largest `DelayFloor` value observed, in milliseconds. `0` until a batch is observed. | Node | | DelayFloorNegatives | long | Number of batches with a negative `DelayFloor` value. | Node | | DelayFloorHistogram | histogram | Distribution of the non-negative `DelayFloor` values, in milliseconds. | Node | | DelayCeilingLast | long | Receive time minus the time of the oldest update in the last batch, in milliseconds. This is the upper bound of the replication delay. | Node | | DelayCeilingMin | long | Smallest `DelayCeiling` value observed, in milliseconds. `0` until a batch is observed. | Node | | DelayCeilingMax | long | Largest `DelayCeiling` value observed, in milliseconds. `0` until a batch is observed. | Node | | DelayCeilingNegatives | long | Number of batches with a negative `DelayCeiling` value. | Node | | DelayCeilingHistogram | histogram | Distribution of the non-negative `DelayCeiling` values, in milliseconds. | Node | | LegacyBatchesOnV2Session | long | Number of batches without update times received over a connection that supports them. Expected to grow during a rolling upgrade and while batches stored by the sender before the upgrade are sent. A steady increase after that indicates a problem. | Node | | V2BatchesOnV1Session | long | Number of batches with update times received over a connection that does not support them. The update times are ignored and the data is applied. Expected to stay at `0`. | Node | The difference between `DelayCeiling` and `DelayFloor` shows how long a batch waited in the sender store. A negative delay means that the clock of the source data center is ahead of the receiver's clock by at least that amount. Negative values are counted in the `*Negatives` metrics and reported by the `*Min` and `*Last` metrics, but are not added to the histograms, so the histograms show delay only. The histograms use the following bucket bounds, in milliseconds: 1, 2, 4, 8, 16, 25, 50, 75, 100, 250, 500, 750, 1000, 3000, 5000, 10000, 25000, and 60000. Histograms are available in the JMX exporter only. For more information, see [Histograms](#histograms). Keep in mind the following when you read these metrics: * The metrics use a cached system clock. Even when the data center clocks are synchronized, `DelayFloor` can show negative values of up to about 10 ms, so clock skew below about 10 ms cannot be measured. * Update times are recorded for incremental replication only. Batches sent during a full state transfer, or when incremental data center replication is disabled, carry no update times. These batches are counted in `Batches` but not in `BatchesWithTrace`. * Update times are exchanged only when both data centers support them. Batches from a data center that runs an older GridGain version are counted in `Batches` only. ### Sender Metrics Register name: `dr.sender.dc.` | Attribute | Type | Description | Scope | | ------------------- | ------ | --------------------------------------------------------------------------------------------------- | ----- | | BytesReceived | long | Total size of batches in bytes received by the sender from data nodes. | Node | | EntriesReceived | long | Total number of entries were received by the sender from data nodes. | Node | | BatchesReceived | int | Total number of batches received by the sender from data nodes. | Node | | BatchesSent | int | Number of sent batches. | Node | | BatchesSentFst | int | Number of full state transfer (FST) batches sent to the receiver hubs. A subset of BatchesSent. | Node | | EntriesSent | long | Number of sent entries. | Node | | BytesSent | long | Number of sent bytes. | Node | | BatchesAcked | int | Number of acknowledged batches. | Node | | BatchesAckedFst | int | Number of acknowledged full state transfer (FST) batches. A subset of BatchesAcked. | Node | | EntriesAcked | long | Number of acknowledged sent entries. | Node | | BytesAcked | long | Number of acknowledged bytes. | Node | | BatchesFailed | int | Number of sent batches that caused an error. | Node | | BatchesFailedFst | int | Number of failed full state transfer (FST) batches. A subset of BatchesFailed. | Node | | EntriesError | long | Number of sent entries that caused an error. | Node | | BytesError | long | Number of sent bytes that caused an error. | Node | | AverageBatchAckTime | double | Total time in milliseconds between sending batches for the first time and receiving acknowledgment. | Node | ### Cache Metrics Register name: `dr.cache.` | Attribute | Type | Description | Scope | | ---------------- | ---- | ------------------------------------------------ | ----- | | pendingQueueSize | long | Number of cache updates in the DR pending queue. | Node | | backupQueueSize | long | Number of cache updates in the DR backup queue. | Node | ## Histograms Metrics that represent histograms are available in the JMX exporter only. Histogram metrics are exported as a set of values where each value corresponds to a specific bucket, and is available through a separate JMX bean attribute. The attribute names of a histogram metric have the following format: ``` {metric_name}_{low_bound}_{high_bound} ``` where * `{metric_name}` - the name of the metric * `{low_bound}` - start of the bound, `0` for the first bound * `{high_bound}` - end of the bound, `inf` for the last bound Examples of the metric names if the bounds are \[10,100]: * histogram\_0\_10 - less than 10 * histogram\_10\_100 - between 10 and 100 * histogram\_100\_inf - more than 100 Histogram metrics exported as JSON strings look like this: ```json { "bounds": [1,2,4,8,16,25,50,75,100,250,500,750,1000,3000,5000,10000,25000,60000], "values": [ {"fromExclusive":0,"toInclusive":1,"value":0}, {"fromExclusive":1,"toInclusive":2,"value":0}, {"fromExclusive":2,"toInclusive":4,"value":0}, {"fromExclusive":4,"toInclusive":8,"value":0}, {"fromExclusive":8,"toInclusive":16,"value":0}, {"fromExclusive":16,"toInclusive":25,"value":0}, {"fromExclusive":25,"toInclusive":50,"value":0}, {"fromExclusive":50,"toInclusive":75,"value":2}, {"fromExclusive":75,"toInclusive":100,"value":0}, {"fromExclusive":100,"toInclusive":250,"value":0}, {"fromExclusive":250,"toInclusive":500,"value":0}, {"fromExclusive":500,"toInclusive":750,"value":0}, {"fromExclusive":750,"toInclusive":1000,"value":0}, {"fromExclusive":1000,"toInclusive":3000,"value":1}, {"fromExclusive":3000,"toInclusive":5000,"value":0}, {"fromExclusive":5000,"toInclusive":10000,"value":0}, {"fromExclusive":10000,"toInclusive":25000,"value":0}, {"fromExclusive":25000,"toInclusive":60000,"value":0}, {"fromExclusive":60000,"value":0} ] } ``` In the histogram, `bounds` is metadata, separate from the actual time values, and it is used to separate the spectrum of data values into buckets. For example, in the histogram above, the `bounds` field shows 18 values. The first bucket always starts with `0`, so in our example, the first bucket is `0 - 1` with the `1` being inclusive. The last bucket always includes infinity, so in our example the final bucket is `60000 and higher`. Any value higher than 60000 is put into that bucket. If a value for the histogram above is `1500` milliseconds, it falls into the `1000 - 3000` bucket and increments the `value` for that bucket (`"value":1`).