> For the complete documentation index, see [llms.txt](https://mariadb.com/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://mariadb.com/docs/gridgain/gridgain-8/tuning-and-troubleshooting/performance-tuning/memory-and-jvm-tuning.md).
# Memory and JVM Tuning
Best practices for memory, JVM, and garbage-collection tuning in GridGain deployments, with and without native persistence or external storage.
This article provides best practices for memory tuning that are relevant for deployments with and without native persistence or an external storage. Even though GridGain stores data and indexes off the Java heap, Java heap is still used to store objects generated by queries and operations executed by your applications. Thus, certain recommendations should be considered for JVM and garbage collection (GC) related optimizations.
{% hint style="info" %}
Refer to [persistence](/docs/gridgain/gridgain-8/tuning-and-troubleshooting/performance-tuning/persistence-tuning.md) tuning article for disk-related optimization practices.
{% endhint %}
## Tune Swappiness Setting
An operating system starts swapping pages from RAM to disk when overall RAM usage hits a certain threshold. Swapping can impact GridGain cluster performance. You can adjust the operating system's setting to prevent this from happening. For Unix, the best option is to either decrease the `vm.swappiness` parameter to `10`, or set it to `0` if native persistence is enabled:
```bash
sysctl -w vm.swappiness=0
```
The value of this setting can prolong GC pauses as well. For instance, if your GC logs show `low user time, high system time, long GC pause` records, it might be caused by Java heap pages being swapped in and out. To address this, use the `swappiness` settings above.
## Share RAM with OS and Apps
An individual machine's RAM is shared among the operating system, GridGain, and other applications. As a general recommendation, if a GridGain cluster is deployed in pure in-memory mode (native persistence is disabled), then you should not allocate more than 90% of RAM capacity to GridGain nodes.
On the other hand, if native persistence is used, then the OS requires extra RAM for its page cache in order to optimally sync up data to disk. If the page cache is not disabled, then you should not give more than 70% of the server's RAM to GridGain.
Refer to [memory configuration](/docs/gridgain/gridgain-8/gridgain8-development/caches-and-memory/memory-configuration/data-regions.md) for configuration examples.
In addition to that, because using native persistence might cause high page cache utilization, the `kswapd` daemon might not keep up with page reclamation, which is used by the page cache in the background. As a result, this can cause high latencies due to direct page reclamation and lead to long GC pauses.
To work around the effects caused by page memory reclamation on Linux, add extra bytes between `wmark_min` and `wmark_low` with `/proc/sys/vm/extra_free_kbytes`:
```bash
sysctl -w vm.extra_free_kbytes=1240000
```
Refer to [this resource](https://events.static.linuxfound.org/sites/events/files/lcjp13_moriya.pdf) for more insight into the relationship between page cache settings, high latencies, and long GC pauses.
## Java Heap and GC Tuning
Even though GridGain and Ignite keep data in their own off-heap memory regions invisible to Java garbage collectors, Java Heap is still used for objects generated by your applications workloads. For instance, whenever you run SQL queries against a GridGain cluster, the queries will access data and indexes stored in the off-heap memory while the result sets of such queries will be kept in Java Heap until your application reads the result sets. Thus, depending on the throughput and type of operations, Java Heap can still be utilized heavily and this might require JVM and GC related tuning for your workloads.
We've included some common recommendations and best practices below. Feel free to start with them and make further adjustments as necessary, depending on the specifics of your applications.
{% hint style="info" %}
Refer to [GC debugging techniques](/docs/gridgain/gridgain-8/tuning-and-troubleshooting/troubleshooting.md#debugging-gc-issues) sections for best practices on GC logs and heap dumps collection.
{% endhint %}
### Heap Sizes Between 32 GB and 47 GB
Do not set the maximum heap size (`-Xmx`) to a value between 32 GB and roughly 47 GB. A heap in that range holds less data than a 31 GB heap does.
Below 32 GB, the JVM stores object references as 32-bit values, a technique called compressed ordinary object pointers, or compressed oops. At 32 GB the JVM turns compressed oops off, every reference grows from 4 to 8 bytes, and every object that holds references grows with them, so fewer objects fit. The heap has to reach roughly 47 GB before the added capacity makes up for the larger references. The exact crossover depends on how many references your data holds.
Set `-Xmx` to 31 GB or less, or to well above 47 GB. Raising a 31 GB heap to 40 GB can cause the very `OutOfMemoryError` it was meant to prevent.
{% hint style="info" %}
The 32 GB threshold assumes the default object alignment of 8 bytes. Raising `-XX:ObjectAlignmentInBytes` moves the threshold higher, at the cost of more padding per object.
{% endhint %}
### Generic GC Settings
Below are sets of example JVM configurations for applications that can utilize Java Heap on server nodes heavily, thus triggering long — or frequent, short — stop-the-world GC pauses.
Use the G1 garbage collector (the default on Java 17). The settings below are a good starting point if 10GB heap is more than enough for your server nodes:
```bash
-server
-Xms10g
-Xmx10g
-XX:+AlwaysPreTouch
-XX:+UseG1GC
-XX:+ScavengeBeforeFullGC
```
If G1 does not work for you, consider using CMS collector and starting with the following settings. Note that 10GB heap is used as an example and a smaller heap can be enough for your use case:
```bash
-server
-Xms10g
-Xmx10g
-XX:+AlwaysPreTouch
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled
-XX:+ScavengeBeforeFullGC
-XX:+CMSScavengeBeforeRemark
```
{% hint style="info" %}
If you use [Ignite native persistence](/docs/gridgain/gridgain-8/architecture/storage/native-persistence.md), we recommend that you set the `MaxDirectMemorySize` JVM parameter to `walSegmentSize * 4`. With the default WAL settings, this value is equal to 256MB.
{% endhint %}
### Advanced Memory Tuning
In Linux and Unix environments, it's possible that an application can face long GC pauses or lower performance due to I/O or memory starvation due to kernel specific settings. This section provides some guidelines on how to modify kernel settings in order to overcome long GC pauses.
{% hint style="warning" %}
All the shell commands given below were tested on RedHat 7. They may differ for your Linux distribution. Before changing the kernel settings, make sure to check the system statistics/logs to confirm that you really have a problem. Consult your IT department before making changes at the Linux kernel level in production.
{% endhint %}
If GC logs show `low user time, high system time, long GC pause` then most likely memory constraints are triggering swapping or scanning of a free memory space.
* Check and adjust the [swappiness settings](#tune-swappiness-setting).
* Add `-XX:+AlwaysPreTouch` to JVM settings on startup.
* Disable NUMA zone-reclaim optimization.
```bash
sysctl -w vm.zone_reclaim_mode=0
```
* Turn off Transparent Huge Pages if RedHat distribution is used.
```bash
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
```
### Advanced I/O Tuning
If GC logs show `low user time, low system time, long GC pause` then GC threads might be spending too much time in the kernel space being blocked by various I/O activities. For instance, this can be caused by journal commits, gzip, or log roll over procedures.
As a solution, you can try changing the page flushing interval from the default 30 seconds to 5 seconds:
```bash
sysctl -w vm.dirty_writeback_centisecs=500
sysctl -w vm.dirty_expire_centisecs=500
```
{% hint style="info" %}
Refer to the [persistence tuning](/docs/gridgain/gridgain-8/tuning-and-troubleshooting/performance-tuning/persistence-tuning.md) section for the optimizations related to disk. Those optimizations can have a positive impact on GC.
{% endhint %}
### NUMA-Aware Memory Allocation
In high-performance environments, the time it takes for the computer to communicate between processor and memory can have negative effects on performance. To further improve performance, you can enable NUMA-aware memory allocation on GridGain. When the option is enabled, processor will access the closest NUMA node, which will increase performance.
Using NUMA-aware allocation leads to load being better spread across the threads.
{% hint style="info" %}
NUMA-aware memory allocation is only available on Linux machines.
{% endhint %}
#### Requirements
* You need to make sure `libnuma 2.0.x` is installed on your Linux machine. It is also recommended to install the `numactl` utility.
* Enable NUMA-aware allocation by setting the `-XX:+UseNUMA` JVM property.
#### Using NUMA Allocation
#### Simple Allocation Strategy
Simple node allocation strategy is best used to attach the data region to a specific NUMA node.
* Allocation with default NUMA policy on all NUMA nodes:
```xml
....
```
* Allocation on specific NUMA node:
```xml
....
```
#### Interleaved Allocation Strategy.
Interleaved allocation is best used to attach the data region to multiple NUMA nodes.
* Interleaved allocation on all NUMA nodes:
```xml
....
```
* Interleaved allocation on specified NUMA nodes:
```xml
....
0
1
```
#### Local Node Allocation Strategy
Allocation on local for process NUMA node, uses `void* numa_alloc_onnode(size_t)` under the hood.
```xml
....
```