/opt/gridgain9cli/bin/gridgain9 cluster status
```
The command output must include the name of your cluster and the number of nodes. The status must be `ACTIVE`.
## Optional: Liveness and Readiness Probes
You can set up configuration for readiness and liveness probes.
* Liveness probe checks if the GridGain node has started in the pod, and responds to REST requests on the liveness endpoint.
* Readiness probe checks if the GridGain node has joined the cluster and is ready to receive data.
{% hint style="warning" %}
When using readiness probes, you must configure both `publishNotReadyAddresses: true` and `podManagementPolicy: Parallel` properties. Without these settings, pods will not be able to discover each other during initialization.
{% endhint %}
The example below shows how to set up liveness and readiness probes with recommended defaults:
```yaml
livenessProbe:
failureThreshold: 3
httpGet:
path: /health/liveness
port: 10300
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 10
readinessProbe:
failureThreshold: 3
httpGet:
path: /health/readiness
port: 10300
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
```
{% hint style="info" %}
Since readiness and liveness probes are highly recommended for kubernetes deployments, the `statefulset.yaml` provided above includes this configuration.
{% endhint %}
For more information about the REST endpoints, see the [OpenAPI specification](https://www.gridgain.com/sdk/gridgain9/latest/openapi.html).
## Optional: KEDA Configuration
You can configure KEDA to automatically scale the cluster based on your needs, ensuring optimal resource optimization and performance. This implementation uses Prometheus to monitor cluster load.
{% hint style="warning" %}
Before configuring KEDA autoscaling based on JVM metrics, you must define the maximum heap and non-heap (Metaspace) memory for the GridGain Java process. If these are not set, threshold values will be calculated incorrectly.
You can set these limits by modifying the `statefulset` to define the `GRIDGAIN9_EXTRA_JVM_ARGS` environment variable and define `-Xmx` (max heap size) and `-XX:MaxMetaspaceSize` (max non-heap/metaspace size) flags.
The total memory for these flags should be less than the container's memory limit (e.g., memory: 4Gi in the limits).
{% endhint %}
To enable KEDA scaling for your cluster:
* Add the necessary Helm repositories, install KEDA and Prometheus:
```bash
helm install keda kedacore/keda --namespace keda --create-namespace
helm install prometheus prometheus-community/prometheus --namespace keda -f prometheus-values.yaml
```
* Deploy the KEDA configurations:
* The `keda-scaled-object.yaml` configuration defines the scaling rules for the GridGain cluster:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: gridgain-autoscale
namespace: gridgain
spec:
scaleTargetRef:
kind: StatefulSet
name: gridgain-cluster
pollingInterval: 30
cooldownPeriod: 120
minReplicaCount: 2 # Set initial number of replics
maxReplicaCount: 5
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleDown:
selectPolicy: Disabled
scaleUp:
stabilizationWindowSeconds: 60 # Increase if needed
selectPolicy: Max
policies:
- type: Pods
value: 1
periodSeconds: 120
triggers:
# Uncomment and modify the following section to enable CPU usage based autoscaling
# - type: prometheus
# metadata:
# name: cpu-usage
# serverAddress: http://prometheus-server.keda.svc.cluster.local:80
# query: sum(os_system_load_average{job="gridgain"})
# threshold: "0.8"
# activationThreshold: "0.6"
- type: prometheus
name: heap-memory-usage
metadata:
serverAddress: http://prometheus-server.keda.svc.cluster.local:80
query: |
sum(jvm_memory_committed_bytes{area="heap", job="gridgain"} / jvm_memory_max_bytes{area="heap", job="gridgain"})
threshold: "0.7"
- type: prometheus
name: nonheap-memory-usage
metadata:
serverAddress: http://prometheus-server.keda.svc.cluster.local:80
query: |
sum(jvm_memory_committed_bytes{area="nonheap", job="gridgain"} / jvm_memory_max_bytes{area="nonheap", job="gridgain"})
threshold: "0.7"
* The `keda-recovery-scaled-job.yaml` configuration handles rebuilding [CMG nodes](/docs/gridgain/gridgain-9/architecture/cluster/cluster-lifecycle.md#cluster-management-group) in the GridGain cluster.
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
name: gridgain-recovery
namespace: gridgain
spec:
jobTargetRef:
template:
spec:
# securityContext:
# runAsUser: 0
# runAsGroup: 0
# fsGroup: 0
containers:
- name: recovery
image: gridgain/gridgain9:9.1.28
command: ["/bin/bash", "/scripts/recovery.sh"]
volumeMounts:
- name: script-vol
mountPath: /scripts
restartPolicy: Never
volumes:
- name: script-vol
configMap:
name: gridgain-recovery-script
defaultMode: 0777
backoffLimit: 1
pollingInterval: 30
successfulJobsHistoryLimit: 2
failedJobsHistoryLimit: 3
maxReplicaCount: 1
triggers:
# - type: prometheus
# metadata:
# serverAddress: http://prometheus-server.keda.svc.cluster.local:80
# query: avg(os_system_load_average{job="gridgain"})
# threshold: "0.8"
# activationThreshold: "0.6"
- type: prometheus
name: heap-memory-usage
metadata:
serverAddress: http://prometheus-server.keda.svc.cluster.local:80
query: |
avg(jvm_memory_committed_bytes{area="heap", job="gridgain"} / jvm_memory_max_bytes{area="heap", job="gridgain"})
threshold: "0.7"
- type: prometheus
name: nonheap-memory-usage
metadata:
serverAddress: http://prometheus-server.keda.svc.cluster.local:80
query: |
avg(jvm_memory_committed_bytes{area="nonheap", job="gridgain"} / jvm_memory_max_bytes{area="nonheap", job="gridgain"})
threshold: "0.7"
* You can deploy the above configurations with the following commands:
```bash
kubectl apply -n gridgain -f keda-scaled-object.yaml
kubectl apply -n gridgain -f keda-recovery-scaled-job.yaml
```
## Installation Troubleshooting
If any issues occur during the installation:
* Check the logs of specific pods:
```bash
kubectl logs -n
```
* Review events in the namespace:
```bash
kubectl get events -n
```
## Troubleshooting via REST
This approach is intended for short-lived debugging and testing sessions, not for long-term or external access.
{% hint style="warning" %}
`kubectl port-forward` opens a local tunnel from your machine to a pod through the Kubernetes API server. Do not expose REST this way permanently.
{% endhint %}
1. Forward the REST port to a specific GridGain 9 replica:
```bash
kubectl port-forward 10300:10300 -n
```
Example:
```bash
kubectl port-forward gridgain-cluster-0 10300:10300 -n gridgain
```
2. Open a new terminal and run REST requests to the `http://localhost:10300`, for example:
```bash
curl 'http://localhost:10300/management/v1/cluster/state'
{"title":"Cluster is not initialized","status":409,"detail":"Cluster is not initialized. Call /management/v1/cluster/init in order to initialize cluster."}
```
```bash
curl 'http://localhost:10300/management/v1/node/info'
{"name":"gridgain-cluster-0","jdbcPort":10800}
```
For more information refer to the documentation:
* Kubernetes: [Use Port Forwarding to Access Applications in a Cluster](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/)
* `kubectl` command reference: [kubectl port-forward](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_port-forward/)
* GridGain 9 REST API docs: [REST API](/docs/gridgain/gridgain-9/gridgain9-development/accessing-data/rest-api.md)
## Limitations and Considerations
When running GridGain 9 in a Kubernetes environment, the node configuration becomes **read-only** and cannot be modified by using the `gridgain9 node config update` CLI command. This is by design, as node configuration is managed via Kubernetes resources. To change your configuration:
1. Manually update the corresponding ConfigMap;
2. Restart all cluster pods by executing `kubectl delete pod` for each replica.
The updated configuration will take effect after the pods are recreated by the Kubernetes controller.
{% hint style="warning" %}
While it is technically possible to make the node configuration writable by using an init container that copies the mounted configuration from one location to another (and pointing GridGain 9 to this new location), we **strongly discourage** this approach. It is not native to the Kubernetes model, and any changes made to the configuration will be lost during pod restarts or re-deployments.
{% endhint %}