Allocate memory to Data Engine

Data Engine uses the following memory types:

• Maximum cumulative memory – To start theData Engine process.

  The maximum cumulative memory cannot be less than the sum of heap limit, Java direct memory, and code cache size.

• Heap limit – To run the server.

• Java direct memory – To execute queries.

• Code cache size – To cache data.

  The value for this parameter is auto-computed based on the heap size and cannot exceed 1 GB. The cache size cannot be configured.

Configuring memory for the Data Engine pod

You can choose one of the following approaches to memory configuration:

Method 1: Use default values

You can leave default values if at least 14 GB of system memory is available:

• Java direct memory = 8 GB

• Heap limit = 4 GB

• Maximum cumulative memory = 13 GB (to cover heap & direct plus 1 GB for cache)

Method 2: Set all memory options manually

You can define all the memory limits manually. For example, you can use the following values if at least 22 GB of memory is available:

• Java direct memory = 16 GB

• Heap limit = 4 GB

• Maximum cumulative memory = 21 GB (to cover heap & direct plus 1 GB for cache)

Method 3: Set only the cumulative memory

You can specify the maximum memory only to autocalculate other values:

• Maximum cumulative memory = 24 GB

• Autoconfigure heap & direct = On

  In this case, the system automatically allocates the best values for heap limit and Java direct memory.

Example – Autoconfiguring heap & direct based on maximum memory

Configuration

Your system memory is 32 GB, and you have the following parameters for the Data Enginee pod:

• Maximum cumulative memory: 20%

• Autoconfigure heap & direct: On

Result

If the available free memory is less than the allocation, you are warned that the operating system can reclaim more free memory when required.

Example – Data Engine not starting due to insufficient memory

Configuration

Your system memory is 32 GB, and you have the following parameters for the Data Engine pod:

• Maximum cumulative memory: 50%

• Heap limit: 3 GB

• Java direct memory: 13 GB

• Cache size: 1024 MB (default, cannot be changed through Configuration Manager)

Result

The available memory of 15 GB is insufficient to start Data Engine.

Example – Unused memory

Configuration

Your system memory is 48 GB, and you have the following parameters for the Data Engine pod:

• Maximum cumulative memory: 50%

• Heap limit: 8 GB

• Java direct memory: 10 GB

• Cache size: 1024 MB (default, cannot be changed through Configuration Manager)

Result

50% of the system memory (48 GB) translates to 24 GB. Therefore, you have an allocation of 4 GB that is currently unused from a total of 24 GB. You can increase your existing memory configuration to use this extra memory.

Check the Data Engine memory allocations in the log

You can check the Data Engine memory allocations by using the log for the respective pod.

1. Navigate to Kubernetes Dashboard.

2. Go to Workloads > Pods.

3. Click the data engine pod.

4. In the upper-right corner of the toolbar, click View logs.

5. Navigate the log file to the beginning and view the memory allocations.