What is out-of-memory and time-out memory error

Skill level: Easy

Time required: 3 minutes

Out-of-memory error

An Out-of-memory (OOMKilled) error occurs when a transaction in Quickwork consumes more memory than its allocated limit. To maintain isolation and efficiency, each transaction is assigned specific CPU and memory resources. By default, Quickwork allocates 256 MB of memory per transaction, which is typically sufficient for most use cases. However, transactions handling extremely large payloads, such as thousands of records, may exceed this limit, resulting in an OOMKilled error.

Best practices to avoid OOMKilled errors

1. Batch Processing: Extract data from source systems in smaller batches instead of retrieving all records at once. This reduces the memory load on individual transactions, preventing errors.

2. Callable journeys: Break large processes into smaller tasks using callable journeys. Dividing payloads into manageable chunks ensures efficient processing without exceeding memory limits.

3. Memory limit increase: If necessary, Quickwork can assist in increasing the memory allocation for specific workflows.

By following these practices, you can optimize resource usage, ensure reliability, and prevent OOMKilled errors, ensuring smooth and efficient workflow execution in Quickwork.

Time-out error

Each transaction has a default 15-minute time span to complete the execution. If the transaction exceeds this execution time, the journey terminates with the time-out error. If necessary, Quickwork can assist to extend the transaction time span. 

How to resolve the issue

1. Optimize time-consuming steps: Identify the steps causing delays and streamline their processes. For example, reduce unnecessary operations or improve data-fetching efficiency.

2. Batch processing: If using loops, send data in smaller batches to reduce execution time.

3. Use callable journeys: Move repetitive or common steps into a separate callable journey. Call this smaller journey from the parent journey to save time and resources.