Decoding challenges with Netflix Conductor

In my last blog, I’ve discussed some core concepts of Netflix Conductor (Conductor OSS), how it works and why we choose this tool as our workflow orchestration engine while building an insurance platform. We’ve seen that Netflix Conductor is a powerful open-source workflow orchestration platform designed for managing complex microservice interactions. While it offers a wealth of benefits, like observability, monitoring and resilience, working with Conductor also presents its own set of challenges. In this blog, let’s delve into some of the challenges we’ve faced while working with Conductor and how we resolved them.

Before starting if you wanna refresh your understanding of how Netflix Conductor works, checkout my previous blog here.

Delay in completion of JOIN tasks

JOIN tasks in Conductor are asynchronous system tasks used in conjunction with a FORK_JOIN task. It waits for a user provided list of zero or more tasks to be completed before moving to the next stage in the workflow.

JOIN tasks leverage a dedicated queue and Conductor’s System Task Worker polls the queue, checking if all JOIN tasks’ dependent tasks (listed in the joinOn attribute) are finished. Unfinished JOIN tasks get re-queued for later evaluation, while completed ones progress the workflow.