Building Workflows using Reactive Programming in NodeJSAmarpreet SinghBlockedUnblockFollowFollowingJan 3Photo by Adam Solomon on UnsplashIntroductionIn this article, I would like to talk about an architecture pattern that I designed, while building a distributed messaging platform.
The BeginningGiven the requirements of the system to be built, I started identifying the various components, needed to solve the problem at hand.
For instance, a server runtime environment, a web application framework, a database, a web-socket library, a queuing system etc.
Also, thought had to be given about the multi-node architecture of the system.
After long nights and many cups of coffee, the tech stack decided was Node.
js, koa.
js, Socket.
io, MySQL, Apache Kafka, Redis.
Given the high-velocity event ingestion and volume of asynchronous tasks, what better run time environment than Node.
js, a non-blocking I/O.
The Pen & PaperAs a usual habit, before I press keys on the keyboard, I put pen to paper, listing down my ideas and preparing a solution.
Given the components in the system, I drafted various workflows, few of which are illustrated below.
It wasn’t this neat, as it looks now ????.
Represents a moduleRepresents the flow of dataWorkflow #1Workflow #2The Beautiful MindAfter setting up the local development environment, it was time to begin with the flow implementation.
With workflow #1 in my mind, I began with the Kafka module, implementing message stream processing.
The next step, was to pass the message.
Initially, I gave power to the current module to delegate message to another(require other module and call one of the method) and so on… to drive the flow.
However, this wasn’t just enough!Various situations came up and I began to ponder on:How to pass the message, once it is processed by the current module?How to handle cyclic dependency ?Cyclic dependencyAlso, if a flow breaks due to an error at some module, how to handle it?Moreover, if a critical connection error occurs with Kafka, Redis or database, which affects all flows, how does our system react to it?How to design a system, which has the possibility of n number of workflows(due to multiple entry points), each involving n number of modules in either sequential or parallel combinations(refer workflow #1, #2)?Multiple entry pointsSolutionTo design a reactive, flexible and robust system which can support/handle:Decoupling of componentsNumerous workflowsThe flow of data among modules(without cyclic dependency)High-velocity event ingestion and a good volume of asynchronous tasksI thought of various solutions, like:Designing a State MachineEvent-Driven ArchitectureIn State Machine Architecture, it would have been a purely functional programming.
Predefining the various flows with the sequence of callbacks/functions to execute.
In Event-Driven Architecture, I thought to design a scalable, decentralised system, by splitting it into various components, each managing their internal logic and failures.
And use events as the mode of communication among them.
In 3 words, a reactive program.
After weighing the pros and cons of each, I chose to go ahead with the event-driven architecture.
As it will help to form a loosely coupled architecture, in which modules become independent of each other and interact with each other via events.
In short, a scalable and reactive system.
The CodeNode.
js EventEmitter class forms the core of the event bus.
Rest of the architecture is my implementation.
The ConnectorA class, extending from Node.
js EventEmitter, assisting in the creation of objects (alias connector/s), which can trigger and listen events.
A connector is required for every module.
The Connector ManagerA central place to manage all connectors.
It acts as a central hub, with which various modules in the application can not only register to listen but also trigger events.
It takes care of:Initialisation of all the connectorsProvides functionality to register and trigger eventsError handlingWhen an event is triggered, it is executed in the check phase of the Node.
js lifecycle(else it might lead to stack overflow) and all the callbacks registered for the event on the connector are invoked.
ConnectorManager.
jsThe Component and it’s ModuleComponents like Redis, Socket.
io, Kafka etc (their Node.
js drivers are available via npm) are exposed as modules.
During the initialisation phase, the module can register for listening events on multiple connectors.
Also, if any state change in module needs to be propagated through the system, the module can trigger event on its connector.
This also provides the means to delegate failures as messages.
Each component is self-contained, encapsulated and isolated from other components.
Application InitialisationFinally, as the Node server is booted up, all the modules are initialised, executing the process of event registration and thereby making the system ready to respond to events and execute workflows.
index.
jsLogsBelow is a snapshot of logs, which demonstrates how modules register, trigger events and how a workflow gets executed.
LogsGithubI have hosted this project architecture on Github for reference.
It’s easy to follow and can be extended easily.
amarpreetsingh29/node-workflowsBuilding workflows in NodeJSThe InternalsTo fully understand and leverage the power of an event-driven architecture, it’s good to have an understanding of the design of event emitter.
The Event emitter is an implementation of the observer pattern.
The observer pattern is a software design pattern in which an object, called the subject, maintains a list of its dependents, called observers, and notifies them automatically of any state changes, usually by calling one of their methods.
Below is a tiny version of the same(keeps track of listener as well), which I drafted.
Subject.
jsConclusionThis architecture in entirety is essence of Reactive programming and can be easily extended and used, where workflows or pipelines are to be built.
To build this program, I choose to use event-based, callback flavour of reactive programming(rather than stream based) and it helped me to solve many problems here and build a solid foundation for this project.
.