Middleware

Messaging Infrastructure

At the heart of the Streaming Integration Platform is an implementation of Apache Kafka, which is usually deployed using one or more Linux servers. To ensure a resilient environment for production, a cluster of three or more Linux hosts is generally recommended.

Rather than actually using Apache Kafka, which is implemented using Java, Synergex recommends using Redpanda, which significantly outperforms Apache Kafka.

Storage Infrastructure

The Streaming Integration Platform operates around snapshots of the original application data, and those snapshots are generally stored in some type of network storage. The preferred solution is to store snapshots using Amazon S3 compatible storage hosted on the local network.

There are several ways to achieve this, typically using third-party storage providers such as SeaweedFS or MinIO. When replicating data using the ISAM Agent, another option is to use an S3 storage provider embedded within that agent. Of course it's also possible to use Amazon S3 cloud storage; however, doing so introduces significant overhead due to the additional latency of internet communication.