Skip to content

Snapshot Agent for Linux & Windows

Part of the Streaming Integration Platform Base Environment, the Snapshot Agent application runs on the same system as your DBL application. It receives requests for a snapshot to be created from the snapshot request topic, creates a new snapshot of your data files, uploads that snapshot data to network S3-compatible storage, and then posts a completion message to the snapshot response topic.

The same Snapshot Agent applciation is used on Linux and Windows, where it usually started as a service during system boot. There is a different Snapshot Agent for OpenVMS systems.

Profile Name

When the Snapshot Agent starts it determines the name of the configuration profile to use, which in turn is used to locate the configuration files to be used. The following mechanisms, listed in order of highest to lowest precedence, can be used to define the name of the profile to use:

  • Command line options --config-profile <name> or -p <name>.
  • Environment variable SIP_CONFIG_PROFILE=<name>.
  • (Linux only) The value of PROFILE=<name> read from /etc/sysconfig/sip.conf
  • The default profile name default.

Configuration Directory

The configuration files required by the Snapshot Agent are located in a configuration directory, the location of which can be determined through several mechanisms, listed in order of highest to lowest precedence:

  • Command line options --config-dir <path> or -d <path>.
  • Environment variable SIP_CONFIG_PATH=<path>.
  • (Windows only) C:\ProgramData\Synergex\SIP
  • Linux default: /etc/synergex/sip

The profile name is appended as a sub directory to the configuration directory. For example, if the configuration profile being used is default and the configuration directory is /etc/synergex/sip, the configuration files should be placed in /etc/synergex/sip/default.

Output Directory

The Snapshot Agent creates output files in a local output directory directory, the location of which can be defined through several mechanisms, listed in order of highest to lowest precedence:

  • Command line option --output-dir <directroy> or -o <directory>.
  • MasterConfig.json value of OriginSystemTools.SnapshotDirectory.

For the Snapshot Agent to start, the output directory must be defined by one of these mechanisms.

Here is an example of the output directory specified via the command line on Linux:

Command line environment variables
snapshot-agent --output-dir /myapp/snapshots

And an example of of th output directory specified via MasterConfig.json on Windows:

OriginSystemTools environment variables
{
  "OriginSystemTools": {
    "SnapshotDirectory": "C:\\MYAPP\\SNAPSHOTS"
  }
}

Environment Variables

To create a snapshot of your data the Snapshot Agent needs to be able to read your DBL applications data files, and this often requires that one or more environment variables are in place to identify the location of those files. If any such environment variables are alredy in place in the system environment (but not via the synergy.ini file on Windows) then the Snapshot Agent will be able to utilize that information, but if not, then environment variable names and values can be specified in these ways, listed in order of highest to lowest precedence:

  • Command line option --env <name>=<value> or -e <name>=<value>.
  • MasterConfig section OriginSystemTools.EnvironmentVariables.

Here is an example of environment variables specified via the command line on Linux:

Command line environment variables
snapshot-agent --env DATA1=/myapp/data1 --env DATA2=/myapp/data2

And an example of of environment variables specified via MasterConfig.json on Windows:

OriginSystemTools environment variables
{
  "OriginSystemTools": {
    "EnvironmentVariables": [
      {
        "Name": "DATA1",
        "Value": "C:\\MYAPP\\DATA1"
      },
      {
        "Name": "DATA2",
        "Value": "C:\\MYAPP\\DATA2"
      }
    ]
  }
}

Configuration Files

The Snapshot Agent reads information from the configuration file MasterConfig.json, which can contain configuration information for various Streaming Integration Platform components and its actual content will depend on which components are being used in any given environment. The Snapshot Agent utilizes information from the following sections within MasterConfig.json:

  • Kafka

    • Describes the location and configuration of the Kafka-compatible server(s) and topics being used.

  • TableDefinitions

    • Describes the schema of the data files being replicated.

  • OriginSystemTools

    • Describes the names and values of environment variables that need to be in place during execution, primarily to access data files.

  • S3StorageExternal

    • Describes the location and configuration of the S3-compatible storage server being used.

Configuration Server

Because, depending on your environment and use case, the same configuration files can be required by several different tools that may be operating on several different servers, and even on different operating systems, the Streaming Integration Platform provides the Configuration Server, which can be used, amongst other things, to centralize the maintenance of these configuration files.

If the Configuration Server is in use, the Snapshot Agent needs to be able to locate and interact with it. For that to happen it needs two pieces of information:

  • Profile name (already described above)
  • Configuration Server URL

The Configuration Server URL can be specified in obe of the following ways, listed in order of highest to lowest precedence:

  • Command line option --config-server-url <url>.
  • Environment variable SIP_CONFIG_SERVER <url>.

If the Configuration Server URL is not specified then the configuration files must be manually provided and maintained in the appropriate configuration directory.

When correctly configured, the Snapshot Agent will, during startup, connect to and authenticate with the Configuration Server and download the latest version of MasterConfig.json which will then be stored in the local configuration directory, overwriting any file that may currently be present. This ensures that the Snapshot Agent always has the latest version of the configuration file each time it starts.

If for any reason the Snapshot Agent is unable to establish a connection to the Configuration Server, it will fall back to using the configuration file that is already present in the local configuration directory.

Note that the TLS certificate used by the Configuration Server to provide secure HTTPS communications must be trusted by the system where your DBL application and the Snapshot Agent are running.

Which Files are Included in a Snapshot

When the Snapshot Agent creates a snapshot, regardless of how that snapshot is initiated, the data files that are processed, and the resulting Parquet files for tables, are determined by information in MasterConfiguration.TableDefinitions.

Specifically, each main definition has one or more sub_table definitions, each of which represents a specific record latout used in the data file. If all the records in a file are defined by the same record latout then there will only be a single sub_table definition. But for files that have multiple record layouts, identified by repository tags, there will be one sub_table definition per record layout. So a file with records matching five record layouts, each defined by a unique tag expression, will have five sub_table definitions.

When a snapshot is created, the Snapshot Afent will consider all files listed under TableDefinitions, but will only produce snapshot output for the logical tables defined by sub_table definitions whose enabled property is set to true.

Creating a Manual Snapshot

The Snapshot Agent generally runs all of the time as a service application, so that it is available whenever an agent applicaiton might need a snapshot to be created. But there are times, during initial environment setup for example, when it is useful to have the ability to create a local snapshot. THis can be achieved by running the agent in and instructing it to immediately create a snapshot, by specifying either the -now or -n command liune option:

Immediate snapshot example
snapshot-agent --now --output-dir /tmp/mysnapshot --config-profile development

You will see messages as the snapshot proceeds:

Example snapshot progress
DEPARTMENT 2026-02-23 13:55:02 unload starting
DEPARTMENT 2026-02-23 13:55:02 successfully processed table
DEPARTMENT 2026-02-23 13:55:02 unload finished: in 13.0272ms
EMPLOYEE 2026-02-23 13:55:02 unload starting
EMPLOYEE 2026-02-23 13:55:02 successfully processed table
EMPLOYEE 2026-02-23 13:55:02 unload finished: in 124.1176ms

Note that when you create a snapshot maually like this:

  • The snapshots files are created in the local output directory.
  • No data is uploaded to the S3-compatible storage server.
  • No completion message is sent to the Kafka snapshot response topic.

If you do want the files to be uploaded and a completion message to be sent, and you have the Snapshot Agent running as a service, you can actieve that my manually posting a snapshot request message to the snapshot request topic.

Example snapshot request
{
  "RequestId": "d7ec641b-6f36-4141-aa23-95e7809f529d",
  "RequestTime": "2026-02-23T21:59:03.2853602Z",
  "Format": "Parquet"
}

When doing so, note that the value of the RequestId property must uniquely identify the snapshot amongst all others, and will be used as the directory name within the S3 storage bucket (we generally use a GUID value), and the RequestTime property must be an accurate and correctly formatted ISO8601/RFC 3339 datetime string expressed in the UTC (or Zulu) timezone:

ISO8601 DateTime
2026-02-23T21:59:03.2853602Z
│          │         │      └─ Z = UTC (Zulu time)
│          │         └──────── Fractional seconds (7 digits)
│          └────────────────── Time (HH:MM:SS)
└───────────────────────────── Date (YYYY-MM-DD)

Other Command Line Options

The Snapshot Agent has the following additional command line options:

Additional options
--verbose  or  -v
    Show verbose messages

--help  or -h
    Display commandline and other help information

Example Snapshot Agent Startup

Example Snapshot Agent startup
===============================================================================
Streaming Integration Platform: Snapshot Agent
===============================================================================

Config profile          : development
MasterConfig source     : Local file
Kafka servers           : 192.168.200.2:9092
CDC topic               : DEFAULT_REPLICATOR
Snapshot topic          : DEFAULT_REQUEST_SNAPSHOT
Snapshot response topic : DEFAULT_SNAPSHOT
S3 endpoint             : https://192.168.200.2:9000
S3 access_key           : sipagent
S3 secret_key           : ********
S3 region               : us-east-1
S3 bucket               : development
Output directory        : C:\DEV\SYNERGEX\IntegrationPlatform\temp
Table definitions       : 2

Ready for messages