Looped projects are harder to maintain than regular one-phase projects. This article describes a method to avoid message looping.
Looped projects are StreamServer configurations where the input (typically paginated, received thru PageIN or PreformatIN) is serialized to a common format using StreamOUT in a single file (phase one) and then read back in again using StreamIN on to the actual process (phase 2). An example:
Input -> PageIN -> StreamOUT -> temp file
Temp file -> StreamIN -> PageOUT -> Output
The purpose of such configurations can vary but typical applications are when you want to merge the input into one message or when you want to clean up “dirty” input before the actual processing starts in the second step. There are also variations where XMLOUT/XMLIN or MessageOUT/MessageIN are used instead of StreamOUT/StreamIN.
Looped projects are harder to maintain and debug compared to regular one-phase since changes to the message configuration requires changes in three places instead of just one (first event, first process, second event). Manually created temp files are usually also more error-prone compared to the internally managed data streams.
So, how can we avoid this? The answer has been available since StreamServer version 4.0 and is called message concatenation. From this version and later, you can add more than one event to a message configuration in Design Center. The message instances from the events (see InDepth #3) can be automatically concatenated to one message instance.
The catch is that all events in a Message configuration must be of the same type. The reason for this is athat the Input Analyser (described in InDepth #3) can only detect one input data type prior to sending input batches to a specific Agent type.
The message concatenation is performed by a "Message Merger" component in the kernel. It receives message instances from an agent and appends message data (field and blocks) to an output message before releasing the output message to the process(es).
You can control how the concatenation should be performed using the settings in the Event Order tab in the Event Settings. This makes it possible to configure what event should start a new output message instance, what events shout added to the output message instance and what event should "flush" the output message instance.
An event set to "First" will always flush any existing output message prior to creating a new output message. There is also a script function called EndMessage() that allows you to flush an output message using a script.
The message merging algorithm works like this:
An output message is created by an event marked a First. The complete message instance from this event is added to the output message. Any repeating events are merged to the output message. Fields on root level (not contained in a block) are added to the root of the output message if they are not allready present in the output message (root level fields are not overwritten in the output message). Blocks are added after all existing blocks in the output message. Events marked as Last will merged with the output message and the output message will finally be released to the process(es)
This functionality is not tied to any particular input data type as it operates on messages. This makes it possible to merge the input from any event/agent type.