Leave feedback
  • Message Concatenation

Write an Article
Monday 15 March, 2010
Stefan Cohen Stefan Cohen StreamServe Employee Administrator
6 likes 3957 views

Looped projects are harder to maintain than regular one-phase projects. This article describes a method to avoid message looping.

Looped projects are StreamServer configurations where the input (typically paginated, received thru PageIN or PreformatIN) is serialized to a common format using StreamOUT in a single file (phase one) and then read back in again using StreamIN on to the actual process (phase 2). An example:

Phase 1:
Input -> PageIN -> StreamOUT -> temp file
Phase 2:
Temp file -> StreamIN -> PageOUT -> Output

The purpose of such configurations can vary but typical applications are when you want to merge the input into one message or when you want to clean up “dirty” input before the actual processing starts in the second step. There are also variations where XMLOUT/XMLIN or MessageOUT/MessageIN are used instead of StreamOUT/StreamIN.

Looped projects are harder to maintain and debug compared to regular one-phase since changes to the message configuration requires changes in three places instead of just one (first event, first process, second event). Manually created temp files are usually also more error-prone compared to the internally managed data streams.

So, how can we avoid this? The answer has been available since StreamServer version 4.0 and is called message concatenation. From this version and later, you can add more than one event to a message configuration in Design Center. The message instances from the events (see InDepth #3) can be automatically concatenated to one message instance.


The catch is that all events in a Message configuration must be of the same type. The reason for this is athat the Input Analyser (described in InDepth #3) can only detect one input data type prior to sending input batches to a specific Agent type.

The message concatenation is performed by a "Message Merger" component in the kernel. It receives message instances from an agent and appends message data (field and blocks) to an output message before releasing the output message to the process(es).



You can control how the concatenation should be performed using the settings in the Event Order tab in the Event Settings. This makes it possible to configure what event should start a new output message instance, what events shout added to the output message instance and what event should "flush" the output message instance.


An event set to "First" will always flush any existing output message prior to creating a new output message. There is also a script function called EndMessage() that allows you to flush an output message using a script.


The message merging algorithm works like this:

An output message is created by an event marked a First. The complete message instance from this event is added to the output message. Any repeating events are merged to the output message. Fields on root level (not contained in a block) are added to the root of the output message if they are not allready present in the output message (root level fields are not overwritten in the output message). Blocks are added after all existing blocks in the output message. Events marked as Last will merged with the output message and the output message will finally be released to the process(es)




This functionality is not tied to any particular input data type as it operates on messages. This makes it possible to merge the input from any event/agent type.


Comments (4)

  • Hi, plese, do you have a sample project doing that ? best regards,

    Monday 02 July, 2012 by Amine Soussou
  • Hi,

    I had one that I used when I created the article back in 2010 but I can unfortunaltelly not find it right now.

    I will look a bit more but it seems like the virtual machine i used might have been lost in a disk crash last year.



    Monday 02 July, 2012 by Stefan Cohen
  • Ok Thanks very Much Stefan,


    What is this Message Merger, is something we have to configure, or it's an automatic process.

    And doe's the 3 events has the same fields and blocks ?



    Tuesday 03 July, 2012 by Amine Soussou
  • The message merger is configured by the Event Order configuration shown above and by adding more than one event (of the same type) in the message.

    The events can all have different fields and blocks so they doesn't have to be the same.

    Tuesday 03 July, 2012 by Stefan Cohen


Post comment