Leave feedback
  • Question

    Data input process

Enter a new topic
  • Jon Jonsson Jon Jonsson
    0 likes 1042 views

    Hi,

    At my customer site we have a middle process between initiated system (telecom billing system) and StreamServe. There is an outstanding question how we can skip that process. Here is the story:

    The initated system creates one XML file for each document. A huge number of XML files are created in each execution. All XML files are saved in an Oracle database as single blobs.

    To process all these XML files directly in StreamServe is out of question very bad in a StreamServe performance point of view (millions of StreamServe jobs). Therefore are the XML files handled by a separated process before StreamServe. This process read out all XML from the database and concatenate them into new XML files with a couple of thousands documents in each new XML file. These new XML files are sent to and processed by StreamServe.

    This is a very common way to handle high volume output from initated systems (at least at the sites I have been at). However, the concat process means a lot of I/O etc. My customer wants to know if there is any optional solution in StreamServe, in order to skip the concat process. So, in order words, can we use any other of the possibilities for input connectors in StreamServe to read directly from the Oracle blobs? Or any other alternative where we can use more of StreamServe and less of the concat process? And still keep the performance. Any example for other likely situations?

    I am very grateful for all comments or ideas (or any kind of out-of-box-thinking). 

    BR Jon

    Wednesday 07 August, 2013
  • Amine Soussou Amine Soussou
    0 likes

    Hi, 

     

    What is the actual config for your server, you can use threads and multithreading to handle this huge amount of data without concat operations.

     

    And it depends of the process you use in StreamServe

    Thursday 15 August, 2013
  • Jon Jonsson Jon Jonsson
    0 likes

    Hi Amine,

    We are using 5 threads for 3 servers in parallel. However, I still think there is a concat process needed, mabye built inside as a sub process the StreamServe project. One idea that came up recently was to use a JAVA input connector with necessary JAVA functionalty that replace the existing separat process (read from DB, concat i suitable XML files etc).

    BR Jon

    Thursday 15 August, 2013
  • Amine Soussou Amine Soussou
    0 likes

    It depends of the complexxity of your initial XML file and the total number for them.

    How your out process work, and may be there some tunning to do to optimize how it works ?

    Thursday 15 August, 2013
  • Jon Jonsson Jon Jonsson
    0 likes

    The XMLs are rather complex (invoice doc. in the telecom area). The total number is approx. 1000' per month.

    At this point I do not see any options to tune the existing StreamServe process. I think the suggestion to have a pre-step in StreamServe with a JAVA-connector is something to go more in detail with.

    Thursday 15 August, 2013
  • Amine Soussou Amine Soussou
    0 likes

    Yes, it can be a solution, or you can prepare some scripts to concatenate all xmls before sending them to StreamServe.

    Friday 16 August, 2013
  • Jon Jonsson Jon Jonsson
    0 likes

    Yes, that is also an option. In fact, this is was it done today, but that process is a little bit unnecessary complicated and very heave for the I/O (also containg charset conversion). The customer prefer to no have anything outside StreamServe, in order to have less steps in process. So, we will see what decision they take.

    Thanks for your reply.

    BR Jon

    Friday 16 August, 2013

 

Latest from the blogs