Leave feedback
  • Question

    Problem with Apache Tomcat and Streamstudio

Enter a new topic
  • Jon Jonsson Jon Jonsson
    0 likes 3995 views

    Hi!

    We have tried a number of different settings for Apache Tomcat, used earlier info from this forum. And for weeks it works good, but suddenly there is problem again today. This is probably caused by an increased number of logged in StreamStudio users. What happens is that StreamStudio crashes, and Apache Tomcat has to be restarted (happend several times today).

    The settings we are using is as following:

    ==============================================================

    -Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 6.0
    -Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 6.0
    -Djava.endorsed.dirs=C:\Program Files\Apache Software Foundation\Tomcat 6.0\endorsed
    -Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 6.0\temp
    -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
    -Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 6.0\conf\logging.properties
    -Xms1024
    -Xmx1024m
    -XX:MaxPermSize=1024m

    Inital memory pool: 1024 MB

    Maximum memory pool: 1024 MB

    ==============================================================

    I have tried using increased values compared with above, with no success.

    The logs are attached.

    We are using:

    StreamServe Persuasion SP5 build 1450.

    Apace Tomcat 6.0

    Java 6 update 30

    Anybody out there that has knowledge of a complete setting that working fine at sites with a large amount of StreamStudio users? Any ideas are SO welcome.

    BR Jon Jonsson, Sogeti

    Monday 10 September, 2012

    Attached files

  • David Shih David Shih StreamServe Employee
    0 likes

    With a 32-bit JVM, the OS can only allocate 2048 MB to the JVM and the Java executable (and then there's additional unusable overhead). In other words, the maximum *total* usable Java Heap + stacks + PermGen is about 1536 MB.

    So on a 32-bit JVM, the most you can allocate is about "-Xmx1280m -XX:MaxPermSize=256m" before the OS refuses to let Tomcat start.

    If you're running a 64-bit OS, switch to a 64-bit JVM, and try increasing the memory allocation, i.e. "-Xmx3072m -XX:MaxPermSize=512m" On Windows, you'd go to the "Java" tab of "tomcat6w.exe" and just point to the jvm.dll of the 64-bit Java. On Unix, you'd edit the $JAVA_HOME environment variable (often in bin/catalina.sh) to point to the new JVM.

    If you're not running a 32-bit OS, then you'd probably need to run multiple instances of Tomcat (i.e. in separate JVMs), and come up with some sort of load-balancing scheme (i.e. using a frontend Apache/IIS web server and modJK). If you host system has enough memory, you can run the multiple JVMs on the same host: make a copy of your CATALINA_HOME directory, and edit the settings in the conf directory...pay attention to the TCP port numbers (8005, 8009, 8080) in the server.xml file, to ensure that each Tomcat instance uses unique ports. Refer to the Tomcat documentation for further details.

    Monday 10 September, 2012
  • Jon Jonsson Jon Jonsson
    0 likes

    Hi David!

    Realized that I forgot to mention that we are running 64-bit OS and also a 64 bit JVM. We will now try your suggestions, related to the 64-bit information above.

    Thanks!

    BR Jon

    Tuesday 11 September, 2012
  • Jon Jonsson Jon Jonsson
    0 likes

    Hi again!

    Well, I tried to set up Tomcat with following settings:

    -Xms1024
    -Xmx3072m
    -XX:MaxPermSize=512m

    Inital memory pool: 1024 MB

    Maximum memory pool: 1024 MB

    This setting caused many more crashes. So, now I am using the inital value, exept for the MaxPermSize:

    -Xms1024
    -Xmx1024m
    -XX:MaxPermSize=512m

    Inital memory pool: 1024 MB

    Maximum memory pool: 1024 MB

    The 64-bit server has 8 GB RAM. Now again, the crashes com VERY rarelym 1-2 times/week. And always at the morning, so it seems to be related to the users login phase. There is not any performance problem when searching in Collector etc.

    Can the problem be related to the ActiveDirectory/SGW? See logging below, the only logg that I can find related to the latest crash today, it is from StreamStudio.log (the Tomcat, and SGW logs does not say anything):

    2012-09-12 08:24:36,105 [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker -1] ERROR com.streamserve.ws.axis2.ExceptionTranslator  - Web Services Error returned by the Service GW!

     

      ==================================================================

     

        A WebService error occured, please follow the trace by viewing

     

        the logs from both the Service GW and the StreamServer.

     

      ==================================================================

     

              Error:

     

                       MessageID:        1

     

                       Message:          Strs::ErrorGeneric::invalidConfigurationError

     

                       ErrorCode:        0x8000001F

     

                       ErrorMessage:     Strs::ErrorGeneric::invalidConfigurationError

     

              Detail:

     

                       Thread ID:        6352

     

                       Source location:  soapstrsschemaserviceobject.cxx, 816, getSchemas

     

                       Error time stamp: Sep 12, 2012 8:24:36 AM

     

                       Call stack:       <NO VALUE>

     

      ==================================================================

     

    2012-09-12 08:24:36,105 [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1 ] WARN  com.streamserve.corrman.business.schema.report.MetaDataImpactDispatche r  - Failed run metadata removal report for application domain [PRODUCTION]!

     

              Reason: getSchemas: Failed to get schemas : : Strs::ErrorGeneric::invalidConfigurationError

    ============================================

    BR Jon Jonsson

    Friday 14 September, 2012
  • David Shih David Shih StreamServe Employee
    0 likes

    Initial Memory Pool = -Xms Java option

    Maximum Memory Pool = -Xmx Java option

    If you're using Windows and the tomcat6w.exe to configure the JVM parameters, get rid of the additional Java Options "-Xms" and "-Xmx". That could be why the JVM failed to start...because it had conflicting options.

    I don't think SGW errors should cause Tomcat to fail. Unless it had a whole bunch of unclosed network socket connections. [Run 'netstat -an | findstr 2718' when Tomcat misbehaves to see how many SGW connections it has open.]

    If there's a problem with the JVM's GC (garbage collection), that could cause Tomcat to crash. In the Windows Task Manager, you'd see the amount of memory increasing and increasing as time goes by. In the ~WEB-INF/spring/properties/log4j.properties file, I think there's a verbosegc logging option that you could enable to help detect if there's a memory leak.

    Friday 14 September, 2012
  • Jon Jonsson Jon Jonsson
    0 likes

    Hi David!

    Very good, we will will try your suggestions.

    Also, we realized that our problem also can be related to antivirus scanning at the server. So, in additional we also has excluced scanning in some directories, also described by your comments in http://streamshare.streamserve.com/Articles/Article/?articleId=422.

    Thanks again!

    BR Jon

    Monday 17 September, 2012