| Wiki Markup |
|---|
{info}[https://jira.kuali.org/browse/OLE-3160]{info}
h2. RequirementRequirements
for
OLE DocStore
{color:#000000}Requirement for OLE DocStore/Discovery system based on IU's current{color}
{color:#000000}library system:{color}
{color:#000000}Store up to 25-50M records, but more close to 25M (currently IU library{color}
{color:#000000}has less than 25M records){color}
{color:#000000}Incremental loading: 10K records{color}
{color:#000000}Concurrent reads: 500{color}
{color:#000000}Concurrent writes: 20 per minute (or 1 per 1-3 second){color}
h4. Search Samples:
This [search sample file|^Search Samples for John.docx] is that John collects for the testing, which will give an idea for the normal searches in library.
h4. Testing use case:
Normally we test non-staff user searching on our systems since that accounts for about 80% to 90% of the load. That is out of scope here since OLE does not provide a non-staff search. So based on our highest load days, approximately 10,000 records are added are updated on our system. Over an 8 hour day, that is about .5 per second. I would round up to 1 create/update per second. Reads are higher so I would place at 5 per second. This is from the docstore. For the Solr index, I would say the searching would be heavier IF some of the display data is stored in the index. This is a question for the Peri and Nianli. I would say the Solr index would be hit at the about 10 searches/second at peak rate. In the middle of all this, a batch load of 10,000 records would not be unusual.
So summarizing
Create/Update Rate on Docstore: 1-2 transactions/second <\- this would also update the solr indexes
Read Rate on Docstore: 5 transactions/second
Read Rate on Solr Indexes: 10 transactions/second. If the display data is coming from Docstore then Docstore read = Solr Read i.e. 10 transactions/second
Batch load of 10,000 records in the midst of testing.
\\
h2. Evaluation Plan
{color:#000000}1) Build and install the app{color}
{color:#000000}2) Use debuggers and profilers to analyze the running app (locally){color}
{color:#000000}3) Build profiling scripts to push the app to the failure point (locally){color}
{color:#000000}4) Run the profiling scripts against the test server and develop an app profile{color}
{color:#000000}5) Check the code in the areas where performance issues are identified{color}
{color:#000000}6) Write up overall recommendations around setup, architecture, performance, and code practices{color}
{color:#000000}7) If time remains, add aspects in the code to do method profiling and generate a report of method performance under load{color}
h2. Problems found by Unicon
h3. DocStore Shutdown Issues:
{color:#000000}These are all memory cleanup issues. If the app is shutdown or{color}
{color:#000000}redeployed without shutting down the server then these will cause the{color}
{color:#000000}server to eventually run out of memory and crash.{color}
{color:#000000}SEVERE: The web application \[/oledocstore\] registered the JDBC driver{color}
{color:#000000}\[org.apache.derby.jdbc.AutoloadedDriver\] but failed to unregister it{color}
{color:#000000}when the web application was stopped. To prevent a memory leak, the{color}
{color:#000000}JDBC Driver has been forcibly unregistered.{color}
{color:#000000}May 21, 2012 2:54:56 PM org.apache.catalina.loader.WebappClassLoader{color}
{color:#000000}clearReferencesThreads{color}
{color:#000000}SEVERE: The web application \[/oledocstore\] appears to have started a{color}
{color:#000000}thread named \[Timer-3\] but has failed to stop it. This is very likely{color}
{color:#000000}to create a memory leak.{color}
{color:#000000}May 21, 2012 2:54:56 PM org.apache.catalina.loader.WebappClassLoader{color}
{color:#000000}clearReferencesThreads{color}
{color:#000000}SEVERE: The web application \[/oledocstore\] appears to have started a{color}
{color:#000000}thread named \[DynamicPooledExecutor\] but has failed to stop it. This{color}
{color:#000000}is very likely to create a memory leak.{color}
{color:#000000}May 21, 2012 2:54:56 PM org.apache.catalina.loader.WebappClassLoader{color}
{color:#000000}clearThreadLocalMap{color}
{color:#000000}SEVERE: The web application \[/oledocstore\] created a ThreadLocal with{color}
{color:#000000}key of type \[org.apache.derby.iapi.sql.dictionary.TableDescriptor$1\]{color}
{color:#000000}(value \[org.apache.derby.iapi.sql.dictionary.TableDescriptor$1@5dbdedf8\]){color}
{color:#000000}and a value of type \[java.util.WeakHashMap\] (value \[{=null, =null,{color}
{color:#000000}={color}
{1, 2, 3}{color:#000000}, =null, =null, =null, ={color}{1, 2, 3}{color:#000000}, =null, =null}\]) but{color}
{color:#000000}failed to remove it when the web application was stopped. This is very{color}
{color:#000000}likely to create a memory leak.{color}
{color:#000000}May 21, 2012 2:54:56 PM org.apache.catalina.loader.WebappClassLoader{color}
{color:#000000}clearThreadLocalMap{color}
{color:#000000}SEVERE: The web application \[/oledocstore\] created a ThreadLocal with{color}
{color:#000000}key of type \[java.lang.ThreadLocal\] (value{color}
{color:#000000}\[java.lang.ThreadLocal@605264dc\]) and a value of type{color}
{color:#000000}\[org.apache.derby.iapi.services.context.ContextManager\] (value{color}
{color:#000000}\[org.apache.derby.iapi.services.context.ContextManager@3499c4cc\]) but{color}
{color:#000000}failed to remove it when the web application was stopped. This is very{color}
{color:#000000}likely to create a memory leak.{color}
h3. {color:#000000}Warnings{color}
{color:#000000}WARN org.apache.jackrabbit.core.query.lucene.SearchIndex.setTextFilterClasses():Line{color}
{color:#000000}2087: The textFilterClasses configuration parameter has been{color}
{color:#000000}deprecated, and the configured value will be ignored:{color}
h3. {color:#000000}Strange dir in tomcat_home (or whatever dir that tomcat is started from){color}
{color:#000000}It looks like someone is trying to create a windows directory path{color}
{color:#000000}(\OLE-SOLR\Indexes). I don't think this is causing a problem but it{color}
{color:#000000}indicates some bad code exists.{color}
{color:#000000}AZMBP:tomcat6 azeckoski$ ls \-al{color}
{color:#000000}total 136{color}
{color:#000000}drwxr-xr-x 15 azeckoski admin 510 21 May 14:50 .{color}
{color:#000000}drwxr-xr-x 7 azeckoski admin 238 21 May 09:47 ..{color}
{color:#000000}\-rw-r\-{color}{color:#000000}{-}r{-}{color}{color:#000000}\-@ 1 azeckoski admin 37951 20 Feb 17:23 LICENSE{color}
{color:#000000}\-rw-r\-{color}{color:#000000}{-}r{-}{color}{color:#000000}\-@ 1 azeckoski admin 558 20 Feb 17:23 NOTICE{color}
{color:#000000}\-rw-r\-{color}{color:#000000}{-}r{-}{color}{color:#000000}\-@ 1 azeckoski admin 8680 20 Feb 17:23 RELEASE-NOTES{color}
{color:#000000}\-rw-r\-{color}{color:#000000}{-}r{-}{color}{color:#000000}\-@ 1 azeckoski admin 6670 20 Feb 17:23 RUNNING.txt{color}
{color:#000000}drwxr-xr-x 6 azeckoski admin 204 21 May 14:50 \OLE-SOLR\Indexes\auth{color}
{color:#000000}drwxr-xr-x@ 25 azeckoski admin 850 21 May 10:00 bin{color}
{color:#000000}drwxr-xr-x@ 9 azeckoski admin 306 20 Feb 17:23 conf{color}
{color:#000000}\-rw-r\-{color}{color:#000000}{-}r{-}{color}{color:#000000}\- 1 azeckoski admin 1565 21 May 14:54 derby.log{color}
{color:#000000}drwxr-xr-x@ 18 azeckoski admin 612 20 Feb 17:23 lib{color}
{color:#000000}drwxr-xr-x@ 11 azeckoski admin 374 21 May 14:50 logs{color}
{color:#000000}drwxr-xr-x@ 4 azeckoski admin 136 21 May 14:50 temp{color}
{color:#000000}drwxr-xr-x@ 6 azeckoski admin 204 21 May 14:26 webapps{color}
{color:#000000}drwxr-xr-x@ 3 azeckoski admin 102 21 May 14:50 work{color}
{color:#000000}AZMBP:tomcat6 azeckoski$ find "\OLE-SOLR\Indexes\auth"{color}
{color:#000000}\OLE-SOLR\Indexes\auth{color}
{color:#000000}\OLE-SOLR\Indexes\auth/index{color}
{color:#000000}\OLE-SOLR\Indexes\auth/index/segments.gen{color}
{color:#000000}\OLE-SOLR\Indexes\auth/index/segments_1{color}
{color:#000000}\OLE-SOLR\Indexes\auth/spellchecker1{color}
{color:#000000}\OLE-SOLR\Indexes\auth/spellchecker1/segments.gen{color}
{color:#000000}\OLE-SOLR\Indexes\auth/spellchecker1/segments_1{color}
{color:#000000}\OLE-SOLR\Indexes\auth/spellchecker2{color}
{color:#000000}\OLE-SOLR\Indexes\auth/spellchecker2/segments.gen{color}
{color:#000000}\OLE-SOLR\Indexes\auth/spellchecker2/segments_1{color}
{color:#000000}\OLE-SOLR\Indexes\auth/spellcheckerFile{color}
{color:#000000}\OLE-SOLR\Indexes\auth/spellcheckerFile/segments.gen{color}
{color:#000000}\OLE-SOLR\Indexes\auth/spellcheckerFile/segments_1{color}
h3. {color:#000000}println in log ("in aadvancedSearch.jsp"){color}
{color:#000000}Looks like someone left in a println statement by accident. Should use{color}
{color:#000000}log.debug instead.{color}
{color:#000000}May 21, 2012 2:50:51 PM org.apache.jk.server.JkMain start{color}
{color:#000000}INFO: Jk running ID=0 time=0/12 config=null{color}
{color:#000000}May 21, 2012 2:50:51 PM org.apache.catalina.startup.Catalina start{color}
{color:#000000}INFO: Server startup in 6749 ms{color}
{color:#000000}in aadvancedSearch.jsp{color}
{color:#000000}in aadvancedSearch.jsp{color}
{color:#000000}May 21, 2012 2:54:54 PM org.apache.coyote.http11.Http11Protocol pause{color}
{color:#000000}INFO: Pausing Coyote HTTP/1.1 on http-8080{color}
h3. Session closing issue:
{color:#000000}2012-05-21 15:36:13,886 \[Finalizer\] WARN{color}
{color:#000000}org.apache.jackrabbit.core.SessionImpl.finalize():Line 1326: Unclosed{color}
{color:#000000}session detected. The session was opened here:{color}
{color:#000000}java.lang.Exception: Stack Trace{color} {color:#000000}at org.apache.jackrabbit.core.SessionImpl.<init>(SessionImpl.java:222){color} {color:#000000}at org.apache.jackrabbit.core.SessionImpl.<init>(SessionImpl.java:239){color} {color:#000000}at org.apache.jackrabbit.core.XASessionImpl.<init>(XASessionImpl.java:99){color} {color:#000000}at org.apache.jackrabbit.core.RepositoryImpl.createSessionInstance(RepositoryImpl.java:1573){color} {color:#000000}at org.apache.jackrabbit.core.RepositoryImpl.createSession(RepositoryImpl.java:923){color} {color:#000000}at org.apache.jackrabbit.core.RepositoryImpl.login(RepositoryImpl.java:1461){color} {color:#000000}at org.apache.jackrabbit.core.TransientRepository.login(TransientRepository.java:381){color} {color:#000000}at org.apache.jackrabbit.commons.AbstractRepository.login(AbstractRepository.java:123){color} {color:#000000}at org.kuali.ole.RepositoryManager.getSession(RepositoryManager.java:71){color} {color:#000000}at org.kuali.ole.RepositoryBrowser.generateNodeCount(RepositoryBrowser.java:488){color} {color:#000000}at org.apache.jsp.nodeCount_jsp._jspService(nodeCount_jsp.java:96){color} {color:#000000}at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70){color} {color:#000000}at javax.servlet.http.HttpServlet.service(HttpServlet.java:717){color} {color:#000000}at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:388){color} {color:#000000}at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313){color} {color:#000000}at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260){color} {color:#000000}at javax.servlet.http.HttpServlet.service(HttpServlet.java:717){color} {color:#000000}at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290){color} {color:#000000}at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206){color} {color:#000000}at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:185){color} {color:#000000}at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:159){color} {color:#000000}at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235){color} {color:#000000}at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206){color} {color:#000000}at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233){color} {color:#000000}at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191){color} {color:#000000}at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127){color} {color:#000000}at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102){color} {color:#000000}at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109){color} {color:#000000}at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293){color} {color:#000000}at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859){color} {color:#000000}at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602){color} {color:#000000}at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489){color} {color:#000000}at java.lang.Thread.run(Thread.java:680){color}
{color:#000000}2012-05-21 15:36:13,888 \[Finalizer\] INFO{color}
{color:#000000}org.apache.jackrabbit.core.TransientRepository.loggedOut():Line 424:{color}
{color:#000000}Session closed{color}
{color:#ff0000}HTC has patched the code at {color}[https://svn.kuali.org/repos/ole/tags/OLE-0.6-1/DocumentStore|https://svn.kuali.org/repos/ole/tags/OLE-0.6-1/DocumentStore]{color:#ff0000}. There are{color}{color:#ff0000} 2 files have {color}{color:#ff0000}changed, RepositoryManager and RepositoryBrowser.{color} |
Page Comparison
General
Content
Integrations