Ticket #410 (closed defect: fixed)

Opened 3 years ago

Last modified 3 years ago

Exporting large excel files causes java.lang.OutOfMemoryError: Java heap space

Reported by: s.h.sikkema@… Owned by: robert@…
Priority: critical Milestone: 0.8.1
Component: Export functionality Version: 0.7.0
Keywords: Cc:
Product: Operating system:
URL: Hardware:

Description

If you try to export an assay with a lot of data, a java.lang.OutOfMemoryError?: Java heap space is thrown. This is probably due to the fact that the excel file is constructed in memory.
A possible solution could be to write the excel to a temporary file and present that to the user as a download.
The problem can be recreated by exporting the Linea study's blood profiling assay on test.nmcdsp.org and selecting everything for export (default). It goes well when you do not export the measurement data.

Relevant stacktrace:

2011-04-12 13:51:51,909 [TP-Processor12] ERROR StackTrace?  - Sanitizing stacktrace:
java.lang.OutOfMemoryError?: Java heap space

at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream?.toByteArray(ByteArrayOutputStream?.java:133)
at org.apache.poi.openxml4j.opc.internal.MemoryPackagePartOutputStream?.flush(MemoryPackagePartOutputStream?.java:76)
at org.apache.poi.openxml4j.opc.internal.MemoryPackagePartOutputStream?.close(MemoryPackagePartOutputStream?.java:51)
at org.apache.poi.xssf.usermodel.XSSFSheet.commit(XSSFSheet.java:2440)
at org.apache.poi.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:196)
at org.apache.poi.POIXMLDocumentPart.onSave(POIXMLDocumentPart.java:200)
at org.apache.poi.POIXMLDocument.write(POIXMLDocument.java:204)
at org.apache.poi.ss.usermodel.Workbook$write.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray?.defaultCall(CallSiteArray?.java:40)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite?.call(AbstractCallSite?.java:116)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite?.call(AbstractCallSite?.java:124)
at dbnp.studycapturing.AssayService?.exportRowWiseDataToExcelFile(AssayService?.groovy:498)
at sun.reflect.NativeMethodAccessorImpl?.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl?.invoke(NativeMethodAccessorImpl?.java:39)
at sun.reflect.DelegatingMethodAccessorImpl?.invoke(DelegatingMethodAccessorImpl?.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite?$PogoCachedMethodSite?.invoke(PogoMetaMethodSite?.java:225)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite?.callCurrent(PogoMetaMethodSite?.java:51)
at org.codehaus.groovy.runtime.callsite.CallSiteArray?.defaultCallCurrent(CallSiteArray?.java:44)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite?.callCurrent(AbstractCallSite?.java:141)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite?.callCurrent(AbstractCallSite?.java:157)
at dbnp.studycapturing.AssayService?.exportRowWiseDataToExcelFile(AssayService?.groovy)
at dbnp.studycapturing.AssayService?$exportRowWiseDataToExcelFile.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray?.defaultCall(CallSiteArray?.java:40)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite?.call(AbstractCallSite?.java:116)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite?.call(AbstractCallSite?.java:128)
at dbnp.studycapturing.AssayController?$_closure11.doCall(AssayController?.groovy:202)
at sun.reflect.NativeMethodAccessorImpl?.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl?.invoke(NativeMethodAccessorImpl?.java:39)
at sun.reflect.DelegatingMethodAccessorImpl?.invoke(DelegatingMethodAccessorImpl?.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)

Change History

Changed 3 years ago by work@…

  • milestone changed from 0.7.1 to 0.8.1

Changed 3 years ago by business@…

Use tab-delimited formats, let user choose:
* Tab-delimited (.txt)
* Comma separated (.csv)
* Semicolon separated (.csv, for Dutch Excel versions)

Changed 3 years ago by s.h.sikkema@…

  • owner changed from s.h.sikkema@… to business@…
  • status changed from new to assigned

Reassigned for testing

Changed 3 years ago by robert@…

A few comments from my side:
- when exporting .txt, empty cells are exported as 'null', instead of remaining empty
- no matter what output type I choose, the file is always exported as tab delimited (maybe because I chose tab delimited the first time?)
- the filename is 'export..txt', with two dots instead of one

Changed 3 years ago by s.h.sikkema@…

  • owner changed from business@… to robert@…

Fixed the issues in commit 1828; reassigned for testing

Changed 3 years ago by robert@…

  • owner changed from robert@… to s.h.sikkema@…

These issues have been solved. Thanks. Two other issues:
- Error messages are non-descriptive. For example, something went wrong and I got this message: "An error occurred while fetching  http://ci.metabolomics.nmcdsp.org/rest/getMeasurementData/query." I have no clue about what I did wrong, or what I should do next. Try again? Call an administrator? Choose another option?
- I've exported the 001LINEA study to european .csv format and it is opened in excel. This is OK. However, the numbers are given as '21.7' instead of 21,7, which should be used in dutch excel. This results in excel not recognizing the numbers as numbers.

Changed 3 years ago by s.h.sikkema@…

  • owner changed from s.h.sikkema@… to robert@…

Fixed, assigning back ...

Changed 3 years ago by work@…

  • status changed from assigned to closed
  • resolution set to fixed

Changed 3 years ago by business@…

  • status changed from closed to reopened
  • resolution fixed deleted

Just tested it with a Dutch Excel version, works mostly, except for zero numbers, they are still '0.0' instead of '0,0' (or maybe just put nothing if it is not filled in, that is better than any '0').

Changed 3 years ago by business@…

  • status changed from reopened to closed
  • resolution set to fixed

Tested on CI, works fine now!

Note: See TracTickets for help on using tickets.