Unable to export large files from MATLAB to SQL server

Smaller tables are fine, but I am unable to send say, 100K rows back to SQL server 2008 from MATLAB 2012a.

MATLAB and SQL server are on different machines - both on my network.

I am using a JDBC connection. The connection is successful, and a partial file gets exported, but not the full file.

I could try batching my queries and breaking the table up, but is there another solution?

Here is the set of commands I use:

conn = database('DataBase','user','password')','com.microsoft.sqlserver.jdbc.SQLServerDriver','jdbc:sqlserver://SERVERNAME:1433;database=DataBase;integratedSecurity=true;') setdbprefs('DataReturnFormat','numeric');

exec(conn,'TRUNCATE TABLE DataBase.dbo.OutputTable') fastinsert(conn, 'DataBase.dbo.OutputTable', {'CustID', 'ClusterID'}, FinalCluster)

Here is the error message. Can someone please help.

ans =

        Attributes: []
              Data: 0
    DatabaseObject: [1x1 database]
          RowLimit: 0
          SQLQuery: 'TRUNCATE TABLE DataBase.dbo.OutputTable'
           Message: []
              Type: 'Database Cursor Object'
         ResultSet: 0
            Cursor: [1x1 com.mathworks.toolbox.database.sqlExec]
         Statement: [1x1 com.microsoft.sqlserver.jdbc.SQLServerStatement]
             Fetch: 0

Error using com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement/setDouble Java exception occurred: java.lang.OutOfMemoryError: GC overhead limit exceeded

Error in database/fastinsert (line 224) StatementObject.setDouble(j,tmp) %NUMERIC, DECIMAL, REAL

Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.Arrays.copyOf(Unknown Source) at java.util.ArrayList.ensureCapacity(Unknown Source) at java.util.ArrayList.add(Unknown Source) at com.mathworks.mlwidgets.explorer.model.actions.ActionRegistryImpl.getActions(ActionRegistryImpl.java:118) at com.mathworks.mlwidgets.explorer.model.actions.ActionManager$4.run(ActionManager.java:354) at com.mathworks.mwswing.MJUtilities.runOnEventDispatchThread(MJUtilities.java:865) at com.mathworks.mlwidgets.explorer.model.actions.ActionManager.refreshActions(ActionManager.java:346) at com.mathworks.mlwidgets.explorer.model.actions.ActionManager.access$000(ActionManager.java:51) at com.mathworks.mlwidgets.explorer.model.actions.ActionManager$1.stateChanged(ActionManager.java:101) at com.mathworks.util.MulticastChangeListener.stateChanged(MulticastChangeListener.java:26) at com.mathworks.mlwidgets.explorer.widgets.table.FileTable$21.valueChanged(FileTable.java:810) at com.mathworks.widgets.grouptable.GroupingTable.fireSelectionListenersIfEnabled(GroupingTable.java:550) at com.mathworks.widgets.grouptable.GroupingTable.access$300(GroupingTable.java:56) at com.mathworks.widgets.grouptable.GroupingTable$17.valueChanged(GroupingTable.java:539) at javax.swing.DefaultListSelectionModel.fireValueChanged(Unknown Source) at javax.swing.DefaultListSelectionModel.fireValueChanged(Unknown Source) at javax.swing.DefaultListSelectionModel.setValueIsAdjusting(Unknown Source) at com.jidesoft.grid.JideTable$15.setValueIsAdjusting(Unknown Source) at com.mathworks.widgets.grouptable.GroupingTable$25.run(GroupingTable.java:776) at com.mathworks.mwswing.MJUtilities.runOnEventDispatchThread(MJUtilities.java:865) at com.mathworks.widgets.grouptable.GroupingTable.restoreSelection(GroupingTable.java:709) at com.mathworks.widgets.grouptable.GroupingTable.access$1000(GroupingTable.java:56) at com.mathworks.widgets.grouptable.GroupingTable$18.run(GroupingTable.java:574) at com.mathworks.widgets.grouptable.GroupingTable$18.run(GroupingTable.java:558) at com.mathworks.widgets.grouptable.GroupingTableModel.refresh(GroupingTableModel.java:689) at com.mathworks.widgets.grouptable.GroupingTableModel.ungroupOrRegroupRootLevelRows(GroupingTableModel.java:569) at com.mathworks.widgets.grouptable.GroupingTableModel.access$900(GroupingTableModel.java:31) at com.mathworks.widgets.grouptable.GroupingTableModel$7.commitRows(GroupingTableModel.java:516) at com.mathworks.widgets.grouptable.RowListTransactionTarget.finishTransaction(RowListTransactionTarget.java:125) at com.mathworks.widgets.grouptable.GroupingTableTransaction$1.run(GroupingTableTransaction.java:113) at java.awt.event.InvocationEvent.dispatch(Unknown Source) at java.awt.EventQueue.dispatchEvent(Unknown Source) Exception in thread "Explorer NavigationContext request queue" java.lang.OutOfMemoryError: GC overhead limit exceeded

Answer 1

[per isakson]

OutOfMemoryError

Did you try increase the allocation of the java heap memory?

File, Preferences, Java Heap Memory

Answer 2

[Dhruv]

Thanks !