java


Reading from a file, running out of memory


I was just recently asked an interview question that held to deal with reading from a CSV file and summing up entries in certain cells. When asked to optimize it, I couldn't answer how to deal with the case of running out of memory if we were given a CSV of size say 100 gigs.
In Java, how exactly does reading from a file work? How do we know when something is too big? How do we deal with that? I was told that you could pass in the intermediate reader object instead of trying to read the entire thing?
The interviewer gave you a hint - BufferedReader. It is an efficient choice for reading a large file line by line.
Small example:
String line;
BufferedReader br = new BufferedReader("c:/test.txt");
while ((line= br.readLine()) != null) {
//do processing
}
br.close();
Here is the documentation
There are several ways to read from a file in Java, some of them involve keeping all of the files lines (or data) in memory as you "read" the data delimited by something like a newline character (reading line by line for example).
For large files you want to process smaller bits at a time using the Scannerclass (or something like it to read specific bytes at a time).
Sample code:
FileInputStream inputStream = new FileInputStream(path);
Scanner sc = new Scanner(inputStream, "UTF-8");
while (sc.hasNextLine()) {
String line = sc.nextLine();
// System.out.println(line);
}
You can use RandomAccessFile to read the file. It may not be the best solution though.

Related Links

Android TCP Client connection freezes iBeacon ranging
How should I fix my code to stop getting errors with the test code?
jni.h: no such file or direcotory on chroot ubuntu
Generating XML with an arbitrary number of nodes
how do I access a BlockingQueue from either Swing or the Console?
How to get conditions in JIRA from transition
authentication — making username and password global in java
Lower running program to administrator
NPath Complexity
How can I avoid our users not having multi tab?
Convert Java string array or JSON string to Javascript Array (Parse.com Cloud Code and Mandrill)
Regular Expression for filetype
Spring Security and REST API?
Spring MVC: use different JSR-303 validators on the same bean?
Smartphone NFC function desktop application
Get Cell according to its number in Board data structure Cell[][] without passing 2 FOR loops

Categories

HOME
ms-access
proxy
react-router
webstorm
razor
paw-app
youtube-api-v3
icloud
disassembler
applepay
gitpitch
cloudkit
dax
qt-creator
nstableview
google-apps-marketplace
circular-dependency
gz
windows-phone-7
orchardcms
ibm-odm
lucene.net
visjs
csrf-protection
realex-payments-api
zurb-foundation-6
wijmo
shopware
pepper
log4js-node
http-status-code-503
create-table
catel
google-qpx-express-api
typed.js
hybridauth
repo
normal-distribution
file-format
x11-forwarding
android-tabhost
hp-ux
acoustics
s
websphere-mq-fte
prolog-setof
promela
togetherjs
setter
tactic
directory-structure
magma
font-size
query-performance
gridpane
httplistener
nsfilemanager
websitepanel
holder.js
pagedlist
superstack
design-by-contract
associative-array
sdhc
t4mvc
cannon.js
kcachegrind
twirl
sorl-thumbnail
camanjs
network-interface
javafx-webengine
codeigniter-routing
wordpress-theme-customize
neolane
document-classification
tld
sgen
dml
marmalade
kgdb
mbr
ruby-datamapper
buster.js
lcs
robotics-studio
browser-detection
path-separator
quickdialog
funscript
jspinclude
asyncfileupload
subscript
data-loss
hirefire
mod-auth
netbeans-7.1
inotifycollectionchanged
blitz++
infrastructure
eqatec
post-build
msdev

Resources

Database Users
RDBMS discuss
Database Dev&Adm
javascript
java
csharp
php
android
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App