java


Reading from a file, running out of memory


I was just recently asked an interview question that held to deal with reading from a CSV file and summing up entries in certain cells. When asked to optimize it, I couldn't answer how to deal with the case of running out of memory if we were given a CSV of size say 100 gigs.
In Java, how exactly does reading from a file work? How do we know when something is too big? How do we deal with that? I was told that you could pass in the intermediate reader object instead of trying to read the entire thing?
The interviewer gave you a hint - BufferedReader. It is an efficient choice for reading a large file line by line.
Small example:
String line;
BufferedReader br = new BufferedReader("c:/test.txt");
while ((line= br.readLine()) != null) {
//do processing
}
br.close();
Here is the documentation
There are several ways to read from a file in Java, some of them involve keeping all of the files lines (or data) in memory as you "read" the data delimited by something like a newline character (reading line by line for example).
For large files you want to process smaller bits at a time using the Scannerclass (or something like it to read specific bytes at a time).
Sample code:
FileInputStream inputStream = new FileInputStream(path);
Scanner sc = new Scanner(inputStream, "UTF-8");
while (sc.hasNextLine()) {
String line = sc.nextLine();
// System.out.println(line);
}
You can use RandomAccessFile to read the file. It may not be the best solution though.

Related Links

Find repeating substring of Length N [closed]
isClickable with Selendroid
Java unicode fonts
How to add similar objects into an array? [closed]
Changing views within an activity
Conditions in while loop
Android RSA and node.js RSA Encryption/Decryption
Which one to choose between SAX and STAX to read large xml files
Why do we need Set and Map for Java Enum
Java manage a process that doesn't exit automatically
Ribbon Load Balancer cannot see list of servers
how to decode bson by java or clojure
ScrollPane inner size limits
Line break on eclipse [closed]
Reading a macro file and writing a excel file inside java application using poi
ICC pin encipherment (VISA)

Categories

HOME
xamarin
arduino-uno
push-notification
hashmap
sqlite3
jxls
disassembler
angular-ui-bootstrap
windows-server
constraint-programming
slick-slider
vault
timeout
designer
pc
angular-ui
beyondcompare
openedx
immutable.js
riot.js
orleans
kvc
numerical-methods
filezilla
ping
google-cloud-nl
airconsole
create-table
vsts-build-task
uiswipegesturerecognizer
wixsharp
git-merge
webix-treetable
unobtrusive-validation
angularjs-factory
azure-ml
email-templates
quadratic-programming
google-closure
http-live-streaming
convertapi
sas-jmp
executenonquery
hibernate-tools
react-scripts
dotnetzip
fancybox-2
lift-json
dtexec
qtwebview
bind9
wdf
angular-cache
intrusion-detection
persist
ios4
photobucket
r-tree
qdialog
nsight
mmc
pretty-print
device-orientation
c++03
ng-animate
socketexception
bitcoinj
typeof
navigationservice
funcunit
apc
easy-install
html4
random-seed
mqx
clipper
selected
dice
whoosh
zpt
iphone-web-app
infrastructure
gamma
vc90
meego
noscript
project-hosting
mdac
ugc
signal-handling

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App