java


Java Reading large files into byte array chunk by chunk


So I've been trying to make a small program that inputs a file into a byte array, then it will turn that byte array into hex, then binary. It will then play with the binary values (I haven't thought of what to do when I get to this stage) and then save it as a custom file.
I studied a lot of internet code and I can turn a file into a byte array and into hex, but the problem is I can't turn huge files into byte arrays (out of memory).
This is the code that is not a complete failure
public void rundis(Path pp) {
byte bb[] = null;
try {
bb = Files.readAllBytes(pp); //Files.toByteArray(pathhold);
System.out.println("byte array made");
} catch (Exception e) {
e.printStackTrace();
}
if (bb.length != 0 || bb != null) {
System.out.println("byte array filled");
//send to method to turn into hex
} else {
System.out.println("byte array NOT filled");
}
}
I know how the process should go, but I don't know how to code that properly.
The process if you are interested:
Input file using File
Read the chunk by chunk of the file into a byte array. Ex. each byte array record hold 600 bytes
Send that chunk to be turned into a Hex value --> Integer.tohexstring
Send that hex value chunk to be made into a binary value --> Integer.toBinarystring
Mess around with the Binary value
Save to custom file line by line
Problem:: I don't know how to turn a huge file into a byte array chunk by chunk to be processed.
Any and all help will be appreciated, thank you for reading :)
To chunk your input use a FileInputStream:
Path pp = FileSystems.getDefault().getPath("logs", "access.log");
final int BUFFER_SIZE = 1024*1024; //this is actually bytes
FileInputStream fis = new FileInputStream(pp.toFile());
byte[] buffer = new byte[BUFFER_SIZE];
int read = 0;
while( ( read = fis.read( buffer ) ) > 0 ){
// call your other methodes here...
}
fis.close();
To stream a file, you need to step away from Files.readAllBytes(). It's a nice utility for small files, but as you noticed not so much for large files.
In pseudocode it would look something like this:
while there are more bytes available
read some bytes
process those bytes
(write the result back to a file, if needed)
In Java, you can use a FileInputStream to read a file byte by byte or chunk by chunk. Lets say we want to write back our processed bytes. First we open the files:
FileInputStream is = new FileInputStream(new File("input.txt"));
FileOutputStream os = new FileOutputStream(new File("output.txt"));
We need the FileOutputStream to write back our results - we don't want to just drop our precious processed data, right? Next we need a buffer which holds a chunk of bytes:
byte[] buf = new byte[4096];
How many bytes is up to you, I kinda like chunks of 4096 bytes. Then we need to actually read some bytes
int read = is.read(buf);
this will read up to buf.length bytes and store them in buf. It will return the total bytes read. Then we process the bytes:
//Assuming the processing function looks like this:
//byte[] process(byte[] data, int bytes);
byte[] ret = process(buf, read);
process() in above example is your processing method. It takes in a byte-array, the number of bytes it should process and returns the result as byte-array.
Last, we write the result back to a file:
os.write(ret);
We have to execute this in a loop until there are no bytes left in the file, so lets write a loop for it:
int read = 0;
while((read = is.read(buf)) > 0) {
byte[] ret = process(buf, read);
os.write(ret);
}
and finally close the streams
is.close();
os.close();
And thats it. We processed the file in 4096-byte chunks and wrote the result back to a file. It's up to you what to do with the result, you could also send it over TCP or even drop it if it's not needed, or even read from TCP instead of a file, the basic logic is the same.
This still needs some proper error-handling to work around missing files or wrong permissions but that's up to you to implement that.
A example implementation for the process method:
//returns the hex-representation of the bytes
public static byte[] process(byte[] bytes, int length) {
final char[] hexchars = "0123456789ABCDEF".toCharArray();
char[] ret = new char[length * 2];
for ( int i = 0; i < length; ++i) {
int b = bytes[i] & 0xFF;
ret[i * 2] = hexchars[b >>> 4];
ret[i * 2 + 1] = hexchars[b & 0x0F];
}
return ret;
}
Edit2 small code typo

Related Links

web app on machine (on local computer works correctly)
Is it possible to sort an arraylist/hashset without the use of a comparator interface?
Java ~ Read external programs memory
Mule - accessing the exception stack and logging a particular part
setSelectionRange is not working properly
How to store user defined class objects in Android App for future use and manipulation?
How may String Objects are created in java (By the following code)
Spring no mapping found for HTTP request with URI[require.js] in DispatcherServlet
Set relative path for LogBack Appender
Java Runtime.exec() not sending emails as from the command line
How can I parse freemarker template's following structure?
Fail when I try to send JSON Object to server
Concat arrays of same type [duplicate]
How to Set a HashMap from jsp to Action
Linked data Jena Eclipse Dbpedia
how to implement the instagram login Oauth using javascript and java

Categories

HOME
image-processing
plone
electron
react-redux
icloud
ckeditor
mvvmcross
ezpublish
android-youtube-api
medical
slurm
spring-xd
izpack
floating-action-button
timeout
circular-dependency
quartz-scheduler
clojurescript
flux
nhibernate-envers
swingx
custom-wordpress-pages
csrf-protection
emgucv
android-widget
buildbot
url-scheme
tooltipster
maxmind
assistant
saas
webkitspeechrecognition
vision
bosh
phonegap
.net-4.0
semantic-versioning
http-referer
wixsharp
code-search-engine
mesos-chronos
galen
texmaker
firebase-admin
reactive-cocoa-5
long-polling
avconv
xml-documentation
youcompleteme
atomicity
aurelia-validation
brightcove
smart-table
root-framework
nested-sets
underscore.js-templating
master-slave
cannon.js
0xdbe
sortedlist
census
dundas
web-controls
e10s
geonetwork
system.reflection
qpainter
websocket4net
bitcoinj
java.nio.file
centos5
imdbpy
xsockets.net
dexterity
tws
qt-faststart
comaddin
delphi-6
free-variable
distutils
hamiltonian-cycle
hgsubversion
mod-auth
xmlspy
xdomainrequest
locationlistener
celltable
visitor-statistic
h.323
user-friendly

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html