java


using kafka-streams to conditionally sort a json input stream


I am new to developing kafka-streams applications. My stream processor is meant to sort json messages based on a value of a user key in the input json message.
Message 1: {"UserID": "1", "Score":"123", "meta":"qwert"}
Message 2: {"UserID": "5", "Score":"780", "meta":"mnbvs"}
Message 3: {"UserID": "2", "Score":"0", "meta":"fghjk"}
I have read here Dynamically connecting a Kafka input stream to multiple output streams that there is no dynamic solution.
In my use-case I know the user keys and output topics that I need to sort the input stream. So I am writing separate processor applications specific to each user where each processor application matches a different UserID.
All the different stream processor applications read from the same json input topic in kafka but each one only writes the message to a output topic for a specific user if the preset user condition is met.
public class SwitchStream extends AbstractProcessor<String, String> {
#Override
public void process(String key, String value) {
HashMap<String, String> message = new HashMap<>();
ObjectMapper mapper = new ObjectMapper();
try {
message = mapper.readValue(value, HashMap.class);
} catch (IOException e){}
// User condition UserID = 1
if(message.get("UserID").equals("1")) {
context().forward(key, value);
context().commit();
}
}
public static void main(String[] args) throws Exception {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "sort-stream-processor");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
TopologyBuilder builder = new TopologyBuilder();
builder.addSource("Source", "INPUT_TOPIC");
builder.addProcessor("Process", SwitchStream::new, "Source");
builder.addSink("Sink", "OUTPUT_TOPIC", "Process");
KafkaStreams streams = new KafkaStreams(builder, props);
streams.start();
}
}
Question 1:
Is it possible to achieve the same functionality easily using the High-Level Streams DSL instead if the Low-Level Processor API? (I admit I found it harder understand and follow the other online examples of the High-Level Streams DSL)
Question 2:
The input json topic is getting input at a high rate 20K-25K EPS. My processor applications don't seem to be able to keep pace with this input stream. I have tried deploying multiple instances of each process but the results are nowhere close to where I want them to be. Ideally each processor instance should be able to process 3-5K EPS.
Is there a way to improve my processor logic or write the same processor logic using the high level streams DSL? would that make a difference?
You can do this in high-level DSL via filter() (you effectively implemented a filter as you only return a message if it's userID==1). You could generalize this filter pattern, by using KStream#branch() (see the docs for further details: http://docs.confluent.io/current/streams/developer-guide.html#stateless-transformations). Also read the JavaDocs: http://kafka.apache.org/0102/javadoc/index.html?org/apache/kafka/streams
KStreamBuilder builder = new KStreamBuilder();
builder.stream("INPUT_TOPIC")
.filter(new Predicate() {
#Overwrite
boolean test(String key, String value) {
// put you processor logic here
return message.get("UserID").equals("1")
}
})
.to("OUTPUT_TOPIC");
About performance. A single instance should be able to process 10K+ records. It's hard to tell without any further information what the problem might be. I would recommend to ask at Kafka user list (see http://kafka.apache.org/contact)

Related Links

Bind Hibernate entity to table at runtime
Can I enable bluetooth on a mobile device with java?
jsp:include , performance, modularization, alternatives and best practices, Part 96
Don't understand the constructor of my View
Zero-copy message-passing on the JVM
Using an empty file to have one application instance
How can I specify the mainClass in the manifest with minijar-maven-plugin?
Configure EJB3 with web.xml like parameters
ajax redirect not working
4 threads as services in Android
Issues with add JButton in the table
Hibernate #OneToOne mapping with a #Where clause
How can I get code assist for jQuery within a JSPX file in eclipse?
Converting to dalvik format failed when I try to add external Jars
Eclipse: Frustration with Java 1.7 (unbound library)
Retrieving Session ID with Spring Security

Categories

HOME
listview
image-processing
adb
sqlite-net-extensions
leon
dxl
ip
ravendb
add
google-translate
numeral.js
collectd
spring-tool-suite
dtrace
ibm-odm
iolanguage
progressive-web-apps
tostring
finite-automata
jsdoc
flux
beyondcompare
alpine
one-hot-encoding
riot.js
abi
facebook-instant-articles
tapestry
tibco-mdm
one-to-many
apache-commons-io
dynamic-featured-image
phpfox
windowbuilder
gitignore
srcset
iframe-resizer
bosh
google-cloud-endpoints-v2
semantic-versioning
ioio
calibre
dartium
lto
sharefile
tasker
ensembles
galen
google-closure
glew
segment
scorm
outlook-2013
osx-mavericks
parentheses
youcompleteme
worksheet
pnotify
pebble-watch
nativeapplication
kbuild
pg-dump
chord-diagram
crosswalk-runtime
mediaelement
word-vba-mac
httplistener
transmitfile
contact-list
qgraphicsview
pervasive-sql
t4mvc
apache-commons-fileupload
composite
nsmutabledictionary
titanium-modules
preferences
xojo
contenttype
wp-query
uv-mapping
farseer
coverflow
quartz-graphics
easy-install
shellexecute
sqlperformance
sharp-repository
ember-app-kit
cassini-dev
distutils
browser-detection
path-separator
bitsharp
word-processor
pysimplesoap
hosts-file
pendrive
asyncfileupload
hgsubversion
doh
data-loss
propagation
xsdobjectgen
multi-tier
microsoft-virtualization
exchange-server-2003

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App