java


Getting Google search results with JSoup


I'm trying to get a list of Google search results via JSoup. The method I'm currently using works perfectly fine for the first page (n), but for n+1 pages it does not work so well. Here is how I'm getting the first page:
doc = Jsoup.connect(search).userAgent("Chrome").get();
links = doc.getElementsByClass("r");
The search string would contain something like: https://www.google.com/search?q=apple for the first page. Then my code looks like this for n+1 pages:
for(int i = 1; i <= pages; i++){
search = "https://www.google.com/#q=" + keyword + "&start=" + (i*10);
doc = Jsoup.connect(search).userAgent("Mozilla").get();
links.addAll(doc.getElementsByClass("r"));
}
The search for n+1 pages would look something like:https://www.google.com/#q=apple&start=10.
The main issue that I'm having is that doc.getElementsByClass("r") in the n+1 searches contains no elements. This means that the class r does not exist in what JSoup returns. I verified this by searching through doc.toString(). Does anyone have any suggestions?
Thank you!

Related Links

how to use the Assist API android m
Missing return statement error with method that has a return in for loop
How can I know whether a java Thread is completed when I use Thread class? [duplicate]
Apache Phoenix (4.3.1 and 4.4.0-HBase-0.98) on Spark 1.3.1 ClassNotFoundException
What does this exception in Android Studio core mean?
autosuggestion is not working in selenium
Prevent invalid date from getting converted into date of next month in jdk6?
Eclipse Java Android - This app won't run without Google Play Services
JavaFX + eGalax Touchscreen + Raspberry Pi
Calling a Stored Procedure Using Oracle SQL Developer
Export DXL to Richtext item causes 64k errors
How to fix [Error] realm = plugin>org.apache.maven.plugins:maven-compiler-plugin:3.x (Happened while working on Spring Boot)
Defining findbugs detector using the existing plugins
Selfcontained non executable jar with maven
Escape \n and \" from Json Response
how to separate 'xml' tag from a given String

Categories

HOME
java
hive
gerrit
azure-data-factory
json.net
layout
sd-card
yarn
rubygems
acquia
bs4
portia
metatrader4
adobe-analytics
export-to-csv
etl
php-7.1
usergrid
phaser
visual-studio-cordova
windows-server-2012
seaborn
after-effects
graphlab
highlight.js
apache-metamodel
undefined
nodatime
physics-engine
tdd
poltergeist
react-css-modules
filezilla
ping
flink-streaming
uisplitview
pdb
caret
protovis
avcapturesession
asset-pipeline
amazon-kinesis-kpl
sequential
hexo
restlet
serverside-rendering
android-browser
fedex
sql-server-agent
unixodbc
gpx
node-sass
eclipse-gef
scorm
logfiles
react-scripts
drupal-6
dtexec
smart-table
setuptools
ado.net-entity-data-model
thrust
nsfilemanager
clang-static-analyzer
react-native-listview
skype4py
hill-climbing
pagerank
revolution-r
retina
wireshark-dissector
wyam
ionic
cakephp-3.1
responsive-images
microbenchmark
jsapi
javax.mail
camanjs
codeigniter-routing
asp.net-web-api-odata
client-side-templating
php-5.4
phpthumb
wp-query
yorick
aqtime
wsdl-2.0
browser-detection
cisco-jtapi
subscript
libc++
w3wp.exe
routedevent
libs
privilege
temporal-database
anti-piracy
uiq3

Resources

Encrypt Message



code
soft
python
ios
c
html
jquery
cloud
mobile