openrefine


OpenRefine - Fill between cells but not at the end of the list


I have a list of stock prices for several stocks. Some of the values are missing due to weekends, holidays and probably other reasons.
The gaps are not consistent. Some are two days and some are more than that.
I want to fill the gaps with the last known value but not at the end of the list.
I have tried in Excel to test a few cells below and if it's now empty, do the fill. The problem is that due to the inconsistency of the gaps, it's a tedious task to change the function for all the cases.
Is there a way to test for the end of a list?
UPDATE - added a screenshot.
See this screenshot. I want to fill where the blue dots are. The red dots are at the end of the list and I don't want to fill those cells.
I am looking for a way to detect the end of the list and stop the filling when the end is detected.
I think this is pretty difficult in OpenRefine and probably a different tool would work better. The main issue is that OpenRefine does not offer the ability to easily work across rows so 'summing a column' (or part of a column) is tricky - this is mentioned in https://github.com/OpenRefine/OpenRefine/issues/200
However, you can do this by forcing OpenRefine in Record mode with the whole project containing a single record. Once you've done this you can access all values in a column using syntax like:
row.record.cells["Column name"].value
This gives an array of all the non-blank values in the column. Since this ignores blank values, in order to have a true view of the values in the column you have to fill in blank cells with a value.
So I think you could probably achieve what you want as follows:
For each column you are going to work with do a cell transform to put a dummy value in empty cells - e.g. if(isBlank(value),"null",value)
Create a new column at the start of your project and put a single value in the very first cell in that column
Switch to Record mode
At this point you should have a single 'Record' in your project - e.g.
You can now access all cells in a column using syntax like row.record.cells["Column 1"].value. You can combine this with 'forRange' to iterate through the contents of this array, using the row.index as the marker for the current row.
I used the following formula to add a new column to the project:
with(row.record.cells["Column 1"].value,w,if(forRange(row.index,w.length(),1,i,w[i].toNumber()).sum()>0,"a","b"))
Then...
Change back to 'Row' mode
Remove the 'null' placeholder from the original column
Create a facet on the 'fill filter' column
In my case I filter to 'a'
Use the 'fill down' option
Remove the filter
And remove the 'record' column
Rather a long winded way of doing it to say the least, but so far I've not been able to find anything better while not going outside OpenRefine. I'm guessing you could probably compress steps 5-11 into a single step or smaller number of steps.
If you want to access the array of cell values using Jython as suggested by iMitwe you need to use:
row["record"]["cells"]["Column 1"]["value"]
instead of
row.record.cells["Column 1"].value
(step 5)
I am doing this on the top of my head, but I think your best chance my be using the fill down option in record mode:
first move your column to the first column and switch to record mode.
then use the following GREL: row.record.cells["data"].value[-1] where data is the name of your column
The [-1] will take the last value and fill the blank. For the case with the red dot, since there is no value it should remains empty. Let us know how it goes.
Unless there's something I am missing or not seeing...
I would have just sorted reverse (date ascending) on the Date column, then individually use Fill Down on each column, except for that last column where you could then use a Date facet on your column Date to specify the exact Date range you wanted to work with, then fill down on that last column, then remove the Date range facet.

Related Links

Appending a specific string in GREL
How to extract ONLY lat, lon values for node “osm_type”:“node” in a Nominatim response using Google Refine
Replace null cell with space character
Open refine by google on private data
Openrefine not working as expected
Open Refine Error Uploading Data?
Open Refine / Google Refine - edit cells in multiple columns
Open Refine : Reconciliation with Freebase data based on ORganization Name
Keep newest duplicate row depending on multiple Columns
multiple filters in google openrefine
Where does openrefine store projects?
Domain Names to Webpage Titles in OpenRefine
How does one run Google refine on a different port than 3333?
OpenRefine - Cross-column clustering
Grel to apply to ALL columns or current column
Open Refine / Google Refine - Remove blank cells in a column

Categories

HOME
twitter
osgi
fluentd
kde
getelementsbytagname
survey
flyway4
gorm
tomcat6
spring-xd
modelica
telephony
ef-migrations
text-rendering
samoa
functional-testing
excel-2007
shopware
status
entitlements
fopen
assistant
elasticsearch-net
compatibility
vision
bpel
http-status-code-503
catel
http-digest
file-format
swisscomdev
hp-ux
gzipstream
estimote
btrace
winscp-net
hls.js
mplayer
windows-iot-core-10
dynamics-sl
objective-c-swift-bridge
infix-notation
thrust
bstr
prettytensor
sigabrt
actionbardrawertoggle
superstack
essence
itextpdf
spim
intellij-14
iis-arr
clipperlib
csquery
block-device
coldbox
winddk
mdt
picturefill
resty-gwt
bluepill
appconkit
subscript
table-footer
file-comparison
yui-datatable
krl
gamequery
dmx512
jquery-ui-droppable
multi-tier
private-members
zune

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App