[Nanocubes-discuss] Question about data feeding and updates

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Nanocubes-discuss] Question about data feeding and updates

salivian
For streaming csv like data into nanocubes, please take a look at the most current



if you specify the input file as '-', the script now reads from stdin, therefore you can write a script to query your oracle database and print the data in a csv like format and pipe that into nanocubes.

eg.
youroracleprogram | python csv2Nanocubes.py --latcol=lat --loncol=lon --catcol=category - | ncserve ...... (the rest of the command)

as long as the pipe from your program is open, new data can be piped into the nanocubes  server.

You can also supply your own nanocube file header with --ncheader=<hdrfile> and let the python script to just parse the csv files and transform the data to the nanocube binaries.

At this point the nanocube does not support random deletion of data,  but we are planning to release a new version that supports sliding temporal windows for streaming Nanocubes.

Stay tune, hope this helps!


Horace
Reply | Threaded
Open this post in threaded view
|

Re: [Nanocubes-discuss] Question about data feeding and updates

Ranjith
Hi Horace, thanks alot  for your response.

I tried to do what you suggested, instead of having a script to read
data from Oracle, i used 2 CSV files just to do a POC for append. Had
the script to CAT file1.csv and then wait for 1 min and then CAT
file2.csv. Piped this script to python csv2Nanocubes.py with '-' option
and this is then piped to NC server. But don't find the data getting
appended into nanoserver properly.  In this case i see the script is
completing all the steps including the wait and then sending all the
data in file1 and file2 as one set to csv2Nanocubes.py. Looks i really
batch doesn't work every hour or so.

Tried without my script, by passing both the csv files as input arg to
csv2Nanocubes.py. It worked well, but looks its processing both the
files as one set and streaming to nanoserver at once time. I tried to
insert a wait before processing the second file in csv2Nanocubes.py,  
but then the data is not getting loaded into NC properly.

The scenario I am trying to do for append is as below.

1. Eg. at 9ET I have one set of records (assume 1000) that I want to
send to NC Server. I am able to do that without any issues.
2. at 10 ET I got 100 more records that I need to append to the NC
Server, without stopping the NC Server or webserver  that are already
running as users are using the GUI.
3. at 11 ET few more records to append similar in #2 above and So on as
I will be getting more data every hour or so.

Is this append possible?


Thanks for you help,
Ranjith

_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net
Reply | Threaded
Open this post in threaded view
|

Re: [Nanocubes-discuss] Question about data feeding and updates

sushil@kratin.co.in
Hi,

Is there any update on this ?

I've almost similar requirement. I need to update on some internal events.

Thanks,
Sushil
Reply | Threaded
Open this post in threaded view
|

Re: [Nanocubes-discuss] Question about data feeding and updates

laurolins
Hi Suhil,

The way to do this now is to start a process that will keep piping data to the nanocube server. Something like this:

monitor-new-data-and-feed-to-nc-process | nanocube-leaf -q 29512

The feeding process should never close the pipe to the nanocube child process.

Lauro

> On Nov 26, 2014, at 2:39 AM, "[hidden email]" <[hidden email]> wrote:
>
> Hi,
>
> Is there any update on this ?
>
> I've almost similar requirement. I need to update on some internal events.
>
> Thanks,
> Sushil
>
>
>
> --
> View this message in context: http://nanocubes-discuss.64146.x6.nabble.com/Nanocubes-discuss-Question-about-data-feeding-and-updates-tp113p134.html
> Sent from the nanocubes-discuss mailing list archive at Nabble.com.
>
> _______________________________________________
> Nanocubes-discuss mailing list
> [hidden email]
> http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net

_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net
Reply | Threaded
Open this post in threaded view
|

Re: [Nanocubes-discuss] Question about data feeding and updates

sushil@kratin.co.in
Hi Lauro,

Just want to be sure that I understood you correctly. 
So simple  bash script will do something like this.

echo "name,address,locality,district,state,latitude,longitude,time" | nanocube-binning-csv --sep=',' --timecol='time' --latcol='latitude' --loncol='longitude' --catcol='locality' - | nanocube-leaf -q 29512

echo "Mr. Dayaram Diwakar Deshmukh,\"At. Kolar, Post. Butibori, Tal. Dist. Nagpur\",Butibori,Nagpur,Maharashtra,20.9311933,79.0056553,05/06/2014 06:25:00 PM"  - | nanocube-leaf -q 29512

Can you please confirm this ?

Thanks,
Sushil

On Wed, Nov 26, 2014 at 5:43 PM, Lauro Lins <[hidden email]> wrote:
Hi Suhil,

The way to do this now is to start a process that will keep piping data to the nanocube server. Something like this:

monitor-new-data-and-feed-to-nc-process | nanocube-leaf -q 29512

The feeding process should never close the pipe to the nanocube child process.

Lauro

> On Nov 26, 2014, at 2:39 AM, "[hidden email]" <[hidden email]> wrote:
>
> Hi,
>
> Is there any update on this ?
>
> I've almost similar requirement. I need to update on some internal events.
>
> Thanks,
> Sushil
>
>
>
> --
> View this message in context: http://nanocubes-discuss.64146.x6.nabble.com/Nanocubes-discuss-Question-about-data-feeding-and-updates-tp113p134.html
> Sent from the nanocubes-discuss mailing list archive at Nabble.com.
>
> _______________________________________________
> Nanocubes-discuss mailing list
> [hidden email]
> http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net

_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net


_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net