Re: [Nanocubes-discuss] Horace, short question about nanocube...

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [Nanocubes-discuss] Horace, short question about nanocube...

salivian
The nanocubes server digests data in a batch mode (for every 100 or 1000s records)  If you want immediate digestion of data, try -b 1 so it will digest every record, but this will hurt the performance a little bit.

Thanks!

Horace

On Sat, Jan 24, 2015 at 1:42 AM, Vjeran Marcinko <[hidden email]> wrote:
Hi Horace,

I'm the one who developed recently java DMP encoder, and was
experimenting a bit with feeding nanocube from java process, and
noticed that last records don't get fed into nanocube, so was
wondering if nanocube does some kind of buffering of the input?

I'm asking this because flushing output buffer to nanocube doesn't
help, but when I close the stream to nanocube, then the records show
up in nanocube..

Regards,
Vjeran


_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net
Reply | Threaded
Open this post in threaded view
|

Re: [Nanocubes-discuss] Horace, short question about nanocube...

Vjeran Marcinko
Hmmm, I enabled this "-b 1" flag, streamed all my data into nanocube,
noticed how whole feeding is al least couple of times slower now,
flushed my output buffer towards nanocube, and still don't receive
some amount of last records.

On Sat, Jan 24, 2015 at 8:46 PM, Horace Ip <[hidden email]> wrote:

> The nanocubes server digests data in a batch mode (for every 100 or 1000s
> records)  If you want immediate digestion of data, try -b 1 so it will
> digest every record, but this will hurt the performance a little bit.
>
> Thanks!
>
> Horace
>
> On Sat, Jan 24, 2015 at 1:42 AM, Vjeran Marcinko <[hidden email]>
> wrote:
>>
>> Hi Horace,
>>
>> I'm the one who developed recently java DMP encoder, and was
>> experimenting a bit with feeding nanocube from java process, and
>> noticed that last records don't get fed into nanocube, so was
>> wondering if nanocube does some kind of buffering of the input?
>>
>> I'm asking this because flushing output buffer to nanocube doesn't
>> help, but when I close the stream to nanocube, then the records show
>> up in nanocube..
>>
>> Regards,
>> Vjeran
>
>

_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net
Reply | Threaded
Open this post in threaded view
|

Re: [Nanocubes-discuss] Horace, short question about nanocube...

Vjeran Marcinko
Wanted to mention that in real production this would hardly present a
problem because if we're talking millions/billions of records, it
hardly makes any difference if last 100 are missing, but still
puzzling. Again, I'm just mentioning this happening when I keep pipe
to nanocube open constantly, without closing it.

On Sun, Jan 25, 2015 at 6:53 AM, Vjeran Marcinko <[hidden email]> wrote:

> Hmmm, I enabled this "-b 1" flag, streamed all my data into nanocube,
> noticed how whole feeding is al least couple of times slower now,
> flushed my output buffer towards nanocube, and still don't receive
> some amount of last records.
>
> On Sat, Jan 24, 2015 at 8:46 PM, Horace Ip <[hidden email]> wrote:
>> The nanocubes server digests data in a batch mode (for every 100 or 1000s
>> records)  If you want immediate digestion of data, try -b 1 so it will
>> digest every record, but this will hurt the performance a little bit.
>>
>> Thanks!
>>
>> Horace
>>
>> On Sat, Jan 24, 2015 at 1:42 AM, Vjeran Marcinko <[hidden email]>
>> wrote:
>>>
>>> Hi Horace,
>>>
>>> I'm the one who developed recently java DMP encoder, and was
>>> experimenting a bit with feeding nanocube from java process, and
>>> noticed that last records don't get fed into nanocube, so was
>>> wondering if nanocube does some kind of buffering of the input?
>>>
>>> I'm asking this because flushing output buffer to nanocube doesn't
>>> help, but when I close the stream to nanocube, then the records show
>>> up in nanocube..
>>>
>>> Regards,
>>> Vjeran
>>
>>

_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net
Reply | Threaded
Open this post in threaded view
|

Re: [Nanocubes-discuss] Horace, short question about nanocube...

salivian
I think it's possible that we can add a timer and trigger an insertion of the records in buffer when no new record has arrived after a certain amount of time. Let me talk to my colleague Lauro on Monday about it but it may not be a priority of the new features. Please let us know if you have a compelling example of this being necessary :)

Thanks!

Horace

On Sun, Jan 25, 2015 at 12:58 AM, Vjeran Marcinko <[hidden email]> wrote:
Wanted to mention that in real production this would hardly present a
problem because if we're talking millions/billions of records, it
hardly makes any difference if last 100 are missing, but still
puzzling. Again, I'm just mentioning this happening when I keep pipe
to nanocube open constantly, without closing it.

On Sun, Jan 25, 2015 at 6:53 AM, Vjeran Marcinko <[hidden email]> wrote:
> Hmmm, I enabled this "-b 1" flag, streamed all my data into nanocube,
> noticed how whole feeding is al least couple of times slower now,
> flushed my output buffer towards nanocube, and still don't receive
> some amount of last records.
>
> On Sat, Jan 24, 2015 at 8:46 PM, Horace Ip <[hidden email]> wrote:
>> The nanocubes server digests data in a batch mode (for every 100 or 1000s
>> records)  If you want immediate digestion of data, try -b 1 so it will
>> digest every record, but this will hurt the performance a little bit.
>>
>> Thanks!
>>
>> Horace
>>
>> On Sat, Jan 24, 2015 at 1:42 AM, Vjeran Marcinko <[hidden email]>
>> wrote:
>>>
>>> Hi Horace,
>>>
>>> I'm the one who developed recently java DMP encoder, and was
>>> experimenting a bit with feeding nanocube from java process, and
>>> noticed that last records don't get fed into nanocube, so was
>>> wondering if nanocube does some kind of buffering of the input?
>>>
>>> I'm asking this because flushing output buffer to nanocube doesn't
>>> help, but when I close the stream to nanocube, then the records show
>>> up in nanocube..
>>>
>>> Regards,
>>> Vjeran
>>
>>


_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net
Reply | Threaded
Open this post in threaded view
|

Re: [Nanocubes-discuss] Horace, short question about nanocube...

Vjeran Marcinko
This is definitely not something necessary, at least from current
point of view. I just stumbled upon this issue when I was developing
continuous streaming from java process, and left wondering whether it
was some issue with my java integration or nanocube had this input
buffering enabled.

On Sun, Jan 25, 2015 at 7:04 AM, Horace Ip <[hidden email]> wrote:

> I think it's possible that we can add a timer and trigger an insertion of
> the records in buffer when no new record has arrived after a certain amount
> of time. Let me talk to my colleague Lauro on Monday about it but it may not
> be a priority of the new features. Please let us know if you have a
> compelling example of this being necessary :)
>
> Thanks!
>
> Horace
>
> On Sun, Jan 25, 2015 at 12:58 AM, Vjeran Marcinko <[hidden email]>
> wrote:
>>
>> Wanted to mention that in real production this would hardly present a
>> problem because if we're talking millions/billions of records, it
>> hardly makes any difference if last 100 are missing, but still
>> puzzling. Again, I'm just mentioning this happening when I keep pipe
>> to nanocube open constantly, without closing it.
>>
>> On Sun, Jan 25, 2015 at 6:53 AM, Vjeran Marcinko <[hidden email]>
>> wrote:
>> > Hmmm, I enabled this "-b 1" flag, streamed all my data into nanocube,
>> > noticed how whole feeding is al least couple of times slower now,
>> > flushed my output buffer towards nanocube, and still don't receive
>> > some amount of last records.
>> >
>> > On Sat, Jan 24, 2015 at 8:46 PM, Horace Ip <[hidden email]> wrote:
>> >> The nanocubes server digests data in a batch mode (for every 100 or
>> >> 1000s
>> >> records)  If you want immediate digestion of data, try -b 1 so it will
>> >> digest every record, but this will hurt the performance a little bit.
>> >>
>> >> Thanks!
>> >>
>> >> Horace
>> >>
>> >> On Sat, Jan 24, 2015 at 1:42 AM, Vjeran Marcinko <[hidden email]>
>> >> wrote:
>> >>>
>> >>> Hi Horace,
>> >>>
>> >>> I'm the one who developed recently java DMP encoder, and was
>> >>> experimenting a bit with feeding nanocube from java process, and
>> >>> noticed that last records don't get fed into nanocube, so was
>> >>> wondering if nanocube does some kind of buffering of the input?
>> >>>
>> >>> I'm asking this because flushing output buffer to nanocube doesn't
>> >>> help, but when I close the stream to nanocube, then the records show
>> >>> up in nanocube..
>> >>>
>> >>> Regards,
>> >>> Vjeran
>> >>
>> >>
>
>

_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net