Quantcast

[Nanocubes-discuss] Updated javascript

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Nanocubes-discuss] Updated javascript

Alex Bongiovanni
I've now been able to start up nanocubes with all sorts of data and query them to my heart's content (though the number of queries that I know is remarkably limited).  So, I've been trying to get a visualization to load, using a subset of the BrightKite data.  The cube loads fine, and can be queried, but when I load the page I get all kinds of Javascript errors.

After a bit of investigation (having never before written any javascript it took a while) what I have concluded is that I shouldn't be querying a cube built from the master branch with a page written for a cube from the 1.0 branch.

So, I started reworking the javascript as best as I can, but it looks like some of the queries have actually been changed (schema_json to schema being one of them).  So I was wondering if you had a newer version of the javascript files that you would be willing to share (and if not that's understandable), or if you could point me in the right direction in the src to figure this out for myself.

--
Alex Bongiovanni
University of Maryland
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Nanocubes-discuss] Updated javascript

laurolins
Hi Alex,

> So, I started reworking the javascript as best as I can, but it looks like some of the queries have actually been changed (schema_json to schema being one of them).  So I was wondering if you had a newer version of the javascript files that you would be willing to share (and if not that's understandable), or if you could point me in the right direction in the src to figure this out for myself.


The file "./js/nanocube2.js” on the master branch has code for querying a “master” nanocube, but we still haven’t pushed javascript code for full demos (like the demos currently on nanocubes.net) using this new query API.

Lauro

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Nanocubes-discuss] Updated javascript

Alex Bongiovanni
Alright, thank you, that's been helping a bit.  Could you maybe explain to me the new syntax for queries?  The examples are good but don't fully explain what each query does, so I'm trying to inference from the paper but I'm not sure if I'm getting the right results.  My understanding is that:

- a "time" query is: .../query/@time=<start bin>:<bucket size>:<number of buckets>  
- a "pos" query is: .../query/@pos=qaddr(x position, y position, zoom level)

But a position query of (0,0,0) gives me the same result as (22485;52342;17).  

And why is there a +8 in the one example?: .../@pos=qaddr(0,1,1)+8/device=0+0?



On Thu, Jan 23, 2014 at 9:34 PM, Lauro Lins <[hidden email]> wrote:
Hi Alex,

> So, I started reworking the javascript as best as I can, but it looks like some of the queries have actually been changed (schema_json to schema being one of them).  So I was wondering if you had a newer version of the javascript files that you would be willing to share (and if not that's understandable), or if you could point me in the right direction in the src to figure this out for myself.


The file "./js/nanocube2.js” on the master branch has code for querying a “master” nanocube, but we still haven’t pushed javascript code for full demos (like the demos currently on nanocubes.net) using this new query API.

Lauro
_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net



--
Alex Bongiovanni
University of Maryland
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Nanocubes-discuss] Updated javascript

laurolins
Hi Alex,

The general idea is that for each dimension you need to describe a set of “bins” in which you are interested:

(1) By default, if you don’t describe anything for a dimension you are implicitly saying: I want the “*” or “Aggregate All” or “Don’t filter anything”-bin.

(2) For all dimensions except for “time" you can use the "find-and-dive” syntax to describe the set of bins you are interested in that dimension:

    [@]<dimension-name>=<bin_address>+<dive_depth>

All dimensions in a nanocube (except for “time" which is special) are internally represented by a tree with a fixed number of levels (e.g. a 26-levels quadtree for spatial bins; a 2-levels tree for a categorical dimension). So, in the case of the "find-and-dive syntax” we are describing the set of bins (i.e. tree nodes) that are descendants of the bin with address <bin_address> exactly <dive_depth> levels. So suppose you want to produce a 256x256 tile for the root bin of a quad-tree dimension named “src”: 

    @src=qaddr(0,0,0)+8

The @ is saying "I want a different report for each bin found in this src dimension” (each pixel will be reported separately: e.g. a different count for each pixel). The qaddr(0,0,0) is a sugar syntax for an address in the “src” dimension. A “bin address” in any nanocube dimension is a 64-bit unsigned integer, and since qaddr(0,0,0) is 0, you could equivalently have written: @src=0+8. The +8 is saying “I am interested in every node that is 8 levels deeper than the root node”. 

In the case of a categorical dimension device, suppose you encode iPhone as the bin with address “1”, and you are interested only in iPhone records. You could write

    @device=1+0             

The 1 is the bin address of iPhone and the 0 is saying we want exactly that bin. Instead, if you want a different report for each device you would write

    @device=<Root_Of_Categorical_Variable>+1

Each child bin of the root bin in the device dimension is a different category. Right now there is a catch on what is the root bin address of a categorical dimension (c1, c2, …, c8). If it is a c1 dimension than the root address is 255, if it is c2 than the root address is 65535, …, if it is cK it is 2^K-1. We need are working on improving this. So for now, in a categorical dimension you can use the numbers 0, …, 2^K - 2 for categories and the number 2^K -1 is reserved as the root bin address of that dimension. So a group by per device in a c1 dimension is described as

    @device=255+1

In this way all different devices will be reported separately in the result set.

(2) For the “time" dimension you can describe a set of bins and how to aggregate them using the syntax:

<time-dim-name>=<start_bin>:<bucket_size>:<no_buckets>

If you defined a nanocube using 2014_1h as the time binning scheme, and you query @time=0:24:10 then you are getting back an aggregate number (one per day) for the first 10 days of 2014.

Here are more examples:

You want a 256x256 tile for the whole spatial domain but only of iPhones (bin address = 1):

query/@src=qaddr(0,0,0)+8/device=1+0

note in this case we haven’t used the @ on the device category (c1) , because we don’t want to split the result set on the device dimension. If you write

query/@src=qaddr(0,0,0)+8/@device=255+1

you would be getting a 256x256 tile for each different device.  The query

query/src=qaddr(1,1,1)+8/device=1+0/@time=0:24:10

would result in a daily time series of 10 days for iPhones on the top-right quadrant of the spatial domain.

Oh, by the way, in the javascript query object when you call .rollup() in one dimension you are saying DON’T USE the (i.e. aggregate all the results of the different bins we want from this dimension),and when you say .drilldown() you are saying USE the @. (i.e. split the result for each bin we want from this dimension).

Best,
Lauro



On Thu, Jan 23, 2014 at 9:34 PM, Lauro Lins <[hidden email]> wrote:
Hi Alex,

> So, I started reworking the javascript as best as I can, but it looks like some of the queries have actually been changed (schema_json to schema being one of them).  So I was wondering if you had a newer version of the javascript files that you would be willing to share (and if not that's understandable), or if you could point me in the right direction in the src to figure this out for myself.


The file "./js/nanocube2.js” on the master branch has code for querying a “master” nanocube, but we still haven’t pushed javascript code for full demos (like the demos currently on nanocubes.net) using this new query API.

Lauro
_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net



--
Alex Bongiovanni
University of Maryland
_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Nanocubes-discuss] Updated javascript

Alex Bongiovanni
Thank you so much, I think I am finally understanding things a bit better.  But I'm still unsure about how to calculate a 'qaddr' for a given position and zoom, or do you calculate that solely based on the bin address?

Somewhat of a different question, but what did the 'tile' query do in the 1.0 cube?  I'm trying to rework the original javascript to be compatible and I've hit a wall with this one (though after all the information from the last email I'm beginning to think it wouldn't be truly terrible to start from scratch with the visualization code).


On Fri, Jan 24, 2014 at 1:54 PM, Lauro Lins <[hidden email]> wrote:
Hi Alex,

The general idea is that for each dimension you need to describe a set of “bins” in which you are interested:

(1) By default, if you don’t describe anything for a dimension you are implicitly saying: I want the “*” or “Aggregate All” or “Don’t filter anything”-bin.

(2) For all dimensions except for “time" you can use the "find-and-dive” syntax to describe the set of bins you are interested in that dimension:

    [@]<dimension-name>=<bin_address>+<dive_depth>

All dimensions in a nanocube (except for “time" which is special) are internally represented by a tree with a fixed number of levels (e.g. a 26-levels quadtree for spatial bins; a 2-levels tree for a categorical dimension). So, in the case of the "find-and-dive syntax” we are describing the set of bins (i.e. tree nodes) that are descendants of the bin with address <bin_address> exactly <dive_depth> levels. So suppose you want to produce a 256x256 tile for the root bin of a quad-tree dimension named “src”: 

    @src=qaddr(0,0,0)+8

The @ is saying "I want a different report for each bin found in this src dimension” (each pixel will be reported separately: e.g. a different count for each pixel). The qaddr(0,0,0) is a sugar syntax for an address in the “src” dimension. A “bin address” in any nanocube dimension is a 64-bit unsigned integer, and since qaddr(0,0,0) is 0, you could equivalently have written: @src=0+8. The +8 is saying “I am interested in every node that is 8 levels deeper than the root node”. 

In the case of a categorical dimension device, suppose you encode iPhone as the bin with address “1”, and you are interested only in iPhone records. You could write

    @device=1+0             

The 1 is the bin address of iPhone and the 0 is saying we want exactly that bin. Instead, if you want a different report for each device you would write

    @device=<Root_Of_Categorical_Variable>+1

Each child bin of the root bin in the device dimension is a different category. Right now there is a catch on what is the root bin address of a categorical dimension (c1, c2, …, c8). If it is a c1 dimension than the root address is 255, if it is c2 than the root address is 65535, …, if it is cK it is 2^K-1. We need are working on improving this. So for now, in a categorical dimension you can use the numbers 0, …, 2^K - 2 for categories and the number 2^K -1 is reserved as the root bin address of that dimension. So a group by per device in a c1 dimension is described as

    @device=255+1

In this way all different devices will be reported separately in the result set.

(2) For the “time" dimension you can describe a set of bins and how to aggregate them using the syntax:

<time-dim-name>=<start_bin>:<bucket_size>:<no_buckets>

If you defined a nanocube using 2014_1h as the time binning scheme, and you query @time=0:24:10 then you are getting back an aggregate number (one per day) for the first 10 days of 2014.

Here are more examples:

You want a 256x256 tile for the whole spatial domain but only of iPhones (bin address = 1):

query/@src=qaddr(0,0,0)+8/device=1+0

note in this case we haven’t used the @ on the device category (c1) , because we don’t want to split the result set on the device dimension. If you write

query/@src=qaddr(0,0,0)+8/@device=255+1

you would be getting a 256x256 tile for each different device.  The query

query/src=qaddr(1,1,1)+8/device=1+0/@time=0:24:10

would result in a daily time series of 10 days for iPhones on the top-right quadrant of the spatial domain.

Oh, by the way, in the javascript query object when you call .rollup() in one dimension you are saying DON’T USE the (i.e. aggregate all the results of the different bins we want from this dimension),and when you say .drilldown() you are saying USE the @. (i.e. split the result for each bin we want from this dimension).

Best,
Lauro



On Thu, Jan 23, 2014 at 9:34 PM, Lauro Lins <[hidden email]> wrote:
Hi Alex,

> So, I started reworking the javascript as best as I can, but it looks like some of the queries have actually been changed (schema_json to schema being one of them).  So I was wondering if you had a newer version of the javascript files that you would be willing to share (and if not that's understandable), or if you could point me in the right direction in the src to figure this out for myself.


The file "./js/nanocube2.js” on the master branch has code for querying a “master” nanocube, but we still haven’t pushed javascript code for full demos (like the demos currently on nanocubes.net) using this new query API.

Lauro
_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net



--
Alex Bongiovanni
University of Maryland
_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net


_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net




--
Alex Bongiovanni
University of Maryland
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Nanocubes-discuss] Updated javascript

laurolins
Hi Alex,

But I'm still unsure about how to calculate a 'qaddr' for a given position and zoom, or do you calculate that solely based on the bin address?

Here is a python script to go from degree coordinates to tile addresses in different levels.

from math import pi, log, tan

# longitude range: [-180, 180]
# latitude range: [-85.05112, +85.05112]
def degrees2mercator(p_degrees):
    lon = p_degrees['x']
    lat = p_degrees['y']
    return { "x": lon/180.0, 
             "y": log( tan( pi/2.0 * (lat/180.0 + 1.0/2.0) ) ) / pi, 
             "type":"mercator" }

# mercator x and y ranges: [-1,1]
def mercator2tile(p_mercator, z):
    mx = p_mercator['x']
    my = p_mercator['y']
    return { \
        "x" : int((mx + 1.0)/2.0 * 2**z),
        "y" : int((my + 1.0)/2.0 * 2**z),
        "z" : z,
        "type": "tile" }

p_newyork_degrees  = { "x": -73.9400, "y": 40.6700, "type": "degrees" }
# >>> p_newyork_degrees
# {'y': 40.67, 'x': -73.94, 'type': 'degrees'}

p_newyork_mercator = degrees2mercator( p_newyork_degrees )
# >>> p_newyork_mercator
# {'y': 0.24772480853320908, 'x': -0.41077777777777774, 'type': 'mercator'}

p_newyork_tile_15  = mercator2tile( p_newyork_mercator, 15 )
# >>> p_newyork_tile_15
# {'y': 20442, 'x': 9653, 'z': 15, 'type': 'tile'}

p_newyork_tile_20  = mercator2tile( p_newyork_mercator, 20 )
# >>> p_newyork_tile_20
# {'y': 654167, 'x': 308922, 'z': 20, 'type': 'tile'}

So, if you want a tile for New York on level 15 you would write qaddr(9653, 20442, 15)and for level 20  qaddr(308922654167, 20)

Somewhat of a different question, but what did the 'tile' query do in the 1.0 cube?  I'm trying to rework the original javascript to be compatible and I've hit a wall with this one (though after all the information from the last email I'm beginning to think it wouldn't be truly terrible to start from scratch with the visualization code).

On the nanocube2.js, the ‘query’ API can be used for any query and the result is a generic tree structure with the result. In order to optimize specific tile queries (smaller than 256x256 tiles), we also have a ‘tile’ API on the master branch. The format of the results of the ‘tile' API in both 1.0 and master branches are the same: a sequence in binary format like: <no_pixels> <pixel_1> … <pixel_n>, where <pixel_i> = <x_coord> <y_coord> <value>.

Best,
Lauro
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Nanocubes-discuss] Updated javascript

Alex Bongiovanni
Degree coordinates to addresses I can do now thank you, but I'm still not sure how to convert 1.0's regions, eg [x=[x1,x1], y=[y1,y2], z], to the new qaddr's.  And what happened to 'shrinkage'?  

Otherwise I seem to have most things working now, I assume (probably incorrectly) that things will draw once I fix the tile queries.

Also, I haven't tried it yet, but I'm not convinced that the 'range' function for the query api works, It doesn't use the parameter 'addr0' and uses an undeclared variable 'offset'.


On Mon, Jan 27, 2014 at 10:11 AM, Lauro Lins <[hidden email]> wrote:
Hi Alex,

But I'm still unsure about how to calculate a 'qaddr' for a given position and zoom, or do you calculate that solely based on the bin address?

Here is a python script to go from degree coordinates to tile addresses in different levels.

from math import pi, log, tan

# longitude range: [-180, 180]
# latitude range: [-85.05112, +85.05112]
def degrees2mercator(p_degrees):
    lon = p_degrees['x']
    lat = p_degrees['y']
    return { "x": lon/180.0, 
             "y": log( tan( pi/2.0 * (lat/180.0 + 1.0/2.0) ) ) / pi, 
             "type":"mercator" }

# mercator x and y ranges: [-1,1]
def mercator2tile(p_mercator, z):
    mx = p_mercator['x']
    my = p_mercator['y']
    return { \
        "x" : int((mx + 1.0)/2.0 * 2**z),
        "y" : int((my + 1.0)/2.0 * 2**z),
        "z" : z,
        "type": "tile" }

p_newyork_degrees  = { "x": -73.9400, "y": 40.6700, "type": "degrees" }
# >>> p_newyork_degrees
# {'y': 40.67, 'x': -73.94, 'type': 'degrees'}

p_newyork_mercator = degrees2mercator( p_newyork_degrees )
# >>> p_newyork_mercator
# {'y': 0.24772480853320908, 'x': -0.41077777777777774, 'type': 'mercator'}

p_newyork_tile_15  = mercator2tile( p_newyork_mercator, 15 )
# >>> p_newyork_tile_15
# {'y': 20442, 'x': 9653, 'z': 15, 'type': 'tile'}

p_newyork_tile_20  = mercator2tile( p_newyork_mercator, 20 )
# >>> p_newyork_tile_20
# {'y': 654167, 'x': 308922, 'z': 20, 'type': 'tile'}

So, if you want a tile for New York on level 15 you would write qaddr(9653, 20442, 15)and for level 20  qaddr(308922654167, 20)

Somewhat of a different question, but what did the 'tile' query do in the 1.0 cube?  I'm trying to rework the original javascript to be compatible and I've hit a wall with this one (though after all the information from the last email I'm beginning to think it wouldn't be truly terrible to start from scratch with the visualization code).

On the nanocube2.js, the ‘query’ API can be used for any query and the result is a generic tree structure with the result. In order to optimize specific tile queries (smaller than 256x256 tiles), we also have a ‘tile’ API on the master branch. The format of the results of the ‘tile' API in both 1.0 and master branches are the same: a sequence in binary format like: <no_pixels> <pixel_1> … <pixel_n>, where <pixel_i> = <x_coord> <y_coord> <value>.

Best,
Lauro

_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net




--
Alex Bongiovanni
University of Maryland
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Nanocubes-discuss] Updated javascript

laurolins
Hi Alex,

Degree coordinates to addresses I can do now thank you, but I'm still not sure how to convert 1.0's regions, eg [x=[x1,x1], y=[y1,y2], z], to the new qaddr's.  And what happened to 'shrinkage'?  


If src is a quadtree dimension and you want to count the number of records in the bottom-left 3x2 rectangle assuming your domain is an 8x8 grid. Then you would do:

query/src=[qaddr(0,0,3),qaddr(3,2,3)]

This range query I had not described before. It takes the form of 

/<dimension-name>=[<bottom-left-addr0>,<top-right-addr1>]

where for quad-trees the zoom level of <bottom-left-addr0> should be the same as the one of <top-right-addr1>


Lauro

P.S. Note that at a zoom level of z the x and y coordinates are integers in the range [0,(2^z) - 1]. So in our example, 2^3 = 8 is an 8x8 grid, and the x, y are in [0,7].

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Nanocubes-discuss] Updated javascript

Alex Bongiovanni
Thank you, that helps immensely.  I seem to have sorted all of the query problems finally (seriously, thank you so much), but the heatmap still does not display.  

I'm supposing that the heatmap is displayed based on data received from the 'tile' queries (based on console output) but as I've checked that the queries themselves are valid (by resending them as 'query' not 'tile') I'm somewhat confused.  Any suggestions on something I might be missing (eg the demo version of Brightkite has controls for the heatmap, which the api version was lacking)?  

It is a very general question, and I don't expect you to know the problems that my code is having, I'm just wondering if you encountered anything similar or if there was some small thing I might overlook.

Thank you again, you've been incredibly helpful.


On Tue, Jan 28, 2014 at 11:04 AM, Lauro Lins <[hidden email]> wrote:
Hi Alex,

Degree coordinates to addresses I can do now thank you, but I'm still not sure how to convert 1.0's regions, eg [x=[x1,x1], y=[y1,y2], z], to the new qaddr's.  And what happened to 'shrinkage'?  


If src is a quadtree dimension and you want to count the number of records in the bottom-left 3x2 rectangle assuming your domain is an 8x8 grid. Then you would do:

query/src=[qaddr(0,0,3),qaddr(3,2,3)]

This range query I had not described before. It takes the form of 

/<dimension-name>=[<bottom-left-addr0>,<top-right-addr1>]

where for quad-trees the zoom level of <bottom-left-addr0> should be the same as the one of <top-right-addr1>


Lauro

P.S. Note that at a zoom level of z the x and y coordinates are integers in the range [0,(2^z) - 1]. So in our example, 2^3 = 8 is an 8x8 grid, and the x, y are in [0,7].


_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net




--
Alex Bongiovanni
University of Maryland
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Nanocubes-discuss] Updated javascript

laurolins
Hi Alex,

I'm supposing that the heatmap is displayed based on data received from the 'tile' queries (based on console output) but as I've checked that the queries themselves are valid (by resending them as 'query' not 'tile') I'm somewhat confused.  Any suggestions on something I might be missing (eg the demo version of Brightkite has controls for the heatmap, which the api version was lacking)?  

Were you able to interpret results from the ‘tile’ API? Check the image on 

to see how to interpret the result of ‘tile’ queries. From there you should have all information to build a heat map tile and overlay it on a map.

Lauro
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Nanocubes-discuss] Updated javascript

Alex Bongiovanni
Hmm...  I'm guessing tiles of 10 bytes are not what I want; based on your link I would guess a size of 80 is correct (for an 8x8 tile, which I'm not really even sure if I have).

After a quick peek at the call stack where tiles are created I do spot that I had added and then changed the tile method to nanocube2.js to make things run error free (depends on the nanocube version, which is undefined for the master branch it seems).  If you don't mind, does anything jump out as being extra special wrong (originally starts about line 170 in 1.0 nanocube.js)?

...
if (_.isUndefined(version)) {
    var record_size = 6;
    var view = new DataView(data);
    var n_records = data.byteLength / record_size;
    // slow, meh
    var x_array = new Uint8Array(n_records);
    var y_array = new Uint8Array(n_records);
    var count_array = new Uint32Array(n_records);

    for (var i=0; i<n_records - 1 /*** ADDED THE '- 1' ***/; ++i) {
        x_array[i] = view.getUint8(record_size*i+1) << (8 /*- resolution*/);
        y_array[i] = 256 - ((1 + view.getUint8(record_size*i)) << (8 /*- resolution*/));

        count_array[i] = view.getInt32(record_size*i+2, true);
    }
    k({x: x_array, y: y_array, count: count_array});
...


On Tue, Jan 28, 2014 at 2:15 PM, Lauro Lins <[hidden email]> wrote:
Hi Alex,

I'm supposing that the heatmap is displayed based on data received from the 'tile' queries (based on console output) but as I've checked that the queries themselves are valid (by resending them as 'query' not 'tile') I'm somewhat confused.  Any suggestions on something I might be missing (eg the demo version of Brightkite has controls for the heatmap, which the api version was lacking)?  

Were you able to interpret results from the ‘tile’ API? Check the image on 

to see how to interpret the result of ‘tile’ queries. From there you should have all information to build a heat map tile and overlay it on a map.

Lauro

_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net




--
Alex Bongiovanni
University of Maryland
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Nanocubes-discuss] Updated javascript

laurolins
>
> Hmm...  I'm guessing tiles of 10 bytes are not what I want; based on your link I would guess a size of 80 is correct (for an 8x8 tile, which I'm not really even sure if I have).

On the 1.0 version at one point we were dealing with two cases: values as uin32_t (4-bytes) and values as doubles (8-bytes). For the master branch we can forget all that for the ‘tile’ API and consider tile descriptions as sequence of 10-bytes-per-pixel: 2 for (i,j) and 8 bytes for value. I believe the case on line 172 of nanocube.js should be the only case needed for the master branch.

Lauro

P.S. Note that we only report a pixel if there is some information about it. If no raw-record touched a certain pixel in some resolution, that pixel is never reported. In the example image you have only 8 pixels were reported when a total of 64 pixels could have been reported in the case there were raw records hitting every pixel.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Nanocubes-discuss] Updated javascript

Alex Bongiovanni
So if only one pixel has any data, then I would expect a 10 byte response?

Also, what should the record size be, and do I need to worry about not having a resolution?  6 is the default record size for the case that you've said should be the one I need, and I commented the resolution out of my code as it was no longer supplied in the 'opts' parameter.


On Tue, Jan 28, 2014 at 3:25 PM, Lauro Lins <[hidden email]> wrote:
>
> Hmm...  I'm guessing tiles of 10 bytes are not what I want; based on your link I would guess a size of 80 is correct (for an 8x8 tile, which I'm not really even sure if I have).

On the 1.0 version at one point we were dealing with two cases: values as uin32_t (4-bytes) and values as doubles (8-bytes). For the master branch we can forget all that for the ‘tile’ API and consider tile descriptions as sequence of 10-bytes-per-pixel: 2 for (i,j) and 8 bytes for value. I believe the case on line 172 of nanocube.js should be the only case needed for the master branch.

Lauro

P.S. Note that we only report a pixel if there is some information about it. If no raw-record touched a certain pixel in some resolution, that pixel is never reported. In the example image you have only 8 pixels were reported when a total of 64 pixels could have been reported in the case there were raw records hitting every pixel.
_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net



--
Alex Bongiovanni
University of Maryland
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Nanocubes-discuss] Updated javascript

laurolins

> So if only one pixel has any data, then I would expect a 10 byte response?

Yes!

> Also, what should the record size be, and do I need to worry about not having a resolution?  6 is the default record size for the case that you've said should be the one I need, and I commented the resolution out of my code as it was no longer supplied in the 'opts' parameter.

Line 172 on https://github.com/laurolins/nanocube/blob/1.0/api/js/nanocube.js shows the 10-bytes record size. That should be the one. Here is the code I see there:
172    } else if (version === "0.0.1") {
173       var record_size = 10;
174        var view = new DataView(data);
175        var n_records = data.byteLength / record_size;
176        // slow, meh
177        var x_array = new Uint8Array(n_records);
178        var y_array = new Uint8Array(n_records);
179        var count_array = new Float64Array(n_records);
180        for (var i=0; i<n_records; ++i) {
181            x_array[i] = view.getUint8(record_size*i+1) << (8 - resolution);
182            y_array[i] = 256 - ((1 + view.getUint8(record_size*i)) << (8 - resolution));
183            count_array[i] = view.getFloat64(record_size*i+2, true);
184        }
185        k({x: x_array, y: y_array, count: count_array});
186    }

Lauro

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Nanocubes-discuss] Updated javascript

Alex Bongiovanni
Alright, this is the code I was using, but I don't know what to use for resolution, that isn't supplied anymore.


On Tue, Jan 28, 2014 at 3:38 PM, Lauro Lins <[hidden email]> wrote:

> So if only one pixel has any data, then I would expect a 10 byte response?

Yes!

> Also, what should the record size be, and do I need to worry about not having a resolution?  6 is the default record size for the case that you've said should be the one I need, and I commented the resolution out of my code as it was no longer supplied in the 'opts' parameter.

Line 172 on https://github.com/laurolins/nanocube/blob/1.0/api/js/nanocube.js shows the 10-bytes record size. That should be the one. Here is the code I see there:
172    } else if (version === "0.0.1") {
173       var record_size = 10;
174        var view = new DataView(data);
175        var n_records = data.byteLength / record_size;
176        // slow, meh
177        var x_array = new Uint8Array(n_records);
178        var y_array = new Uint8Array(n_records);
179        var count_array = new Float64Array(n_records);
180        for (var i=0; i<n_records; ++i) {
181            x_array[i] = view.getUint8(record_size*i+1) << (8 - resolution);
182            y_array[i] = 256 - ((1 + view.getUint8(record_size*i)) << (8 - resolution));
183            count_array[i] = view.getFloat64(record_size*i+2, true);
184        }
185        k({x: x_array, y: y_array, count: count_array});
186    }

Lauro
_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net



--
Alex Bongiovanni
University of Maryland
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Nanocubes-discuss] Updated javascript

laurolins
Note on the image that I sent before how we specify the ‘spatial bins’ in the ‘src’ dimension:

http://localhost:29512/tile/@src=0+3

The +3 there is the resolution of the tile (find-and-dive syntax). If you want the maximum resolution you would ask for 


That could bring up to 256*256 pixels if there is data in all of them. The general format:


Lauro
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Nanocubes-discuss] Updated javascript

Alex Bongiovanni
Ah, okay.  Thanks.


On Tue, Jan 28, 2014 at 3:49 PM, Lauro Lins <[hidden email]> wrote:
Note on the image that I sent before how we specify the ‘spatial bins’ in the ‘src’ dimension:

http://localhost:29512/tile/@src=0+3

The +3 there is the resolution of the tile (find-and-dive syntax). If you want the maximum resolution you would ask for 


That could bring up to 256*256 pixels if there is data in all of them. The general format:


Lauro

_______________________________________________
Nanocubes-discuss mailing list
[hidden email]
http://mailman.nanocubes.net/mailman/listinfo/nanocubes-discuss_mailman.nanocubes.net




--
Alex Bongiovanni
University of Maryland
Loading...