SSDB benchmarks of version 12.3 ERROR

[iwlam525]

I came across a problem about SSDB on version 12.3. First, I successfully compiled the whole source code of 12.3, so did SSDB. Second, I successfully run the SciDB, for example, create, load and scan operations are all ok. But, when I run the SSDB benchmarks as README said. It failed. Here is the output when I run the command : ./run_tiny.sh -g >run_tiny.sh_g_log_3 2>&1
And the content of the file run_tiny.sh_g_log_3 is : (The red content I marked is ERROR)


SSDB:
[.... Data Generation ....]
[..... Initialization ....]
Loading libs
Query was executed successfully
Query was executed successfully
Create Tiny Array
UserQueryException in file: src/query/parser/ALTranslator.cpp function: createArrayReferenceParam line: 800
Error id: scidb::SCIDB_SE_QPROC::SCIDB_LE_ARRAY_DOESNT_EXIST
Error description: Query processor error. Array 'tiny' does not exist.
remove (tiny)
^^^^
Failed query id: 1100847293314
UserQueryException in file: src/query/parser/ALTranslator.cpp function: createArrayReferenceParam line: 800
Error id: scidb::SCIDB_SE_QPROC::SCIDB_LE_ARRAY_DOESNT_EXIST
Error description: Query processor error. Array 'tiny_space' does not exist.
remove (tiny_space)
^^^^^^^^^^
Failed query id: 1100847293315
Query was executed successfully
Query was executed successfully
Loading Tiny data ..
Loading Time: 0 seconds
Cooking Tiny data into tiny_obs array ..
Cooking Time: 0 seconds
Grouping Tiny observations into tiny_groups array ..
SystemException in file: src/network/BaseConnection.cpp function: sendAndReadMessage line: 367
Error id: scidb::SCIDB_SE_NETWORK::SCIDB_LE_CANT_SEND_RECEIVE
Error description: Network error. Cannot send or receive network messages.
Grouping Time: 0 seconds
Pre-Reparting the Array
3 [0xb696c710] FATAL scidb.services.network null - Error #system:111 when connecting to localhost:1239
3 [0xb696c710] FATAL scidb.services.network null - Error #system:111 when connecting to localhost:1239
iquery SystemException in file: src/network/BaseConnection.cpp function: connect line: 267
Error id: scidb::SCIDB_SE_NETWORK::SCIDB_LE_CONNECTION_ERROR
Error description: Network error. Error #system:111 when connecting to localhost:1239.
Pre-Observation spliting
store(reshape(slice(tiny_obs,Z,0),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=499967:499976,10,0,I=500048:500057,10,0]),tiny_obs_0)
store(reshape(slice(tiny_obs,Z,1),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=499990:499999,10,0,I=499961:499970,10,0]),tiny_obs_1)
store(reshape(slice(tiny_obs,Z,2),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=666073:666082,10,0,I=402182:402191,10,0]),tiny_obs_2)
store(reshape(slice(tiny_obs,Z,3),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=500036:500045,10,0,I=500007:500016,10,0]),tiny_obs_3)
store(reshape(slice(tiny_obs,Z,4),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=499949:499958,10,0,I=500030:500039,10,0]),tiny_obs_4)
store(reshape(slice(tiny_obs,Z,5),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=499972:499981,10,0,I=499942:499951,10,0]),tiny_obs_5)
store(reshape(slice(tiny_obs,Z,6),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=499994:500003,10,0,I=499965:499974,10,0]),tiny_obs_6)
store(reshape(slice(tiny_obs,Z,7),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=707670:707679,10,0,I=443779:443788,10,0]),tiny_obs_7)
store(reshape(slice(tiny_obs,Z,8),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=500040:500049,10,0,I=500011:500020,10,0]),tiny_obs_8)
store(reshape(slice(tiny_obs,Z,9),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=499953:499962,10,0,I=500034:500043,10,0]),tiny_obs_9)
store(reshape(slice(tiny_obs,Z,10),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=499967:499976,10,0,I=500048:500057,10,0]),tiny_obs_10)
store(reshape(slice(tiny_obs,Z,11),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=499990:499999,10,0,I=499961:499970,10,0]),tiny_obs_11)
store(reshape(slice(tiny_obs,Z,12),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=666073:666082,10,0,I=402182:402191,10,0]),tiny_obs_12)
store(reshape(slice(tiny_obs,Z,13),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=500036:500045,10,0,I=500007:500016,10,0]),tiny_obs_13)
store(reshape(slice(tiny_obs,Z,14),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=499949:499958,10,0,I=500030:500039,10,0]),tiny_obs_14)
store(reshape(slice(tiny_obs,Z,15),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=499972:499981,10,0,I=499942:499951,10,0]),tiny_obs_15)
store(reshape(slice(tiny_obs,Z,16),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=499994:500003,10,0,I=499965:499974,10,0]),tiny_obs_16)
store(reshape(slice(tiny_obs,Z,17),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=707670:707679,10,0,I=443779:443788,10,0]),tiny_obs_17)
store(reshape(slice(tiny_obs,Z,18),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=500040:500049,10,0,I=500011:500020,10,0]),tiny_obs_18)
store(reshape(slice(tiny_obs,Z,19),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=499953:499962,10,0,I=500034:500043,10,0]),tiny_obs_19)
Done!
[Begin]
Run [0]:
Q1: 0 seconds
Q2: 0 seconds
UserQueryException in file: src/query/parser/ALTranslator.cpp function: createArrayReferenceParam line: 800
Error id: scidb::SCIDB_SE_QPROC::SCIDB_LE_ARRAY_DOESNT_EXIST
Error description: Query processor error. Array 'tiny_reparted' does not exist.
thin(window(subarray(tiny_reparted,0,0,0,19,9,9),1,4,4,avg(a)),0,1,2,3,2,3)
^^^^^^^^^^^^^
Failed query id: 1100848043342
Q3: 0 seconds
UserQueryException in file: src/query/parser/ALTranslator.cpp function: createArrayReferenceParam line: 800
Error id: scidb::SCIDB_SE_QPROC::SCIDB_LE_ARRAY_DOESNT_EXIST
Error description: Query processor error. Array 'tiny_obs_0' does not exist.
avg(filter(subarray(tiny_obs_0,503000,503000,503000+50,503000+50),center is not null),sumPixel)
^^^^^^^^^^
Failed query id: 1100848063343
UserQueryException in file: src/query/parser/ALTranslator.cpp function: createArrayReferenceParam line: 800
Error id: scidb::SCIDB_SE_QPROC::SCIDB_LE_ARRAY_DOESNT_EXIST
Error description: Query processor error. Array 'tiny_obs_2' does not exist.
avg(filter(subarray(tiny_obs_2,503000,503000,503000+50,503000+50),center is not null),sumPixel)
^^^^^^^^^^
Failed query id: 1100848073346
UserQueryException in file: src/query/parser/ALTranslator.cpp function: createArrayReferenceParam line: 800
Error id: scidb::SCIDB_SE_QPROC::SCIDB_LE_ARRAY_DOESNT_EXIST
Error description: Query processor error. Array 'tiny_obs_7' does not exist.
avg(filter(subarray(tiny_obs_7,503000,503000,503000+50,503000+50),center is not null),sumPixel)
^^^^^^^^^^
Failed query id: 1100848063344

[dedcode]

- How many instances do you have? Tiny is not tested for a distributed setting.

- Then can you execute the INIT commands in the script one by one and check their outcome. I see:
"Loading Tiny data ..
Loading Time: 0 seconds
Cooking Tiny data into tiny_obs array ..
Cooking Time: 0 seconds"
Did anything get loaded in the recipient arrays tiny, tiny_obs ??

- The script works for me, the only issue is the python connector that tries to split the array don't seem to store the results, I'll open a ticket for that. The immediate work arround is to run the split commands manually in iquery:
AFL% store(reshape(slice(tiny_obs,Z,0),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=499967:499976,10,0,I=500048:500057,10,0]),tiny_obs_0);
AFL% store(reshape(slice(tiny_obs,Z,1),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=499990:499999,10,0,I=499961:499970,10,0]),tiny_obs_1);
etc ...

ded

[iwlam525]

- How many instances do you have? Tiny is not tested for a distributed setting.

- Then can you execute the INIT commands in the script one by one and check their outcome. I see:
"Loading Tiny data ..
Loading Time: 0 seconds
Cooking Tiny data into tiny_obs array ..
Cooking Time: 0 seconds"
Did anything get loaded in the recipient arrays tiny, tiny_obs ??

- The script works for me, the only issue is the python connector that tries to split the array don't seem to store the results, I'll open a ticket for that. The immediate work arround is to run the split commands manually in iquery:
AFL% store(reshape(slice(tiny_obs,Z,0),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=499967:499976,10,0,I=500048:500057,10,0]),tiny_obs_0);
AFL% store(reshape(slice(tiny_obs,Z,1),<oid:int64 NULL,center:bool NULL,polygon:int32 NULL,sumPixel:int64 NULL,avgDist:double NULL,point:bool NULL>[J=499990:499999,10,0,I=499961:499970,10,0]),tiny_obs_1);
etc ...

ded

I have only one instance. And I try to execute the INIT commands in the script one by one, finding that there is something wrong at "groupstars" phase. When executing iquery -r /dev/null -aq "store(groupstars(filter(tiny_obs,oid is not null and center is not null),tiny_space,0.025,20),tiny_groups)" , the SciDB suddenly became disconnected. Then I checked the scidb.log , I found the errors in tests/benchmarks/ss-db/src/operators/groupstars/grouper.cpp :

Code: Select all

void getObservationsInImage(std::vector<ObsPos> &observations, int imageId) {
    std::map<int, int>::const_iterator it = _timeIndexes.find(imageId);

    assert (it != _timeIndexes.end()); // ERROR
    assert (it->second >= 0);
    assert (it->second < _allObsCount);
   
    int indexx = it->second;
    for (int index = it->second; index < _allObsCount && _allObs[index].time == imageId; ++index) {
       observations.push_back (_allObs[index]);
    }
  }

The code with ERROR commented causes SciDB disconnected. Because "it == _timeIndexes.end()" will be true!

And I also found that the python connector that tries to split the array don't seem to store the results.

Actually, the benchmarks confused me....

[dedcode]

Can you check out the latest version ? the python connector is now fixed and I have modified the run_tiny.sh script to address you issue.

[iwlam525]

Can you check out the latest version ? the python connector is now fixed and I have modified the run_tiny.sh script to address you issue.

I'm always working on the version of 12.3--the latest version.

Could you give me the run_tiny.sh script that you've modified to me ? My email adress is iwlam525@163.com.

Thank you!~