Writing a chunk to an array

[danielw]

Hello,

I'm having trouble putting chunks into an array.
I've created an array using:

scidb::ArrayID aid = catalog.addArray(*dptr, scidb::psLocalNode); // dptr is a pointer to a scidb::ArrayDesc
scidb::DBArray array(aid);

It is a one-attribute array.

boost::shared_ptr<scidb::ArrayIterator> ai = array.getIterator(0);
scidb::Chunk& outChunk = ai->newChunk(c); // c is an instance of scidb::Coordinates --> [0,0,0]
outChunk.allocate(i.byteSize());
outChunk.setSparse(false); // Never sparse
i.readInto(outChunk.getData()); // Write i.byteSize() bytes into outChunk.getData()

Is this a reasonable approach? I ask since I keep getting a segfault during the write, so I wanted to double-check. I haven't ruled out my own faulty code that is supposed to fill the chunk.

The array I am writing to has dimensions and attributes as follows:

AFL% dimensions(image);
[("unnamed0",0,1024,1024,0,0,1023,"int64"),("unnamed1",0,1024,1024,0,0,1023,"int64"),("unnamed2",0,9223372036854775808,42247,0,0,42246,"int64")]
AFL% attributes(image);
[("","uint16",false)]
AFL%

Does anything seem obviously wrong?

Thanks for your help,
-Daniel

[apoliakov]

Hello, Daniel.

I'm not the authority on the storage pathway and I've forwarded this to the right developer.

In the meantime, the best pattern to look at is the one inside the store operator.

PhysicalStore.cpp says:

Code: Select all

                    ConstChunk const& srcChunk = srcArrayIterators[i]->getChunk();
                    Coordinates const& first = srcChunk.getFirstPosition(false);
                    Coordinates const& last = srcChunk.getLastPosition(false);
                    for (size_t j = 0; j < nDims; j++) {
                        if (last[j] > highBoundary[j]) {
                            highBoundary[j] = last[j];
                        }
                        if (first[j] < lowBoundary[j]) {
                            lowBoundary[j] = first[j];
                        }
                    }
                    dstArrayIterators[i]->copyChunk(srcChunk);
                    for (size_t j = step; j != 0 && !srcArrayIterators[i]->end(); ++(*srcArrayIterators[i]), --j);


And it looks like there's some magic inside copyChunk having to do with the disk chunk versus memory chunk distinction. From Storage.cpp:

Code: Select all

    void LocalStorage::DBArrayIterator::copyChunk(ConstChunk const& srcChunk)
    {
        if (version > 1) { // first version of delta-array should always be allocated - not cloned
            addr.coords = srcChunk.getFirstPosition(false);
            if (storage->patchChunk(addr, srcChunk)) {
                return;
            }
        } else if (version == 0) {   
            DBChunk const* diskChunk = srcChunk.getDiskChunk();
            if (diskChunk != NULL) {
                AttributeDesc const& srcAttrDesc = diskChunk->getAttributeDesc();
                if (srcAttrDesc.getReserve() == 0 && attrDesc.isNullable() == srcAttrDesc.isNullable()) {
                    addr.coords = srcChunk.getFirstPosition(false);
                    if (storage->cloneChunk(addr, *diskChunk)) {
                        return;
                    }
                }
            }
        }
        ArrayIterator::copyChunk(srcChunk);
    }


Hopefully our storage specialist will reply soon as well.

[apoliakov]

This is what Konstantin, our storage guru had to add:

boost::shared_ptr<scidb::ArrayIterator> ai = array.getIterator(0);
scidb::Chunk& outChunk = ai->newChunk(c); // c is an instance of scidb::Coordinates --> [0,0,0]
outChunk.allocate(i.byteSize());
outChunk.setSparse(false); // Never sparse
i.readInto(outChunk.getData()); // Write i.byteSize() bytes into outChunk.getData()

Yes, this is correct way of filling chunk.
You only should not forget to call chunk.write() after filling its data.
So complete sequence of creating of new chunk is:

boost::shared_ptr<scidb::ArrayIterator> ai = array.getIterator(0);
scidb::Chunk& outChunk = ai->newChunk(c); // c is an instance of scidb::Coordinates --> [0,0,0]
outChunk.allocate(i.byteSize());
outChunk.setSparse(false); // Never sparse
i.readInto(outChunk.getData()); // Write i.byteSize() bytes into outChunk.getData()
outChunk.write();

Please make sure that readInto reads exactly i.byteSize() bytes.

Also definition of third coordinate (unamed2) seems to be very suspicious: it has length 9223372036854775808
which is actually negative 64-bit value and is not MAX_COORDINATE:

AFL% dimensions(image);
[("unnamed0",0,1024,1024,0,0,1023,"int64"),("unnamed1",0,1024,1024,0,0,1023,"int64"),("unnamed2",0,9223372036854775808,42247,0,0,42246,"int64")]
AFL% attributes(image);
[("","uint16",false)]
AFL%

May be it is the reason of fault.
In any case I need to see stack trace of the crashed thread.