Loading array from binary files

Get help using the SciDB system, discuss existing features, report bugs and problems.

Loading array from binary files

Postby ls_ram » Tue Sep 20, 2011 3:41 pm

From the wiki, it looks like SciDB only supports loading from csv, or text. Could I know if there is any way to load values into an array directly from binary files.
Posts: 1
Joined: Wed Sep 14, 2011 8:36 pm

Re: Loading array from binary files

Postby plumber » Mon Sep 26, 2011 3:13 pm

This is a tough problem, because the term binary can be used to cover a lot of things.

Here's where we are:

1. We have the basic CSV load utility, and a "proprietary" (read - convenient for us) external format. These are designed for ease of use over either performance or adherence to any kind of standard. The idea is that in a lot of places passing around CSV files is the "way these things are handled".

2. We have a couple of people working on loaders / file access layers for specific "binary" formats: the ones that are common in the science data community. For example HDF-5, and FITS. Why these? Because there are people using SciDB with data in these formats.

See: https://github.com/wangd/SciDB-HDF5

And the FITS loader is going be added to the trunk as an example.

3. The design of the engine is such that adding another external format isn't that hard. 2 groups (at least) have already done this for their formats with (relatively) little help from anyone here. We also prototype ideas for storage manager designs by writing drivers that talk to external (ie. Non-SciDB storage manger) files.
Posts: 191
Joined: Mon Sep 27, 2010 6:03 pm

Return to SciDB Support and Discussion

Who is online

Users browsing this forum: Yahoo [Bot] and 2 guests