An IDataset implementation that loads subsets of continuous data in serial to conserve memory. More...

#include <dataset.h>

Inheritance diagram for Sleipnir::CDataSubset:

Public Member Functions
bool	Initialize (const char szDataDirectory, const IBayesNet pBayesNet, size_t iGeneSize)
	Construct a data subset corresponding to the given Bayes net using data files from the given directory; loads only the requested number of genes at a time.
bool	Initialize (const std::vector< std::string > &vecstrDataFiles, size_t iGeneSize)
	Construct a data subset corresponding using the given data files; loads only the requested number of genes at a time.
bool	Open (size_t iGeneOffset)
	Load data for the subset's configured number of genes starting at the given offset.
bool	IsHidden (size_t iNode) const
	Returns true if the requested experimental node is hidden (does not correspond to a data file).
size_t	GetDiscrete (size_t iY, size_t iX, size_t iNode) const
	Return the discretized value at the requested position.
float	GetContinuous (size_t iY, size_t iX, size_t iNode) const
	Return the continuous value at the requested position.
const std::string &	GetGene (size_t iGene) const
	Returns the gene name at the requested index.
size_t	GetGenes () const
	Returns the number of genes in the dataset.
bool	IsExample (size_t iY, size_t iX) const
	Returns true if some data file can be accessed at the requested position.
const std::vector< std::string > &	GetGeneNames () const
	Return a vector of all gene names in the dataset.
size_t	GetExperiments () const
	Return the number of experimental nodes in the dataset.
size_t	GetGene (const std::string &strGene) const
	Return the index of the given gene name, or -1 if it is not included in the dataset.
size_t	GetBins (size_t iNode) const
	Return the number of discrete values in the requested experimental node; -1 if the node is hidden or continuous.
void	Remove (size_t iY, size_t iX)
	Remove all data for the given dataset position.
void	FilterGenes (const CGenes &Genes, CDat::EFilter eFilter)
	Remove values from the dataset based on the given gene set and filter type.

Detailed Description

An IDataset implementation that loads subsets of continuous data in serial to conserve memory.

A data subset loads data continuously, like CDataset; this can be very expensive in memory usage for large datasets. To make this more tractable, CDataSubset allows subsets of genes of a specific size to be loaded in serial. A data subset is initialized with a specific size; each time a subset is opened, all data for this many genes (the size) will be loaded, starting with the gene at the requested offset. This allows the user to first load and use genes 0 through S-1, then S through 2S-1, and so forth. Typical usage resembles:

 CDataSubset Data;
 iGeneSize = 1000;
 Data.Initialize( "data/dir/path/", pBayesNet, iGeneSize );
 for( iGeneOffset = 0; iGeneOffset < iTotalGeneCount; iGeneOffset += iGeneSize ) {
   Data.Open( iGeneOffset );
   // do stuff with Data
 }

Definition at line 811 of file dataset.h.

Member Function Documentation

void Sleipnir::CDataSubset::FilterGenes	(	const CGenes &	Genes,
		CDat::EFilter	eFilter
	)		`[inline, virtual]`

Remove values from the dataset based on the given gene set and filter type.

Parameters:

Genes	Gene set used to filter the dataset.
eFilter	Way in which to use the given genes to remove values.

Remove values and genes (by removing all incident edges) from the dataset based on one of several algorithms. For details, see CDat::EFilter.

Remarks:: Generally implemented using Remove, so may not be supported by all implementations and may either mask or unload the filtered data.

See also:: CDat::FilterGenes

Implements Sleipnir::IDataset.

Definition at line 866 of file dataset.h.

size_t Sleipnir::CDataSubset::GetBins ( size_t iNode ) const [inline, virtual]

Return the number of discrete values in the requested experimental node; -1 if the node is hidden or continuous.

Parameters:

iNode Experimental node for which bin number should be returned.

Returns:: Number of discrete values taken by the given experimental node; -1 if the node is hidden or continuous.

See also:: GetDiscrete

Implements Sleipnir::IDataset.

Definition at line 858 of file dataset.h.

float Sleipnir::CDataSubset::GetContinuous	(	size_t	iY,
		size_t	iX,
		size_t	iNode
	)		const `[inline, virtual]`

Return the continuous value at the requested position.

Parameters:

iY	Data row.
iX	Data column.
iNode	Experimental node from which to retrieve the requested pair's value.

Returns:: Continuous value from the requested position and data file; not-a-number (NaN) if the value is missing.

Remarks:: Equivalent to using CDataPair::Get on the encapsulated data file with the appropriate indices. Behavior not defined when the corresponding data node is inherently discrete.

See also:: GetDiscrete

Implements Sleipnir::IDataset.

Definition at line 827 of file dataset.h.

References Sleipnir::CFullMatrix< tType >::Get(), and Sleipnir::CMeta::GetNaN().

size_t Sleipnir::CDataSubset::GetDiscrete	(	size_t	iY,
		size_t	iX,
		size_t	iNode
	)		const `[inline, virtual]`

Return the discretized value at the requested position.

Parameters:

iY	Data row.
iX	Data column.
iNode	Experimental node from which to retrieve the requested pair's value.

Returns:: Discretized value from the requested position and data file using that file's quantization information; -1 if the value is missing.

Remarks:: Equivalent to using CDataPair::Quantize and GetContinuous or CDataPair::Get on the encapsulated data file with the appropriate indices. Behavior not defined when no discretization information is available for the requested data node.

Implements Sleipnir::IDataset.

Definition at line 821 of file dataset.h.

References Sleipnir::CFullMatrix< tType >::Get().

size_t Sleipnir::CDataSubset::GetExperiments ( ) const [inline, virtual]

Return the number of experimental nodes in the dataset.

Returns:: Number of experimental nodes in the dataset.

Remarks:: For most datasets (those not containing hidden nodes), this will be equal to the number of encapsulated data files.

Implements Sleipnir::IDataset.

Definition at line 850 of file dataset.h.

const std::string& Sleipnir::CDataSubset::GetGene ( size_t iGene ) const [inline, virtual]

Returns the gene name at the requested index.

Parameters:

iGene Index of gene name to return.

Returns:: Gene name at the requested index.

Remarks:: For efficiency, no bounds checking is performed.

See also:: GetGenes

Implements Sleipnir::IDataset.

Definition at line 833 of file dataset.h.

Referenced by GetGene().

size_t Sleipnir::CDataSubset::GetGene ( const std::string & strGene ) const [inline, virtual]

Return the index of the given gene name, or -1 if it is not included in the dataset.

Parameters:

strGene Gene name to retrieve.

Returns:: Index of the requested gene name, or -1 if it is not in the dataset.

See also:: GetGeneNames

Implements Sleipnir::IDataset.

Definition at line 854 of file dataset.h.

References GetGene().

const std::vector<std::string>& Sleipnir::CDataSubset::GetGeneNames ( ) const [inline, virtual]

Return a vector of all gene names in the dataset.

Returns:: Vector of gene names in the dataset.

See also:: GetGenes | GetGene

Implements Sleipnir::IDataset.

Definition at line 846 of file dataset.h.

size_t Sleipnir::CDataSubset::GetGenes ( ) const [inline, virtual]

Returns the number of genes in the dataset.

Returns:: Number of genes in the dataset.

Remarks:: Equal to the union of all genes in encapsulated data files.

See also:: GetGene

Implements Sleipnir::IDataset.

Definition at line 837 of file dataset.h.

bool Sleipnir::CDataSubset::Initialize	(	const char *	szDataDirectory,
		const IBayesNet *	pBayesNet,
		size_t	iGeneSize
	)

Construct a data subset corresponding to the given Bayes net using data files from the given directory; loads only the requested number of genes at a time.

Parameters:

szDataDirectory	Directory from which data files are loaded.
pBayesNet	Bayes nets whose nodes will correspond to files in the data subset.
iGeneSize	Number of genes to be loaded at once; all data for these genes will be loaded.

Returns:: True if data subset was constructed successfully.

Creates a data subset with nodes corresponding to the given Bayes net structure. Data is loaded continuously or discretely as indicated by the Bayes net, and nodes for which a corresponding data file (i.e. one with the same name followed by an appropriate CDat extension) cannot be located are marked as hidden. No data is loaded during initialization; this instead occurs during Open.

See also:: CDataset::Open

Definition at line 749 of file dataset.cpp.

References Sleipnir::IBayesNet::GetNodes(), Sleipnir::CFullMatrix< tType >::Initialize(), and Sleipnir::IBayesNet::IsContinuous().

bool Sleipnir::CDataSubset::Initialize	(	const std::vector< std::string > &	vecstrDataFiles,
		size_t	iGeneSize
	)

Construct a data subset corresponding using the given data files; loads only the requested number of genes at a time.

Parameters:

vecstrDataFiles	Vector of file paths to load.
iGeneSize	Number of genes to be loaded at once; all data for these genes will be loaded.

Returns:: True if data subset was constructed successfully.

Creates a dataset with nodes corresponding to the given data files; all files are assumed to be continuous. No data is loaded during initialization; this instead occurs during Open.

See also:: CDataset::Open

Definition at line 791 of file dataset.cpp.

References Sleipnir::CFullMatrix< tType >::Initialize().

bool Sleipnir::CDataSubset::IsExample	(	size_t	iY,
		size_t	iX
	)		const `[inline, virtual]`

Returns true if some data file can be accessed at the requested position.

Parameters:

iY	Data row.
iX	Data column.

Returns:: True if a data file can be accessed at the requested position.

A dataset position is a usable example if at least one data file can be accessed at that position; that is, if some data file provides a non-missing value for that gene pair. Implementations that filter pairs in some manner can also prevent particular positions from being usable examples.

Implements Sleipnir::IDataset.

Definition at line 841 of file dataset.h.

References Sleipnir::CFullMatrix< tType >::Get().

bool Sleipnir::CDataSubset::IsHidden ( size_t iNode ) const [inline, virtual]

Returns true if the requested experimental node is hidden (does not correspond to a data file).

Parameters:

iNode Experimental node to investigate.

Returns:: True if the requested experimental node is hidden.

Since a dataset can be constructed either directly on a collection of data files or by tying a model such as a Bayes net to data files, IDataset can determine which model nodes are hidden by testing whether a data file exists for them. If no such file exists, the node is hidden and, for example, can be treated specially during Bayesian learning.

Remarks:: Datasets constructed directly from data files will never have hidden nodes.

Implements Sleipnir::IDataset.

Definition at line 817 of file dataset.h.

bool Sleipnir::CDataSubset::Open ( size_t iGeneOffset )

Load data for the subset's configured number of genes starting at the given offset.

Parameters:

iGeneOffset First gene ID to load into the data subset.

Returns:: True if data was loaded successfully.

For a data subset configured with a particular size, loads all data for that many genes beginning at the given index.

See also:: Initialize

Definition at line 823 of file dataset.cpp.

References Sleipnir::CFullMatrix< tType >::Get(), Sleipnir::CFullMatrix< tType >::GetColumns(), Sleipnir::CFullMatrix< tType >::GetRows(), and Sleipnir::CDataPair::Open().

void Sleipnir::CDataSubset::Remove	(	size_t	iY,
		size_t	iX
	)		`[inline, virtual]`

Remove all data for the given dataset position.

Parameters:

iY	Data row.
iX	Data column.

Unloads or masks data from all encapsulated files for the requested gene pair.

Remarks:: For efficiency, bounds checking is not performed; the given row and column should both be less than GetGenes. Not supported by all implementations.

Implements Sleipnir::IDataset.

Definition at line 862 of file dataset.h.

References Sleipnir::CFullMatrix< tType >::Get().

The documentation for this class was generated from the following files:

src/dataset.h
src/dataset.cpp

Public Member Functions

Detailed Description

Member Function Documentation