Class for gathering data, from failed path analysis, and from completed runs. More...

#include <datagatherer.hpp>

Collaboration diagram for datagatherer:

Public Member Functions
	datagatherer ()

	datagatherer (boost::filesystem::path bdir, boost::filesystem::path fundir, int numvar)

	datagatherer (int numvar, boost::filesystem::path loc)

int	num_reals ()

void	slave_init ()

void	add_file_to_save (std::string filename)

boost::filesystem::path	MakeTargetFilename (std::string filename)

void	SlaveSetup (ProgSettings &paramotopy_settings, runinfo &paramotopy_info, int myid, boost::filesystem::path called_dir)

bool	SlaveCollectAndWriteData (double *current_params, ProgSettings &paramotopy_settings, timer &process_timer)

void	AppendOnlyPosReal (std::string orig_filename, double *current_params, ProgSettings &paramotopy_settings)

void	AppendData (std::string orig_filename, double *current_params)

void	WriteAllData ()

void	WriteData (std::string outstring, boost::filesystem::path target_file)

bool	GatherDataForFails (std::vector< point > terminal_fails, std::vector< point > &successful_resolves)

void	GatherDataFromMenu ()

std::vector< point >	CompareInitial_Gathered (std::vector< point > terminal_fails, std::vector< point > current_data)

std::vector< point >	GatherFinalizedDataToMemory (boost::filesystem::path folder_with_data)

void	WriteUnsuccessfulResolves (std::map< int, point > successful_resolves)

std::map< int, point >	ReadSuccessfulResolves ()

void	CheckForResolvedFailedPoint (std::string file_to_gather, int next_index, std::string &next_data, std::map< int, point > successful_resolves)

boost::filesystem::path	GetAvailableRuns ()

void	CollectSpecificFiles (std::string file_to_gather, std::vector< boost::filesystem::path > folders_with_data, boost::filesystem::path run_to_analyze, int parser_index, bool mergefailed)

std::vector < boost::filesystem::path >	GetFoldersForData (boost::filesystem::path dir)

void	IncrementOutputFolder (boost::filesystem::path &output_folder_name, boost::filesystem::path base_output_folder_name, int &output_folder_index)

void	MergeFolders (std::string file_to_gather, boost::filesystem::path left_folder, boost::filesystem::path right_folder, boost::filesystem::path output_folder_name, int parser_index)

void	finalize_run_to_file (std::string file_to_gather, boost::filesystem::path source_folder, boost::filesystem::path base_output_folder_name, int parser_index, bool mergefailed)

void	rest_of_files (std::ifstream &datafile, std::string &output_buffer, std::ofstream &outputfile, std::vector< boost::filesystem::path > filelist, int file_index, int parser_index)

bool	endoffile_stuff (std::ifstream &datafile, int &file_index, std::vector< boost::filesystem::path > filelist)

bool	ReadPoint (std::ifstream &fin, std::string &data, int parser_index)

bool	ReadPoint (std::ifstream &fin, int &next_index, std::string &data, int parser_index)

std::string	ParseSolutionsFile (std::ifstream &fin)

std::vector< std::vector < std::pair< std::string, std::string > > >	ParseSolutionsFile_ActualSolutions (std::ifstream &fin)

std::string	ParseFailedPaths (std::ifstream &fin)
	the parser for the failed_paths file type, which is output from bertini. More...

Public Attributes
boost::filesystem::path	fundamental_dir

boost::filesystem::path	base_dir

int	numvariables

boost::filesystem::path	run_to_analyze

std::vector< std::string >	gather_savefiles

std::vector< int >	gather_parser_indices

std::vector< std::string >	ParamNames

int	numfiles

boost::filesystem::path	DataCollectedbase_dir

int	buffersize

int	newfilethreshold

int	myid

std::string	real_filename

std::map< std::string, fileinfo >	slavemap

int	standardstep2

Detailed Description

Class for gathering data, from failed path analysis, and from completed runs.

contains methods for gathering data from the bfiles_filename/run0/step2/DataCollected/c0 folders (or in another respective location).

Definition at line 82 of file datagatherer.hpp.

Constructor & Destructor Documentation

datagatherer::datagatherer ( )

inline

default constructor

Definition at line 90 of file datagatherer.hpp.

References base_dir, fundamental_dir, and numvariables.

datagatherer::datagatherer	(	boost::filesystem::path	bdir,
		boost::filesystem::path	fundir,
		int	numvar
	)

inline

constructor with instantiation of base_dir, fundamental_dir, and numvar. intended for use in the collection of data for a complete run, including merging of failed path successes.

Definition at line 97 of file datagatherer.hpp.

References base_dir, fundamental_dir, numvariables, and run_to_analyze.

datagatherer::datagatherer	(	int	numvar,
		boost::filesystem::path	loc
	)

inline

constructor with instantiation of numvariables, and run_to_analyze. intended for use in failed path analysis

Definition at line 105 of file datagatherer.hpp.

References fundamental_dir, numvariables, and run_to_analyze.

Member Function Documentation

void datagatherer::add_file_to_save ( std::string filename )

inline

Definition at line 161 of file datagatherer.hpp.

References GetFileParserIndex(), and fileinfo::parser_index.

Referenced by SlaveSetup().

Here is the call graph for this function:

Here is the caller graph for this function:

void datagatherer::AppendData	(	std::string	orig_filename,
		double *	current_params
	)

creates a string to append read-in data to the running file.

Parameters

orig_filename	Name of the data file to be read.
current_parameter_values	Current parameter values.

Definition at line 259 of file datagatherer.cpp.

References ParamNames, and slavemap.

Referenced by SlaveCollectAndWriteData().

Here is the caller graph for this function:

void datagatherer::AppendOnlyPosReal	(	std::string	orig_filename,
		double *	current_params,
		ProgSettings &	paramotopy_settings
	)

sorts the input file for positive real solutions creates a string to append read-in data to the running file.

Parameters

orig_filename	Name of the data file to be read.
current_parameter_values	Current parameter values.
paramotopy_settings	the settings structure, passed by reference

Definition at line 155 of file datagatherer.cpp.

References numvariables, ParamNames, ParseSolutionsFile_ActualSolutions(), ProgSettings::settings, and slavemap.

Referenced by SlaveCollectAndWriteData().

Here is the call graph for this function:

Here is the caller graph for this function:

void datagatherer::CheckForResolvedFailedPoint	(	std::string	file_to_gather,
		int	next_index,
		std::string &	next_data,
		std::map< int, point >	successful_resolves
	)

Parameters

file_to_gather	The name of the file we are sorting and merging.
next_index	The index. Used in lookup in the map successful_resolves.
next_data	A mutable string containing the data. If successful_resolves has a point with the same index, and it was successfully resolved, we replace the next_data with that in the map.
successful_resolves	Map of points containing the data produced by failed path analysis

Definition at line 882 of file datagatherer.cpp.

Referenced by finalize_run_to_file().

Here is the caller graph for this function:

void datagatherer::CollectSpecificFiles	(	std::string	file_to_gather,
		std::vector< boost::filesystem::path >	folders_with_data,
		boost::filesystem::path	run_to_analyze,
		int	parser_index,
		bool	mergefailed
	)

Main loop for gathering a particular file in a particular completed run.

Parameters

file_to_gather	The name of the bertini output file to gather.
folders_with_data	Vector with the names of the folders containing the data.
run_to_analyze	The name of the run.
parser_index	The index corresponding to the filename. Different file types have different parsers.
mergefailed	Bool determining whether to merge data from failed path analysis. This is false if collecting from a run DURING failed path analysis (it would break things), but is true if doing a menu collection.

Definition at line 521 of file datagatherer.cpp.

References finalize_run_to_file(), IncrementOutputFolder(), MergeFolders(), and run_to_analyze.

Referenced by GatherDataForFails(), and GatherDataFromMenu().

Here is the call graph for this function:

Here is the caller graph for this function:

std::vector< point > datagatherer::CompareInitial_Gathered	(	std::vector< point >	terminal_fails,
		std::vector< point >	current_data
	)

Compares initially run data to re-solved during failed path analysis. Reads in data from the successful_resolves file.

Returns: vector of successfully resolved points, complete with data

Definition at line 376 of file datagatherer.cpp.

Referenced by GatherDataForFails().

Here is the caller graph for this function:

bool datagatherer::endoffile_stuff	(	std::ifstream &	datafile,
		int &	file_index,
		std::vector< boost::filesystem::path >	filelist
	)

Reads the remainder of a file into memory, after a point at which we are certain there is not more sorting to be done.

Parameters

datafile	The data file being read.
file_index	The integer index of the file being read.
filelist	The vector containing the list of the files to read.

Definition at line 1206 of file datagatherer.cpp.

Referenced by MergeFolders().

Here is the caller graph for this function:

void datagatherer::finalize_run_to_file	(	std::string	file_to_gather,
		boost::filesystem::path	source_folder,
		boost::filesystem::path	base_output_folder_name,
		int	parser_index,
		bool	mergefailed
	)

Write the total collected data to a single file, and verify that the data is in order.

Parameters

file_to_gather	The name of the Bertini output file we are gathering currently.
source_folder	The name of the source folder.
base_output_folder_name.	The name of the bfiles_filename/run#/gathered_data. We append with /finalized.
parser_index	The parser choice to use. Corresponds to file_to_gather.
mergefailed	Boolean indicating whether the data checks out.

Definition at line 727 of file datagatherer.cpp.

References CheckForResolvedFailedPoint(), FindFiles(), ReadPoint(), ReadSuccessfulResolves(), and WriteUnsuccessfulResolves().

Referenced by CollectSpecificFiles().

Here is the call graph for this function:

Here is the caller graph for this function:

bool datagatherer::GatherDataForFails	(	std::vector< point >	terminal_fails,
		std::vector< point > &	successful_resolves
	)

outermost method for gathering data during failure analysis mode. keeps track of which points were successful (or not).

Parameters

terminal_fails	vector containing the indices of the attempted resolves.
successful_resolves	Vector of points for holding the successful data.

Definition at line 339 of file datagatherer.cpp.

References CollectSpecificFiles(), CompareInitial_Gathered(), gather_parser_indices, gather_savefiles, GatherFinalizedDataToMemory(), GetFilesToParse(), GetFoldersForData(), and run_to_analyze.

Referenced by failinfo::find_successful_resolves().

Here is the call graph for this function:

Here is the caller graph for this function:

void datagatherer::GatherDataFromMenu ( )

outermost method for gathering data from main menu. no input necessary

Definition at line 485 of file datagatherer.cpp.

References CollectSpecificFiles(), gather_parser_indices, gather_savefiles, GetAvailableRuns(), GetFilesToParse(), GetFoldersForData(), and run_to_analyze.

Referenced by DataManagementMainMenu().

Here is the call graph for this function:

Here is the caller graph for this function:

std::vector< point > datagatherer::GatherFinalizedDataToMemory ( boost::filesystem::path folder_with_data )

gather the data from a completed, gathered, finalized step2, to memory.

Parameters

folder_with_data Path having data in it.

Definition at line 417 of file datagatherer.cpp.

References point::collected_data, gather_parser_indices, gather_savefiles, point::index, and ReadPoint().

Referenced by GatherDataForFails().

Here is the call graph for this function:

Here is the caller graph for this function:

boost::filesystem::path datagatherer::GetAvailableRuns ( )

search for completed runs with this filename.

Returns: path of choice for run to collect. this is automatically determined if there is exactly one completed run with the filename.

Definition at line 909 of file datagatherer.cpp.

References FindDirectories(), fundamental_dir, get_int_choice(), run_to_analyze, and TestIfFinished().

Referenced by GatherDataFromMenu().

Here is the call graph for this function:

Here is the caller graph for this function:

std::vector< boost::filesystem::path > datagatherer::GetFoldersForData ( boost::filesystem::path dir )

Read the folders containing the data. The folder names are contained in a plain text file.

Parameters

dir	String containing the name of the directory in which to look.

Definition at line 1452 of file datagatherer.cpp.

Referenced by GatherDataForFails(), and GatherDataFromMenu().

Here is the caller graph for this function:

void datagatherer::IncrementOutputFolder	(	boost::filesystem::path &	output_folder_name,
		boost::filesystem::path	base_output_folder_name,
		int &	output_folder_index
	)

A lower-level function. Increases the output folder index by 1, during a merge.

Parameters

output_folder_name	MUTABLE string to hold the new folder name.
base_output_folder_name	The string from which we build output_folder_name
output_folder_index	MUTABLE integer, which herein is incremented, and contatenated to base_... to make the output_folder_name.

Definition at line 1433 of file datagatherer.cpp.

Referenced by CollectSpecificFiles().

Here is the caller graph for this function:

boost::filesystem::path datagatherer::MakeTargetFilename ( std::string filename )

Definition at line 9 of file datagatherer.cpp.

References DataCollectedbase_dir, and slavemap.

Referenced by SlaveCollectAndWriteData(), and WriteAllData().

Here is the caller graph for this function:

void datagatherer::MergeFolders	(	std::string	file_to_gather,
		boost::filesystem::path	left_folder,
		boost::filesystem::path	right_folder,
		boost::filesystem::path	output_folder_name,
		int	parser_index
	)

A lower-level function. Actually merges two folders, left and right, into the output_folder_name, using the parser determined by parser_index.

Parameters

file_to_gather	The name of the bertini output file we are gathering.
left_folder	The name of the left folder currently being merged.
output_folder_name	The name of the folder into which to place the merged data.
parser_index	The index of the parser to use on this filetype.

Definition at line 953 of file datagatherer.cpp.

References endoffile_stuff(), FindFiles(), ReadPoint(), and rest_of_files().

Referenced by CollectSpecificFiles().

Here is the call graph for this function:

Here is the caller graph for this function:

int datagatherer::num_reals ( )

inline

Definition at line 143 of file datagatherer.hpp.

References real_filename.

Referenced by slave_process::LoopSearch().

Here is the caller graph for this function:

std::string datagatherer::ParseFailedPaths ( std::ifstream & fin )

the parser for the failed_paths file type, which is output from bertini.

function for parsing the failed_paths file (just one instance of it) gets called repeatedly to see if any points had failed paths (each point has one failed_paths file)

Parameters

fin	The handle to the file.

Returns: The data in string form.

Definition at line 1405 of file datagatherer.cpp.

References numvariables.

Referenced by ReadPoint().

Here is the caller graph for this function:

std::string datagatherer::ParseSolutionsFile ( std::ifstream & fin )

function for parsing ***_solutions files, as output from bertini.

Parameters

fin	The file handle.

Returns: A string with the data

Definition at line 1357 of file datagatherer.cpp.

References numvariables.

Referenced by ReadPoint().

Here is the caller graph for this function:

std::vector< std::vector< std::pair< std::string, std::string > > > datagatherer::ParseSolutionsFile_ActualSolutions ( std::ifstream & fin )

Function for getting the solutions and parameter values, for sorting

Parameters

fin

Returns: vector of data

Definition at line 1300 of file datagatherer.cpp.

References numvariables.

Referenced by AppendOnlyPosReal().

Here is the caller graph for this function:

bool datagatherer::ReadPoint	(	std::ifstream &	fin,
		std::string &	data,
		int	parser_index
	)

function for parsing ***_solutions files, as output from bertini.

Parameters

fin	The file handle.
data	The string to which to read the data.
parser_index	The parser to use.

Definition at line 1227 of file datagatherer.cpp.

References ParseFailedPaths(), and ParseSolutionsFile().

Referenced by finalize_run_to_file(), GatherFinalizedDataToMemory(), MergeFolders(), ReadSuccessfulResolves(), and rest_of_files().

Here is the call graph for this function:

Here is the caller graph for this function:

bool datagatherer::ReadPoint	(	std::ifstream &	fin,
		int &	next_index,
		std::string &	data,
		int	parser_index
	)

function for parsing ***_solutions files, as output from bertini.

Parameters

fin	The file handle.
next_index	The index of the point just read.
data	The string to which to read the data.
parser_index	The parser to use.

Definition at line 1261 of file datagatherer.cpp.

References ParseFailedPaths(), and ParseSolutionsFile().

Here is the call graph for this function:

std::map< int, point > datagatherer::ReadSuccessfulResolves ( )

Reads data produced by failed path analysis into memory.

Returns: A map of points, complete with collected data.

Definition at line 607 of file datagatherer.cpp.

References base_dir, point::collected_data, point::index, and ReadPoint().

Referenced by finalize_run_to_file(), and failinfo::RecoverProgress().

Here is the call graph for this function:

Here is the caller graph for this function:

void datagatherer::rest_of_files	(	std::ifstream &	datafile,
		std::string &	output_buffer,
		std::ofstream &	outputfile,
		std::vector< boost::filesystem::path >	filelist,
		int	file_index,
		int	parser_index
	)

While merging, we will get to a point when there is no more sorting to do. The rest of the data is in set of files, and in order, so all that remains is to copy the rest of the file in to the output file. This function does that.

Parameters

datafile	The stream for reading the data.
output_buffer	The string we read into.
outputfile	The output stream we write to.
filelist	Vector containing the names of the files to be read.
file_index	Index integer of the file currently being read.
parser_index	The integer index of the parser being used.

Definition at line 1152 of file datagatherer.cpp.

References ReadPoint().

Referenced by MergeFolders().

Here is the call graph for this function:

Here is the caller graph for this function:

void datagatherer::slave_init ( )

inline

Definition at line 147 of file datagatherer.hpp.

Referenced by SlaveSetup().

Here is the caller graph for this function:

bool datagatherer::SlaveCollectAndWriteData	(	double *	current_params,
		ProgSettings &	paramotopy_settings,
		timer &	process_timer
	)

reads data from bertini output files, and writes it to files in DataCollected folder, if the threshold is big enough

Parameters

linenumber	Parameter point index
AllParams	Current parameter values.
paramotopy_settings	the settings structure, passed by reference
process_timer	Timer object for collecting timing data.

Definition at line 83 of file datagatherer.cpp.

References timer::add_time(), AppendData(), AppendOnlyPosReal(), buffersize, MakeTargetFilename(), timer::press_start(), ProgSettings::settings, slavemap, and WriteData().

Referenced by slave_process::LoopBasic(), and slave_process::LoopSearch().

Here is the call graph for this function:

Here is the caller graph for this function: