now, dataloaders can return arrays of data, which will be handled as separate runs by the remaining pipeline. Therefore, the structure of the loaders had to be modified in the following way:
- evaluator objects no longer have access to the dataloader object itself, but rather to the data returned by them. This data is now more general and more sanity check must be implemented by the evaluators.
- existing dataloaders and evaluators should work as before. If a dataloader does not overwrite the
run_countpropery, it is handled as single-run. - dataloaders overwriting the
run_countproperty can provide a separate configuration for each run
Please check if this change would break too many custom evaluators/dataloaders from your side.