Iterative Data Loading with XGBoost

Datawhale Insights Source: Coggle Data Science

During the process of reading and training on large-scale datasets, iterative data loading is a very suitable choice, and Pytorch supports this method of iterative reading. Next, we will introduce the iterative loading method of XGBoost.

Memory Data Loading

class IterLoadForDMatrix(xgb.core.DataIter):
    def __init__(self, df=None, features=None, target=None, batch_size=256*1024):
        self.features = features
        self.target = target
        self.df = df
        self.batch_size = batch_size
        self.batches = int( np.ceil( len(df) / self.batch_size ) )
        self.it = 0 # set iterator to 0
        super().__init__()

    def reset(self):
        '''Reset the iterator'''  
        self.it = 0

    def next(self, input_data):
        '''Yield next batch of data.'''  
        if self.it == self.batches:
            return 0 # Return 0 when there's no more batch.  
        
        a = self.it * self.batch_size
        b = min( (self.it + 1) * self.batch_size, len(self.df) )
        dt = pd.DataFrame(self.df.iloc[a:b])
        input_data(data=dt[self.features], label=dt[self.target]) #, weight=dt['weight'])
        self.it += 1
        return 1

Method of calling (this method is more suitable for GPU training):

Xy_train = IterLoadForDMatrix(train.loc[train_idx], FEATURES, 'target')
dtrain = xgb.DeviceQuantileDMatrix(Xy_train, max_bin=256)

Reference Document:

https://xgboost.readthedocs.io/en/latest/python/examples/quantile_data_iterator.html

External Data Iterative Loading

class Iterator(xgboost.DataIter):
  def __init__(self, svm_file_paths: List[str]):
    self._file_paths = svm_file_paths
    self._it = 0
    super().__init__(cache_prefix=os.path.join(".", "cache"))

  def next(self, input_data: Callable):
    if self._it == len(self._file_paths):
      # return 0 to let XGBoost know this is the end of iteration
      return 0

    X, y = load_svmlight_file(self._file_paths[self._it])
    input_data(X, y)
    self._it += 1
    return 1

  def reset(self):
    """Reset the iterator to its beginning"""
    self._it = 0

Method of calling (this method is more suitable for CPU training):

it = Iterator(["file_0.svm", "file_1.svm", "file_2.svm"])
Xy = xgboost.DMatrix(it)

# Other tree methods including ``hist`` and ``gpu_hist`` also work, but has some caveats
# as noted in following sections.
booster = xgboost.train({"tree_method": "approx"}, Xy)

Reference Document:

https://xgboost.readthedocs.io/en/stable/tutorials/external_memory.html

It’s not easy to organize,pleaselikeand share↓