- tags: AI, HuggingFace, Fine-Tune
- source: https://discuss.huggingface.co/t/how-to-save-a-mapped-dataset/42303/4
I have a Dataset.map
that runs every time I run the script:
train_ds.map(prepare_dataset, num_proc=1)
I solved it by setting cache_file_name
:
train_ds = train_ds.map(prepare_dataset, num_proc=1, cache_file_name="trans.ds.cache")