- tags: AI
HuggingFace
Links to this note
Set cache_file_name to avoid running Dataset.map every time
tags: AI, HuggingFace, Fine-Tune source: https://discuss.huggingface.co/t/how-to-save-a-mapped-dataset/42303/4 I have a Dataset.map that runs every time I run the script: train_ds.map(prepare_dataset, num_proc=1) I solved it by setting cache_file_name : train_ds = train_ds.map(prepare_dataset, num_proc=1, cache_file_name="trans.ds.cache")