# Data not passed to run\_summarization.py

**URL:** <https://discuss.huggingface.co/t/data-not-passed-to-run-summarization-py/15355>\
**Category:** Amazon SageMaker\
**Created:** [March 4, 2022, 2:28pm UTC](https://discuss.huggingface.co/t/data-not-passed-to-run-summarization-py/15355 "2022-03-04T14:28:41Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![thusken](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thusken/32/27787_2.png) [@thusken](https://discuss.huggingface.co/u/thusken)\
**Post date:** [March 4, 2022, 2:28pm UTC](https://discuss.huggingface.co/t/data-not-passed-to-run-summarization-py/15355/1 "2022-03-04T14:28:42Z")

</div>

Hello!

I’m in the process of trying to finetune the `NbAiLab/nb-t5-base` model on my own dataset. The training job starts fine when calling `huggingface_estimator.fit()`, but gives a `ValueError` after the training image has been downloaded:

```auto
ValueError: Need either a dataset name or a training/validation file.

```

So it appears that the training data that is passed in `huggingface_estimator.fit()` is not  
passed correctly to the `run_summarization.py` script. Any pointers? Simplified script is below.

```auto
CHECKPOINT = "NbAiLab/nb-t5-base"

train_dataset = Dataset.from_pandas(train_df) # df with columns 'text' and 'labels'
test_dataset = Dataset.from_pandas(test_df)

tokenizer = AutoTokenizer.from_pretrained(CHECKPOINT)

def tokenize(batch):
    return tokenizer(batch['text'], padding='max_length', truncation=True)
   
train_dataset = train_dataset.map(tokenize, batched=True)
test_dataset = test_dataset.map(tokenize, batched=True)

train_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'labels'])
test_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'labels'])

s3 = S3FileSystem()  

# save train_dataset to s3
training_input_path = f's3://{sess.default_bucket()}/data/train'
train_dataset.save_to_disk(training_input_path,fs=s3)

# save preprocessed test data to S3
test_input_path = f"s3://{session.default_bucket()}/data/test"
test_dataset.save_to_disk(test_input_path, fs=s3)

hyperparameters = {
	"model_name_or_path":CHECKPOINT,
	"output_dir":"/opt/ml/model",
        "epochs": 1,
        "train_batch_size": 32
}

# git configuration to download our fine-tuning script
git_config = {'repo': 'https://github.com/huggingface/transformers.git','branch': 'v4.6.1'}

# creates Hugging Face estimator
huggingface_estimator = sagemaker.huggingface.HuggingFace(
	entry_point='run_summarization.py',
	source_dir='./examples/pytorch/summarization',
	instance_type='ml.p3.2xlarge',
	instance_count=1,
        checkpoint_s3_uri=f's3://{sess.default_bucket()}/checkpoints',
        use_spot_instances=True,
        max_wait=3600,
        max_run=1000,
	role=role,
	git_config=git_config,
	transformers_version='4.6.1',
	pytorch_version='1.7.1',
	py_version='py36',
	hyperparameters = hyperparameters
)

```

---

<div class="post-metadata">

**Author:** ![anwarika](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@anwarika](https://discuss.huggingface.co/u/anwarika)\
**Post date:** [March 4, 2022, 4:21pm UTC](https://discuss.huggingface.co/t/data-not-passed-to-run-summarization-py/15355/2 "2022-03-04T16:21:01Z")

</div>

how are you calling `fit()` method?

---

<div class="post-metadata">

**Author:** ![thusken](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thusken/32/27787_2.png) [@thusken](https://discuss.huggingface.co/u/thusken)\
**Post date:** [March 7, 2022, 8:12am UTC](https://discuss.huggingface.co/t/data-not-passed-to-run-summarization-py/15355/3 "2022-03-07T08:12:30Z")

</div>

So, looks like it was a typo causing the problems. I have used both `sess.default_bucket()` and `sessìon.default_bucket()`, while `sess` is never defined. My bad!

For reference, I called the fit method as follows:

```auto
#starting the train job
huggingface_estimator.fit(
    inputs={
        "train": training_input_path,
        "test": test_input_path
    }
)

```

---

<div class="post-metadata">

**Author:** ![thusken](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thusken/32/27787_2.png) [@thusken](https://discuss.huggingface.co/u/thusken)\
**Post date:** [March 8, 2022, 1:48pm UTC](https://discuss.huggingface.co/t/data-not-passed-to-run-summarization-py/15355/4 "2022-03-08T13:48:05Z")

</div>

Unfortunately, the problem persists even after fixing the typo’s in the default buckets.

Calling the `.fit()` method as follows still results in the same error. I have verified that `training_input_path` and `test_input_path` point to the correct location on S3 (the location that is passed to the `.save_to_disk()` call from two `Dataset` objects`)

```auto
huggingface_estimator.fit(
    {
        "train": training_input_path,
        "test": test_input_path
    }
)

```

This still results in the following `ValueError`, any help is still appreciated!

```auto
ValueError: Need either a dataset name or a training/validation file.

```

---

<div class="post-metadata">

**Author:** ![thusken](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thusken/32/27787_2.png) [@thusken](https://discuss.huggingface.co/u/thusken)\
**Post date:** [March 8, 2022, 1:57pm UTC](https://discuss.huggingface.co/t/data-not-passed-to-run-summarization-py/15355/5 "2022-03-08T13:57:15Z")

</div>

Looking at the source code for `run_summarization.py`, the error seems to be in the ` __post_init__ ` call of `DataTrainingArguments` here:

```auto
if self.dataset_name is None and self.train_file is None and self.validation_file is None:
            raise ValueError("Need either a dataset name or a training/validation file.")

```

So it looks like the data from S3 is not parsed to `train_file`, looking at the `train_file` argument it seems to expect a csv or json, rather than a location on S3

```auto
train_file: Optional[str] = field(
        default=None, metadata={"help": "The input training data file (a jsonlines or csv file)."}
    )

```

Am I right in conluding from this that the `run_summarization` script cannot be used with data from S3?

---

<div class="post-metadata">

**Author:** ![anwarika](https://avatars.discourse-cdn.com/v4/letter/a/e0b2c6/32.png) [@anwarika](https://discuss.huggingface.co/u/anwarika)\
**Post date:** [March 8, 2022, 2:15pm UTC](https://discuss.huggingface.co/t/data-not-passed-to-run-summarization-py/15355/6 "2022-03-08T14:15:26Z")

</div>

If you are using run\_summarization.py you need to tell it where to look. Sagemaker will load your s3 data into `/opt/ml/input/data/train/whatever.csv` or `/opt/ml/input/data/test/whatever.csv`, so pass in another parameter in the hyperparameters. ie

```auto
hyperparameters={
    'model_name_or_path': MODEL,
    'train_file': "/opt/ml/input/data/train/whatever_train.csv",
    'test_file': "/opt/ml/input/data/test/whatever_test.csv",
etc

```

---

<div class="post-metadata">

**Author:** ![thusken](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thusken/32/27787_2.png) [@thusken](https://discuss.huggingface.co/u/thusken)\
**Post date:** [March 8, 2022, 2:20pm UTC](https://discuss.huggingface.co/t/data-not-passed-to-run-summarization-py/15355/7 "2022-03-08T14:20:02Z")

</div>

Thanks for the tip! I was using the `save_to_disk()` method from the `Dataset` object which (I believe) writes the entire `Dataset` object to S3 as an `.arrow` file. I’ll try it out with a simple csv file.

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [March 8, 2022, 2:35pm UTC](https://discuss.huggingface.co/t/data-not-passed-to-run-summarization-py/15355/8 "2022-03-08T14:35:53Z")

</div>

@thusken if you would like to use your dataset, which was saved with `save_to_disk` you would need to fork the `examples/` script you use and replace the `load_dataset` method with `load_from_disk` method. [[REF](https://huggingface.co/docs/datasets/master/en/package_reference/loading_methods#datasets.load_from_disk)].

---

<div class="post-metadata">

**Author:** ![thusken](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thusken/32/27787_2.png) [@thusken](https://discuss.huggingface.co/u/thusken)\
**Post date:** [March 8, 2022, 2:42pm UTC](https://discuss.huggingface.co/t/data-not-passed-to-run-summarization-py/15355/9 "2022-03-08T14:42:02Z")

</div>

Good to know @philschmid ! I’ll try it out with the CSV approach first, but might give it a go with the `load_from_disk` method if that doesn’t work.