# ValueError: Expected input batch\_size (1024) to match target batch\_size (4)

**URL:** <https://discuss.huggingface.co/t/valueerror-expected-input-batch-size-1024-to-match-target-batch-size-4/134607>\
**Category:** Beginners\
**Created:** [January 7, 2025, 9:10am UTC](https://discuss.huggingface.co/t/valueerror-expected-input-batch-size-1024-to-match-target-batch-size-4/134607 "2025-01-07T09:10:05Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![adenaiml](https://avatars.discourse-cdn.com/v4/letter/a/c37758/32.png) [@adenaiml](https://discuss.huggingface.co/u/adenaiml)\
**Post date:** [January 7, 2025, 9:10am UTC](https://discuss.huggingface.co/t/valueerror-expected-input-batch-size-1024-to-match-target-batch-size-4/134607/1 "2025-01-07T09:10:05Z")

</div>

I want to perform LoRA Fine-tuning on an llama3 model, using the following dataset:  
[sst2\_sent.json](https://github.com/xlhex/emnlp2023_z-defence/blob/main/sst2_sent.json)

**sst2\_sent.json example**

```auto
{"sentence": "hide new secretions from the parental units", "label": 0}
{"sentence": "contains no wit , only labored gags", "label": 0}
{"sentence": "that loves its characters and communicates something rather beautiful about human nature", "label": 1}
{"sentence": "remains utterly satisfied to remain the same throughout", "label": 0}
........

```

I have split the data set into training data and testing data.

But I get the following error while training:

> ValueError: Expected input batch\_size (1024) to match target batch\_size (4).

Here is my code:

```auto
def tokenize_dataset(dataset, tokenizer):
    def preprocess_function(examples):
        inputs = tokenizer(examples["sentence"], padding="max_length", truncation=True, max_length=128)
        inputs["labels"] = examples["label"]
        return inputs
    return dataset.map(preprocess_function, batched=True)

```

```auto
def load_model(model_name):
    """Load the model and configure LoRA
    """
    model = AutoModelForTokenClassification.from_pretrained(
        model_name,
        quantization_config=quantization_config,
        device_map='cuda:0',
        num_labels=2
    )

    lora_config = LoraConfig(
        r=16,  
        lora_alpha=8,  
        target_modules=['q_proj', 'k_proj', 'v_proj', 'o_proj'],
        lora_dropout=0.05,  
        bias='none', # wether to train bias weights, set to 'none' for attention layers
        task_type='SEQ_CLS'
    )
    model = prepare_model_for_kbit_training(model)
    return get_peft_model(model, lora_config)

```

```auto
def train_model(model, train_dataset, test_dataset, tokenizer, output_dir):
    """Set training parameters and fine-tune them"""
    training_args = TrainingArguments(
        output_dir=output_dir,
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        logging_steps=10,
        num_train_epochs=3,
        save_steps=100,
        learning_rate=1e-4,
        save_on_each_node=True,
        gradient_checkpointing=True
    )
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=test_dataset,
        data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer, padding=True),
    )

    print("Staring to training...")
    trainer.train()
    print("Finished to training...")
    trainer.save_pretrained(output_dir) # Save model
    tokenizer.save_pretrained(output_dir) # Save tokenizer

```

**main.py :**

```auto
# 1. Load dataset
print("Loading and processing dataset...")
dataset = load_dataset("json", data_files={
    "train": "train.json",
     "test": "test.json"})
train_dataset = dataset["train"]
test_dataset = dataset["test"]

# 2. Tokenization
print("------------------------------------------------------------")
print("Start the Tokenization dataset...")
tokenizer = AutoTokenizer.from_pretrained(llm_name, add_prefix_space=True)
tokenizer.pad_token_id = tokenizer.eos_token_id
tokenizer.pad_token = tokenizer.eos_token
tokenized_train = tokenize_dataset(train_dataset, tokenizer)
tokenized_test = tokenize_dataset(test_dataset, tokenizer)

```

```auto
# 3. Model fine-tuning
print("------------------------------------------------------------")
print("Loading model and starting fine-tuning...")
model = load_model("meta-llama/Meta-Llama-3-8B-Instruct")
model.config.pad_token_id = tokenizer.pad_token_id
model.config.use_cache = False
train_model(model, tokenized_train, tokenized_test, tokenizer, "./llm-lora-fine-tuned")

```

I’ve seen a few posts similar, but none of the proposed answers were useful for this case. Has this ever happened to you?  
Would love some insight.  
Thanks in advance.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [January 7, 2025, 4:28pm UTC](https://discuss.huggingface.co/t/valueerror-expected-input-batch-size-1024-to-match-target-batch-size-4/134607/2 "2025-01-07T16:28:00Z")

</div>

I tried to reproduce it, but no matter what I did, DataCollatorForSeq2Seq kept failing…  
If you use CausalLM and DataCollatorForLanguageModeling together, it will work, but it probably won’t be the training you want.  
The fastest thing to do might be to inherit from DataCollatorForSeq2Seq and rewrite part of it.

> <https://github.com/tloen/alpaca-lora/issues/412>
>
> AFAIK, llama models are decoder-only so we should use a collator for language mo…del and set \`mlm=False\`. I am wondering what the reason is for picking \`DataCollatorForSeq2Seq\`? It seems to me that \`DataCollatorForSeq2Seq\` is for a encoder-decoder model like T5.

> <https://stackoverflow.com/questions/75854700/how-to-fine-tune-a-huggingface-seq2seq-model-with-a-dataset-from-the-hub>