Fine-tuning LLM with QLoRA on Single GPU: Training Falcon-7b on ChatBot Support FAQ Dataset

Fine-tuning Falcon 7B on a single GPU is practical even with a tiny, FAQ-style dataset—using QLoRA to train only a small slice of parameters. The workflow takes roughly minutes end-to-end on a Google Cloud Tesla T4 setup and produces noticeably more on-dataset, support-like answers after training. The key takeaway is that parameter-efficient fine-tuning can steer a general-purpose open model toward a specific chatbot domain without needing large clusters or massive datasets.

The process starts by setting up the environment for 4-bit QLoRA training: bitsandbytes with PyTorch 2.0.1, Hugging Face Transformers, Accelerate, and a QLoRA configuration that enables double quantization while computing in 16-bit precision. Falcon 7B weights are loaded in 4-bit, and a tokenizer is initialized from the Falcon 7B checkpoint, with the padding token aligned to the EOS token. Instead of updating the full model, QLoRA freezes the base model and trains low-rank adapter matrices (LoRA) on top—so only about 0.13% of parameters become trainable. Gradient checkpointing is enabled to trade compute for reduced GPU memory usage.

Training data comes from the “e-commerce-fake-Q-chatbot” dataset on Hugging Face, credited to Muhammad Mahmud. The example used here is intentionally small: 79 question–answer pairs (about 20 KB) extracted from a JSON file and flattened into a simple question/answer structure. Each example is formatted into an instruction-style prompt using a “Human:” / “Assistant:” template, then tokenized with truncation and padding. The dataset is shuffled and mapped into model-ready features (input IDs, attention masks, and related fields).

The fine-tuning run uses a single epoch with batch size 1, a cosine learning-rate scheduler, and a paged AdamW optimizer in 8-bit—an efficiency-focused choice integrated through the Transformers stack. Training is performed with a language-modeling objective that predicts next tokens (no masked language modeling), and TensorBoard tracks loss/learning-rate behavior. Loss decreases smoothly and the learning rate converges close to zero, with the run reported at about 7.5 minutes for this small dataset.

Before training, the base Falcon 7B tends to fall into a repetitive, nonsensical loop for the “create an account” prompt, repeatedly asking for an email and password. After applying the saved QLoRA adapter (stored as a small ~19 MB adapter artifact plus configuration), the same prompt yields a more coherent, support-style response: directing the user to the signup page and describing the registration/confirmation-email flow. Additional FAQ prompts—tracking orders, return policy, and clearance/final-sale returns—also produce more aligned answers, often referencing typical policy constraints (e.g., return windows, condition requirements) and escalating to customer support when appropriate.

Overall, the experiment demonstrates that QLoRA can turn Falcon 7B into a more reliable FAQ chatbot on a single GPU, provided the prompt format matches the training data and the adapter is applied correctly at inference time.