Fine-tuning Tiny LLM on Your Data | Sentiment Analysis with TinyLlama and LoRA on a Single GPU

Fine-tuning a “tiny” LLM on a custom dataset can deliver strong sentiment and topic predictions using a single GPU—provided the training setup is tuned carefully. The core workflow pairs TinyLlama with LoRA adapters, formats each example so it fits within the model’s context window, trains only the adapter weights, then merges the adapter back into the base model for straightforward inference. The payoff is practical: faster training and inference than larger models, plus the ability to keep private company data inside the fine-tuning pipeline.

The case study targets two outputs from crypto news: (1) sentiment (positive/neutral/negative) and (2) subject (Bitcoin, altcoin, blockchain, Ethereum, NFT, DeFi). Instead of relying on prompt engineering alone, the approach assumes tiny models won’t match larger systems on benchmarks without adaptation. Fine-tuning is also framed as a way to reduce prompt length—using a compact template that elicits structured outputs—while avoiding data exposure by training on internal datasets.

Dataset preparation centers on quality and distribution. The transcript recommends more than a thousand high-quality examples and uses a stratified train/validation/test split to preserve label frequencies across subjects and sentiment classes. The crypto dataset (Crypton News) is structured with title, text, subject, and sentiment fields, while additional sentiment metadata (like subjectivity) is used only for analysis, not prediction. A key warning emerges from the real-world distribution: the data is biased toward Bitcoin and blockchain, and sentiment is skewed toward neutral/positive. The example keeps the original distributions rather than applying rebalancing techniques, but notes oversampling/undersampling as possible remedies.

On the modeling side, the setup loads TinyLlama (non-chat variant) and adds a padding token to the tokenizer, then resizes token embeddings so the model can pad correctly. Incorrect padding can cause repetition artifacts, so padding configuration is treated as a critical detail. The transcript also checks token counts after applying a custom template (title + text + a structured “prediction” section containing subject and sentiment) to ensure inputs stay well under the model’s 2048 context window.

LoRA is configured to make training feasible on a free-tier Google Colab T4 GPU. Rather than updating the full model, LoRA injects trainable low-rank matrices into targeted layers—specifically self-attention and MLP linear layers—while the base weights remain frozen. The example uses a LoRA rank (r) of 128 and LoRA alpha scaled accordingly, with a small dropout. Training uses fp16 and a standard AdamW optimizer (no quantized optimizer), runs for one epoch, and employs a completion-only loss strategy via a data collator that masks tokens outside the “prediction” template.

Evaluation shows the practical results: on a test set of about 1,242 examples, subject prediction accuracy lands around 78.6% using a rough metric, while sentiment accuracy reaches just over 90%. The transcript includes qualitative samples where label noise appears—sometimes the model’s sentiment or subject seems more reasonable than the provided labels—suggesting that real-world datasets can be imperfect even when they look structured. After training, the adapter is merged back into the base model, saved with the tokenizer, and used through a text-generation pipeline to produce structured outputs from new titles and article text.