HomeAIWonderful Tuning LLMs on a Single Shopper Graphic Card | by Naser...

Wonderful Tuning LLMs on a Single Shopper Graphic Card | by Naser Tamimi | Jan, 2024


Learnings from superb tuning a big language mannequin on a single shopper GPU

Picture by Creator (Midjourney).

After we take into consideration Giant Language Fashions or another generative fashions, the primary {hardware} that involves thoughts is GPU. With out GPUs, many developments in Generative AI, machine studying, deep studying, and information science would’ve been inconceivable. If 15 years in the past, players had been enthusiastic concerning the newest GPU applied sciences, at the moment information scientists and machine studying engineers be a part of them and pursue the information on this subject too. Though often players and ML customers are two completely different sorts of GPUs and graphic playing cards.

Gaming customers often use shopper graphic playing cards (resembling NVIDIA GeForce RTX Collection GPUs), whereas ML and AI builders often observe information about Knowledge Heart and Cloud Computing GPUs (resembling V100, A100, or H100). Gaming graphic playing cards often have a lot much less GPU reminiscence (at most 24GB as of January 2024) in comparison with Knowledge Heart GPUs (within the vary of 40GB to 80GB often). Additionally, their value is one other important distinction. Whereas most shopper graphic playing cards could possibly be as much as $3000, most Knowledge Heart graphic playing cards begin from that value and may go tens of hundreds of {dollars} simply.

Since many individuals, together with myself, might need a shopper graphic card for his or her gaming or day by day use, they could be to see if they will use the identical graphic playing cards for coaching, fine-tuning, or inference of LLM fashions. In 2020, I wrote a complete article about whether or not we will use shopper graphic playing cards for information science tasks (hyperlink to the article). At the moment, the fashions had been largely small ML or Deep Studying fashions and even a graphic card with 6GB of reminiscence might deal with many coaching tasks. However, on this article, I’m going to make use of such a graphic card for giant language fashions with billions of parameters.

For this text, I used my Geoforce 3090 RTX card which has 24GB of GPU reminiscence. On your reference, information middle graphic playing cards resembling A100 and H100 have 40GB and 80GB of reminiscence respectively. Additionally, a typical AWS EC2 p4d.24xlarge occasion has 8 GPUs (V100) with a complete of 320GB of GPU reminiscence. As you may see the distinction between a easy shopper…

Supply hyperlink

latest articles

Head Up For Tails [CPS] IN
ChicMe WW

explore more