Inethiwekhi ye-Neural YaLM 100B ekusebenzeni.

Ekupheleni kukaJuni, i-Yandex
ikhuphe i- neural network kunye ne-100 yeebhiliyoni zeeparamitha ezibizwa ngokuba yi-YaLM 100B kuluntu . Yeyona nethiwekhi inkulu efana ne-GPT ye-neural kwindawo yoluntu. Isixelela ngendlela abafundise ngayo, babonise eyona mizekelo ibalaseleyo kunye nokuba yintoni ekwaziyo ukuyenza i-neuron. Kodwa ngaba ilungile ekusebenzeni kwaye iyasebenza ekhaya? Inqaku lithe cwaka malunga noku, ngaphezu koko, akulula kangako ukuyiqhuba kwaye uyijonge, kuba malunga ne-200 Gb ye-GPU RAM iyafuneka. La magqabantshintshi ngoHabré atyhila
imeko ngokuchaneke kakhulu
.

Kuyacaca ukuba, kwiYandex, bonke abantu abanjalo abahlakaniphile, kwaye abazange bathumele i-How-to eqhelekileyo. Ayikho i-api yemodeli enkulu, akukho modeli eyenziwe yahluthwa ephakathi okanye encinci yabantu abaqhelekileyo (kwiGoogle Colab). Akukho mzekelo unikiweyo malunga nendlela yokuseta imodeli, indlela yokuvelisa umbhalo. Kuphela nje ukuba inqaku libonisa isibini nuances for Nerds kwaye yiyo. Kwanele ukujonga ngokuthe ngqo indlela ibhanki eyenze ngayo ngeleta “C” kwaye wenze okufanayo. Ndafumana ingcamango yokuba lo mzekelo ngomnye nje imifuniselo engaphumelelanga ukuba lusizi ukulahla kwinkunkuma, ngoko ke iposelwe kuMthombo ovulekileyo ukubonisa ukuba yeyiphi imifuziselo emikhulu Yandex idala, kwaye ngaphezu koko, ngumthombo ovulekileyo!

Kukho imibuzo emininzi kwi-Intanethi ukuba ungayiqhuba njani i-yalm okanye uzame i-intanethi, kodwa akukho zimpendulo kule nto. Ndandiphakathi kwabasebenzisi ababuza le mibuzo. Kwaye qalisa ukucinga. Ekubeni ndandiyidinga ngokwenene indlela yokuvelisa izicatshulwa zeerobhothi zemali. Ukuze bakwazi ukuqikelela kuphela amaxabiso, kodwa baphinde baphawule ngesicatshulwa, ngokusekelwe kwiingxelo zemali. Ngokwenene, kuya kufana nento eyenziwa ngabahlalutyi bezemali, kuphela ngokusetyenziswa kwengqondo yokwenziwa. Kukho iindlela ezimbini zokusebenzisa i-yam.
Qasha iseva efininge-200+ Gb GPU RAM okanye uguqule ikhowudi kwaye uqhube nge-deepspeed zero offload (xa i-GPU iqhuba ngokulandelelana inxalenye yenethiwekhi ye-neural, kwaye enye igcinwe kwi-CPU RAM okanye i-NVMe). Eyokuqala ibiza kakhulu, malunga nama-ruble angama-2500 ngeyure okanye i-1.7 yezigidi ngenyanga. Eyesibini engaziwa, kuba ikhowudi kwindawo yokugcina ayibonelelwanga,
iingcebiso kuphela kumcimbi wogcino, okungekho nzima ukwenza. Masiqale ngokulula.

YaLM 100B Imiyalelo yokuQalisa

1. Sirenta i-200 GB GPU RAM, umzekelo apha .

Udinga ubuncinci be-200 GB yememori yevidiyo iyonke. 8×40 = 320 GB. Inye kuphela ifanelekile. Ngaphantsi kwe-200 ayinakwenzeka, ngakumbi kunokwenzeka. Utolo lubonisa i-RAM ye-CPU, asiyijongi. Unokuba nabani na.

Sibonisa idiski malunga ne-300 GB, ukwenzela ukuba kunye ne-spare kwaye ngokukhethekileyo idiski ekhawulezayo, kuba. amashumi eegigabhayithi zedatha ziya kuthunyelwa kwaye zisuka kuyo.

Xa udala kwimithombo, khetha Ubuntu ML (Ukufunda ngoMatshini). Oku kunyanzelekile ukwenzela ukuba amakhadi evidiyo aqwalaselwe kwaye akukho nto idinga ukufakwa ukongeza.

Xa udala iseva, kukho ama-nuances anee-quotas, unokufumana imvakalelo yokuba izixhobo azikho, kodwa ngokwenene ufuna nje ukwandisa i-quotas kwizicwangciso. Emva kokuba umncedisi evuliwe (kungathatha imizuzu emi-5-10), qhagamshela kumncedisi nge-ssh okanye ngokuthe ngqo kwi-console yewebhu kwiphepha leseva kwaye wenze umyalelo.

nvidia-smi

Isiphumo kufuneka sibe yitafile enamakhadi evidiyo, inguqulo yomqhubi kunye ne-cuda. Ngokumalunga noku.
Kwisihloko soguqulelo lomqhubi nalapho. Kwicala lasekhohlo kukho iinombolo zefowuni, embindini ubukhulu bememori yesixhobo. Ukuba awunalo olu lwazi, ngoko uqokelele umncedisi kumthombo ongalunganga. Ubuntu ML (Ukufunda ngoMatshini) kuyafuneka, njengoko kuchaziwe ngasentla.

2. Clone indawo yokugcina ngeYaLM

sudo git clone https://github.com/yandex/YaLM-100B/ yalm
cd yalm

Vala kwifolda yakho yasekhaya ukuze ungahleleli i-docker config emva koko. Ukuba yenziwe kwenye indawo,
yiya apha kwaye wongeze indlela eya apho yenziwe khona.

3. Khuphela iindawo zokukhangela (ingcaciso yoqeqesho olusisiseko)

sudo chmod +x ./download/download.sh
sudo bash ./download/download.sh

Oku kuyakuthatha malunga neyure. Ukuze singachithi ixesha ngelize, senza uxhulumaniso olutsha lwe-ssh kwaye ngokufanayo siqala ukwakha isitya se-docker.

4. Faka i- nvidia – docker 2

Idocker eqhelekileyo ayifanelekanga,
i-nvidia-docker2 iyafuneka .
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#setting-up-nvidia-container-toolkit

5. Ukwakha isitya seYaLM

cd yalm
sudo chmod +x ./docker/*
sudo bash ./docker/build.sh

Kwakhona malunga neyure.

I-Hack yobomi. Unokukhuphela iindawo zokukhangela, ufake i-docker kwaye wakhe isitya kwi-server ephantsi ngekhadi levidiyo enye. Kuya kufana ngexesha, ngoko unokonga kancinci. Emva kokuhlanganisana kumncedisi wexabiso eliphantsi, siyayicima, kwaye senze iseva yokulwa sisebenzisa idiski kwiseva ephantsi. Emva koko awuyi kulihlawula ngaphezulu ixesha lokulinda indibano kunye nokupompa iindawo zokuhlola.

6. Lungisa umxholo

6.1 Iindawo zokuhlola

Emva kokuba ukhuphelo lwee-checkpoints luphelile, kufuneka uzifake kwi-configs. Kukho iindlela ezimbini, iiparamitha ezichanekileyo okanye iindawo zokutshekisha ezichanekileyo. Kuyo yonke indawo kulindeleke ukuba iindawo zokutshekisha ziya kuba kuluhlu oluphambili lweprojekthi, ngokulandelelanayo, oko kukhutshelweyo makukhutshelwe kwifolda yokukhuphela engasentla. Ukuba kwifolda yeyam phumeza

mv ./download/yalm100b_checkpoint ./

Okanye utshintshe iindlela kwiifayile kwiifayile
zomzekelo https://github.com/yandex/YaLM-100B/blob/c91b7d7fe8dbf39c9e307d6d324446d0df136a23/examples/generate_interactive.sh#L8-L9

6.2 Amakhadi evidiyo

Sijonga ukuba amakhadi evidiyo abekwe ngokuchanekileyo. Ukuba unamakhadi evidiyo asibhozo, akukho nto ifuna ukutshintshwa. Ukuba inani lihlukile, ngoko sitshintsha le migca
Kumgca wesibini, amanani ezixhobo ezisetyenzisiweyo (unokuzijonga kwi-nvidia-smi, osele uqalise). Kweyesine, inani labo.

7. Qhuba isikhongozeli sedokhi

Ukuba kwifolda yeyam, yenza umyalelo

sudo bash ./docker/run.sh

Ukuba yonke into ilungile, ke uya kusiwa kwisikhongozeli apho kufuneka uye kwifolda ye-yalm kulawulo lwakho lwasekhaya.

cd ~/yalm

8. Qhuba umzekelo kwi-YaLM 100B

Sikulungele ukusungula omnye wemizekelo. Zichazwe
apha .

chmod +x ./examples/generate_interactive.sh
./examples/generate_interactive.sh

Yiba nomonde, uhlala ulinde enye imizuzu eyi-10-15 de imodeli ye-GPT yenziwe kwaye izisindo ezivela kwiindawo zokutshekisha zilayishiwe.

Xa ukwakhiwa kugqityiwe, iMegatronML iya kukukhuthaza ukuba ufake umxholo ukuvelisa umbhalo. Lumka xa uchwetheza. Ngaphantsi kweemeko ezithile, impazamo iyenzeka, inkqubo iyaphazamiseka kwaye kufuneka uqalise indibano kwakhona. Ngoko ke, kungcono ukusebenzisa imizekelo ethatha umbhalo kwifayile.

9. Iziphumo zomsebenzi

Ijongeka inika umdla. Kakade ke, le yimizekelo nje emihle. Ndaluqhuba uvavanyo kwiisampuli ezahlukeneyo. Njengoko kulindelekile, kokukhona umxholo ungcono, kokukhona isicatshulwa esinentsingiselo ngakumbi siya kuveliswa. Iseti epheleleyo yezizukulwana zovavanyo zinokujongwa kumakhonkco:

• generate_conditional_sampling.sh
• generate_conditional_greedy.sh
• generate_unconditional.sh

Ngexabiso, lindibiza malunga ne-9 lamawaka ee-ruble zokurenta amaseva amanqanaba ahlukeneyo ukusuka kuqeqesho kunye nokulungiselela ukuya kwisizukulwana. Ukuphoxeka okuthile yayikukuba awukwazi ukuvelisa yonke into kwangoko. Kuthatha ixesha elide kakhulu ukuqalisa kwaye okubhaliweyo akuvelisi ngokukhawuleza njengoko besinokuthanda, kunikwe ixabiso lomncedisi ngeyure.
 

Uyiqhuba njani iYaLM ngaphandle kwe200Gb GPU RAM?

Kufuneka udibanise i-zero enzulu yokukhuphela kwi-config. Kwabo bayaziyo le nto sithetha ngayo, kuya kuba lula kakhulu ukuyenza. Kwabanye, lo asingomsebenzi omncinane kwaphela. Kubalulekile ukwazi ukuba ukhuphelo lunokuba kwi-CPU RAM okanye iNVMe. Unokulibala malunga neNVMe okwangoku, kuba. isixa esikhulu kakhulu sedata siqhubekekiswa kwaye idiski ayikwazi ukumelana nayo. I-Zero yokukhuphela i-CPU iyinyani ngakumbi. Kuyinyani, oku kufuneka ube ne-200+ Gb CPU RAM kwisitokhwe, nayo ingabizi. Kwaye isicatshulwa esinye siya kuveliswa malunga nemizuzu engama-20 ukuya kwengama-40, kuba akukabikho nto inokwenzeka ukuyifanisa kumakhadi amabini evidiyo. Njengoko unokubona kwi-skrini engezantsi, kuphela ikhadi levidiyo elilodwa elibandakanyekayo kwisizukulwana, kwaye emva koko kuphela kwikota yememori. Kuya kuhlala kubonakala ukuba kutheni yonke i-24 GB ingasetyenziswanga,
Ewe, ekuqukumbeleni, ndiza kuthetha ukuba kunokwenzeka ukubaleka nakwi-RTX 3070 TI enye. Kodwa akukho ngqiqo ethile kule nto, kuba. I-NVMe ayiyi kukuvumela ukuba usebenze ngokukhawuleza i-150 GB yedatha kwi-swap, ekwi-appendage ye-96 GB ye-RAM.

Ukushwankathela

Ewe kunjalo, ndisezakuzama ukufumana iindlela ezifanelekileyo zokuqalisa. Kodwa ukuza kuthi ga ngoku ndifikelele kwisigqibo sokuba i-YaLM 100b ibiza kakhulu / icotha kakhulu kwimisebenzi yam. Kwimali efanayo, abantu baya kubhala ngakumbi kwaye ngcono kakhulu. Kodwa ndicinga ukuba yinto yexeshana, siza kubona. Ukuba ufuna uncedo ngokuqalisa, ukuseta i-yalm, okanye ufuna ukubona iziphumo kwimizekelo yomxholo wakho, bhalela iposi okanye itelegram.