Neural network YaLM 100B mukuita.

Pakupera kwaChikumi, Yandex yakaburitsa
neural network ine 100 bhiriyoni paramita inonzi YaLM 100B kune veruzhinji . Ndiyo yakakura GPT-yakafanana neural network munzvimbo yeruzhinji. Iyo inotaura nezve madzidzisiro avakaita, yakaratidza yakanakisa mienzaniso uye izvo neuron inokwanisa. Asi yakanaka kwazvo mukuita uye inoshanda kumba? Chinyorwa chacho chinyararire pamusoro peizvi, uyezve, hazvisi nyore kumhanya uye kuzvitarisa, sezvo ingangoita 200 Gb yeGPU RAM inodiwa. Uku kutaura kwaHabré
kunoratidza mamiriro acho nenzira kwayo
.

Sezvineiwo, muYandex, vanhu vese vakangwara vakadaro, uye havana kana kutumira yakajairika How-to. Iko hakuna api yemhando hombe, hapana yakagadzirira-yakagadzirwa yakabviswa-pasi yepakati kana diki modhi yevanhuwo zvavo (muGoogle Colab). Hapana muenzaniso unopiwa wekuti ungamisa sei modhi, maitiro ekugadzira zvinyorwa. Ingori iyo chinyorwa chinoratidza akati wandei nuances kune nerds uye ndizvozvo. Zvakakwana kunyatsotarisa kuti bhangi rakaita sei netsamba “C” uye kuita zvakafanana. Ndakawana fungidziro yekuti iyi modhi ingori imwe yekuedza kwakakundikana yaive tsitsi yekukanda mumarara, saka yakatumirwa muOpen Source kuratidza kuti mamodeli makuru Yandex anogadzira, uyezve, yakavhurika sosi!

Pane mibvunzo yakawanda paInternet maitiro ekumhanya yalm kana kuedza online, asi hapana mhinduro kune izvi. Ndakanga ndiri pakati pevashandisi vakabvunza mibvunzo iyi. Uye tanga kufunga nezvazvo. Sezvo ndaida chaizvo nzira yekugadzira zvinyorwa zvemarobhoti emari. Saka kuti vagone kufanotaura kwete chete kukosha, asiwo kutaura pamusoro payo mumagwaro, zvichienderana nemishumo yemari. Muchidimbu, zvichave zvakafanana nezvinoitwa nevanoongorora zvemari, chete nekushandiswa kweuchenjeri hwekugadzira. Pane nzira mbiri dzekumhanyisa yalm.
Renda sevha mugoreine 200+ Gb GPU RAM kana shandura kodhi uye kumhanya neyakadzama zero kuburitsa (apo iyo GPU inoteedzera chikamu cheneural network, uye imwe yese ichichengetwa muCPU RAM kana NVMe). Yokutanga inodhura zvikuru, inenge 2500 rubles paawa kana mamiriyoni 1.7 pamwedzi. Yechipiri isingazivikanwe, nekuti iyo kodhi mune repository haina kupihwa, inongoratidzira
mune iyo nyaya yekuchengetera, iyo isiri yakaoma kuita. Ngatitange nyore.

YaLM 100B Yekutanga Mirayiridzo

1. Isu tinokwereta 200 GB GPU RAM, semuenzaniso pano .

Iwe unoda inokwana 200 GB yehuwandu hwevhidhiyo memory. 8×40 = 320 GB. Iyi chete inokodzera. Zvisingasviki mazana maviri hazvigoneke, zvakawanda zvinogoneka. Museve unoratidza iyo CPU RAM, isu hatizvitarise. Anogona kuva chero munhu.

Isu tinoratidza dhisiki rinenge 300 GB, kuitira kuti neasina uye zviri nani inokurumidza dhisiki, nekuti. makumi emagigabytes e data achaendeswa uye kubva kwairi.

Paunenge uchigadzira mune masosi, sarudza Ubuntu ML (Muchina Kudzidza). Izvi zvinosungirwa kuitira kuti makadhi evhidhiyo agadziriswe uye hapana chinoda kuiswazve.

Paunenge uchigadzira sevha, pane nuances ine quotas, unogona kuwana manzwiro ekuti michina haisipo, asi muchokwadi iwe unongoda kuwedzera quotas muzvirongwa. Mushure mekunge sevha yaitwa (zvinogona kutora 5-10 maminetsi), batanidza kune sevha kuburikidza ne ssh kana zvakananga muwebhu console pane peji revha uye ita murairo.

nvidia-smi

Mhedzisiro yacho inofanira kunge iri tafura ine makadhi evhidhiyo, mutyairi shanduro uye cuda. Zvinenge sezvizvi.
Mudhiraivha vhezheni musoro uye kupi. Kurutivi rworuboshwe kune nhamba dzemudziyo, pakati pane saizi yendangariro yemudziyo. Kana iwe usina ruzivo urwu, saka watora sevha kubva kune isiriyo sosi. Ubuntu ML (Machine Learnong) inodiwa, sezvatsanangurwa pamusoro.

2. Clone repository neYaLM

sudo git clone https://github.com/yandex/YaLM-100B/ yalm
cd yalm

Clone kune yako folda yekumba kuti haufanirwe kugadzirisa iyo docker config mushure. Kana yakaumbwa kumwe kunhu, wobva
waenda pano wowedzera nzira inoenda kwayakagadzirwa.

3. Dhawunirodha nzvimbo dzekutarisa (yekutanga modhi yekudzidziswa ruzivo)

sudo chmod +x ./download/download.sh
sudo bash ./download/download.sh

Izvi zvinotora inenge awa. Kuti tisatambise nguva pasina, tinogadzira nyowani ssh yekubatanidza uye pamwe chete tinotanga kuvaka docker mudziyo.

4. Isa nvidia – docker 2

Yakajairika docker haina kukodzera,
nvidia-docker2 inodiwa .
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#setting-up-nvidia-container-toolkit

5. Kuvaka mudziyo weYaLM

cd yalm
sudo chmod +x ./docker/*
sudo bash ./docker/build.sh

Inenge zvakare awa.

Hupenyu hack. Unogona kudhawunirodha macheki, isa docker uye kuvaka mudziyo pane yakachipa sevha ine imwe vhidhiyo kadhi. Zvichava zvakafanana munguva, saka unogona kuchengetedza zvishoma. Mushure mekuungana pane yakachipa sevha, tinoidzima, uye tinogadzira sevha yekurwa tichishandisa dhisiki kubva kune yakachipa server. Ipapo hauzobhadhare nguva yekumirira gungano nekupomba macheki.

6. Gadzirira zviri mukati

6.1 Checkpoints

Mushure mekudhawunirodha kwekutarisa kwapera, iwe unofanirwa kutsvedza iwo mumagadzirirwo. Pane nzira mbiri, dzakaringana paramita kana kutamisa cheki. Kwese kwese kunotarisirwa kuti nzvimbo dzekutarisa dzichange dziri mudhairekitori guru repurojekiti, zvichiteerana, izvo zvakadhawunirwa zvinofanirwa kutamiswa kubva kudhawunirodha folda iri pamusoro. Kuve mune yalm folda ita

mv ./download/yalm100b_checkpoint ./

Kana kuti shandura nzira dzemafaira ari mumuenzaniso mafaera
https://github.com/yandex/YaLM-100B/blob/c91b7d7fe8dbf39c9e307d6d324446d0df136a23/examples/generate_interactive.sh#L8-L9

6.2 Vhidhiyo makadhi

Tinotarisa kuti makadhi evhidhiyo akanyatsogadzirirwa. Kana uine makadhi masere evhidhiyo, saka hapana chinoda kuchinjwa. Kana nhamba yacho yakasiyana, saka tinoshandura mitsara iyi
Mumutsara wechipiri, nhamba dzezvishandiso zvinoshandiswa (unogona kuzvitarisa mu nvidia-smi, iyo yawakatotanga). Muchina, nhamba yavo.

7. Mhanya iyo docker mudziyo

Kuve mune yalm folda, ita murairo

sudo bash ./docker/run.sh

Kana zvese zvakanaka, ipapo iwe uchaendeswa kune mudziyo waunoda kuenda kune yalm folda mune yako dhairekitori repamba.

cd ~/yalm

8. Mhanyai muenzaniso kubva kuYaLM 100B

Isu takagadzirira kuburitsa mumwe wemienzaniso. Vanotsanangurwa
pano .

chmod +x ./examples/generate_interactive.sh
./examples/generate_interactive.sh

Iva nemoyo murefu, inoramba ichimirira imwe 10-15 maminetsi kusvika iyo GPT modhi yagadzirwa uye uremu kubva kunzvimbo dzekutarisa dzatakurwa.

Kana iyo kuvaka yapera, MegatronML inokukurudzira kuti uise mamiriro ekugadzira zvinyorwa. Ngwarira paunonyora. Mune mamwe mamiriro ezvinhu, kukanganisa kunoitika, chirongwa chinopunzika uye iwe unofanirwa kutanga gungano zvakare. Naizvozvo, zviri nani kushandisa mienzaniso inotora zvinyorwa kubva mufaira.

9. Migumisiro yebasa

Zvinotaridzika zvinonakidza. Zvechokwadi, iyi inongova mienzaniso yakanaka. Ndakamhanya bvunzo pamasampuli akasiyana. Sezvinotarisirwa, zviri nani mamiriro ezvinhu, zvinyorwa zvine zvazvinoreva zvichagadzirwa. Iyo yakazara seti yezvizvarwa zvekuedza inogona kutariswa pane zvinongedzo:

• generate_conditional_sampling.sh
• generate_conditional_greedy.sh
• generate_unconditional.sh

Pamutengo, zvakandidhura nezve 9 zviuru zveRubles zvekurenda maseva ehunyanzvi hwakasiyana kubva pakudzidziswa uye kubva pakugadzirira kuenda kune chizvarwa. Kunyanya kuodzwa mwoyo kwaive kuti haugone kugadzira zvese ipapo ipapo. Zvinotora nguva yakareba kwazvo kuti utange uye zvinyorwa hazviburitse nekukurumidza sezvatinoda, tichipihwa mutengo weserver paawa.
 

Maitiro ekumhanyisa YaLM pasina 200Gb GPU RAM?

Iwe unofanirwa kuwedzera yakadzika zero kurodha kune iyo config. Kune avo vanoziva zvatiri kutaura nezvazvo, zvichava nyore kwazvo kuzviita. Kune vamwe, iri harisi basa diki zvachose. Izvo zvakakosha kuti uzive kuti kuburitsa kunogona kunge kuri muCPU RAM kana NVMe. Unogona kukanganwa nezveNVMe parizvino, nekuti. yakawanda kwazvo data iri kugadziriswa uye dhisiki haigone kurarama nayo. Zero offload CPU inonyanya kuitika. Ichokwadi, nokuda kweizvi unoda kuva ne200+ Gb CPU RAM muzvitoro, izvo zvakare zvisingadhuri. Uye chinyorwa chimwe chete chichagadzirwa kweanenge 20-40 maminetsi, sezvo zvisati zvave kukwanisa kufananidza pamakadhi maviri evhidhiyo. Sezvauri kuona muiyo skrini pazasi, kadhi revhidhiyo rimwe chete raibatanidzwa muchizvarwa, uyezve kwechikamu chechina chendangariro. Izvo zvinoramba zvichionekwa kuti sei ese 24 GB asina kushandiswa,
Zvakanaka, mukupedzisa, ndichataura kuti zvinokwanisika kumhanya kunyange pane imwe RTX 3070 TI. Asi hapana pfungwa chaiyo mune izvi, nekuti. NVMe haizokubvumiri iwe kukurumidza kugadzirisa 150 GB yedata mukuchinjana, iyo iri mune appendage ye96 GB ye RAM.

Kupfupisa

Ehe, ini ndicharamba ndichiedza kutsvaga akakwana ekutanga nzira. Asi kusvika zvino ndasvika pamhedziso yekuti YaLM 100b inodhura zvakanyanya / inononoka kumabasa angu. Pamari imwe chete, vanhu vanonyora zvakanyanya uye zvirinani. Asi ndinofunga ndezvenguva pfupi, tichaona. Kana iwe uchida rubatsiro nekutangisa, kumisikidza yalm, kana kuti uchida kuona mibairo pamienzaniso yemamiriro ako ezvinhu, nyorera kune tsamba kana teregiramu.