Neural network YaLM 100B mukuchita.

Kumapeto kwa June, Yandex
idatulutsa neural network yokhala ndi magawo 100 biliyoni otchedwa YaLM 100B kwa anthu . Ndilo neural network yayikulu kwambiri ya GPT pagulu la anthu. Imafotokoza za momwe adaphunzitsira, adawonetsa zitsanzo zabwino kwambiri komanso zomwe neuron imatha. Koma kodi ndizochita bwino komanso zogwiritsidwa ntchito kunyumba? Nkhaniyi ili chete pa izi, kuwonjezera apo, sikophweka kuyendetsa ndikuyiyang’ana, popeza pafupifupi 200 Gb ya GPU RAM ikufunika. Ndemanga iyi ya Habré
ikuwonetsa momwe zinthu zilili molondola
.

Zachidziwikire, ku Yandex, anthu onse anzeru oterowo, ndipo sanatumizenso momwe angachitire. Palibe api yachitsanzo chachikulu, palibe mtundu wokonzeka kuvula kapena wawung’ono wa anthu wamba (mu Google Colab). Palibe chitsanzo chomwe chimaperekedwa momwe mungakhazikitsire chitsanzo, momwe mungapangire malemba. Kungoti nkhaniyi ikuwonetsa ma nuances angapo a nerds ndipo ndi momwemo. Ndikokwanira kuyang’anitsitsa momwe banki inachitira ndi chilembo “C” ndikuchita zomwezo. Ndinaona kuti chitsanzo ichi ndi chimodzi mwa mayesero olephera omwe anali achifundo kutaya zinyalala, kotero adayikidwa mu Open Source kuti asonyeze zitsanzo zazikulu zomwe Yandex imapanga, komanso, ndizotseguka!

Pali mafunso ambiri pa intaneti momwe mungayendetsere yalm kapena kuyesa pa intaneti, koma palibe mayankho pa izi. Ndinali m’gulu la ogwiritsa ntchito omwe adafunsa mafunso awa. Ndipo anayamba kuzilingalira izo. Popeza ndimafunikira njira yopangira zolemba zama robot azachuma. Kotero kuti athe kuneneratu osati zikhalidwe zokha, komanso ndemanga pa izo m’malemba, pogwiritsa ntchito malipoti a zachuma. Kwenikweni, zidzakhala zofanana ndi zomwe akatswiri a zachuma amachita, pokhapokha pogwiritsa ntchito nzeru zopanga. Pali njira ziwiri zoyendetsera yalm.
Perekani seva mumtambondi 200+ Gb GPU RAM kapena sinthani kachidindo ndikuthamanga ndikutsitsa zero kwambiri (pamene GPU imayendetsa motsatizana gawo la neural network, ndipo zotsalazo zimasungidwa mu CPU RAM kapena NVMe). Yoyamba ndi yokwera mtengo kwambiri, pafupifupi 2500 rubles pa ola kapena 1.7 miliyoni pamwezi. Chachiwiri chosadziwika, chifukwa kachidindo m’nkhokwe sanaperekedwe, amangosonyeza
pa nkhani ya chosungira, amene si kovuta kuchita. Tiyeni tiyambe mophweka.

YaLM 100B Launch Malangizo

1. Timabwereka 200 GB GPU RAM, mwachitsanzo apa .

Mufunika osachepera 200 GB ya kukumbukira mavidiyo onse. 8×40 = 320 GB. Ichi chokha chikugwirizana. Zochepera 200 sizingatheke, zambiri ndizotheka. Muvi ukuwonetsa CPU RAM, sitikuyang’ana. Iye akhoza kukhala aliyense.

Timawonetsa diski ya pafupifupi 300 GB, kotero kuti ndi yopuma komanso makamaka diski yofulumira, chifukwa. makumi a ma gigabytes a data adzasamutsidwa kupita ndi kuchokera pamenepo.

Mukapanga magwero, sankhani Ubuntu ML (Kuphunzira Pamakina). Izi ndizovomerezeka kuti makadi akanema akonzedwe ndipo palibe chomwe chiyenera kuyikidwanso.

Popanga seva, pali ma nuances okhala ndi ma quotas, mutha kumva kuti zida sizikupezeka, koma kwenikweni mumangofunika kuwonjezera magawo pazosintha. Seva ikatsegulidwa (zitha kutenga mphindi 5-10), kulumikizana ndi seva kudzera pa ssh kapena mwachindunji pa intaneti patsamba la seva ndikuchita lamulo.

nvidia-smi

Zotsatira zake ziyenera kukhala tebulo lokhala ndi makadi a kanema, mtundu wa driver ndi cuda. Pafupifupi chonchi.
Mu dalaivala Baibulo chamutu ndi kumene. Kumanzere kuli manambala a chipangizo, chapakati ndi kukula kwa kukumbukira kwa chipangizocho. Ngati mulibe chidziwitso ichi, ndiye kuti mwasonkhanitsa seva kuchokera kugwero lolakwika. Ubuntu ML (Machine Learnong) ndiyofunika, monga tafotokozera pamwambapa.

2. Lumikizani chosungira ndi YaLM

sudo git clone https://github.com/yandex/YaLM-100B/ yalm
cd yalm

Tsatirani chikwatu chakunyumba kwanu kuti musamasinthe ma docker pambuyo pake. Ngati apangidwa kwinakwake, ndiye
pitani apa ndikuwonjezera njira yomwe idapangidwa.

3. Tsitsani cheke (zachitsanzo zophunzitsira)

sudo chmod +x ./download/download.sh
sudo bash ./download/download.sh

Izi zitenga pafupifupi ola limodzi. Kuti tisataye nthawi pachabe, timapanga kulumikizana kwatsopano kwa ssh ndipo mofananamo timayamba kupanga chidebe cha docker.

4. Ikani nvidia – docker 2

Docker wamba sizoyenera ,
nvidia-docker2 ndiyofunikira .
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#setting-up-nvidia-container-toolkit

5. Kumanga chidebe cha YaLM

cd yalm
sudo chmod +x ./docker/*
sudo bash ./docker/build.sh

Ndi pafupi ola limodzi.

Moyo kuthyolako. Mutha kutsitsa macheke, kukhazikitsa docker ndikumanga chidebe pa seva yotsika mtengo ndi khadi limodzi lamavidiyo. Zidzakhala chimodzimodzi mu nthawi, kotero inu mukhoza kusunga pang’ono. Pambuyo pa msonkhano pa seva yotsika mtengo, timachotsa, ndikupanga seva yolimbana ndi disk kuchokera pa seva yotsika mtengo. Ndiye simudzawononga nthawi yodikirira msonkhano ndikupopa malo oyendera.

6. Konzani zomwe zili

6.1 Macheke

Mukamaliza kutsitsa ma checkpoints, muyenera kuwalowetsa mu configs. Pali njira ziwiri, zolondola magawo kapena kusamutsa cheke. Kulikonse kumayembekezeredwa kuti ma checkpoints azikhala mu bukhu lalikulu la polojekitiyi, motsatira, zomwe zatsitsidwa ziyenera kusamutsidwa kuchokera ku foda yotsitsa pamwambapa. Pokhala mu chikwatu yalm yambitsani

mv ./download/yalm100b_checkpoint ./

Kapena sinthani njira zamafayilo omwe ali m’mafayilo
achitsanzo https://github.com/yandex/YaLM-100B/blob/c91b7d7fe8dbf39c9e307d6d324446d0df136a23/examples/generate_interactive.sh#L8-L9

6.2 Makadi avidiyo

Timayang’ana kuti makadi amakanema adayikidwa bwino. Ngati muli ndi makadi asanu ndi atatu a kanema, ndiye kuti palibe chomwe chiyenera kusinthidwa. Ngati chiwerengerocho ndi chosiyana, ndiye timasintha mizere iyi
Mu mzere wachiwiri, manambala a zipangizo zomwe zimagwiritsidwa ntchito (mukhoza kuziyang’ana mu nvidia-smi, zomwe mwayambitsa kale). Chachinayi, chiwerengero chawo.

7. Thamangani chidebe cha docker

Pokhala mufoda ya yalm, perekani lamulo

sudo bash ./docker/run.sh

Ngati zonse zili bwino, ndiye kuti mudzatengedwera ku chidebe momwe muyenera kupita ku chikwatu cha yalm m’ndandanda yanu yakunyumba.

cd ~/yalm

8. Thamangani chitsanzo kuchokera ku YaLM 100B

Ndife okonzeka kuyambitsa chimodzi mwa zitsanzo. Iwo akufotokozedwa
apa .

chmod +x ./examples/generate_interactive.sh
./examples/generate_interactive.sh

Khalani oleza mtima, imatsaliranso kudikirira mphindi 10-15 mpaka mtundu wa GPT upangidwe ndipo zolemera zochokera kumalo ochezera zimatsitsidwa.

Kumangako kukamaliza, MegatronML idzakulimbikitsani kuti mulowetse mawu kuti mupange malemba. Samalani pamene mukulemba. Nthawi zina, cholakwika chimachitika, pulogalamuyo imawonongeka ndipo muyenera kuyambitsanso msonkhano. Chifukwa chake, ndikwabwino kugwiritsa ntchito zitsanzo zomwe zimatengera zolemba kuchokera pafayilo.

9. Zotsatira za ntchito

Zikuwoneka zosangalatsa. Inde, izi ndi zitsanzo zabwino chabe. Ndinayesa mayeso pazitsanzo zosiyanasiyana. Monga momwe zimayembekezeredwa, nkhaniyo ikakhala yabwinoko, mawu omveka bwino amapangidwa. Mndandanda wonse wa mibadwo yoyesera ukhoza kuwonedwa pa maulalo:

• generate_conditional_sampling.sh
• generate_conditional_greedy.sh
• generate_unconditional.sh

Pamtengo wake, zidanditengera pafupifupi ma ruble 9,000 pakubwereketsa ma seva amitundu yosiyanasiyana kuyambira pakuphunzitsidwa komanso kukonzekera kupita ku mibadwo. Chokhumudwitsa china chinali chakuti simungathe kupanga chilichonse nthawi yomweyo. Zimatenga nthawi yayitali kwambiri kuti tiyambe ndipo malembawo sapanga mofulumira monga momwe timafunira, chifukwa cha mtengo wa seva pa ola limodzi.
 

Momwe mungayendetsere YaLM popanda 200Gb GPU RAM?

Muyenera kuwonjezera zotsitsa zero zozama ku config. Kwa iwo omwe akudziwa zomwe tikukamba, zidzakhala zosavuta kuchita. Kwa ena, iyi si ntchito yaing’ono konse. Ndikofunika kudziwa kuti kutsitsa kumatha kukhala mu CPU RAM kapena NVMe. Mutha kuyiwala za NVMe pakadali pano, chifukwa. deta yochuluka kwambiri ikukonzedwa ndipo litayamba silingathe kupirira nazo. Zero kutsitsa CPU ndi yeniyeni. Zowona, chifukwa cha izi muyenera kukhala ndi 200+ Gb CPU RAM m’gulu, zomwenso sizotsika mtengo. Ndipo lemba limodzi lidzapangidwa kwa mphindi pafupifupi 20-40, popeza sizinali zotheka kuzifananitsa pamakhadi awiri avidiyo. Monga mukuwonera pachithunzichi pansipa, khadi imodzi yokha ya kanema idakhudzidwa ndi m’badwo, ndiyeno kotala la kukumbukira. Zikuwonekerabe chifukwa chake ma 24 GB onse sagwiritsidwa ntchito,
Chabwino, pomaliza, ndikunena kuti ndizotheka kuthamanga ngakhale pa RTX 3070 TI imodzi. Koma palibe tanthauzo lenileni mu izi, chifukwa. NVMe sikukulolani kuti musinthe mwachangu 150 GB ya data mukusinthana, zomwe zili mu 96 GB ya RAM.

Kufotokozera mwachidule

Zachidziwikire, ndiyesetsabe kupeza njira zabwino zoyambira. Koma mpaka pano ndazindikira kuti YaLM 100b ndiyokwera mtengo kwambiri / yochedwa kwambiri pantchito zanga. Kwa ndalama zomwezo, anthu adzalemba zambiri komanso zabwino kwambiri. Koma ndikuganiza kuti ndi zakanthawi, tiwona. Ngati mukufuna thandizo poyambitsa, kukhazikitsa yalm, kapena mukufuna kuwona zotsatira pazitsanzo zanu, lemberani makalata kapena telegalamu.