I have first results to share of my re-implementation of Apple work on training specific LoRA's on top of small LM to perform specific tasks, for example summarization.
Github:
Huggingface:
I have implemented Mixture-of-Depths and it shows significant memory reduction during training and 10% speed increase. I will verify if it achieves the same quality with 12.5% active tokens.
thanks
@haeggee
for initial code
I have tried schedule_free optimizer from FAIR:
The quality looks similar, but look at the variations of metric! It's very smooth, so the training is also very stable. Same learning rate
I believe this is a big deal for LLM pre-training.
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
very interesting!!! Train multi-scale vq-vae, predict whole residual to the vq-vae token matrix in 1 step of decoding
@LevanKvirkvelia
on coding and math tasks benchmarks show that it's better than cmd-r+. for other tasks no instruct version is here yet. So gpt4 level quality
@norpadon
у тебя мнение правильное, но ты уже отстал от жизни)) за 7 лет многое в мышлении и в самом процессе егэ поменялось. Чтобы подготовиться недостаточно просто читать учебник, ты его итак на уроках и после них читаешь) Но репетиторы необязательная часть подготовки к экзамену.
In order to use it, you have to add optimizer.eval() and optimizer.train() in your code in train/eval sections and disable your regular scheduler. It supports warmup steps as well.
I have usage example in my research repo:
I published the newly released CodeQwen1 1.5 7B () to the Ollama!
ollama run thepowerfuldeez/code_qwen:7b "Write python script to compute fibonacci numbers"
there's a nice concept in perfomance optimization, that reminded me of
@UnslothAI
's recent gradient accumulation trick with async offloading.
It's called recursive doubling 🧵
mixture-of-experts with asymetric experts
From 9 experts, 4 are full size, 4 are 0.25 size, 1 is identity function (mixture-of-depths). This could be easily configured with mlp_width
what makes me, me:
compassion to technologies
architecture
making coffee
high tech food industry
vegan
exploring everywhere
deep thinker
finding new ways to old problems
high achiever
digital detox
love playful discomfort
yoga
vipassana
scooter riding in asia
what makes me, me:
- Ukraine
- software
- lsd
- alan watts
- 10 hour streams of uninterrupted train frontcam footage
- LED RGB lights
- making coffee
- "women power" songs
- blockchain and tech
- sofas are great
- amateur rapping and singing
- English language as a lifelong
@Teknium1
@haeggee
in theory we should get shorter seq_len every other decoder layer, so with 12.5% capacity_factor overall attention complexity is 75% lower
but most of the complexity lies in MLPs :(
Wanna build usable RAG application using Cmd-R+ model. Currently downloading whole arxiv, then process all documents, store vectors, build retriever.
After that, use LLM providers to run model with retrieved documents and perform grounding
Update on Mixture-of-Depths performance.
Time to achieve 10B tokens:
- With MoD: 47.3h
- Without MoD: 55.3h
Speed boost: 17%
As you can see on plots, quality degrades in average compared to baseline. Although when compared avg of Piqa/Arc_easy/Sciq – no difference
Okay, converted and pushed GGUF version to the HF hub here:
And pushed that to ollama:
ollama run thepowerfuldeez/code_qwen:7b-base '<PRE> def compute_gcd(x, y): <SUF>return result <MID>'
When I develop project locally & remotely I have 2 problems:
- For some reason I don’t get the same GitHub key in 2 diff machines, so when I commit I show as 2 different persons
- If I push from local + work at remote, I need to sync and override changes - why this hasn’t fixed?
Another relevant paper – use LOMO optimizer that is based on SGD. Authors say 1) SGD is enough for LLM pretraining because loss surface doesn't have large curvature (problem of SGD in general) and 2) local optimum is good enough (this is fine-tune related)
DPO loss modified with SFT objective on chosen prompts to increase probability of responses
Next, newly proposed RPO (Reward-aware pref. opt.). Helps mitigate overfitting and undertraining on high quality rejected responses. Difference is scaled by reward difference. 300k data
Here’s my take on how Apple
intelligence works in iOS 18:
Semantic index is used as RAG provider for small LLM with function calling behavior:
1. Apple bought several companies that do semantic indexing, providing Siri with realtime data, now this integrated across all personal
So new small LLaVA-like model is trained for 45% of total steps on 1 GPU (RTX4090) and I am already seeing good progress in terms of MMMU score.
For comparison:
1.8B: 27.9%
7B : 33.1%
34B: 44.7%
Yesterday I benchmarked LLaVA1.6-Mistral
I got 31.6% on MMMU without beam search and 32.8% with num_beams=5.
It's noticeably higher however inference time & memory reqs are 5x higher too...
@reach_vb
currently training reproduction of apple summarization lora for small llm (using Qwen2-1.5B).
Trained a couple of variations on my synthetic preference dataset, will update my profile with dataset and models soon!
@TheZachMueller
great! i've put 2x4090 into ASUS ROG Maximus Z690 for $500 with 1500W corsair HX1500i and case Fractal Design Torrent.
Have you been able to run fp8 on your setup?
I thought that latest LLaVA repository lacks some easy to start code, so I made small one-file utility library with instruction on how to run latest LLaVA-1.6 models!
You can access it here:
Found this library which could enhance speed of training a lot.
It uses Mixture-of-Experts, Mixture-of-Attention heads.
Combined with recent Mixture-of-Depths I think I could get really good performance
minecraft with mods
trying something new only because it’s “new”
easily attached to other ppl
no headphones in the city
raves and hard electronic music
dominant
dogs
It's very cool that I could still run 34B multimodal LLMs with num_beams=3 on 24G RTX4090 in 4bit without loss of quality.
And I guess I could do fine-tuning with qlora and deepspeeed mode 3 as well
Released dataset from LLaVA only has 665k entries. ShareGPT4V paper has enhanced subset. Authors also added DocVQA, ChartQA, DVQA, AI2D and laion/gpt4v-dataset but didn't share code for processing.
This was long day with crumbling data, increased instruct ds 665k->944k
Update on Summarization LoRA for Qwen2-1.5B:
SFT works better than DPO (and consumes a lot less memory at training, I might be actually run 4096-8192 context on 2x4090)
I just took 'chosen' key in dpo dataset and re-run experiment
BERTScore 0.389 vs 0.269
There is a new library for extreme quantization in town! allows for 2bit quantizing with almost the same quality as for 4bit. It's quite easy to try as all code and instructions are here, and interface is the same as for HF models. I will try it
ok I plan to re-implement work of Apple for making small llm perform various tasks, such as summarization.
I heard people tried phi3-small and it performs poorly after fine-tuning. What about qwen2?
@hiddentao
So sorry this is happened for you. In which area do you live? If you don’t want to specify that’s ok, maybe just north, or southeast will be fine
@how_uhh
Actually when I tried to increase lr I started getting nans
So this is still underexplored, however with exp_sq and momentum I doubt it makes much difference
btw стрим потом нарежу и будет контент на канал спустя 2 (3?) года неактивности )
Так что еще будет 3 анонса, чтобы хоть минимально расшевелить 3 моих подписчика )
Siri doesn’t have world knowledge and reasoning abilities, because it’s still a small LLM trained to perform actions and respond to short queries.
This will improve in the future
@rachelclif
no I usually express love in a very straight and fastest way possible cause instead I would feel fatigued by long haul of undisclosed communication
@vikhyatk
but relu dies out quickly no? More research needed. Imo gelu fused implementation works as fast as relu.
I generally think the next big thing is to determine a way to use any function in each layer through NAS
Also I plan to implement LLaVA-MoE approach, as I've seen one work in this domain. I believe approach with many small image agents could achieve performance of larger image llms.
Here's the paper:
Mixture-of-Experts for Large Vision-Language Models
Ollama is so smooth, so easy to setup to start running local llms. It's like a git v3 (v2 is huggingface, haha).
I really like the concept of Modelfile and how easy gguf models gets integrated
Basically Apple let you add agentic behavior to your apps by adding new App Intents and take actions. It’s a very smart choice, because it’s not LLM who decides to take actions, but a developer! Super excited about new Siri!