“We won’t run it in digital because we’re purists and maniacs.”

@dessatel @handleym99 Those two papers have definitely made me discount perplexity more. Most semi-recent quant papers I’ve read show a variety of benchmarks which I think is better signal. This is also new and interesting, different impact for context vs weight knowledge:

@MaxWinebach sent 🍿

Despite the general messiness, it wrote one line of code in a different and simpler way than my solution. Neat.

@fleetwood___ Oh yeah. If this looks good for normal (scale/bias) quants, I am hoping it helps non-uniform/LUT methods too.

Possible silver lining: code could be a jumpstart if you want to try quantizing ModernBERT with SpinQuant.

@ivanchanavinah Dope! Text extraction is pretty good for the few I tried but formulas could use some work.

@dessatel Haha if tool == “LLM wrapper” I’m ok with that

Tried a lot of prompt variations with o1-mini/preview so I do think it is just beyond them.

Might test this out. Seems low effort/fairly interesting. GPTQ, AWQ, llama.cpp, maybe HQQ. Anything others?

Read about it here:

@dessatel Yeah! So many cool things in the paper. Lossless 8-bit RTN too.

SpinQuant: QuaRot: