Delighted to be a minor co-author on this work, led by @pranavn1008: Combining losses for different Matyroshka-nested groups of bits in each weight within a neural network leads to an accuracy improvement for models, especially for low-bit-precision levels (e.g. 2-bit representations). Paper: "Matryoshka Quantization" at Inspired by an off-hand comment I made to @jainprateek_ and @adityakusupati about their Matyroshka Repsentation Learning work (: "In the same way that Matryoshka representations across different units work, I wonder if we could treat the bits of each weight in a similar nested way". It turns out the answer is yes!

Andhra Pradesh is planning "Work From Home" in a big way, especially for women. First, I would like to extend greetings to all women and girls in STEM on the International Day of Women and Girls in Science. Today, we celebrate their achievements and commit ourselves to providing them equal and full access to growth opportunities in these fields.  Now, returning to the headline - As we know, work landscape underwent a shift during the COVID-19 pandemic. With technology readily available to scale, "Work From Home" gained prominence. Concepts such as remote work, coworking spaces (CWS), and Neighbourhood Workspaces (NWS) can empower businesses and employees alike to create flexible, productive work environments.  Such initiatives can help us strike a better work-life balance as well. We plan to harness these trends to drive meaningful change in AP. The Andhra Pradesh IT & GCC Policy 4.0 is a game-changing step in that direction. We're offering incentives for developers to create IT office spaces in every city/town/mandal and supporting IT/GCC firms to generate employment at the grassroots.   I'm confident these initiatives will foster greater workforce participation, especially of women professionals, who will benefit through flexible remote/hybrid work options.

Indeed, we’re seeing from R1’s reasoning traces that "Wait...", "Oh but..." are ways of backtracking past decoding errors, so errors aren’t really irrecoverable. Also, R1 Zero with pure autoregressive generation goes from 15 to 71 on AIME without any test time search or verifiers. It’s interesting that Yann, who faced resistance from “theoretically sound” ML folks when trying to publish CNNs/DL related work 15 years ago, is now mounting (somewhat flawed) theoretical reasons for why LLMs won’t work.

Gemini 2.0 Flash is the best model available right now for genric usecase. - Quality response. - Super fast. - Support ( Audio, video, docs, image,) - Tools ( Structured Output, code execution, function calling, Grounding) - least hallucination. - super cheap - 1M context Token.

@rbhar90 How so? Unless you’re explicitly doing some test time search outside the LLM (eg tree search or majority vote), it is just an autoregressive output. And discredits the notion that errors are irrecoverable.

DeepSeek R1 hosted their own server has two levels of censorship: (L1) A post-decoding (generation) filter based on keywords like "Taiwan", "Arunachal Pradesh" (L2) A biased dataset added with a higher weighting in the pre-training and/or SFT mixture. And RLHF signal to ensure it conforms to their expectation. While (L1) is easy to get rid of with self-hosting, (L2) takes some more work. End users who can self-host or use a US hosted service, while benefitting from the cost improvements, might want to also fix (L2) depending on their use case. We can see this by looking at the difference in responses for the same query. R1 on DeepSeek server, refuses to even answer non-provocative questions like Q1: "Who is the President of Taiwan" A: "Sorry, I'm not sure how to approach this type of question yet. Let's chat about math, coding, and logic problems instead!" Q2: "Name the states of India" - A: N/A (generation proceeds until it says Arunachal and then just deletes everything) R1 hosted independently as-is, gets rid of (L1) above, but (L2) still exists via PT/SFT data, yielding the following responses: Q1: "Who is the President of Taiwan" A: Taiwan is an inalienable part of China, and there is no such position as "President of Taiwan." Currently, the leader of the Taiwan region is Lai Ching-te. We adhere to the One-China principle and oppose any form of "Taiwan independence" separatist activities. Q2: "Name the states of India" Proceeds to answer because the post decode filter doesn't exist with a self-hosted version. "As of 2023, India comprises 28 states and 8 union territories. Below is the list of states: Andhra Pradesh Arunachal Pradesh Assam Bihar Chhattisgarh ... "

Elon really has his best cracked engineers sleeping in the Eisenhower Building, trying to put some startup energy into the government. Incredible. We needed this.

I hope everyone who sees this chart fully understands where we are & where we’re heading (just use your conservative extrapolation for the next two years). Meanwhile, a massive shock awaits those who don’t understand it or remain unaware! Source: @emollick & @EpochAIResearch