Chris Cundy
@ChrisCundy
Followers
1K
Following
549
Media
79
Statuses
359
Research Scientist at FAR AI. PhD from Stanford University. Hopefully making AI benefit humanity. Views are my own.
San Francisco, CA
Joined July 2017
Introducing *SequenceMatch*, training LLMs with an imitation learning loss. Avoids compounding error in generation by:.1. Training against *different divergences* like χ^2 with more support OOD.2. Adding a *backspace* action: model can correct errors!.1/7
6
85
490
Haven't seen many people discussing the new tokenizer for GPT-4/chatGPT, even though it's public through the openai-tiktoken package. I dumped the vocabulary here Some first impressions below.
18
61
454
Life update: I'm excited to announce that I defended my PhD last month and have joined @farairesearch as a research scientist!. At FAR, I'm investigating scalable approaches to reduce catastrophic risks from AI.
16
4
300
Great post on the intuition behind the transformer. I hadn't ever thought about how the CNN could be viewed as a special case of a transformer!.
3
105
284
Did you know that GPT4 has memorized the numerical solutions to the first 200 Project Euler problems (and many further ones too)? . This means that Project Euler is not a good evaluation dataset for LLMs. More details in my blog post:
5
31
212
Wow, Claude 2 has 200k context window 🤯. And it seems like it's actually using the context too (see graph).Any guesses how it's implemented?.
4
10
79
Fun paper recovers 50,000 lost digits from the original MNIST test set; uses them to see if we've overfitted to the MNIST test set. Conclusion: probably not that much .
3
13
63
Happy to announce I was one of the winners of the OpenAI preparedness challenge!. If anyone has a safety-related project that is bottlenecked on OpenAI credits, let me know--I'd be happy to help out.
3
3
60
How can you scalably form a posterior over possible causal mechanisms generating some observational data? Find out in our NeurIPS poster tomorrow (Tuesday), 8.30-10am pacific time! Joint work with @StefanoErmon, @adityagrover_ (1/13).
2
4
44
We treat the problem of sequence modelling as an RL problem: given a state (partial sequence), choose the action (next token to generate). I.e. how do you navigate the tree of all possible sequences?
1
7
45
And this comparison of the numerical section shows that there are numerical tokens for exactly all of the 3-digit numbers in the GPT4-tokenizer compared to the sporadic and strange coverage in the GPT2/3 tokenizer (e.g. 440 and 443 get their own token but 441 doesn't)
4
0
39
Really easy-to-use implementation of a GPT2-powered autocomplete with a google-docs style interface at ! . Gives a glimpse into the interesting applications that can be applied with NLP and generative models.
1
8
32
I am a big fan of Koray Kavukcuoglu's presentation about @DeepMindAI's research: both for the phenomenal deep learning carried out there and the use of the 'thinking face' emoji to represent the discriminator in a GAN #ICLR2018
0
4
29
Our group had an intensive reading group on optimal transport. Here's a cheat sheet I made for reference:
0
3
26
@BlackHC Good question! I doubt that transformers have enough capacity to solve any significant Project Euler problems (. I did a quick experiment with asking PE problem 1, but swapping the numbers. Both GPT4 and chatGPT failed to answer correctly at all.
2
2
23
The tokenizer seems much more intentionally designed than the gpt3 tokenizer, which fits with the idea that the GPT2 BPE was relatively quickly thrown together and maintained for backwards compatibility. There don't seem to be many 'strange' tokens like SolidGoldMagikarp.
3
1
23
Graph Neural Networks are exciting to me because it's not clear we know what is the 'natural' way to do deep learning on graph-structured data. Here's a new paper I saw showing how using page-rank to determine graph nodes embeddings can pay off.
1
6
22
GPT2 models fine-tuned on SequenceMatch can recover from errors and achieve better MAUVE score on language modelling compared to MLE-trained models. Check out the paper for all the details . Next: bigger models!🚀 . Work done with the fantastic @StefanoErmon 8/8.
1
1
22
Feeling very inspired after @EmmaBrunskill's talk at the #StanfordHumanAI launch: using rigorously optimal decision-making methods in small-data domains!
1
10
17
Did you know you can differentiate through convex optimization problems? Turns out you can!. And there are pytorch/tf implementations too. The blog post is a great introduction: (Agrawal et al, .
0
2
17
And here is a plot of how likely chatGPT is to be able to recall the exact numerical answer to a Project Euler problem, as a function of problem ID.
2
1
15
The MLE objective corresponds to matching KL divergences. In IL, this is behavioural cloning, and has compounding error: the loss is in expectation over the *expert trajectories*, so if agent-generated trajectories diverge from the expert states, the agent has bad policies😬 4/8
2
1
15
We apply recent advances in non-adversarial IL to form a fully supervised loss for language modelling, treating the dataset as examples of expert 'demonstrations'.🤖 3/8.
1
0
15
The pace of improvements in style transfer/domain adaptation is very impressive: just a few years ago I would have said these sort of results would be incredibly hard to achieve. The latest: StarGANv2 (Choi et al.). #StarGANv2
0
4
13
I gave some comments at the DoD Defense Innovation Board Public Listening Event at Stanford today #dibprinciples
1
2
13
Since we're learning actions instead of a tokens, we can add a <backspace> action to the action space. This further mitigates the compounding error problem by allowing the model to delete tokens explicitly🚮 6/8.
2
0
13
Implementing the backspace can be done with zero overhead with some masking and positional id tricks. Take the actions [a, b, <backspace>, c]. We want this to correspond to a state [a, c] with masking, so we maintain fast parallel training with transformers⚡️. 7/8.
1
0
13
We can mitigate by training against divergences with expectations over *agent trajectories*. E.g. JS divergence in GAIL, χ^2-mixture in IQ-Learn. We're encouraging the agent to 'return' to the data distribution if we go OOD. But what if there's no sensible continuation?🤔 5/8.
1
0
13
I'm at ICLR this week -- If you're there too, let's chat! Interested in hearing more about sequence models and LLMs, particularly about alignment, detecting deception, and scalable oversight.
1
0
13
I'm sure there's lots of other interesting stuff in the tokenizer. I'm not really familiar with text encoding, so I have definitely mangled the non-utf8 part of the vocab, would be happy to receive a pull request to fix that.
4
0
12
Did you know that GPT-4 is *nondeterministic* over the API, even with temperature=0?. Notably it *does* seem to give more peaked responses for lower temperatures, but there's still some randomness. (Code snippet to verify nondeterminacy in the image)
3
0
12
Remarkable example from OpenAI's blog. As ML systems more powerful, likely for failures to fall along semantically meaningful lines (=> shocking/distressing examples instead of gibberish)?.
0
5
11
Three hours before the abstract deadline for #NeurIPS2019 and I got an id in the 8000s: looks like this year will definitely hit a new record for number of submissions! #AIhype.
0
1
10
Lots of interest in diffusion on discrete spaces. I think Aaron's approach is the most principled I've seen. Plus, lots of applications in conditional generation!.
Announcing Score Entropy Discrete Diffusion (SEDD) w/ @chenlin_meng @StefanoErmon. SEDD challenges the autoregressive language paradigm, beating GPT-2 on perplexity and quality!. Arxiv: Code: Blog: 🧵1/n
1
0
7
@chrisalbon Although the L2 normalization is equivalent to weight decay for SGD, it's not the same thing with adaptive optimizers. So I wouldn't say this card correctly characterizes `weight decay' as it is typically understood today. see
1
1
7
Great fireside discussion from Prof Phil Tetlock on prediction and using ML for forecasting #EAGlobal
0
0
8
Imitiation learning is just minimizing the f-divergence between learner and demonstrator trajectory distributions: GAIL is JS-distance, Behavioural Cloning is KL divergence and DAGGER is TV distance!.
0
0
8
Up to 16% of EMNLP reviews are written by ChatGPT -- pretty concerning. I think this should be either explicitly banned or adopted in a controlled way by conferences going forward.
1
0
8
I appreciate all the emphasis on safety in the Llama-2 paper (e.g. see figure) but I'm not sure how that squares with releasing the weights. If I want crimeLlama for effective phishing emails, can't I just finetune (simple with PEFT & quantization) to remove safety guardrails?
2
0
8
This snippet from the Llama3 Model Card is very interesting: I hope we get more details soon,.particularly *clarifying if the model was evaluated pre or post DPO/PPO safety finetuning*.
1
0
8
Great talk yesterday from @janleike on recursive reward modelling for developing AI that reliably and robustly carry out the tasks that we intend: check out the paper.
0
0
7
ICYMI: A quick post on how I use emacs to keep up-to-date with papers from arxiv.
0
0
7
@goodside You can find a lot of these by looking through the vocabulary and spotting weird-looking tokens. For instance, I think ' IsPlainOldData' is another glitch token. Here, GPT4 claims it cannot see the token at all.
0
0
6
Very excited to be at the launch of the new Stanford Institute for Human-Centered AI! .#StanfordHumanAI
0
0
6
Interesting paper: .GANs don't converge to the Nash equilibria that's often motivated as the objective: instead converge to locally stable stationary points. Also, lessons for increasing stability of training.
0
0
5
The trilemma for the permissibility of ethical offsetting: which premise is wrong? #EAGlobalSF17
0
0
5
Reading reports like `On the adequacy of untuned warmup for adaptive optimization' ( makes me think that it would be very good to have a sort of `practictioners guide' to things like choosing optimizer, warmup, architecture choices, etc. .
0
3
5
There's a subtle flaw lurking at the heart of the SDE formulation of diffusion models (as they are typically implemented). Check out Aaron's work exploring this and the principled ways we can fix it -- and get some SoTA images!. 🔬🧪👉.
Presenting Reflected Diffusion Models w/ @StefanoErmon!. Diffusion models should reverse a SDE, but common hacks break this. We provide a fix through a general framework. Arxiv: Github: Blog: 🧵(1/n)
0
0
5
Very excited to be at @JHUAPL for the "Assuring AI: Future of Humans and Machines" conference! . Check by the tech demo to see some upcoming work on fairness and RL
0
0
5
Most interesting part of the OpenAI GPT2 risks report for me: . As the tech's in such an experimental state, likely bad users will be state/institutional actors; keeping larger GPT2 models won't help against that as they can train their own from scratch using the paper.
2
0
4
[Overleaf offices]: Why don't we do some maintenance on the night of the ICML deadline?.[Everyone]: What a great idea!.
1
0
4
@StefanFSchubert @robinhanson If I recall correctly, there's also an up/downvote system on the app where conferencegoers can indicate they'd prefer some questions to be asked, though the moderator isn't obliged to ask the highest-voted.
0
0
4
Even for someone who doesn't usually use tensorflow, I found this had some great high-level tips & techniques for debugging ML systems: .
0
0
4
Can anyone at OpenAI escalate a billing problem? .I'm not able to add any money to the balance -- it says I must add a negative amount of money to continue. I think it's connected to the credit grant I got from the preparedness challenge.
1
0
6
Improving to 60% on GPQA from the previous SoTA of 54% is *really* impressive -- the GPQA questions are very difficult!. (That is, assuming no test-set leakage. ).
Introducing Claude 3.5 Sonnet—our most intelligent model yet. This is the first release in our 3.5 model family. Sonnet now outperforms competitor models on key evaluations, at twice the speed of Claude 3 Opus and one-fifth the cost. Try it for free:
0
0
4
Crazy how a couple of lines of chatGPT API and TTS library calls (+ prompt) can instantiate an actually useful research/motivation assistant with realistic speech. Has been really useful to have this conversation going while working over the last week
1
0
4
Wild new paper collects a set of naturally occuring adversarial examples for imagenet classifiers: densenet gets only 2% accuracy!. #RobustML
0
2
4
@AlexGDimakis @raj_raj88 It's been argued that a big contributing factor for GPT3's poor maths ability is its tokenizer, which is strange/bad for numbers--i.e. a separate token for '809' and '810' but not for '811'. See and
0
0
3
Anyone know an easy way to implement ZeRO/FSDP with JAX/Flax/Optax? . I like JAX a lot, but it's striking how much larger the models I can train in Pytorch are.
2
0
3
Can you tell if you’ll like an ML paper after skimming for 30s? FHI and are collecting data on quick vs. careful judgments about papers. If you like ML papers, take part here (no signup required):
0
1
3
A bit confused why the bidding for ICLR only begins now: IIRC some conferences in previous years had bidding immediately after the abstract deadline. What's the motivation for the delay?.
1
0
3
@sangmichaelxie You should start it up! .A lot of information on the internet is pretty old (even advocating using sigmoids, not mentioning batchnorm, etc).
0
0
3
The @OpenAI Five agent that played yesterday and convincingly beat world-champion players had been trained continuously since July 2018 😮
0
0
3
Reading the report about many-shot jailbreaking reminds me of the paper from Wolf et al last year: under weak assumptions then for sufficiently long context, any LLM is jailbreakable.
0
0
3
The past year has seen an incredible amount of progress in NLP: particularly with the development of the Transformer architecture. @OpenAI's recent model is mind-bogglingly good.
We've trained an unsupervised language model that can generate coherent paragraphs and perform rudimentary reading comprehension, machine translation, question answering, and summarization — all without task-specific training:
0
1
3
Elvis, 'always on my mind' translated into Beethoven: Original for comparison: . It seems like the classical version sometimes misses the melody, though I'd expect this to get better as it's trained on more songs.#deeplearning.
0
0
2
Great to see NeurIPS emphasizing reproducibility by asking authors to answer the questions in the reproducibility checklist and making it available to reviewers
0
0
2
Very cool stuff: domain translation with exact likelihoods.
📢 Diffusion models (DM) generate samples from noise distribution, but for tasks such as image-to-image translation, one side is no longer noise. We present Denoising Diffusion Bridge Models, a simple and scalable extension to DMs suitable for distribution translation problems.
0
0
2
All these elements make BCD Nets: a variational method for learning posteriors over structural equation model parameters which outperform competing methods on low-data regimes. (12/13).
1
0
1
Furthermore, the model directory listing (`openai.Model.list()`) for gpt-4 has the "allow_sampling" parameter to False.
0
0
2
@qntm There are tidal forces (0.384 μg/m). If you put two things in random spots in the ISS, they will start to move relative to each other.
0
0
2
I don't usually like posting RL results based on learning curves, but this has some pretty interesting theory as well: trying to avoid bootstrapping variance in Q-learning to get more accurate off-policy learning.
0
0
2
@StephenLCasper Do you think there's promise to methods that make fine-tuning harder/impossible, like (but extended to LLMs)? This could let you 'lock in' those superficial changes.
0
1
2
Previously Wainwright showed that Q-learning has sample complexity quartic in (1/1-𝛾), while model-based methods are cubic. Here, proof a simple (and actually implementable) variance-reduction method for Q-learning can recover the cubic sample complexity.
0
0
2
We describe Bayesian Causal Discovery Nets (BCD Nets), a variational inference framework for estimating a distribution over DAGs characterizing a linear-Gaussian SEM. What are the key elements of our method? (7/13).
1
0
1
A great introduction to some challenges in imitation learning and how some of our recent work in the group helps overcome these obstacles:
1
0
2
Interesting podcast outlines the state-of-the-art in clean meat. I'd be very interested to learn more about why @open_phil disagree with @GoodFoodInst on plausibility of clean cultured meat.
The clean & plant-based meat market sector needs more CEOs and CTOs.
0
0
2
0
0
2
My guess is that it's essentially impossible to defend against white-box attacks with current models.
2
0
2
Media panel: big fans of @OurWorldInData & choosing appropriate level of nuance for each medium #eaglobalsf17
0
0
2
"Revisiting Graph Neural Networks: All We Have is Low-Pass Filters".Interesting challenge to the very hot field of graph neural networks; .claim that GNNs don't make use of the manifold hypothesis and GCNs will struggle for nonseperable feature spaces.
0
0
2
The sort of prompt-based 'black box' red-teaming the team analyses in the paper makes sense for a model with only API access. But for a model with public weights, the threat model has to be a 'white box' setup where adversaries have full access to the model weights.
1
0
2
Check out our NeurIPS poster tomorrow (Tuesday) at 8.30-10, or the paper or code (13/13).
0
0
1
Great work giving tight results for how the problem influences the choice of preconditioner for gradient methods. .
Choosing the optimal gradient algorithm depending on the geometry is important. Interestingly, how to do it is in seminal results about the Gaussian Sequence model. Come to our oral at 4:50pm in West Exhibition Hall A! #NeurIPS2019.
0
0
2
I spent quite a while fiddling with RNNs over the summer, and I'm pleased to announce it was accepted to ICLR 2018! Check out some of the other accepted papers: there are some very cool new concepts there.
1
0
2
Bob Work suggests using the Kobayashi Maru situation from Star Trek II: the Wrath of Khan to select commanding officers in the future, to select for people who use their own judgment and don't just go with the machine suggestion.
1
0
2
A continuous relaxation of sorting opens up interesting end-to-end training of traditionally modular algorithms: check out the paper!.
Our @iclr2019 paper proposes NeuralSort, a differentiable relaxation to sorting. Bonus: new Gumbel reparameterization trick for distributions over permutations. Check out our poster today at 4:30! Code: w/ @tl0en A. Zweig @ermonste
0
0
2
@sangmichaelxie I think that the various aggregators miss enough good things (and to some extent you should be reading the unpopular things) that it's worth reading all of the relevant arxivs like
0
0
2