Do you want expressive mixture models that can subtract probability mass? you can square circuits!
But by doing so we can also lose expressiveness 🚨
We introduce sum of squares 🆘 PCs to overcome this limitation.
🆘 PCs also explain why complex parameters boost performance!
One trick that I like to use when training my neural networks is to add some noise ε~Laplace(time(), sqrt(time())) to the gradients of the 13th layer at epoch 3 for batch 7.
Is there a reason why a BSc, MSc or PhD Student in
#AI
or
#ML
should not attend
#UAI2021
@UncertaintyInAI
on July 27-30?
The conference is top tier, will be virtual and registration will be free for students!!!
👇👇👇
Classical mixture models are limited to positive weights and this requires learning very large mixtures!
Can we learn (deep) mixtures with negative weights?
Answer in our
#ICLR2024
spotlight by
@loreloc_
Aleks, Martin, Stefan, Nicolas
@arnosolin
📜
After the score-based models, I decided to take a step back and cover the basics in a new blog post: Probabilistic modeling and Mixture Models. Additionally, I make a brief intro to ✨Probabilistic Circuits✨ Check:
📄Post:
🖥️ Code:
I'm reading a bit about visual explainations for
#ML
#XAI
and I found
"Sanity Checks for Saliency Maps"
which says that many visualizations are invariant to random weights, i.e. they are just detecting edges not visualizing what the model is looking at!
Protip to all PhD students struggling to write last minute their
@NeurIPSConf
#NeurIPS2022
papers:
large language models such as GPT3 () and OPT-175B () are now publicly available for you to write your missing sections!
When conference
#proceedings
were printed in a single book and submissions did not follow a single
#LaTeX
template:
behold the papers accepted at
#NIPS89
the 2nd Conference on advances in Neural Information Processing Systems!
🧵👇
💥Breakthrough work for the tractable probabilistic modeling community!💥
👉
Continuous latent variable models can be turned into expressive tractable discrete mixtures via numerical integration
by
@ahc_correia
@gengala13
@cassiopc
@equaeghe
@ropeharz
As I joined the
@ELLISforEurope
Unit at
@EdinburghUni
@InfAtEd
I'm looking for PhD students and other
#ELLIS
comrades for co-supervision!
Topics: efficient and reliable generative models in the wild; automating learning, reasoning and their combination!
Please share 🔁 ❤️ 🙏 !
Are you considering a PhD in
#machinelearning
or a related research area? The next round of applications for the ELLIS PhD program is coming up. Stay tuned at and apply by November 15!
+ LINK to call:
Soon the rebuttal phase for
@NeurIPSConf
#NeurIPS2020
will start . On a Friday (evening in Europe).
It is not the first time
#ML
#AI
authors are asked to work on weekends.
This is a bad cultural and working habit that can only go worse in times of
#COVID19
. When for a change?
TIL there are 3-10% of training images of CIFAR-10/0 that are identical or very similar to test set images.
Replacing them with new samples (from same distribution) makes the test set accuracy drop by 9-14%
Often we take our image datasets for granted. There’s value in actually looking at the pixels. Check out:
Do We Train on Test Data? Purging CIFAR of Near-Duplicates, Björn Barz, Joachim Denzler
Paper:
@skoularidou
@_sam_sinha_
@zacharylipton
but that uses adversarial learning in one variant (and MMD in another).
Let me shamelessly plug RAEs
The catch is that ex-post density estimation on the latent space gets you better aggregate posterior estimation and hence sample quality
A formal definition of what an inductive bias is in
#ML
from
@tommmitchell
book (1996)
and yes, it can be any sort of background knowledge or prior (we can express in logical rules about the input, and mod at least for Tom's version space)
I met a kid today at the beach who was struggling to make a castle with pebbles. After the nth failure, I taught him the bitter lesson.
I told him that scaling is all he needs, that he should not waste time reasoning and that symbols are not well defined.
He burst into tears.
📜
"""
Inconsistent results based on hyperparameter optimization configurations are widespread in
#ML
. When comparing two algorithms J and K searching one subspace can yield that J outperforms K, but searching another can entail the opposite.
"""
people are all of a sudden reading one of my favorite papers (also the wackiest) from grad school—about non-determinism, arbitrariness, reliable measurement, and hyperparameter optimization (HPO)
a quick thread on this, why I think it’s esp. interesting in this current moment
Dear reviewers,
if a method is very simple and yet yields big advantages (time/space efficiency, sota performance, ...) and it is novel (no one tried it even if it is simple), it is TOTALLY worth a publication.
(I'm talking in general, here)
#NeurIPS2024
writing tips
(part 1)
✨make a very clear example of what is the issue/problem you are trying to solve✨
👉what is obvious for you is not for others
👉even experts in the field need to "see it"
👉this can be your Fig 1
👉it helps scope your solution
#ELLIS
PhD Applications are now open!
Consider applying to my group at
@EdinburghUni
@InfAtEd
if you want to work in probabilistic ML investigating efficient and reliable models, combining learning and reasoning and automating them!
ELLIS PhD Program in
#machinelearning
now accepting applications (deadline: November 15, 2021). Details on the program and the application process here:
from the
@NeurIPSConf
guide to ACs:
"""
Try to counter biases you perceive in the reviews.
Unfashionable subjects should be treated fairly but often aren't, to the advantage of the papers on more mainstream approaches.
To help the NeurIPS community move faster [1/2]
Consider submitting works on normalizing flows, probabilistic circuits and tractable inference at large at the 4th Tractable Probabilistic Modeling Workshop
#TPM2021
to be held at
@UncertaintyInAI
#UAI2021
!
Deadline: 28 May
Please RT and share! 👇👇👇
Congrats to Dr.
@andreasgrv
who just defended his
#PhD
thesis on unargmaxability and constraints in deep learning, examined by Vlad Niculae and
@driainmurray
🎉
A pity you could not (yet) see Andreas' beautifully designed thesis!
In the meantime, some pics from the pre-viva 👇
Mixtures are *A M A Z I N G*!
They let you:
1) marginalize in O(k) (k = # components) if components can marginalize (eg. Gaussian)
2) approximate well enough *any* density for k → +∞
i.e., asymptotically...but in practice?
How many Gaussians to fit the densities in the pic?
when they ask me why I won't be going back to
#Italy
anytime soon
a Vatican priest with a PhD in moral theology and some interests in ethical aspects of AI is now leading the Italian task force of
#AI
(previously led by a 90 years old guy with no experience in computer science)
Il teologo francescano Paolo
#Benanti
è il nuovo Presidente della Commissione sull'Intelligenza Artificiale per l'informazione, dopo le dimissioni di Giuliano
#Amato
@ultimora_pol
I am Vienna for
@iclr_conf
#ICLR2024
for a week presenting work done with
@loreloc_
@diegocalanzone
& Christian Jimenez.
Ping me if you want to chat about
- realiable and complex probabilistic reasoning 🎯📊
- neuro-symbolic integration 🤖🧠
- fast inference⚡️🔬
Mixture models that subtract probability density can be exponentially more expressive in representing complex distributions
We prove this when they are learned by squaring tensorized networks, and show how they represent a unifying modelling framework in our
#ICLR2024
spotlight!
thanks
@laurent_dinh
for sharing more than just research successes and achievements at
@iclr_conf
#ICLR2020
: a more human and fuller journey from rejections at conferences to collective advances as a community
join
@loreloc_
#ICLR2024
@iclr_conf
at Poster
#210
Thursday 10.45 (today in 20 mins!)
to learn how to "cut" distributions while preserving distribution semantics and tractability
Mixture models that subtract probability density can be exponentially more expressive in representing complex distributions
We prove this when they are learned by squaring tensorized networks, and show how they represent a unifying modelling framework in our
#ICLR2024
spotlight!
De-hyping
#ML
and
#AI
, and super-human performance claims + (finally!) treating ImageNet as multi-label classification
I am in love with the rigorous and skeptikal empirical science by
@Vaishaal
Rebecca Roelofs
@HoriaMania
@beenwrekt
@lschmidt3
👇👇👇
Superhappy to discuss the Atlas paper
📜
tomorrow at the
@MITCSAIL
#Programming
#Languages
Review
👉
as one of the papers that "that may substantially transform the
#PL
community and beyond"!
Miracolo!
Just 13 hrs before the
#NeurIPS20
deadline Our Lady of the Well in Capurso (La Madonna del Pozzo di Capurso) appeared to show me what were wrong in the experiments!
Now we have a submission we can submit to
@NeurIPSConf
we just need to rerun all and write a paper!
I am excited to announce that my book, "Deep Generative Modeling", is available online and in print (
@SpringerNature
):
Code used in the book is freely available online: (1/4)
@shaohua0116
honestly...
this tells a long story about
#authorship
and the current state of
#ML
and
#AI
fields
32 papers is 1 paper every 12 days
20 papers is 1 paper every 18 days
16 papers is 1 paper every 23 days
"I cannot emphasize enough that you should not interpret
#Bayesian
methods as just regularizers of maximum likelihood
#MLE
learning"
@andrewgwils
in the "Bayesian Deep Learning and a Probabilistic Perspective of Model Construction" at
@icmlconf
👇👇👇
A good metaphor for some
@iclr_conf
#iclr2021
submissions I came across 👇👇👇
Some good ingredients (ideas) are strangely placed on a plate (literature) and hidden by an overabundance of sauce (details, claims) no one is going to eat through (hard to check)
Bon appetit!
After 8 months of long coding nights ☕️ we finally officially release Pythae 🥳, a python library unifying generative autoencoder implementations including vaegan🥗, vqvae or RAEs.
🖥️ github repo:
👉paper:
A couple more lectures for the
#MLPR
course on probabilistic
#ML
@InfAtEd
next week...
While going through the recordings of past lectures, I found your cube materializing on my slides,
@ZoubinGhahrama1
!
"""
There is a tendency to classify work as Bayesian or not Bayesian, with very stringent criteria for what qualifies as Bayesian [...] We believe this mentality encourages tribalism, which is not conductive to the best research
"""
words of wisdom by
@andrewgwils
@Pavel_Izmailov
Our new paper "Bayesian Deep Learning and a Probabilistic Perspective of Generalization": . Includes (1) benefits of BMA; (2) BMA <-> Deep Ensembles; (3) new methods; (4) BNN priors; (5) generalization in DL; (6) tempering in BDL. With
@Pavel_Izmailov
. 1/19
5 postdoc positions in
#AI
and
#ML
in
#AI4Science
, Climate and Earth Sciences at the Image and Signal Processing group
@isp_uv_es
in Valencia, Spain 🇪🇸🇪🇺
👇👇
Germany 🇩🇪, Italy 🇮🇹 and Austria 🇦🇹, wtf?
do you really need to see Kyiv carpet bombed before deciding to effectively isolate Putin and exclude Russia from SWIFT?
On this day, 422 years ago, Giordano Bruno was burned alive by the Catholic Church for saying that stars were distant suns surrounded by planets, and that Earth was not the center of this infinite universe.
#GiordanoBruno
#1600
we evaluate
#continual
learning models on "baby" benchmarks, where it is easy to show no catastrophic forgetting!
we propose a new simple benchmark that glues simple but still challenging tasks in a curriculum: from MNIST to Imagenet and back!
📜
my advice to
#PhD
applicants👇
write a
#research
statement to highlight how your past experience & current interests in
#ML
overlap with those of the labs you are applying to!
I'll be hiring via
#ELLIS
too, you can read about research @
#APRIL
here 👉
Just a few days are left until our
#ELLISPhD
application portal will close for this year! Take a look at our FAQs if you still have questions and don’t forget to submit your application by November 15!
#machinelearning
#AI
#phdprogram
@ai_elise
15 authors and no one who knows that pretty tables have no column separators...
(ps. is this a new trend to have only abstract and figures on the 1st page?)
(pps. did we really need to coin the term "Foundation Transformers"?)
#ICML2023
@icmlconf
deadlines are out:
🚨Paper Submissions Open on
@openreviewnet
:
🗓️Jan 09 '23 02:00 PM UTC
🚨Full Paper Submission Deadline:
Jan 26 '23 08:00 PM UTC
out of local minima, it is important to encourage risk and recognize that new approaches can't initially yield state-of-the-art competitive results.
Nor are they always sold according to the recipes we are used to.
[2/2]
"""
thanks
@NeurIPSConf
for the reminder 🫶
AISTATS is one of those venues in
#AI
and
#ML
with high quality in reviews, submitted content and discussions!
Plus,
#AISTATS2022
will be live in Valencia, Spain!
We invite submissions to the AISTATS 2022 conference and welcome paper submissions on artificial intelligence, machine learning, statistics, and related areas!
Are there any flexible probabilistic model classes (neural or non-neural) that permit tractable computation of expectations, information & distributional properties generally (bonus points for conditional variants)
@guyvdb
@martin_trapp
@DaniloJRezende
?
"""
There is no true data distribution.
Even if it were, it would not be learnable.
And even if it were learnable, we could not perform inference over it.
And even if we could, it would not help humans' decisions.
"""
#Gorgias
the nihilist (400 BC) on probabilistic
#ML
#AI
I am thrilled to announce that 2/2 bars have been accepted by this COVID test, making it one of the most positive results it can provide.
(no worries, I will be back to boast about
#NeurIPS
hopefully soon)
Even more remarkable than the publicly available lib in 🐍 are the benchmarks run in the paper (👉 ).
TL;DR: deterministic AEs can be comparable and often much better generative models than VAEs and variants!
A short 🧵👇
I'll be giving a talk at the ICML 2024 Differentiable Almost Everything Workshop!
⏲️ Friday 9:10 in Stolz 0
Join us if you're interested in learning about differentiating through logic and other algorithms!
This paper will get a thread at a certain point as it has many stories to tell, beyond its interesting technical content.
For example, it was brutally rejected at
@icmlconf
earlier this year before getting accepted as oral now at
@NeurIPSConf
#NeurIPS2021
! 👀
Amazing art and explanation of the underlying math.
A mixture of increasingly more complex polynomials to "fit" a realistic terrain!🔥
This can be a nice recipe for a boosting algorithm for density estimation...👀
what is going on with
@NeurIPSConf
early registration for conference and workshop costing 850$?
Isn't it almost double the pre-pandemic cost of an in-person conference?
"we show that a simple greedy algorithm, running in almost linear time, can find solutions of much better quality than the GNN. The greedy algorithm is faster by a factor of 10**4 for problems with a million variables."
🔥🔥🔥